Feature point processing method and related apparatus

Abstract

The application discloses a feature point processing method, and is applied to the technical field of image processing. In the method, rotation information of an image acquisition device when collecting an image frame is acquired, rotation transformation processing is performed on feature point pairs in the image, rotation information between the feature point pairs is eliminated, motion components of a rotation dimension introduced by the image acquisition device are fundamentally offset, the effectiveness of a feature point pair method based on threshold filtering of a translation dimension is improved, and the accuracy of the feature point pairs obtained after filtering is ensured.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a feature point processing method and related apparatus. Background Technology

[0002] Feature extraction and matching of images is a crucial task in many computer vision applications, widely used in object detection, image retrieval, image recognition, video tracking, image stitching, and 3D reconstruction. Since the accuracy of feature matching significantly impacts subsequent image processing tasks, and preliminary feature matching results often contain numerous mismatches, effectively filtering out mismatched feature points is a critical task in the feature matching process.

[0003] Currently, most traditional filtering algorithms take into account the small motion changes between frames, consider the feature regionality in the feature point extraction and matching stages, and then use a fixed threshold to filter feature point pairs that are too far apart.

[0004] However, in complex scenarios where image acquisition devices move quickly or where there are moving objects, the changes between image frames are significant, making it difficult for traditional filtering algorithms to effectively filter feature point pairs, resulting in low accuracy of the final feature point pairs. Summary of the Invention

[0005] This application provides a feature point processing method and related apparatus. By acquiring the rotation information of the image acquisition device when acquiring image frames, the feature point pairs in the image are subjected to rotation transformation processing to eliminate the rotation information between the feature point pairs. This fundamentally cancels out the motion component of the rotation dimension introduced by the image acquisition device, thereby improving the effectiveness of the threshold filtering feature point pair method based on the translation dimension and ensuring the accuracy of the filtered feature point pairs.

[0006] A first aspect of this application provides a feature point processing method, comprising: performing feature point matching on a first image and a second image to obtain multiple feature point pairs, wherein each feature point pair includes a feature point in the first image and a feature point in the second image. The first image and the second image are two images for which feature point matching is to be performed. For example, the first image and the second image are two consecutive image frames acquired by an image acquisition device; or, the first image and the second image are two consecutive image frames in a video segment acquired by the image acquisition device.

[0007] A first rotation angle and a second rotation angle are obtained. The first rotation angle is the rotation angle of the image acquisition device when acquiring the first image, and the second rotation angle is the rotation angle of the image acquisition device when acquiring the second image. Both the first and second rotation angles include rotation angles in three directions, i.e., three-dimensional rotation angles.

[0008] A pose transformation matrix is ​​determined based on the first rotation angle and the second rotation angle. The pose transformation matrix is ​​used to eliminate rotation between feature point pairs. Specifically, the pose transformation matrix can be used to transform the pose of feature points in the first image to the pose of feature points in the second image, i.e., to transform the pose of feature points in the first image; or it can be used to transform the pose of feature points in the second image to the pose of feature points in the first image, i.e., to transform the pose of feature points in the second image.

[0009] The pose transformation matrix is ​​used to transform the multiple feature point pairs to obtain multiple pose-transformed feature point pairs. Each feature point pair in the multiple pose-transformed feature point pairs is in the same pose in the rotation dimension.

[0010] The multiple pose-transformed feature point pairs are filtered based on a first translation threshold to obtain multiple filtered feature point pairs. The first translation threshold can be a pre-set threshold, adaptable to most motion scenarios and scenarios with moving objects.

[0011] In this scheme, by acquiring the rotation information of the image acquisition device when acquiring image frames, the feature point pairs in the image are subjected to rotation transformation processing to eliminate the rotation information between the feature point pairs. This fundamentally cancels out the motion component of the rotation dimension introduced by the image acquisition device, thereby improving the effectiveness of the threshold filtering feature point pair method based on the translation dimension and ensuring the accuracy of the filtered feature point pairs.

[0012] Furthermore, regarding the motion information affecting the distance between feature point pairs in the image acquisition device, this motion information includes three-dimensional rotation components and three-dimensional translation components. The three-dimensional rotation components are the motion components that mainly affect the distance between feature point pairs. Therefore, in this embodiment, by eliminating the rotation information of feature point pairs, most of the motion of the image acquisition device is canceled out, ensuring accurate filtering of feature point pairs in the translation dimension in complex motion scenes.

[0013] In one possible implementation, when the image acquisition device uses a rolling shutter image sensor, the electronic device can acquire the rotation angles of each feature point in the first image and the second image, so as to facilitate subsequent pose transformation of the feature point pairs.

[0014] Specifically, the first rotation angle includes multiple first sub-rotation angles, each of which is the rotation angle when the image acquisition device acquires feature points belonging to the first image in the multiple feature point pairs. The second rotation angle includes multiple second sub-rotation angles, each of which is the rotation angle when the image acquisition device acquires feature points belonging to the second image in the multiple feature point pairs. Determining the pose transformation matrix based on the first and second rotation angles includes: obtaining a reference rotation angle, which may be, for example, the rotation angle of a reference point in the first or second image; determining the pose transformation matrix corresponding to the feature points belonging to the first image in the multiple feature point pairs based on the multiple first sub-rotation angles and the reference rotation angle; and determining the pose transformation matrix corresponding to the feature points belonging to the second image in the multiple feature point pairs based on the multiple second sub-rotation angles and the reference rotation angle.

[0015] In one possible implementation, obtaining the first rotation angle and the second rotation angle includes: obtaining a first angular velocity of the image acquisition device when acquiring the first image and a second angular velocity of the image acquisition device when acquiring the second image; determining the first rotation angle based on the rotation angle of the image acquisition device when acquiring the third image, the first angular velocity, and a first time interval, wherein the third image is the frame preceding the first image, and the first time interval is the acquisition interval between the third image and the first image; and determining the second rotation angle based on the rotation angle of the image acquisition device when acquiring the fourth image, the second angular velocity, and a second time interval, wherein the fourth image is the frame preceding the second image, and the second time interval is the acquisition interval between the fourth image and the second image.

[0016] In one possible implementation, determining the pose transformation matrix based on the first rotation angle and the second rotation angle includes: obtaining the intrinsic parameter matrix of the image acquisition device; and determining the pose transformation matrix based on the intrinsic parameter matrix, the first rotation angle, and the second rotation angle.

[0017] In one possible implementation, the first translation threshold is related to the motion state of the image acquisition device when acquiring images related to the second image; wherein, the images related to the second image include the second image, the preceding N frames of the second image, and the following M frames of the second image, where N and M are both integers greater than or equal to 0. Specifically, if the motion intensity of the image acquisition device is high when acquiring images related to the second image, the first translation threshold can be a higher value; if the motion intensity of the image acquisition device is low when acquiring images related to the second image, the first translation threshold can be a lower value.

[0018] Since the amplitude of movement of the same image acquisition device may vary at different times, meaning the distance the feature points move in different images acquired by the device will also differ, using a fixed threshold to filter feature point pairs may result in insufficient or incorrect filtering. This solution, by binding the value of the first translation threshold used for filtering feature point pairs to the motion state of the image acquisition device, allows for the selection of an appropriate filtering threshold based on the device's motion state, thereby improving the accuracy of feature point pair filtering.

[0019] In one possible implementation, the method further includes: acquiring the three-axis rotational angular velocity of the image acquisition device when acquiring an image related to the second image and a first motion degree value of the image acquisition device when acquiring the first image; wherein the first motion degree value is related to the motion state of the image acquisition device when acquiring an image related to the first image.

[0020] Based on the three-axis rotational angular velocity and the first motion degree value, a second motion degree value is determined by the image acquisition device when acquiring the second image. For example, the second motion degree value can be the sum of the three-axis rotational accelerations corresponding to the second image and the first motion degree value; or the second motion degree value can be the sum of the average three-axis rotational accelerations corresponding to images related to the second image and the first motion degree value.

[0021] A first translation threshold corresponding to the second motion intensity value is determined from multiple thresholds. That is, multiple thresholds can be preset for filtering, with different filtering thresholds corresponding to different levels of motion intensity. Since the second motion intensity value can be used to represent the motion intensity of the image acquisition device, a corresponding threshold can be selected from multiple thresholds based on the second motion intensity value.

[0022] In this scheme, the intensity of motion of the image acquisition device is determined based on the motion information of the image acquisition device when acquiring images, and the filtering threshold is adaptively selected based on the intensity of motion of the image acquisition device to filter feature point pairs, which can effectively improve the accuracy of filtering feature point pairs.

[0023] In one possible implementation, the method further includes: acquiring the triaxial acceleration of the image acquisition device when acquiring an image related to the second image; determining the second motion degree value of the image acquisition device in the second image based on the triaxial acceleration and the first motion degree value includes: determining the second motion degree value of the image acquisition device in the second image based on the triaxial acceleration, the triaxial rotational angular velocity and the first motion degree value.

[0024] This scheme considers both the three-axis angular velocity and three-axis acceleration of the image acquisition device to determine the intensity of its motion, thereby improving the accuracy of motion intensity assessment and providing support for improving the accuracy of filtering feature point pairs.

[0025] In one possible implementation, the step of matching feature points in the first image and the second image to obtain multiple feature point pairs includes: extracting feature points from the first image and the second image to obtain multiple feature points; matching the multiple feature points to obtain multiple pairs of original feature point pairs; and filtering the multiple pairs of original feature point pairs according to a feature point pair filtering method to obtain the multiple feature point pairs; wherein the feature point pair filtering method includes one or more of the following: standard deviation filtering method, fixed threshold filtering method, and random sampling consistency filtering method.

[0026] A second aspect of this application provides a feature point processing apparatus, comprising: a processing module for performing feature point matching on a first image and a second image to obtain multiple feature point pairs, each of the multiple feature point pairs including a feature point in the first image and a feature point in the second image; an acquisition module for acquiring a first rotation angle and a second rotation angle, the first rotation angle being the rotation angle of the image acquisition device when acquiring the first image, and the second rotation angle being the rotation angle of the image acquisition device when acquiring the second image; the processing module further for determining a pose transformation matrix based on the first rotation angle and the second rotation angle, the pose transformation matrix being used to eliminate rotation between feature point pairs; the processing module further for performing pose transformation on the multiple feature point pairs according to the pose transformation matrix to obtain multiple pose-transformed feature point pairs, each of the multiple pose-transformed feature point pairs being in the same pose in the rotation dimension; and the processing module further for filtering the multiple pose-transformed feature point pairs based on a first translation threshold to obtain multiple filtered feature point pairs.

[0027] In one possible implementation, the first rotation angle includes multiple first sub-rotation angles, each of which is a rotation angle when the image acquisition device acquires feature points belonging to the first image from the multiple feature point pairs. The second rotation angle includes multiple second sub-rotation angles, each of which is a rotation angle when the image acquisition device acquires feature points belonging to the second image from the multiple feature point pairs. The processing unit is specifically configured to: obtain a reference rotation angle; determine the pose transformation matrix corresponding to the feature points belonging to the first image from the multiple feature point pairs based on the multiple first sub-rotation angles and the reference rotation angle; and determine the pose transformation matrix corresponding to the feature points belonging to the second image from the multiple feature point pairs based on the multiple second sub-rotation angles and the reference rotation angle.

[0028] In one possible implementation, the acquisition module is further configured to acquire a first angular velocity of the image acquisition device when acquiring the first image and a second angular velocity of the image acquisition device when acquiring the second image; the processing module is further configured to determine a first rotation angle based on the rotation angle of the image acquisition device when acquiring a third image, the first angular velocity, and a first time interval, wherein the third image is the previous frame of the first image, and the first time interval is the acquisition interval between the third image and the first image; the processing module is further configured to determine a second rotation angle based on the rotation angle of the image acquisition device when acquiring a fourth image, the second angular velocity, and a second time interval, wherein the fourth image is the previous frame of the second image, and the second time interval is the acquisition interval between the fourth image and the second image.

[0029] In one possible implementation, the processing module is further configured to: obtain the intrinsic parameter matrix of the image acquisition device; and determine the pose transformation matrix based on the intrinsic parameter matrix, the first rotation angle, and the second rotation angle.

[0030] In one possible implementation, the first translation threshold is related to the motion state of the image acquisition device when acquiring images associated with the second image; wherein, the images associated with the second image include the second image, the first N frames of the second image, and the last M frames of the second image, where N and M are both integers greater than or equal to 0.

[0031] In one possible implementation, the acquisition module is further configured to acquire the three-axis rotational angular velocity of the image acquisition device when acquiring an image related to the second image and the first motion degree value of the image acquisition device when acquiring the first image; the processing module is further configured to determine a second motion degree value of the image acquisition device in the second image based on the three-axis rotational angular velocity and the first motion degree value; the processing module is further configured to determine a first translation threshold corresponding to the second motion degree value among multiple thresholds.

[0032] In one possible implementation, the acquisition module is further configured to acquire the triaxial acceleration of the image acquisition device when acquiring an image related to the second image; the processing module is further configured to determine a second motion degree value of the image acquisition device when acquiring the second image based on the triaxial acceleration, the triaxial rotational angular velocity, and the first motion degree value.

[0033] In one possible implementation, the processing module is further configured to: extract feature points from the first image and the second image to obtain multiple feature points; match the multiple feature points to obtain multiple pairs of original feature points; filter the multiple pairs of original feature points according to a feature point pair filtering method to obtain the multiple feature point pairs; wherein the feature point pair filtering method includes one or more of a standard deviation filtering method, a fixed threshold filtering method, and a random sampling consistency filtering method.

[0034] A third aspect of this application provides a feature point processing apparatus, including a memory and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the feature point processing apparatus performs the method as described in any implementation of the first aspect above.

[0035] A fourth aspect of this application provides a computer storage medium, characterized in that the computer storage medium stores instructions that, when executed by a computer, cause the computer to perform the method described in any of the implementations of the first aspect.

[0036] The fifth aspect of this application provides a computer program product, characterized in that the computer program product stores instructions that, when executed by a computer, cause the computer to perform the method described in any of the implementations of the first aspect.

[0037] A sixth aspect of this application provides a chip including a processor and a communication interface for communicating with a module other than the chip, the processor for running computer programs or instructions such that an apparatus on which the chip is mounted can perform the method of any implementation of the first aspect described above.

[0038] The technical effects of any of the implementation methods in aspects two through six can be found in the technical effects of different implementation methods in aspect one above, and will not be repeated here. Attached Figure Description

[0039] Figure 1 This is a schematic diagram illustrating an application scenario of a feature processing method provided in an embodiment of this application.

[0040] Figure 2 This is a schematic diagram of the structure of an electronic device 101 provided in an embodiment of this application;

[0041] Figure 3 A flowchart illustrating a feature point processing method provided in an embodiment of this application;

[0042] Figure 4 This is a schematic diagram of the exposure of a rolling shutter image sensor and a global shutter image sensor.

[0043] Figure 5a A schematic diagram of pose transformation provided in an embodiment of this application;

[0044] Figure 5b A schematic diagram of an image sequence provided in an embodiment of this application;

[0045] Figure 6 A flowchart illustrating the feature point processing method provided in this application embodiment;

[0046] Figure 7 This is a schematic diagram of the structure of a feature point processing device 700 provided in an embodiment of this application;

[0047] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0048] Figure 9 This is a schematic diagram of a computer-readable storage medium provided in an embodiment of this application. Detailed Implementation

[0049] The embodiments of this application are described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. As those skilled in the art will recognize, with the development of technology and the emergence of new scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0050] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein.

[0051] Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or modules is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not explicitly listed or inherent to those processes, methods, products, or apparatuses. The naming or numbering of steps appearing in this application does not imply that the steps in the method flow must be performed in the chronological / logical order indicated by the naming or numbering. The execution order of named or numbered process steps can be changed according to the desired technical purpose, as long as the same or similar technical effect can be achieved.

[0052] To facilitate understanding, the technical terms used in this embodiment will be introduced below.

[0053] A feature is a piece of information relevant to solving a computational task associated with an application; it is informative and non-redundant. A feature might be a specific structure in an image, such as a point, edge, or object. Features can also be the result of general neighborhood operations or feature detection applied to an image. For example, features at specific locations in an image, such as mountain peaks, building corners, doorways, or interestingly shaped snow blocks. These localized features are often called keypoint features, and they are typically described as patches of pixels surrounding the point location, often referred to as image patches.

[0054] Feature extraction refers to a series of operations that construct features from raw data. It is a step of reducing dimensionality, reducing the initial dataset to more manageable groups (features) to facilitate learning, while maintaining the accuracy and completeness of describing the original dataset.

[0055] Feature Point: A form of expression for image feature extraction and matching results, where features are labeled with key points.

[0056] Feature matching (Pairing): In computer vision tasks, similar features are identified by comparing feature point descriptors in an image. That is, the distance between feature vectors is calculated, which can group feature points that express the same information in different image frames into pairs, thus obtaining feature point pairs.

[0057] The main steps of feature extraction and matching are as follows.

[0058] 1. Detection: Identifying feature points.

[0059] 2. Description: Describes the local appearance around each feature point. This description is invariant to changes in lighting, translation, scale, and in-plane rotation. Generally, each feature point is provided with a descriptor vector.

[0060] 3. Mataching: Identifying similar features by comparing descriptors in images. For two images, a pair (Xi, Yi)->(Xi', Yi') can be obtained, where (Xi, Yi) is a feature of one image and (Xi', Yi') is a feature of the other image.

[0061] Scale-Invariant Feature Transform (SIFT): A computer vision algorithm used to detect and describe local features of an image. It finds extreme points in spatial scale and extracts their position, scale, and rotation invariants (features). Therefore, its features are independent of the size and rotation of the image and have a high tolerance for changes in lighting, noise, and slight changes in viewpoint.

[0062] ORB (Oriented FAST and Rotated BRIEF): A fast feature point extraction and description algorithm that combines the accelerated segment test (FAST) detection method with the optimized binary robust independent elementary features (BRIEF) feature descriptor to quickly obtain reliable paired feature points.

[0063] Speeded Up Robust Features (SURF) is a robust local feature point detection and description algorithm that largely improves the feature extraction and description methods based on the SIFT algorithm.

[0064] Random Sampling Consensus (RANSAC): An iterative method for estimating mathematical model parameters from a set of data containing outliers. In computer vision, it is often used to identify erroneous feature point matching pairs.

[0065] Image recognition: A common task in computer vision, it is a technique that uses computers to process, analyze, and understand images in order to identify targets and objects of various patterns.

[0066] Deep Learning (DL): A branch of machine learning, it is an algorithm that uses artificial neural networks as its architecture and attempts to learn representations of data using multiple processing layers with complex structures or multiple nonlinear transformations. Based on the dataset labeling and the corresponding research problem, it can be divided into supervised learning, semi-supervised learning, and unsupervised learning.

[0067] Neural Network (NN): A model inspired by the structure of the brain, consisting of multiple layers (at least one of which is a hidden layer), each layer containing simply connected units or neurons (with non-linear relationships).

[0068] Convolutional Neural Network (CNN): A type of feedforward neural network whose artificial neurons can respond to a portion of the surrounding units within their coverage area, and it performs exceptionally well in large-scale image processing.

[0069] Image warping is a digital image processing procedure that achieves point-to-point mapping of images based on functional mapping relationships. Pure image warping only changes the pixel position, but the mapping result is often a floating-point number, which cannot satisfy the one-to-one mapping result for all pixel positions. Therefore, an interpolation algorithm is needed to satisfy the mapping of each pixel. According to the mapping direction, it can be divided into forward warping (the input image is directly warped to the output image) and backward warping (the output image looks up the corresponding pixel position in the input image in reverse).

[0070] An Inertial Measurement Unit (IMU) is a device that measures the three-axis angular velocity and acceleration of an object. Typically, an IMU contains three single-axis accelerometers and three single-axis gyroscopes. The accelerometers detect the acceleration signals of the object along the three coordinate axes of the carrier's coordinate system, while the gyroscopes detect the angular velocity signals of the carrier relative to the navigation coordinate system. Based on the IMU, it is possible to measure the angular velocity and acceleration of an object in three-dimensional space and use this to calculate the object's attitude.

[0071] Complementary Metal Oxide Semiconductor (CMOS): A commonly used semiconductor device characterized by low power consumption, good anti-interference performance, and a wide tolerance range for power supply voltage fluctuations. It has read / write capabilities and is frequently used in devices such as microcomputers and portable instruments for data storage.

[0072] Rolling shutter: A shutter mode for CMOS sensors. Before exposure begins, the image sensor is zeroed, and then exposures are performed line by line. After each exposure, the image signal value is read out. Because the start and end times of each line exposure are different, distortion may occur when imaging fast-moving objects, but it greatly improves sensor read / write efficiency.

[0073] Global Shutter: A shutter mode for CMOS sensors that resets the image sensor before exposure begins and collects light from all pixels, allowing the entire image to be exposed simultaneously. The more pixels transmitted through the sensor, the slower the camera's overall frame rate.

[0074] Image feature extraction and matching is a crucial task in many computer vision applications, widely used in object detection, image retrieval, and other fields. Whether using traditional methods or recent deep learning-based feature extraction methods, filtering out erroneous feature point pairs is a key challenge in improving matching accuracy. Traditional filtering algorithms are mostly designed based on the understanding of inter-frame motion changes, considering feature regionality during feature point extraction and matching, and then using fixed thresholds to narrow down the set of valid point pairs. For example, the SIFT algorithm requires setting corresponding empirical thresholds in steps such as locating feature points, removing extreme points, and calculating feature point orientation, balancing computational accuracy and efficiency. Similarly, the classic ORB (Oriented FAST and Rotated BRIEF) algorithm uses the FAST (Features from Accelerated Segment Test) algorithm to detect feature points, also requiring a grayscale threshold to detect feature points within the detection range.

[0075] Without considering camera pose and motion, image data is acquired and stably imaged frame by frame, so the preset thresholds in traditional filtering algorithms can be pre-calibrated and applied. However, in real continuous shooting processes, imaging is not stable, and the motion of the image acquisition device itself includes rotational and translational dimensions. In complex scenarios such as fast-moving image acquisition devices or the presence of moving objects, traditional filtering algorithms cannot adaptively adjust the thresholds, making it difficult to guarantee the acquisition of correct feature point matching pairs.

[0076] Similarly, deep learning can complete the extraction step through a feature network, and then determine the spatial feature similarity through a metric network. In computer vision tasks, the industry often uses Convolutional Neural Networks (CNNs) to extract feature maps from complex image preprocessing steps, making them suitable for large-scale data processing tasks. Compared with traditional algorithms, deep learning has a strong self-learning and evolutionary advantage, and its feature extraction descriptions are highly interpretable, leading to its widespread use in everyday specific tasks (such as face recognition). However, deep learning models have limited generalization ability; the accuracy and coverage of the model results are strongly correlated with the training dataset, making it difficult to adaptively adjust to unknown data. Since most datasets currently used to train deep learning models are images collected in static environments, deep learning models trained on these datasets are difficult to apply to complex scenarios such as fast-moving image acquisition devices or the presence of moving objects, meaning they cannot obtain correct feature point matching pairs.

[0077] In view of this, embodiments of this application provide a feature point processing method, which obtains the rotation information of the image acquisition device when acquiring image frames, performs rotation transformation processing on the feature point pairs in the image, so as to eliminate the rotation information between the feature point pairs, fundamentally canceling the motion component of the rotation dimension introduced by the image acquisition device, thereby improving the effectiveness of the threshold filtering feature point pair method based on the translation dimension and ensuring the accuracy of the feature point pairs obtained after filtering.

[0078] Please refer to Figure 1 , Figure 1 This is a schematic diagram illustrating an application scenario of a feature processing method provided in an embodiment of this application. For example... Figure 1 As shown, turning on the video recording device (101) is the starting point of this scene and the foundation for video acquisition. Acquiring video data (102) is the process of the device capturing images and converting light signals into digital signals. Acquiring IMU data (103) is the process of simultaneously acquiring the three-axis rotational angular velocity data of the gyroscope and the three-directional acceleration data of the accelerometer during video acquisition. Video data processing (104) is the process of processing the raw images captured by the video recording device to improve image quality and obtain video-related information; the intermediate process uses the three-axis rotational angular velocity data and acceleration data acquired in step 103. Outputting video (105) outputs the processed video to a designated purpose. The product implementation of the feature processing method provided in this application embodiment can be program code included in video recording processing software and deployed on a video recording terminal. Figure 1 Taking the application scenario shown as an example, the program code provided in this application embodiment mainly functions in the video data processing (104) stage.

[0079] In the video data processing (104) stage, by executing the feature point processing method provided in the embodiments of this application, it is possible to obtain accurately matched feature point pairs, and to perform tasks such as image matching, target detection or video tracking based on the obtained feature point pairs, and finally output the corresponding video data.

[0080] It is worth noting that the above example illustrates the application of the feature point processing method in a video shooting scenario, illustrating the scenarios in which the method provided in this application is applied. In practical applications, the feature point processing method provided in this application can also be applied to other scenarios that require feature point matching, and this application does not specifically limit this application.

[0081] The above describes the application scenarios of the feature point processing method provided in the embodiments of this application. The following will describe the devices to which the feature point processing method is applied.

[0082] Specifically, the feature processing method provided in this application embodiment can be applied to an electronic device, which may be an electronic device with image acquisition function.

[0083] Exemplary examples show that the electronic devices in this application can be smartphones, personal computers (PCs), laptops, tablets, smart TVs, mobile internet devices (MIDs), wearable devices (such as smartwatches, smart glasses, or smart helmets), virtual reality (VR) devices, augmented reality (AR) devices, wireless electronic devices in industrial control, wireless electronic devices in self-driving, wireless electronic devices in remote medical surgery, wireless electronic devices in smart grids, wireless electronic devices in transportation safety, wireless electronic devices in smart cities, and wireless electronic devices in smart homes. The following embodiments do not impose any special limitations on the specific form of the electronic device.

[0084] Please refer to Figure 2 , Figure 2 This is a schematic diagram of the structure of an electronic device 101 provided in an embodiment of this application. Figure 2 As shown, electronic device 101 includes processor 103, which is coupled to system bus 105. Processor 103 can be one or more processors, each of which can include one or more processor cores. A video adapter 107 drives a display 109, which is coupled to system bus 105. System bus 105 is coupled to input / output (I / O) bus via bus bridge 111. I / O interface 115 is coupled to the I / O bus. I / O interface 115 communicates with various I / O devices, such as input device 117 (e.g., touchscreen), external storage 121 (e.g., hard disk, floppy disk, optical disk, or USB flash drive), multimedia interface, etc. A transceiver 123 (capable of sending and / or receiving radio communication signals), a camera 155 (capable of capturing still and moving digital video images), and an external USB port 125. Optionally, the interface connected to I / O interface 115 can be a USB interface.

[0085] The processor 103 can be any conventional processor, including reduced instruction set computing (RISC) processors, complex instruction set computing (CISC) processors, or combinations thereof. Optionally, the processor can be a special-purpose device such as an ASIC.

[0086] Electronic device 101 can communicate with software deployment server 149 via network interface 129. Exemplarily, network interface 129 is a hardware network interface, such as a network interface card (NIC). Network 127 can be an external network, such as the Internet, or an internal network, such as Ethernet or a virtual private network (VPN). Optionally, network 127 can also be a wireless network, such as a WiFi network or a cellular network.

[0087] Hard disk drive interface 131 is coupled to system bus 105. Hardware driver interface is connected to hard disk drive 133. Internal memory 135 is coupled to system bus 105. Data running in internal memory 135 may include operating system (OS) 137, applications 143, and schedules of electronic device 101.

[0088] The processor 103 can communicate with the internal memory 135 via the system bus 105, and retrieve instructions and data from the application program 143 from the internal memory 135 to execute the program.

[0089] The operating system consists of Shell 139 and kernel 141. Shell 139 is an interface between the user and the operating system kernel. Shell 139 is the outermost layer of the operating system. Shell 139 manages the interaction between the user and the operating system: waiting for user input, interpreting user input for the operating system, and handling various operating system outputs.

[0090] Kernel 141 consists of the parts of the operating system used to manage memory, files, peripherals, and system resources. Kernel 141 interacts directly with the hardware. The operating system kernel typically runs processes and provides inter-process communication, CPU time-slice management, interrupts, memory management, I / O management, and so on.

[0091] For example, in the case where the electronic device 101 is a smartphone, the application 143 includes instant messaging related programs. In one embodiment, when the application 143 needs to be executed, the electronic device 101 can download the application 143 from the software deployment server 149.

[0092] Please refer to Figure 3 , Figure 3 This is a flowchart illustrating a feature point processing method provided in an embodiment of this application. Figure 3 As shown, the feature point processing method includes the following steps 301-305.

[0093] Step 301: Perform feature point matching on the first image and the second image to obtain multiple feature point pairs. Each feature point pair includes a feature point in the first image and a feature point in the second image.

[0094] In this embodiment, the first image and the second image are two images for which feature point matching needs to be performed. For example, the first image and the second image are two consecutive image frames captured by the image acquisition device; or, the first image and the second image are two consecutive image frames in a video segment captured by the image acquisition device.

[0095] During the feature point matching process between the first image and the second image, the electronic device can first extract feature points from the first image and the second image, respectively obtaining multiple feature points in the first image and multiple feature points in the second image. For example, the electronic device can extract feature points from the first image and the second image based on feature point extraction algorithms such as ORB or SIFT to obtain the corresponding feature points.

[0096] The electronic device then matches feature points in the first image with feature points in the second image, obtaining multiple feature point pairs. Each feature point pair obtained by the electronic device includes one feature point from the first image and one feature point from the second image. For example, the electronic device can match feature points in the first and second images based on feature point matching algorithms such as BFMatcher or knnMatch.

[0097] Optionally, when multiple original feature point pairs are obtained based on the feature point matching algorithm, the electronic device can perform preliminary filtering on the multiple original feature point pairs based on a feature point pair filtering method to obtain multiple feature point pairs, which are used to perform subsequent steps. For example, the electronic device can perform preliminary filtering on multiple original feature point pairs based on one or more of the standard deviation filtering method, the fixed threshold filtering method, and the random sampling consensus filtering method. When the electronic device uses the above-mentioned multiple feature point pair filtering methods to perform preliminary filtering on the original feature point pairs, the electronic device can use the above-mentioned multiple feature point pair filtering methods serially to filter the original feature point pairs, so that each feature point pair method can filter out a portion of abnormal feature point pairs.

[0098] Because raw feature point pairs are subject to many uncontrollable factors in natural environments, some raw feature point pairs may contain outliers. If these outlier raw feature point pairs are analyzed directly without initial filtering, they may affect the accuracy of subsequent filtering. Furthermore, filtering raw feature point pairs using a feature point pair-based filtering method can minimize the number of feature point pairs, thereby reducing the computational load of subsequent pose transformation and secondary filtering. In other words, minimizing the number of feature point pairs requiring pose transformation effectively improves algorithm performance.

[0099] Step 302: Obtain a first rotation angle and a second rotation angle, wherein the first rotation angle is the rotation angle of the image acquisition device when acquiring the first image, and the second rotation angle is the rotation angle of the image acquisition device when acquiring the second image.

[0100] Understandably, during the process of acquiring the first and second images, the image acquisition device may move. This motion can typically be decomposed into six dimensions: three-dimensional rotation and three-dimensional translation. Simply put, three-dimensional space can be divided into three coordinate axes: the X-axis, the Y-axis, and the Z-axis. The image acquisition device can rotate around any one or more of these axes, or it can translate along any one or more of them. Therefore, the motion of the image acquisition device can be described using three-dimensional rotation and three-dimensional translation.

[0101] In this embodiment of the application, an IMU (Integrated Measurement Unit) may be provided in the image acquisition device used to acquire the first image and the second image. This IMU can measure the triaxial acceleration and triaxial angular velocity of the image acquisition device in real time during the image acquisition process. Based on the triaxial angular velocity acquired by the IMU, the rotation angle of the image acquisition device at different times can be determined, that is, the first rotation angle of the image acquisition device when acquiring the first image and the second rotation angle of the image acquisition device when acquiring the second image can be determined respectively. Both the first and second rotation angles include rotation angles in three directions, i.e., three-dimensional rotation angles.

[0102] Specifically, when an image acquisition device completes the acquisition of all pixels in the entire image at the same time, for example, when the image acquisition device uses a global shutter image sensor to acquire images, the electronic device can determine the rotation angle of the image acquisition device when acquiring each image based on the acquired three-axis angular velocity and the acquisition interval between images.

[0103] For example, the electronic device can acquire the first angular velocity of the image acquisition device when acquiring the first image and the second angular velocity of the image acquisition device when acquiring the second image. Since the image acquisition device typically records the time of acquiring the current image when acquiring each image, and the IMU in the image acquisition device also records the time of acquiring angular velocity data, the first angular velocity corresponding to the first image and the second angular velocity corresponding to the second image can be determined based on the time information of the image acquisition device acquiring the first and second images, and the time information of the IMU in the image acquisition device acquiring the angular velocity data.

[0104] Then, the electronic device determines the first rotation angle based on the rotation angle of the image acquisition device when acquiring the third image, the first angular velocity, and the first time interval, wherein the third image is the frame preceding the first image, and the first time interval is the acquisition interval between the third image and the first image. Specifically, the first angular velocity can be understood as the rotation speed of the image acquisition device during the first time interval between acquiring the third image and the first image. Therefore, based on the first time interval and the first angular velocity, the angle rotated by the image acquisition device from the completion of acquiring the third image to the start of acquiring the first image can be determined. Thus, by adding the angle rotated by the image acquisition device during the first time interval to the rotation angle of the image acquisition device when acquiring the third image, the rotation angle of the image acquisition device when acquiring the first image can be obtained.

[0105] Similarly, the electronic device can determine the second rotation angle based on the rotation angle of the image acquisition device when acquiring the fourth image, the second angular velocity, and the second time interval, wherein the fourth image is the previous frame of the second image, and the second time interval is the acquisition interval between the fourth image and the second image.

[0106] The first image and the fourth image can be the same image, or they can be different images. For ease of explanation, the following will use the first image and the fourth image as the same image.

[0107] Specifically, the rotation angle of the image acquisition device when acquiring a certain image can be determined based on the following formula 1.

[0108] θ t =θ t-1 +h*ω t Formula 1

[0109] Where, θ t θ represents the rotation angle of the image acquisition device at time t. t-1The rotation angle of the image acquisition device at time t-1 is represented by ω; h represents the time interval between time t and time t-1; t This represents the angular velocity at time t. Based on Formula 1, the rotation angles along the three coordinate axes can be determined.

[0110] Furthermore, if the first image is the first frame image captured by the image acquisition device, that is, the image acquisition device did not capture any other images before capturing the first image, then the first rotation angle corresponding to the first image can be determined to be 0.

[0111] It should be noted that the electronic device used to perform the feature point processing method provided in the embodiments of this application and the image acquisition device described above can be the same device or different devices.

[0112] Step 303: Determine a pose transformation matrix based on the first rotation angle and the second rotation angle. The pose transformation matrix is ​​used to eliminate rotation between feature point pairs.

[0113] After determining the first rotation angle corresponding to the first image and the second rotation angle corresponding to the second image, a pose transformation matrix can be determined based on these two rotation angles so that feature points from different images in a feature point pair can be in the same pose.

[0114] The pose transformation matrix can be used to transform the pose of feature points in the first image to the pose of feature points in the second image, that is, to transform the pose of feature points in the first image; or the pose transformation matrix can be used to transform the pose of feature points in the second image to the pose of feature points in the first image, that is, to transform the pose of feature points in the second image.

[0115] For example, the pose transformation matrix can be represented by the following formula 2 or formula 3.

[0116]

[0117] Among them, H n-1 R represents the pose transformation matrix corresponding to the (n-1)th frame of the image; K represents the intrinsic parameter matrix of the image acquisition device; n This represents the rotation angle of the nth frame image; K is the inverse matrix representing the rotation angle of the (n-1)th frame image; -1 This represents the inverse of the intrinsic parameter matrix.

[0118]

[0119] Among them, H n R represents the pose transformation matrix corresponding to the nth frame of the image; K represents the intrinsic parameter matrix of the image acquisition device; n-1This represents the rotation angle of the (n-1)th frame image; K is the inverse matrix representing the rotation angle of the nth frame image; -1 This represents the inverse of the intrinsic parameter matrix.

[0120] Assuming the first image is the (n-1)th frame image mentioned above, and the second image is the nth frame image mentioned above, then the pose transformation matrix corresponding to the first image is the pose transformation matrix shown in Formula 2, and the pose transformation matrix corresponding to the second image is the pose transformation matrix corresponding to Formula 3.

[0121] Step 304: Perform pose transformation on the plurality of feature point pairs according to the pose transformation matrix to obtain a plurality of pose-transformed feature point pairs, wherein each feature point pair in the plurality of pose-transformed feature point pairs is in the same pose in the rotation dimension.

[0122] After obtaining the pose transformation matrix, the electronic device can perform pose transformation on each feature point pair based on the pose transformation matrix to eliminate the rotation between the two feature points in the feature point pair.

[0123] For example, after obtaining the pose transformation matrix corresponding to the first image, the electronic device can perform pose transformation on the feature points belonging to the first image in the feature point pair based on the pose transformation matrix, that is, multiply the pose transformation matrix by the matrix corresponding to the feature point to obtain the pose-transformed feature points. In this way, the feature points belonging to the first image in the feature point pair are in the same pose as the feature points belonging to the second image after pose transformation, thereby eliminating the rotation between feature points.

[0124] Alternatively, after obtaining the pose transformation matrix corresponding to the second image, the electronic device can perform pose transformation on the feature points belonging to the second image in the feature point pair based on the pose transformation matrix. This involves multiplying the pose transformation matrix by the matrix corresponding to the feature point to obtain the pose-transformed feature points. In this way, the feature points belonging to the second image in the feature point pair are in the same pose as the feature points belonging to the first image after pose transformation, thus eliminating the rotation between feature points.

[0125] Step 305: Filter the multiple pose-transformed feature point pairs based on the first translation threshold to obtain multiple filtered feature point pairs.

[0126] In this embodiment, because the interval between two adjacent images is short, even in fast-moving scenarios, the range of motion of the image acquisition device is limited within a short period. Given this limited range of motion, the distance an object can move across two adjacent images acquired by the image acquisition device is also limited.

[0127] In other words, for two matching feature points, the distance between them will not exceed a certain threshold. For example, suppose there is feature point 1 in the first image and feature point 2 in the second image, and feature point 1 matches feature point 2. During the time interval between the acquisition of the first image and the acquisition of the second image, the image acquisition device moves, causing the position of the same object to change in the two images, that is, the positions of the two matching feature points change in the first and second images. Since the range of motion of the image acquisition device is limited, the distance between feature point 1 and feature point 2 is also limited, for example, the distance between feature point 1 and feature point 2 will not exceed 100 pixels. Therefore, in this embodiment, a certain distance threshold can be used to filter feature point pairs, considering feature point pairs with excessively large distances as abnormal feature point pairs, thereby filtering out some abnormal feature point pairs.

[0128] The first translation threshold can be a pre-set threshold, adaptable to most motion scenarios and scenarios with moving objects. In the process of filtering feature point pairs based on the first translation threshold, for example, feature point pairs whose distance is greater than the first translation threshold can be filtered. In this case, the distance between feature points in each of the multiple pose-transformed feature point pairs can be calculated first. That is, the distance between the two feature points in a feature point pair is determined based on their positions in the first and second images, respectively. After obtaining the distance between the feature points in each feature point pair, the distance between the feature points is compared with the first translation threshold. If the distance between the feature points is greater than the first translation threshold, the corresponding feature point pair is filtered, ultimately resulting in multiple filtered feature point pairs.

[0129] In addition, in some scenarios, when it is necessary to extract moving objects in an image, feature point pairs whose distance between feature points is less than the first translation threshold can be filtered out.

[0130] It should be noted that in some cases, the feature point pair matching process needs to output the original feature point pairs, while the filtered feature point pairs in this embodiment are feature point pairs after pose transformation. Therefore, the electronic device can find multiple corresponding feature point pairs from the initially filtered feature point pairs based on multiple filtered feature point pairs, and output the found multiple feature point pairs.

[0131] In this scheme, by acquiring the rotation information of the image acquisition device when acquiring image frames, the feature point pairs in the image are subjected to rotation transformation processing to eliminate the rotation information between the feature point pairs. This fundamentally cancels out the motion component of the rotation dimension introduced by the image acquisition device, thereby improving the effectiveness of the threshold filtering feature point pair method based on the translation dimension and ensuring the accuracy of the filtered feature point pairs.

[0132] Furthermore, regarding the motion information affecting the distance between feature point pairs in the image acquisition device, this motion information includes three-dimensional rotation components and three-dimensional translation components. The three-dimensional rotation components are the motion components that primarily influence the distance between feature point pairs. Therefore, in this embodiment, by eliminating the rotation information of feature point pairs, most of the motion of the image acquisition device can be offset, ensuring accurate filtering of feature point pairs in complex motion scenes.

[0133] As can be understood, the above describes the pose transformation process for multiple feature point pairs when all pixels in an image are acquired at the same time. When all pixels in an image are acquired at the same time, the acquisition time for feature points belonging to the same image is the same; therefore, the rotation angle of all feature points in the same image is the same.

[0134] However, when all pixels in an image are not acquired at the same time, feature points within the same image may have the same acquisition time. Therefore, the rotation angles of different feature points in the same image may be different. In practical applications, it is necessary to first determine the rotation angle corresponding to each feature point in order to achieve pose transformation of feature point pairs.

[0135] Specifically, when an image acquisition device uses a global shutter image sensor, all pixels in the image acquired by the device are acquired at the same time; when an image acquisition device uses a rolling shutter image sensor, all pixels in the image acquired by the device are not acquired at the same time.

[0136] Global shutter image sensors employ a global shutter control method. When the shutter opens, all photosensitive units on the global shutter image sensor begin exposure simultaneously; when the shutter closes, all photosensitive units on the global shutter image sensor cease exposure simultaneously. The global shutter image sensor has a built-in buffer that stores the signals collected by the photosensitive units after exposure, facilitating continuous signal readout. Therefore, global shutter image sensors support simultaneous start and end of exposure for all photosensitive units. The photosensitive units are used to convert light signals into electrical signals during exposure to record light signal information. For example, the photosensitive units in an image sensor can be photoelectric sensors. Photosensitive unit exposure refers to light shining onto the photosensitive unit, causing the photosensitive unit to convert the light signal into an electrical signal.

[0137] Rolling shutter image sensors employ a rolling shutter control method. After the shutter opens, the photosensitive units on the rolling shutter image sensor are exposed row by row, and the signal acquired by each row is continuously read out after the exposure of each row is complete. For rolling shutter image sensors, the start and end points of the exposure time for different rows of photosensitive units are different, but the exposure time is the same.

[0138] For details, please refer to Figure 4 , Figure 4 This is a schematic diagram illustrating the exposure of a rolling shutter image sensor and a global shutter image sensor. (Example) Figure 4 As shown in (a), the rolling shutter image sensor uses a line-by-line exposure method. The exposure time for the first row of photosensitive units is [t0, t1]. The start and end points of the exposure time for each subsequent row of photosensitive units are different. The time difference between the end times of exposure for two adjacent rows of photosensitive units is exactly the time required to read out the signal from one row. That is, the exposure of the next row of photosensitive units ends just as the signal acquired by the previous row is read out, allowing for the continued reading of the signal acquired by the next row. Because the exposure time periods for photosensitive units in different rows are different, the rotation angles corresponding to pixels in different rows are different in images captured using a rolling shutter image sensor.

[0139] like Figure 4 As shown in (b), the global shutter image sensor uses a method where all photosensitive units start and end exposure simultaneously, with the exposure time period for each row of photosensitive units being [t0, t1]. Since the exposure time period for all photosensitive units is the same, the rotation angle corresponding to all pixels in the image captured using the global shutter image sensor is the same.

[0140] Based on this, when the image acquisition device uses a rolling shutter image sensor, the electronic device can acquire the rotation angle of each feature point in the first image and the second image, so as to facilitate subsequent pose transformation of the feature point pairs.

[0141] For example, for the first rotation angle corresponding to the first image acquired by the electronic device, the first rotation angle includes a plurality of first sub-rotation angles, which are the rotation angles when the image acquisition device acquires feature points belonging to the first image among the plurality of feature point pairs.

[0142] The second rotation angle corresponding to the second image acquired by the electronic device includes multiple second sub-rotation angles, which are the rotation angles when the image acquisition device acquires feature points belonging to the second image among the multiple feature point pairs.

[0143] Specifically, since the pixels in the first and second images are acquired through line-by-line exposure, the electronic device can obtain the exposure time of each line in the first and second images, thereby determining the acquisition time corresponding to each row of pixels. Thus, by determining the exposure line to which the feature points in the first and second images belong, the electronic device can determine the acquisition time corresponding to the feature points in the first and second images, thereby determining the angular velocity of each feature point at its corresponding acquisition time. Then, the electronic device can determine the rotation angle corresponding to each feature point based on Formula 1 above, thereby obtaining the aforementioned multiple first sub-rotation angles and multiple second sub-rotation angles.

[0144] Since multiple feature points in the first image may have different rotation angles, and multiple feature points in the second image may also have different rotation angles, the electronic device can select a reference point in one of the images and convert multiple feature points in the first image and multiple feature points in the second image into poses corresponding to the reference point, thereby uniformly eliminating the rotation between feature points in all feature point pairs.

[0145] For example, the electronic device can acquire a reference rotation angle, which may be, for example, the rotation angle of a reference point in the first image or the second image. For instance, the reference point may be the center point in the first image or the center point in the second image. Furthermore, the reference point may also be a point in other images; this embodiment does not specifically limit the reference point.

[0146] Based on the plurality of first sub-rotation angles and the reference rotation angle, a pose transformation matrix is ​​determined for the feature points belonging to the first image in the plurality of feature point pairs. This pose transformation matrix is ​​used to transform the pose of the feature points belonging to the first image to the pose corresponding to the reference point.

[0147] Based on the plurality of second sub-rotation angles and the reference rotation angle, a pose transformation matrix is ​​determined for the feature points belonging to the second image in the plurality of feature point pairs. This pose transformation matrix is ​​used to transform the pose of the feature points belonging to the second image to the pose corresponding to the reference point.

[0148] For example, see Figure 5a , Figure 5a This is a schematic diagram of pose transformation provided for an embodiment of this application. Figure 5aAs shown, when the center point of the nth frame image (i.e., the second image) is selected as the reference point, the pose transformation matrix corresponding to the feature points in the (n-1)th frame image (i.e., the first image) can be determined based on the rotation angles of the feature points in the nth frame image and the rotation angle of the center point of the nth frame image. Similarly, the pose transformation matrix corresponding to the feature points in the nth frame image can also be determined based on the rotation angles of the feature points in the nth frame image and the rotation angle of the center point of the nth frame image.

[0149] Specifically, the process of obtaining the pose transformation matrix corresponding to the feature points in the first image and the pose transformation matrix corresponding to the feature points in the second image can be represented by the following formulas 4 and 5.

[0150]

[0151] Among them, H n-1 R represents the pose transformation matrix corresponding to the (n-1)th frame of the image; K represents the intrinsic parameter matrix of the image acquisition device; n_center This represents the rotation angle of the center point of the nth frame image; K is the inverse matrix representing the rotation angle of the (n-1)th frame image; -1 This represents the inverse of the intrinsic parameter matrix.

[0152]

[0153] Among them, H n R represents the pose transformation matrix corresponding to the nth frame of the image; K represents the intrinsic parameter matrix of the image acquisition device; n_center This represents the rotation angle of the center point of the nth frame image; K is the inverse matrix representing the rotation angle of the nth frame image; -1 This represents the inverse of the intrinsic parameter matrix.

[0154] Assuming the first image is the (n-1)th frame image mentioned above, and the second image is the nth frame image mentioned above, then the pose transformation matrix corresponding to the feature points in the first image is the pose transformation matrix shown in Formula 4, and the pose transformation matrix corresponding to the feature points in the second image is the pose transformation matrix corresponding to Formula 5.

[0155] Furthermore, the process of pose transformation of feature points can be represented by the following formula 6 or formula 7.

[0156] P′ n-1 =H n-1 *P n-1 Formula 6

[0157] P′ n =H n *P n Formula 7

[0158] Among them, P n-1 Let P be a feature point on the (n-1)th frame of the image. n P′ is a feature point on the nth frame of the image. n-1 P′ represents the feature point on the (n-1)th frame of the image after pose transformation. n P′ represents the feature point on the nth frame image after pose transformation. n-1 and P′ n Located in the same rotational pose, the rotation between feature points has been eliminated.

[0159] Since the range of motion of the same image acquisition device may vary at different times, meaning that the distance the feature points move in different images acquired by the same device will also vary, if a fixed threshold is always used to filter the feature point pairs, there may be insufficient or incorrect filtering.

[0160] Based on this, the embodiments of this application provide multiple filtering thresholds. In practical applications, the corresponding filtering threshold can be selected based on the intensity of motion of the image acquisition device, thereby dynamically adjusting the filtering threshold based on the motion state of the image acquisition device and improving the filtering accuracy of feature point pairs.

[0161] Optionally, the first translation threshold is related to the motion state of the image acquisition device when acquiring images related to the second image; wherein, the images related to the second image include the second image, the first N frames of the second image, and the last M frames of the second image, where N and M are both integers greater than or equal to 0. Specifically, if the motion intensity of the image acquisition device is high when acquiring images related to the second image, the first translation threshold can be a higher value; if the motion intensity of the image acquisition device is low when acquiring images related to the second image, the first translation threshold can be a lower value.

[0162] Please refer to Figure 5b , Figure 5b This is a schematic diagram of an image sequence provided in an embodiment of this application. For example... Figure 5bAs shown, the image sequence includes N+M+1 images. For the nth frame, the images associated with the nth frame are these N+M+1 images, namely, the nth frame itself, the N frames preceding the nth frame, and the M frames following the nth frame. Here, N is an integer greater than or equal to 0, and M can also be an integer greater than or equal to 0. For example, when both N and M are 0, the images associated with the nth frame are the nth frame itself; when N is 1 and M is 0, the images associated with the nth frame are the nth frame itself and the (n-1)th frame; when both N and M are greater than 0, the images associated with the nth frame are the nth frame itself, the N frames preceding the nth frame, and the M frames following the nth frame. Figure 5b The nth frame image can be the second image mentioned above. Figure 5b The (n-1)th frame image can be the first image mentioned above.

[0163] For example, the electronic device can acquire the three-axis rotational angular velocity of the image acquisition device when acquiring an image related to the second image and the first motion degree value of the image acquisition device when acquiring the first image; wherein, the first motion degree value is related to the motion state of the image acquisition device when acquiring an image related to the first image.

[0164] Based on the three-axis rotational angular velocity and the first motion degree value, a second motion degree value of the image acquisition device in the second image is determined. The second motion degree value represents the intensity of motion of the image acquisition device when acquiring the second image. A larger second motion degree value indicates a higher intensity of motion of the image acquisition device when acquiring the second image; a smaller second motion degree value indicates a lower intensity of motion of the image acquisition device when acquiring the second image. Furthermore, since the motion of the image acquisition device is usually continuous, i.e., the motion state of the image acquisition device is not usually abrupt, the first motion degree value of the preceding frame image (i.e., the first image) is also considered when determining the second motion degree value corresponding to the second image.

[0165] For example, if the images associated with the second image only include the second image, the three-axis rotational angular velocity corresponding to the second image can be added to the first motion value to obtain the second motion value.

[0166] For example, if the images associated with the second image include multiple images, after obtaining the three-axis rotational angular velocities corresponding to each of the multiple images, the average value of the three-axis rotational angular velocities corresponding to the multiple images is calculated. Then, the difference between the average value and the three-axis rotational angular velocity of the second image is used to obtain the deviation value. Finally, the deviation value is added to the first motion degree value to obtain the second motion degree value.

[0167] For example, when the images associated with the second image include multiple images, after obtaining the three-axis rotational angular velocities corresponding to each of the multiple images, a corresponding weight is assigned to each of the multiple images. The three-axis rotational angular velocities of each of the multiple images are multiplied by their respective weights and then summed to obtain the median value of the three-dimensional rotational angular velocity. The difference between the median value of the three-dimensional rotational angular velocity and the three-axis rotational angular velocity of the second image is then used to obtain a deviation value. Finally, this deviation value is added to the first motion degree value to obtain the second motion degree value. The weights corresponding to each of the multiple images can be set according to actual conditions, and this embodiment is not limited to this. For example, images closer to the second image have a larger weight, and images farther away from the second image have a smaller weight.

[0168] Finally, a first translation threshold corresponding to the second motion intensity value is determined from a set of preset thresholds. In other words, multiple thresholds can be preset for filtering, with different thresholds corresponding to different levels of motion intensity. Since the second motion intensity value can represent the motion intensity of the image acquisition device, a corresponding threshold can be selected from among the multiple thresholds based on the second motion intensity value. Generally, the larger the second motion intensity value, the larger the threshold corresponding to it; conversely, the smaller the second motion intensity value, the smaller the threshold corresponding to it.

[0169] For example, the preset thresholds may include 50, 100, 200, and 500. When the second motion level value is less than 5, the first translation threshold corresponding to the second motion level value is determined to be 50; when the second motion level value is greater than or equal to 5 and less than 10, the first translation threshold corresponding to the second motion level value is determined to be 100; when the second motion level value is less than or equal to 10 and less than 15, the first translation threshold corresponding to the second motion level value is determined to be 200; and when the second motion level value is greater than or equal to 15 and less than 20, the first translation threshold corresponding to the second motion level value is determined to be 500.

[0170] It is understood that in practical applications, the number of preset thresholds can be determined according to the actual situation, and this embodiment does not impose a specific limitation on this. In addition, the electronic device can also establish a mapping relationship between motion level values ​​and thresholds, and after obtaining the second motion level value, determine the threshold corresponding to the second motion level value.

[0171] In some embodiments, the three-axis rotational angular velocity and three-axis acceleration of the image acquisition device can be considered simultaneously to determine the degree of motion of the image acquisition device.

[0172] For example, the electronic device may also be the device that acquires the three-axis rotational acceleration of the image acquisition device when acquiring an image related to the second image.

[0173] Then, the electronic device determines the second motion value of the image acquisition device in the second image based on the triaxial acceleration, the triaxial rotational angular velocity, and the first motion degree value.

[0174] For example, the electronic device may determine the second degree of motion value based on the following formulas 8-10.

[0175] V_R n =a1*V_R n-1 +b1*R_S n Formula 8

[0176] V_T n =a2*V_T n-1 +b2*T_S n Formula 9

[0177] V n =a3*V_R n +b3*V_T n Formula 10

[0178] Where a1, a2, a3, b1, b2, b3 are coefficients; V_R n V_R represents the degree of drastic rotation in the image acquisition device when acquiring the nth frame; n-1 R_S represents the degree of drastic rotation in the image acquisition device when acquiring the (n-1)th frame; n V_T represents the sum of the three-axis rotational angular velocities of the image acquisition device when acquiring the nth frame of the image; n V_T represents the degree of drastic translation in the image acquisition device when acquiring the nth frame; n-1 T_S represents the degree of drastic translation in the image acquisition device when acquiring the (n-1)th frame; n V represents the sum of the three-axis accelerations of the image acquisition device when acquiring the nth frame of the image; n This indicates the degree of motion intensity of the image acquisition device when acquiring the nth frame image, which is the second motion intensity value mentioned above.

[0179] In addition, R_S n It can also represent the deviation between the average three-axis rotational angular velocity of the image associated with the nth frame and the three-axis rotational angular velocity of the nth frame; T_S n It can also represent the deviation between the average triaxial acceleration of the image associated with the nth frame and the triaxial acceleration of the nth frame. R_S nIt can also represent the deviation between the weighted average of the three-axis rotational angular velocities corresponding to the image associated with the nth frame and the three-axis rotational angular velocity of the nth frame; T_S n Alternatively, it can represent the deviation between the weighted average of the triaxial accelerations corresponding to the images associated with the nth frame and the triaxial accelerations of the nth frame image.

[0180] To facilitate understanding, the feature point processing method provided in the embodiments of this application will be described in detail below with specific examples.

[0181] Please refer to Figure 6 , Figure 6 This is a flowchart illustrating the feature point processing method provided in an embodiment of this application. Figure 6 As shown, the feature point processing method includes the following steps 601-608.

[0182] Step 601, Feature Extraction.

[0183] First, the electronic device acquires an input image sequence, which is the image sequence for which feature point matching needs to be performed. Then, the electronic device extracts feature points from each image in the image sequence based on a feature point extraction algorithm, obtaining the original feature points in each image of the image sequence.

[0184] Step 602, Feature matching.

[0185] After obtaining the original feature points in the image, the electronic device performs feature point matching on the original feature points in two adjacent images to obtain multiple pairs of original feature points.

[0186] Step 603: Preliminary filtering of feature point pairs.

[0187] For the multiple original feature point pairs obtained, the electronic device performs preliminary filtering on the multiple original feature point pairs based on one or more feature point pair filtering algorithms, such as standard deviation filtering method, fixed threshold filtering method and random sampling consistency filtering method, to obtain the preliminary filtered feature point pairs.

[0188] Step 604, 3D rotation data calculation.

[0189] In addition, the electronic device acquires IMU data from the image acquisition device during the acquisition of the aforementioned image sequence, and determines the three-dimensional rotation angle corresponding to the feature points of each image in the image sequence based on the three-dimensional angular velocity data in the IMU data and the acquisition time information of the image sequence.

[0190] Step 605, 3D pose transformation calculation.

[0191] After obtaining the three-dimensional rotation angles corresponding to each feature point in the image, the electronic device determines the pose transformation matrix corresponding to each feature point in the image based on the image frame rate, image information, and camera information related to the image sequence.

[0192] Specifically, the image frame rate indicates the frequency at which the image acquisition device acquires the image sequence. Therefore, based on the image frame rate, the electronic device can determine the acquisition time corresponding to each image in the image sequence. Image information indicates the size of the image and the position of each feature point in the image. Camera information indicates the image sensor used by the image acquisition device to acquire the image sequence. For example, camera information indicates that the image acquisition device uses a rolling shutter image sensor, and the number of exposure rows of the rolling shutter image sensor is N. Therefore, based on the image information and camera information, the electronic device can determine the acquisition time corresponding to each feature point in the image sequence. Combining the three-dimensional rotation data from step 604, the rotation angle corresponding to each feature point can be determined, thereby further determining the pose transformation matrix corresponding to each feature point.

[0193] Step 606, feature point pose transformation.

[0194] Based on the pose transformation matrix corresponding to the feature points obtained in step 605, the electronic device performs pose transformation on the feature points in the feature point pair to obtain the pose-transformed feature point pair.

[0195] Step 607, translation threshold evaluation.

[0196] Based on the IMU data of the image acquisition device during the acquisition of image sequences, the electronic device can assess the intensity of motion of the image acquisition device when acquiring images, thereby obtaining the translation threshold corresponding to each image. This translation threshold is used to represent the intensity of motion of the image acquisition device when acquiring images.

[0197] Step 608, feature point pair filtering.

[0198] Finally, the electronic device filters the feature point pairs after pose transformation based on the translation threshold, filtering out feature point pairs whose distance between feature points is greater than the translation threshold, thus obtaining filtered feature point pairs.

[0199] The foregoing primarily describes the solutions provided in the embodiments of this application from a methodological perspective. It is understood that, in order to achieve the aforementioned functions, the model training device includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the modules and algorithm steps of the various examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0200] The apparatus used to perform the above method embodiments in the present application will now be described in detail.

[0201] Please refer to Figure 7 , Figure 7 This is a schematic diagram of the structure of a feature point processing device 700 provided in an embodiment of this application. Figure 7 As shown, the feature point processing device 700 includes: a processing module 701, configured to perform feature point matching on a first image and a second image to obtain multiple feature point pairs, each of the multiple feature point pairs including a feature point in the first image and a feature point in the second image; an acquisition module 702, configured to acquire a first rotation angle and a second rotation angle, the first rotation angle being the rotation angle of the image acquisition device when acquiring the first image, and the second rotation angle being the rotation angle of the image acquisition device when acquiring the second image; the processing module 701 is further configured to determine a pose transformation matrix based on the first rotation angle and the second rotation angle, the pose transformation matrix being used to eliminate rotation between feature point pairs; the processing module 701 is further configured to perform pose transformation on the multiple feature point pairs according to the pose transformation matrix to obtain multiple pose-transformed feature point pairs, each of the multiple pose-transformed feature point pairs being in the same pose in the rotation dimension; the processing module 701 is further configured to filter the multiple pose-transformed feature point pairs based on a first translation threshold to obtain multiple filtered feature point pairs.

[0202] In one possible implementation, the first rotation angle includes multiple first sub-rotation angles, each of which is a rotation angle when the image acquisition device acquires feature points belonging to the first image from the multiple feature point pairs. The second rotation angle includes multiple second sub-rotation angles, each of which is a rotation angle when the image acquisition device acquires feature points belonging to the second image from the multiple feature point pairs. The processing unit is specifically configured to: obtain a reference rotation angle; determine the pose transformation matrix corresponding to the feature points belonging to the first image from the multiple feature point pairs based on the multiple first sub-rotation angles and the reference rotation angle; and determine the pose transformation matrix corresponding to the feature points belonging to the second image from the multiple feature point pairs based on the multiple second sub-rotation angles and the reference rotation angle.

[0203] In one possible implementation, the acquisition module 702 is further configured to acquire a first angular velocity of the image acquisition device when acquiring the first image and a second angular velocity of the image acquisition device when acquiring the second image; the processing module 701 is further configured to determine a first rotation angle based on the rotation angle of the image acquisition device when acquiring a third image, the first angular velocity, and a first time interval, wherein the third image is the previous frame of the first image, and the first time interval is the acquisition interval between the third image and the first image; the processing module 701 is further configured to determine a second rotation angle based on the rotation angle of the image acquisition device when acquiring a fourth image, the second angular velocity, and a second time interval, wherein the fourth image is the previous frame of the second image, and the second time interval is the acquisition interval between the fourth image and the second image.

[0204] In one possible implementation, the processing module 701 is further configured to: obtain the intrinsic parameter matrix of the image acquisition device; and determine the pose transformation matrix based on the intrinsic parameter matrix, the first rotation angle, and the second rotation angle.

[0205] In one possible implementation, the first translation threshold is related to the motion state of the image acquisition device when acquiring images associated with the second image; wherein, the images associated with the second image include the second image, the first N frames of the second image, and the last M frames of the second image, where N and M are both integers greater than or equal to 0.

[0206] In one possible implementation, the acquisition module 702 is further configured to acquire the three-axis rotational angular velocity of the image acquisition device when acquiring an image related to the second image and the first motion degree value of the image acquisition device when acquiring the first image; the processing module 701 is further configured to determine a second motion degree value of the image acquisition device in the second image based on the three-axis rotational angular velocity and the first motion degree value; the processing module 701 is further configured to determine a first translation threshold corresponding to the second motion degree value among multiple thresholds.

[0207] In one possible implementation, the acquisition module 702 is further configured to acquire the triaxial acceleration of the image acquisition device when acquiring an image related to the second image; the processing module 701 is further configured to determine a second motion degree value of the image acquisition device when acquiring the second image based on the triaxial acceleration, the triaxial rotational angular velocity and the first motion degree value.

[0208] In one possible implementation, the processing module 701 is further configured to: extract feature points from the first image and the second image to obtain multiple feature points; match the multiple feature points to obtain multiple pairs of original feature points; filter the multiple pairs of original feature points according to a feature point pair filtering method to obtain the multiple feature point pairs; wherein the feature point pair filtering method includes one or more of a standard deviation filtering method, a fixed threshold filtering method, and a random sampling consistency filtering method.

[0209] The following describes an electronic device provided by an embodiment of this application. Please refer to [link / reference]. Figure 8 , Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 800 can specifically be a mobile phone, tablet, laptop, smart wearable device, server, etc., and is not limited thereto. The electronic device 800 may be equipped with... Figure 7 The feature point processing device described in the corresponding embodiment is used to implement Figure 7 The corresponding embodiment describes the feature point processing function. Specifically, the electronic device 800 includes: a receiver 801, a transmitter 802, a processor 803, and a memory 804 (wherein the electronic device 800 may have one or more processors 803). Figure 8 (Taking a processor as an example), the processor 803 may include an application processor 8031 ​​and a communication processor 8032. In some embodiments of this application, the receiver 801, transmitter 802, processor 803, and memory 804 may be connected via a bus or other means.

[0210] Memory 804 may include read-only memory and random access memory, and provides instructions and data to processor 803. A portion of memory 804 may also include non-volatile random access memory (NVRAM). Memory 804 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0211] The processor 803 controls the operation of electronic devices. In specific applications, the various components of electronic devices are coupled together through a bus system. This bus system includes not only data buses but also power buses, control buses, and status signal buses. However, for clarity, all buses in the diagram are referred to as a bus system.

[0212] The methods disclosed in the embodiments of this application can be applied to or implemented by processor 803. Processor 803 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above methods can be completed by integrated logic circuits in the hardware of processor 803 or by instructions in software form. Processor 803 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and may further include application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Processor 803 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 804, and processor 803 reads the information from memory 804 and, in conjunction with its hardware, completes the steps of the above method.

[0213] Receiver 801 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of electronic devices. Transmitter 802 can be used to output digital or character information through the first interface; transmitter 802 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; transmitter 802 may also include a display device such as a display screen.

[0214] Please refer to Figure 9 This application also provides a computer-readable storage medium, in some embodiments of which the above-described... Figure 3 The disclosed method can be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or articles of art.

[0215] Figure 9 A conceptual partial view of an example computer-readable storage medium arranged according to at least some of the embodiments shown herein is illustrated schematically. The example computer-readable storage medium includes a computer program for executing computer processes on a computing device.

[0216] In one embodiment, the computer-readable storage medium 900 is provided using a signal carrying medium 901. The signal carrying medium 901 may include one or more program instructions 902, which, when executed by one or more processors, can provide the functions or parts thereof described above with reference to FIG. 5. Furthermore, Figure 9 The program instruction 902 in the document also describes example instructions.

[0217] In some examples, the signal carrying medium 901 may include a computer-readable medium 903, such as, but not limited to, a hard disk drive, a compact disc (CD), a digital video optical disc (DVD), a digital magnetic tape, a memory, ROM, or RAM, etc.

[0218] In some embodiments, the signal-bearing medium 901 may comprise a computer-recordable medium 904, such as, but not limited to, a memory, a read / write (R / W) CD, a R / W DVD, and so on. In some embodiments, the signal-bearing medium 901 may comprise a communication medium 905, such as, but not limited to, digital and / or analog communication media (e.g., fiber optic cables, waveguides, wired communication links, wireless communication links, and so on). Therefore, for example, the signal-bearing medium 901 may be transmitted by a wireless form of communication medium 905 (e.g., a wireless communication medium conforming to the IEEE 802 standard or other transmission protocols).

[0219] One or more program instructions 902 may be, for example, computer-executable instructions or logical implementation instructions. In some examples, the computing device may be configured to provide various operations, functions, or actions in response to one or more program instructions 902 conveyed to the computing device via a computer-readable medium 903, a computer-recordable medium 904, and / or a communication medium 905.

[0220] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0221] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device to execute the methods described in the various embodiments of this application.

[0222] In the above embodiments, the implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of a computer program product.

[0223] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, communication device, computing device, or data center to another website, computer, communication device, computing device, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a communication device or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0224] It should be understood that the phrase "an embodiment" or "one embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in one embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence number of the above-described processes does not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0225] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

Claims

1. A feature point processing method, characterized in that, include: Feature point matching is performed on the first image and the second image to obtain multiple feature point pairs. Each feature point pair includes a feature point in the first image and a feature point in the second image. Obtain a first rotation angle and a second rotation angle, wherein the first rotation angle is the rotation angle of the image acquisition device when acquiring the first image, and the second rotation angle is the rotation angle of the image acquisition device when acquiring the second image; A pose transformation matrix is ​​determined based on the intrinsic parameter matrix of the image acquisition device, the first rotation angle, and the second rotation angle. The pose transformation matrix is ​​used to eliminate rotation between feature point pairs. The pose transformation is performed on the multiple feature point pairs according to the pose transformation matrix to obtain multiple pose-transformed feature point pairs. Each feature point pair in the multiple pose-transformed feature point pairs is in the same pose in the rotation dimension. Based on the first translation threshold, the multiple pose-transformed feature point pairs are filtered to obtain multiple filtered feature point pairs.

2. The method according to claim 1, characterized in that, The first rotation angle includes multiple first sub-rotation angles, each of which is the rotation angle when the image acquisition device acquires feature points belonging to the first image in the multiple feature point pairs. The second rotation angle includes multiple second sub-rotation angles, each of which is the rotation angle when the image acquisition device acquires feature points belonging to the second image in the multiple feature point pairs. The step of determining the pose transformation matrix based on the first rotation angle and the second rotation angle includes: Obtain the reference rotation angle; Based on the plurality of first sub-rotation angles and the reference rotation angle, determine the pose transformation matrix corresponding to the feature points belonging to the first image in the plurality of feature point pairs; Based on the plurality of second sub-rotation angles and the reference rotation angle, the pose transformation matrix corresponding to the feature points belonging to the second image in the plurality of feature point pairs is determined.

3. The method according to claim 1, characterized in that, The process of obtaining the first rotation angle and the second rotation angle includes: The first angular velocity of the image acquisition device when acquiring the first image and the second angular velocity of the image acquisition device when acquiring the second image are obtained; The first rotation angle is determined based on the rotation angle of the image acquisition device when acquiring the third image, the first angular velocity, and the first time interval, wherein the third image is the previous frame of the first image, and the first time interval is the acquisition interval between the third image and the first image. The second rotation angle is determined based on the rotation angle of the image acquisition device when acquiring the fourth image, the second angular velocity, and the second time interval, wherein the fourth image is the previous frame of the second image, and the second time interval is the acquisition interval between the fourth image and the second image.

4. The method according to any one of claims 1-3, characterized in that, The first translation threshold is related to the motion state of the image acquisition device when acquiring an image related to the second image; The images associated with the second image include the second image, the first N frames of the second image, and the last M frames of the second image, where N and M are both integers greater than or equal to 0.

5. The method according to claim 4, characterized in that, The method further includes: The three-axis rotational angular velocity of the image acquisition device when acquiring an image related to the second image and the first degree of motion value of the image acquisition device when acquiring the first image are obtained; Based on the three-axis rotational angular velocity and the first motion degree value, the second motion degree value of the image acquisition device in the second image is determined; A first translation threshold corresponding to the second motion level value is determined from multiple thresholds.

6. The method according to claim 5, characterized in that, The method further includes: The triaxial acceleration of the image acquisition device when acquiring an image related to the second image is obtained; The step of determining the second motion value of the image acquisition device in the second image based on the triaxial acceleration and the first motion value includes: Based on the triaxial acceleration, the triaxial rotational angular velocity, and the first motion degree value, the second motion degree value of the image acquisition device in the second image is determined.

7. The method according to any one of claims 1-3, characterized in that, The feature point matching of the first image and the second image yields multiple feature point pairs, including: Feature points are extracted from the first image and the second image to obtain multiple feature points; The multiple feature points are matched to obtain multiple pairs of original feature point pairs; The multiple pairs of original feature points are filtered according to the feature point pair filtering method to obtain the multiple feature point pairs; The feature point pair filtering method includes one or more of the following: standard deviation filtering method, fixed threshold filtering method, and random sampling consistency filtering method.

8. A feature point processing device, characterized in that, include: The processing module is used to perform feature point matching on the first image and the second image to obtain multiple feature point pairs, wherein each feature point pair includes a feature point in the first image and a feature point in the second image; The acquisition module is used to acquire a first rotation angle and a second rotation angle, wherein the first rotation angle is the rotation angle of the image acquisition device when acquiring the first image, and the second rotation angle is the rotation angle of the image acquisition device when acquiring the second image; The processing module is further configured to determine a pose transformation matrix based on the intrinsic parameter matrix of the image acquisition device, the first rotation angle, and the second rotation angle, wherein the pose transformation matrix is ​​used to eliminate rotation between feature point pairs; The processing module is further configured to perform pose transformation on the plurality of feature point pairs according to the pose transformation matrix to obtain a plurality of pose-transformed feature point pairs, wherein each feature point pair in the plurality of pose-transformed feature point pairs is in the same pose in the rotation dimension. The processing module is further configured to filter the multiple pose-transformed feature point pairs based on a first translation threshold to obtain multiple filtered feature point pairs.

9. The apparatus according to claim 8, characterized in that, The first rotation angle includes multiple first sub-rotation angles, each of which is the rotation angle when the image acquisition device acquires feature points belonging to the first image in the multiple feature point pairs. The second rotation angle includes multiple second sub-rotation angles, each of which is the rotation angle when the image acquisition device acquires feature points belonging to the second image in the multiple feature point pairs. The processing module is specifically used to: obtain a reference rotation angle; Based on the plurality of first sub-rotation angles and the reference rotation angle, determine the pose transformation matrix corresponding to the feature points belonging to the first image in the plurality of feature point pairs; Based on the plurality of second sub-rotation angles and the reference rotation angle, the pose transformation matrix corresponding to the feature points belonging to the second image in the plurality of feature point pairs is determined.

10. The apparatus according to claim 8, characterized in that, The acquisition module is further configured to acquire the first angular velocity of the image acquisition device when acquiring the first image and the second angular velocity of the image acquisition device when acquiring the second image; The processing module is further configured to determine the first rotation angle based on the rotation angle of the image acquisition device when acquiring the third image, the first angular velocity, and the first time interval, wherein the third image is the previous frame image of the first image, and the first time interval is the acquisition interval duration between the third image and the first image. The processing module is further configured to determine the second rotation angle based on the rotation angle of the image acquisition device when acquiring the fourth image, the second angular velocity, and the second time interval, wherein the fourth image is the previous frame image of the second image, and the second time interval is the acquisition interval duration between the fourth image and the second image.

11. The apparatus according to any one of claims 8-10, characterized in that, The first translation threshold is related to the motion state of the image acquisition device when acquiring an image related to the second image; The images associated with the second image include the second image, the first N frames of the second image, and the last M frames of the second image, where N and M are both integers greater than or equal to 0.

12. The apparatus according to claim 11, characterized in that, The acquisition module is further configured to acquire the three-axis rotational angular velocity of the image acquisition device when acquiring an image related to the second image and the first motion degree value of the image acquisition device when acquiring the first image; The processing module is further configured to determine a second motion degree value of the image acquisition device in the second image based on the three-axis rotational angular velocity and the first motion degree value; The processing module is further configured to determine a first translation threshold corresponding to the second motion degree value among multiple thresholds.

13. The apparatus according to claim 12, characterized in that, The acquisition module is also used to acquire the triaxial acceleration of the image acquisition device when acquiring an image related to the second image; The processing module is further configured to determine a second motion degree value of the image acquisition device in the second image based on the triaxial acceleration, the triaxial rotational angular velocity, and the first motion degree value.

14. The apparatus according to any one of claims 8-10, characterized in that, The processing module is further configured to: Feature points are extracted from the first image and the second image to obtain multiple feature points; The multiple feature points are matched to obtain multiple pairs of original feature point pairs; The multiple pairs of original feature points are filtered according to the feature point pair filtering method to obtain the multiple feature point pairs; The feature point pair filtering method includes one or more of the following: standard deviation filtering method, fixed threshold filtering method, and random sampling consistency filtering method.

15. A feature point processing device, characterized in that, It includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the feature point processing device performs the method as described in any one of claims 1 to 7.

16. A computer storage medium, characterized in that, The computer storage medium stores instructions that, when executed by the computer, cause the computer to perform the method according to any one of claims 1 to 7.

17. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1 to 7.

Images