A method for image registration
By combining subpixel correction of feature extraction and descriptor matching with optical flow method and applying multiple constraint filtering, the problem of low registration accuracy of three-dimensional visual image is solved, and efficient and accurate matching effect is achieved.
Patent Information
- Application Number
- CN202110051778.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-01-15
- Publication Date
- 2025-01-10
- Estimated Expiration
- 2041-01-15
AI Technical Summary
The existing three-dimensional visual image registration technology has the problems of low feature point positioning accuracy, sparse features, and non-maximum value suppression processing, which is particularly prone to errors in scenarios where viewing angles, lighting, and time spans vary greatly.
Feature extraction and descriptor matching are used to obtain the initial matching set, subpixel accuracy correction is performed through the optical flow method, and filtered in combination with descriptor distance, local space, and two-view projection geometric consistency constraints to improve matching accuracy.
It realizes efficient and accurate matching of image registration, reduces matching errors, and maintains excellent matching accuracy especially in scenes with large changes.
Smart Images
Figure CN112734817B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the field of three-dimensional vision, and in particular to an image registration method. Background Art
[0002] Three-dimensional vision is one of the traditional research directions in the field of computer vision. Its goal is to enable computers to perceive and understand environmental information in the three-dimensional world through two-dimensional images. Its main theoretical basis is multi-view geometry, that is, the camera takes multiple two-dimensional images of the same scene from multiple perspectives, and the positions of the same three-dimensional point on different two-dimensional images are subject to geometric constraints. Therefore, how to accurately obtain the same three-dimensional point on different two-dimensional images is a technical problem that needs to be solved.
[0003] At present, a large number of image registration technologies have emerged in the field of 3D vision, but they have the following traditional disadvantages: 1. Feature point positioning is basically at the pixel level; 2. The extracted features are generally sparse and will be processed through non-maximum suppression, resulting in the possibility that the two matched features are just close points in the three-dimensional space; 3. During the registration process, matching errors are prone to occur in scenes with large changes in viewing angle, lighting, time span, etc. Summary of the invention
[0004] The purpose of the present invention is to provide an image registration method to solve the problem of low image registration accuracy.
[0005] To achieve the above object, the present invention provides an image registration method, comprising the following steps:
[0006] S1. Extract features from the two acquired frames of images respectively, and obtain an initial matching set by using descriptor matching based on the extracted features;
[0007] S2. Filter the initial matching set to obtain a first matching set;
[0008] S3. Using the optical flow method to perform sub-pixel precision correction on the first matching set;
[0009] S4. Filter the modified first matching set to obtain a second matching set, and complete image registration.
[0010] According to one aspect of the present invention, in step S1, in the step of extracting features from the two acquired frames of images respectively, the features are manual features or deep features based on a neural network; wherein the extracted features include: feature position, feature orientation, feature extraction area radius and feature descriptor information.
[0011] According to one aspect of the present invention, in step S1, in the step of obtaining an initial matching set by using descriptor matching based on the extracted features, the initial matching set is Among them, x is the feature extracted from the first frame image, and y is the feature extracted from the second frame image.
[0012] According to one aspect of the present invention, in step S1, in the step of obtaining an initial matching set by using descriptor matching based on the extracted features, a non-maximum suppression algorithm is used to filter densely appearing features, and only the features with the largest response values are retained; then, nearest neighbor brute force matching and fast approximate nearest neighbor matching are performed based on the distance between feature descriptors to obtain the initial matching set.
[0013] According to one aspect of the present invention, in step S2, the initial matching set is filtered, and in the step of obtaining the first matching set, filtering is performed using at least one of a descriptor distance constraint, a matching local space constraint, and a two-view projection geometric consistency constraint.
[0014] According to one aspect of the present invention, in step S2, during the process of filtering the initial matching set, if multiple processing methods are used, the initial matching set is filtered in sequence based on the selected processing methods.
[0015] According to one aspect of the present invention, the first matching set is
[0016] In step S3, the step of using the optical flow method to perform sub-pixel accuracy correction on the first matching set includes:
[0017] S31. Obtain a match C to be corrected in the first match set. i =(x i ,y i ), and obtain the features x contained therein respectively. i ,y i The feature position P on the two-dimensional pixel plane xi , P yi , the feature orientation θ xi ,θ yi and the feature extraction area radius r xi 、r yi ;
[0018] S32. A window is set around the two features in proportion to the radius of the feature extraction area, and if there is a sub-pixel position vertex in the window, the grayscale value of the pixel at the sub-pixel position is obtained by interpolation. Under the assumption of relative photometric invariance, the feature position in the window and the relative photometric error of the feature orientation are iteratively optimized to obtain a sub-pixel precision position.
[0019] According to one aspect of the present invention, in step S32, the relative photometric error is obtained by summing the relative photometric errors of each element in the window, and the relative photometric error formula of each element is:
[0020]
[0021] Among them, P xij With P yij Match C respectively i The coordinates of the jth point in the window in the two images, T is the current estimated match C i The rotation and translation transformation of the two end points on the two-dimensional plane, I1(P) and I2(P) are the grayscale values of the first image and the second image at the pixel coordinate P, respectively. and Match C respectively i The average gray value of the pixel in the window of the two images, and Match C respectively i The collection of elements in the window at the two endpoints.
[0022] According to one aspect of the present invention, in step S4, in the step of filtering the first corrected matching set to obtain the second matching set, the matching set to be corrected is determined by the obtained sub-pixel precision position. i Is the displacement correction amount greater than the preset value? If so, the matching C is eliminated. i ;
[0023] Complete the correction of all the matches C in the first matching set i The second matching set is obtained.
[0024] According to one aspect of the present invention, in step S4, in the step of filtering the first corrected matching set to obtain the second matching set, the matching set to be corrected is determined by the obtained sub-pixel precision position. i Is it still within the image range after correction? If not, the match C is removed. i ;
[0025] Complete the correction of all the matches C in the first matching set i The second matching set is obtained.
[0026] According to a solution of the present invention, by combining the feature method and the optical flow method for sub-pixel image registration, efficient and accurate matching between different images is achieved, effectively solving the problem of large image registration errors.
[0027] According to a solution of the present invention, an initial matching set is obtained based on a feature method, and after filtering the initial matching set, an optical flow method is further used to perform sub-pixel accuracy correction on the matching set so that the matching errors in the matching set are effectively quantified, and the final matching set obtained after further filtering has higher matching accuracy.
[0028] According to a solution of the present invention, excellent matching accuracy can be achieved in scenes with large changes in viewing angle, illumination, time span, etc., and the registration result is better. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 is a flowchart schematically showing a method for image registration according to an embodiment of the present invention;
[0030] Figure 2 is a principle diagram schematically showing a matching filtering method based on spatial local constraints according to an embodiment of the present invention;
[0031] Figure 3 is a schematic diagram showing a principle diagram of a matched filtering method based on geometric consistency of two-view projections according to an embodiment of the present invention;
[0032] Figure 4 is a diagram schematically showing a window structure used in an optical flow method according to an embodiment of the present invention;
[0033] Figure 5 The figure schematically shows the window scaling at both ends of the optical flow matching method according to an embodiment of the present invention. DETAILED DESCRIPTION
[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0035] When describing the embodiments of the present invention, the orientation or positional relationship expressed by the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside" and "outside" are based on the orientation or positional relationship shown in the relevant drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the above terms should not be understood as limiting the present invention.
[0036] The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments. The embodiments cannot be described one by one here, but the embodiments of the present invention are not therefore limited to the following embodiments.
[0037] Combination Figure 1 and Figure 2 As shown, according to one embodiment of the present invention, an image registration method of the present invention comprises the following steps:
[0038] S1. Extract features from the two acquired frames of images respectively, and obtain an initial matching set by using descriptor matching based on the extracted features;
[0039] S2. Filter the initial matching set to obtain a first matching set;
[0040] S3. Using the optical flow method to perform sub-pixel precision correction on the first matching set;
[0041] S4. Filter the corrected first matching set to obtain a second matching set, and complete the image registration.
[0042] According to one embodiment of the present invention, in step S1 , an image pair I1 , I2 is read in, wherein I1 is a first image and I2 is a second image.
[0043] It should be pointed out that the core task of image feature matching is to extract local features. Features are composed of key points and descriptors, and have four main properties: the first is locality, that is, the features are only related to a small area around the key points; the second is efficiency, that is, in an image, the number of features should be much smaller than the number of pixels; the third is repeatability, that is, the same "area" needs to have as similar descriptors as possible after being detected in different images; the fourth is distinguishability, that is, different "areas" need to have as different descriptors as possible, so that mismatching is less likely to occur.
[0044] According to one embodiment of the present invention, in step S1, in the step of extracting features from the two acquired frames of images respectively, the features are manual features or deep features based on neural networks; in this embodiment, if the features are manual features, they can be obtained by methods such as SIFT, SURF, ORB, etc., and if the features are deep features, they can be obtained by methods such as LIFT, SuperPoint, D2-Net, etc. In this embodiment, the features can be selected as manual features and feature extraction can be performed based on the SIFT algorithm to obtain a set of key points and a set of descriptors. In this embodiment, each feature will contain feature position, feature orientation, feature extraction area radius and feature descriptor information after extraction.
[0045] According to an embodiment of the present invention, in step S1, in the step of obtaining an initial matching set by using descriptor matching based on the extracted features, the initial matching set is Among them, x is the feature extracted from the first frame image I1, and y is the feature extracted from the second frame image I2.
[0046] According to one embodiment of the present invention, in step S1, in the step of obtaining an initial matching set by using descriptor matching based on the extracted features, a non-maximum suppression algorithm is used to filter densely appearing features, and only the features with the largest response values are retained; then, nearest neighbor brute force matching and fast approximate nearest neighbor matching are performed based on the distance between feature descriptors to obtain the initial matching set.
[0047] According to one embodiment of the present invention, in step S2, the initial matching set is filtered, and in the step of obtaining the first matching set, it is used to filter out wrong matches. In many scenes, the image matching obtained by relying solely on the similarity of descriptors is often unreliable. Therefore, this step filters wrong matches based on the initial matching set obtained in the previous step to increase the reliability of the matching. In this embodiment, filtering is performed using at least one processing method based on descriptor distance constraints, matching local space constraints, and two-view projection geometric consistency constraints. The specific processing method is as follows:
[0048] When using the filtering method based on descriptor distance constraints, there are two specific processing methods. One is to directly eliminate matches whose descriptor distance is greater than the preset threshold σ1, and the other is to first calculate the ratio of the nearest neighbor descriptor distance to the next nearest neighbor descriptor distance, and then eliminate matches whose ratio is greater than the preset threshold σ2.
[0049] When using filtering methods based on local spatial constraints of matching sets, see Figure 2As shown in the figure, one image can get many image matching pairs through step 1. For one correct match, such as the three matches in the dog's nose area in the figure (the match is represented by a straight line segment), there are often other matches around it that are basically consistent with its direction. For an incorrect match, such as the match from the cat's paw to the dog's chin in the figure, it is difficult to find a match around it that is consistent with its direction. This is the local spatial constraint of image matching. Through this constraint, messy random mismatches can be effectively eliminated.
[0050] When using filtering methods based on geometric consistency constraints between two view projections between images, see Figure 3 As shown in the figure, given two camera optical centers c and c', a projection point p is observed on the imaging plane of the camera optical center c, then the three-dimensional space point P corresponding to the projection point must be on the ray cp. Therefore, the projection point of P on the imaging plane of the camera optical center c' must be on l' (the projection of the ray cx on the imaging plane). According to such geometric constraints, matches that do not meet the constraints can be filtered out. This type of filtering method is implemented with RANSAC and its variant algorithms.
[0051] According to an embodiment of the present invention, in step S2, during the filtering process of the initial matching set, if multiple processing methods are used, the initial matching set is filtered sequentially based on the selected processing methods. Specifically, when three filtering methods are selected, in order to ensure the reliability of the obtained initial matching, the above three filtering methods are applied sequentially to the initial matching set. Of course, if two filtering methods are selected, the two selected filtering methods are applied sequentially.
[0052] According to an embodiment of the present invention, after filtering the initial matching set, the first matching set obtained is
[0053] In this embodiment, the locality and efficiency of local features can provide the algorithm with a more robust matching effect, but it also brings some inherent disadvantages. First, the locality makes the final match actually a match from image block to image block, rather than a point-to-point match. After the viewing angle changes, due to the efficiency criterion, it is difficult to guarantee that the points detected in the same "area" are the same point. Even if two image blocks belonging to the same "area" are matched in the end, the points representing the two image blocks are not the same point.
[0054] Furthermore, it is necessary to correct the matching endpoint position in the present invention. Then, in step S3, the matching position is corrected by the optical flow method. The optical flow method relies on three assumptions. The first point is that the displacement of the midpoints between the previous and next frames is not large; the second point is the grayscale invariance assumption, which requires that the external light intensity remains basically constant; the third point is the spatial correlation assumption, and the movement of each point is similar to that of the adjacent points; among these three assumptions, the first one makes the optical flow method unable to work well for scenes with large motion, and the second one makes the optical flow method unable to adapt well to scenes with large changes in illumination.
[0055] In the present invention, after the aforementioned steps S1 to S2, the obtained matching basically satisfies the first assumption of the optical flow method.
[0056] Furthermore, in step S3, the step of using the optical flow method to perform sub-pixel accuracy correction on the first matching set includes:
[0057] S31. Obtain a match C to be corrected in the first match set i =(x i , yi), and obtain the features x contained in them respectively i ,y i The feature position P on the two-dimensional pixel plane xi , P yi , the feature orientation θ xi ,θ yi and the feature extraction area radius r xi 、r yi In this embodiment, the feature direction θ xi ,θ yi That is, the angle information of the feature on the two-dimensional pixel plane, so that the feature orientation θ xi ,θ yi The feature position P of the feature on the two-dimensional pixel plane xi , P yi The transformation is converted into a rotation transformation plus a displacement transformation on a two-dimensional plane to improve the effect of the optical flow method in angle-changing scenes.
[0058] S32. Set a window around the two features in proportion to the radius of the feature extraction area. If there is a sub-pixel position vertex in the window, obtain the pixel grayscale value of the sub-pixel position by interpolation. Under the assumption of relative photometric invariance, iteratively optimize the feature position and feature orientation relative photometric error in the window to obtain a sub-pixel precision position. Figure 4As shown, based on the rotation transformation and displacement transformation of the features in the above steps, the neighborhood window shape is designed to be a polygonal form that is approximately circular. In order to improve the performance of the optical flow method in the scenario of scale transformation, the solution of the present invention adjusts the size of the window according to the radius ratio of the neighborhood of the matching features at both ends, and obtains the grayscale of the new window element through bilinear interpolation. The neighborhood window form of the two features that are finally matched is as follows Figure 5 shown.
[0059] According to an embodiment of the present invention, in step S32, for each matching C i , the correction amount is obtained by iterative optimization through minimizing the relative photometric error of the two end point regions. The relative photometric error calculation formula for each element in the window is as follows:
[0060]
[0061] in, and Match C respectively i The coordinates of the jth point in the window in the two images, T is the current estimated match C i The rotation and translation transformation of the two end points on the two-dimensional plane, I1(P) and I2(P) are the grayscale values of the first image and the second image at the pixel coordinate P, respectively. and Match C respectively i The average grayscale value of the pixel in the two images. and Match C respectively i The collection of elements in the window at the two endpoints.
[0062] According to an embodiment of the present invention, in step S4, in the step of filtering the modified first matching set to obtain the second matching set, after the iterative optimization of the above steps is completed, a correction amount is generated for each match. In this embodiment, the matching C to be modified is determined by the obtained sub-pixel precision position. i Is the displacement correction amount greater than the preset value (for example, the preset value is 3 pixels)? If so, the matching C is eliminated. i ;
[0063] Complete all matches C in the modified first matching set i The judgment is used to filter the matching set, and the remaining matches are corrected to obtain the second matching set.
[0064] According to another embodiment of the present invention, in step S4, in the step of filtering the first corrected matching set to obtain the second matching set, after the above steps are iteratively optimized, a correction amount is generated for each match. In this embodiment, the matching C to be corrected is determined by the obtained sub-pixel precision position. i Is it still within the image range after correction? If not, the match C is removed. i ;
[0065] Complete the correction of all the matches C in the first matching set i The judgment is used to filter the matching set, and the remaining matches are corrected to obtain the second matching set.
[0066] The above contents are merely examples of specific solutions of the present invention. For devices and structures not described in detail therein, it should be understood that they can be implemented by adopting general devices and general methods available in the art.
[0067] The above is only one solution of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. An image registration method, comprising the following steps: S1. Extract features from the two acquired frames of images respectively, and obtain an initial matching set by using descriptor matching based on the extracted features; wherein the extracted features include: feature position, feature orientation, feature extraction area radius and feature descriptor information; S2. Filter the initial matching set to obtain a first matching set; wherein the filtering is performed by using a plurality of processing methods based on a descriptor distance constraint, a matching local space constraint, and a two-view projection geometric consistency constraint, and the initial matching set is filtered in sequence based on the selected processing methods; S3. Use the optical flow method to perform sub-pixel precision correction on the first matching set; wherein the first matching set is Then step S3 includes: S31. Obtain a match C to be corrected in the first match set. i =(x i ,y i ), and obtain the features x contained therein respectively. i ,y i The feature position P on the two-dimensional pixel plane xi , P yi , the feature orientation θ xi ,θ yi and the radius r of the feature extraction area xi 、r yi ; S32. A window is set around the two features in proportion to the radius of the feature extraction area. If there is a sub-pixel position vertex in the window, the grayscale value of the pixel at the sub-pixel position is obtained by interpolation. Under the assumption of relative photometric invariance, the relative photometric error of the feature position and the feature orientation in the window is iteratively optimized to obtain a sub-pixel precision position; wherein the relative photometric error of each match is obtained by summing the relative photometric errors of each element in its window. For a match C i =(x i ,y i ), the relative photometric error formula of each element in the window is: in, and Match C respectively i The coordinates of the jth point in the window in the two images, T is the current estimated match C i The rotation and translation transformation of the two end points on the two-dimensional plane, I1(P) and I2(P) are the grayscale values of the first image and the second image at the pixel coordinate P, respectively. and Match C respectively i The average grayscale value of the pixel in the two images. and Match C respectively i The set of elements in the window at the two endpoints; S4. Filter the first corrected matching set to obtain a second matching set, and complete the image registration; wherein the matching set to be corrected is determined by the obtained sub-pixel precision position. i Is the displacement correction amount greater than the preset value? If so, the matching C is eliminated. i ; Complete the correction of all the matching C in the first matching set i to obtain the second matching set; or, in the step of filtering the modified first matching set to obtain the second matching set, the matching C to be modified is determined by the obtained sub-pixel precision position. i Is it still within the image range after correction? If not, the match C is removed. i ; Complete the correction of all the matching C in the first matching set i The second matching set is obtained.
2. The image registration method according to claim 1, characterized in that: In step S1, in the step of extracting features from the two acquired frames of images respectively, the features are manual features or deep features based on a neural network.
3. The image registration method according to claim 2, characterized in that: In step S1, in the step of obtaining an initial matching set by using descriptor matching based on the extracted features, the initial matching set is Among them, x is the feature extracted from the first frame image, and y is the feature extracted from the second frame image.
4. The image registration method according to claim 2, characterized in that: In step S1, in the step of obtaining an initial matching set by using descriptor matching based on the extracted features, a non-maximum suppression algorithm is used to filter densely appearing features, and only the features with the largest response values are retained; then, the nearest neighbor brute force matching and fast approximate nearest neighbor matching are performed based on the distance between the feature descriptors to obtain the initial matching set.