Image feature matching method and device based on local consistency

Abstract

The present invention provides an image feature matching method and device based on local consistency, the method comprising: obtaining two images to be matched; detecting feature points in the two images by SIFT algorithm and establishing feature descriptors; constructing a set of initial matching sets according to the similarity of feature descriptors in the two images; deleting mismatches from the initial matching set by neighborhood consistency constraint to obtain a second matching set; calculating the deviation of the motion vector of each feature point by the second matching set; optimizing a preset image feature matching model by constraining the deviation of the motion vector by motion vector consistency, and determining hyperparameters; performing feature matching on any two images to be matched by the optimized image feature matching model to obtain an image feature matching result. The present invention has the beneficial effects of: combining the local consistency of neighborhood consistency and motion vector consistency to remove and optimize mismatches in image feature matching, thereby improving matching accuracy.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, in particular to image matching technology, and proposes an image feature matching method and device based on local consistency. Background Art

[0002] Feature matching is a basic and important work in computer vision, and is of great significance to a variety of visual tasks. Feature matching refers to establishing reliable feature point correspondences between two images of the same scene. It is essentially a complex combinatorial optimization problem of NPC (Non-deterministic Polynomial Complete problem, NPC), which requires a lot of computation. Most current methods remove false matches from a set of initial matching sets based on estimating the underlying image transformation model. However, the transformation model may vary for different image data sets, which severely limits the universality of the algorithm. Moreover, since the matched image pairs are susceptible to external noise, outliers, and non-rigid transformations, the performance of traditional feature matching algorithms often drops significantly, which seriously restricts their use in engineering. Therefore, studying a universal, more robust and efficient feature matching algorithm is of great significance and value in both theory and practical applications.

[0003] Based on the research in recent years, image matching methods can be generally divided into three categories: region-based methods, feature-based methods, and learning-based methods. Among them, feature-based methods have been widely and deeply studied and have great advantages: (1) Features can better reflect the essential structure of the image and can reduce the impact of sensor and environmental noise. (2) Compared with traditional region-based methods, feature-based methods simplify the image expression form, which greatly improves the algorithm's running speed and reduces the computational cost of processing images. Although traditional methods such as scale-invariant feature transform (SIFT) are widely used in various fields, there are still many problems that need to be solved.

[0004] First, because remote sensing images may have ground undulations, changes in imaging viewpoints, or more complex non-rigid transformations, predefined parametric (such as affine, monogenic, and polar geometry) or non-parametric (such as non-rigid) models cannot fit well. Second, traditional local descriptors only consider some weak features of feature points; when the remote sensing image information is more complex, or the image is affected by nonlinear radiation differences, speckle noise, light changes, etc., the accuracy of the local descriptor will be greatly reduced, resulting in a large number of false matches. Third, due to the combinatorial nature of the matching problem, the algorithm usually has a high computational complexity. Even if the interference of outliers is excluded, simply matching the feature points of two remote sensing images will result in a complex arrangement. Summary of the invention

[0005] In order to solve the problems of low robustness, slow speed and high complexity in existing image matching methods, the technical solution adopted by the present invention is to provide an image feature matching method and device based on local consistency.

[0006] According to one aspect of the present invention, there is provided an image feature matching method based on local consistency, comprising the following steps:

[0007] S1: Get two images to be matched;

[0008] S2: Detect feature points in the two images using the SIFT algorithm and establish feature descriptors;

[0009] S3: construct an initial set of matches based on the similarity of feature descriptors in the two images;

[0010] S4: deleting mismatches from the initial matching set by using a neighborhood consistency constraint to obtain a second matching set;

[0011] S5: Calculate the deviation of the motion vector of each feature point through the second matching set;

[0012] S6: performing motion vector consistency constraints on the deviation of the motion vector, thereby optimizing the preset image feature matching model and determining hyperparameters;

[0013] S7: Perform feature matching on the two images to be matched using the optimized image feature matching model to obtain an image feature matching result.

[0014] Preferably, the S4 includes:

[0015] S4.1: Obtain the attribute variables of each initial match in the initial match set through neighborhood consistency constraints

[0016] S4.2: Set the threshold η by comparing η with Compare and filter the feature points that do not meet the neighborhood consistency constraints to obtain the second matching set U:

[0017] U={(x i ,y i )∈S|Rst i >η}

[0018] Among them, S represents the initial matching set, i represents the sequence number of the initial matching, (x i ,y i ) represents the i-th initial match, Rst i Represents all the neighborhoods corresponding to the i-th initial match The sum of M represents the number of neighbors taken by each initial match, j represents the sequence number of the neighborhood, that is, the jth neighborhood, K j represents the size of the jth neighborhood.

[0019] Preferably, the S5 includes:

[0020] S5.1: after performing vector conversion on the second matching set, spatially gridding the second matching set into a plurality of non-overlapping units, and calculating an estimated motion vector of each unit by Gaussian kernel convolution operation;

[0021] S5.2: Calculate the deviation between the initial motion vector and the estimated motion vector in each cell.

[0022] Preferably, the S5.1 includes:

[0023] S5.1.1: Convert the second matching set U into the initial motion vector set S′:

[0024]

[0025] Among them, x i and i They represent the i-th feature point in the two images that constitute the initial match, i represents the sequence number of the feature point or the initial match, and m i =y i -x i Indicates that the i-th initial match (x i ,y i ), where N represents the number of initial matches in the initial matching set;

[0026] S5.1.2: Divide each dimension of the initial motion vector set S′ into n c non-overlapping parts, and then we get G = n c ×n c units, so the initial motion vector set S′ can be divided into G units, becoming the set

[0027] Among them, n c represents the number of divisions of each dimension of the initial motion vector set S′, G represents the number of divisions of the initial motion vector set S′ as a whole, that is, S′ is divided into G units, C j,k represents the cell in row j and column k, where j and k represent cell C respectively. j,k The row and column numbers;

[0028] S5.1.3: Each unit C is obtained by Gaussian kernel convolution j,k The estimated motion vector

[0029] Preferably, the S5.2 includes:

[0030] S5.2.1: Calculate a numerical deviation between the initial motion vector and the estimated motion vector;

[0031] S5.2.2: Calculate the length ratio deviation between the initial motion vector and the estimated motion vector;

[0032] S5.2.3: Calculate the angular deviation between the initial motion vector and the estimated motion vector;

[0033] S5.2.4: Calculate the total deviation of the motion vector based on the numerical deviation, the length ratio deviation and the angle deviation.

[0034] Preferably, after S5.2.4, the method further includes:

[0035] Performing quantization processing on the total deviation of the motion vector to obtain a quantization result;

[0036] The quantization result is used to reflect the consistency between the initial motion vector and the estimated motion vector in each unit.

[0037] According to another aspect of the present invention, there is also provided an image feature matching device based on local consistency, comprising the following modules:

[0038] An image acquisition module, used for acquiring two images to be matched;

[0039] Feature point detection and descriptor creation module, used to detect feature points in two images through SIFT algorithm and create feature descriptors;

[0040] An initial matching set construction module, used to construct a set of initial matching sets according to the similarity of feature descriptors in two images;

[0041] A neighborhood consistency constraint module, used to delete false matches from the initial matching set through neighborhood consistency constraints to obtain a second matching set;

[0042] A motion vector deviation calculation module, used to calculate the deviation of the motion vector of each feature point through the second matching set;

[0043] A motion vector consistency constraint module, used to optimize a preset image feature matching model and determine hyperparameters by performing motion vector consistency constraints on the deviation of the motion vector;

[0044] The feature matching module is used to perform feature matching on two images to be matched through an optimized image feature matching model to obtain an image feature matching result.

[0045] The present invention uses a primary filtering strategy based on the consistency of the feature point neighborhood to remove a part of outliers with obvious errors, purify the neighborhood to obtain an approximate internal point set, and thus construct a more realistic neighborhood. The image space is gridded into multiple non-overlapping units, and an estimated motion vector is calculated for each unit through Gaussian kernel convolution operation to represent the potential motion characteristics of each unit; finally, by judging the consistency between the initial motion vector and the estimated motion vector in each unit, cunning mismatches are further eliminated to obtain a more accurate match.

[0046] The technical solution provided by the present invention has the following beneficial effects:

[0047] 1. It can cope with the ground undulations, imaging viewpoint changes or more complex non-rigid transformations of remote sensing images.

[0048] 2. It can effectively remove a large number of mismatches caused by complex remote sensing image information or when the image is affected by nonlinear radiation difference, speckle noise, light changes, etc.

[0049] 3. The method of the present invention has linear complexity and can solve thousands of matching problems in a few milliseconds, ensuring the speed of the algorithm. This is useful for many real-time applications and can quickly provide a good initialization of the matching algorithm for specific complex problems, especially remote sensing images. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The specific effects of the present invention will be further described below in conjunction with the accompanying drawings and embodiments, in which:

[0051] Figure 1 It is a flow chart of an image feature matching method based on local consistency of the present invention;

[0052] Figure 2 It is the overall framework diagram of the local consistency algorithm of the present invention;

[0053] Figure 3 It is a neighborhood consistency schematic diagram of the present invention;

[0054] Figure 4 is the cumulative distribution diagram of the total motion deviation of the feature points before and after the quantization processing of the present invention, where Figure 4 (a) is before quantization processing, Figure 4 (b) after quantization processing;

[0055] Figure 5 It is a schematic diagram of the hyperparameter setting experiment of the present invention;

[0056] Figure 6 It is the qualitative result of the feature matching of the present invention;

[0057] Figure 7 It is the quantitative experimental result of the feature matching of the present invention. DETAILED DESCRIPTION

[0058] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, specific embodiments of the present invention are now described in detail with reference to the accompanying drawings.

[0059] refer to Figure 1 This embodiment provides an image feature matching method based on local consistency, which mainly includes the following steps:

[0060] S1: Get two images to be matched;

[0061] S2: Detect feature points in the two images using the SIFT algorithm and establish feature descriptors;

[0062] S3: construct an initial set of matches based on the similarity of feature descriptors in the two images;

[0063] S4: deleting mismatches from the initial matching set by using a neighborhood consistency constraint to obtain a second matching set;

[0064] Step S4 specifically includes:

[0065] (1) Neighborhood consistency model

[0066] For image pairs describing the same scene or object, when the viewpoint changes (such as stereo disparity) or non-rigid transformations (such as dynamic scenes), the absolute distance between two corresponding feature points may change significantly; however, the spatial neighborhood relationship between feature points representing the local neighborhood structure of the image scene is very stable and will not be affected, and can generally be well preserved. Taking non-rigid facial images as an example, due to the physical constraints of bones and muscles, changes in expression and perspective will not lead to local structural changes in the face, such as the relative positions of the eyes, nose, mouth, chin, etc.

[0067] For the initial matching set S n An initial match (x i ,y i ), if it is an inlier (a feature point that can be correctly matched), then the distribution of its neighborhood elements should be similar. On the contrary, for an outlier (a feature point that is incorrectly matched), the corresponding neighborhood distribution will be very different. This is the neighborhood consistency constraint, such as Figure 3 As shown in the figure, the initial matching of the inliers and outliers is shown respectively (x i ,y i), and the five initial matches in its neighborhood. It is easy to see that the neighborhood of the inliers contains many common initial matches, but the neighborhood of the outliers has almost no common initial matches (correct matches are represented by black lines, and incorrect matches are represented by gray lines). In order to capture this property mathematically, the present invention defines a set of size The neighborhood of and Among them Represents point x i Neighborhood, M represents the number of neighborhoods, which is composed of all feature points of X, with a distance of x i The most recent K j feature points, K j represents the size of the jth neighborhood. This strategy is called the multi-K-Nearest Neighbor strategy (multi-KNN). and The neighborhood consistency between can be characterized by the following formula:

[0068]

[0069] in, for and The number of common feature points in two neighborhoods, is the ratio of the number of common feature points in the jth neighborhood of the ith initial match to the number of all feature points. Obviously, if the corresponding feature point is an inlier, The value of will be large, and vice versa. Note that the true common element cannot be determined without GroundTruth. However, S can be applied n Included in and The reason for using this approximation is that if the initial match (x i ,y i ) is an interior point, then it appears in the local neighborhood at the same time and Most of the initial matches of the feature points in are also likely to be inliers; on the contrary, if (x i ,y i ) is an outlier point, then in its neighborhood and It is very likely that the initial matches will not appear at the same time, that is, x i Neighborhood The matching point of a point in y is likely to be i Neighborhood Besides.

[0070] (2) Preliminary filtering of neighborhood consistency constraints

[0071] When the image is subjected to complex non-rigid transformations or is affected by external noise, the initial matching set will contain a large number of outliers. In order to remove outliers with obvious errors to a certain extent and purify the local area, so that the estimated motion vector closer to the potential true motion vector can be obtained in the subsequent processing, the attribute variables obtained by the neighborhood consistency constraint are To perform preliminary filtering.

[0072] To achieve this goal, a simple threshold η can be set by dividing η by By comparison, we can filter out the feature points that do not meet the neighborhood consistency constraint and obtain a second matching set U after preliminary filtering:

[0073] U={(x i ,y i )∈S|Rst i >η}, (2)

[0074] Where U is the second matching set after preliminary filtering, η is the threshold used for comparison, Rst i Represents all the neighborhoods corresponding to the i-th initial match The sum of M represents the number of neighborhoods. The second matching set U after preliminary filtering is usually clean enough to be used for the construction of the motion vector field. According to a large number of experimental results, the number of neighborhoods is set to 3 in the multiple K-nearest neighbor strategy, and the size of each neighborhood is K1=9, K2=10 and K3=11. If necessary, the initial matching set can be iteratively filtered by setting different thresholds η, but it is sufficient for the method of the present invention to perform filtering only once.

[0075] S5: Calculate the deviation of the motion vector of each feature point through the second matching set;

[0076] Step S5 specifically includes:

[0077] (1) Numerical deviation of motion vector

[0078] In order to effectively remove the noise interference in the image, according to the theory of image denoising, for a noisy image, the present invention fully considers the pixels in the local area (determined by the size of the convolution kernel) and adopts the motion vector method to obtain the real pixel information to remove the noise. First, the second matching set U is converted into the initial motion vector set Among them, m i =y i -x i represents the i-th initial match (x i ,yi ), for the potential true matches in the image of the matching set, their motion vectors should be regular and smooth, that is, they should satisfy the consistency of motion vectors. In other words, correct matches will have similar motion behaviors in the local neighborhood, while false matches are usually randomly distributed. Figure 2 As shown in the figure, it can be seen that each matching has three steps, namely, establishing the initial matching set, purifying the neighborhood using neighborhood consistency, and further filtering out outliers using motion vector consistency. Each step has two pictures, the left picture represents the gridded initial motion vector field, and the right picture represents the estimated motion vector field of each unit (generated by Gaussian kernel convolution). It can be seen that as the denoising goes deeper, the outliers in the vector field become fewer and fewer, and the estimated vector field becomes more and more regular and smooth, removing some abrupt vectors.

[0079] In order to obtain the estimated motion vector, each dimension of the initial motion vector set S′ is first divided into n c non-overlapping parts, and then the initial motion vector set S′ is divided into G=n c ×n c units (G is the number of units) becomes a set Among them C j,k represents the cell in row j and column k, where j and k represent cell C respectively. j,k The initial motion vector set after gridding is as follows: Figure 2 As shown, Figure 2 n c =20. Now, is defined as the average motion vector matrix, where is the (j,k)th unit C j,k The average motion vector within:

[0080]

[0081] where |·| represents the modulo operation, |C j,k |It is Unit C j,k The size of unit C j,k The number of motion vectors in .

[0082] In order to make full use of the interaction between adjacent units, the present invention uses an efficient convolution theory because it can comprehensively consider the local n k ×n k Here, the convolution of the motion vector Defined as:

[0083]

[0084] in The size generated after Gaussian kernel convolution is n c ×n c ×2 matrix, is the estimated motion vector of the (j, k)th cell, W is a counting matrix, indicating the number of motion vectors in each cell, W j,k =|C j,k | represents the cell C at row j and column k j,k The denominator in formula (4) is used for weight compensation to keep the scale of the convolution result from changing too much. When there is 0 in , ε is an infinitesimal positive number. K is n k ×n k The Gaussian kernel distance matrix, also known as the Gaussian convolution kernel, represents the distance between a unit and its surroundings (n k ×n k -1) units, through this parameter, the connection relationship between a unit and its neighboring units can be established. Each element K i,j Defined as:

[0085]

[0086] Among them, K i,j represents the Gaussian kernel distance of the cell in row i and column j, s i,j =(i,j) T , are the position of each unit in the Gaussian convolution kernel K (i.e., row i, column j) and the center position of the Gaussian convolution kernel, and [·] rounds the element to the nearest integer not less than itself. Because the center position inside the convolution kernel needs to be determined, n k Must be a positive odd number.

[0087] In addition, in order to avoid the influence of isolated samples, isolated samples can be excluded in the convolution process by subtracting the corresponding average vector from the numerator and denominator, and then adjusting the weight of each unit. In formula (4), B(W) represents the binary form of W, whose value is 0 or 1. i,j >0, B(W i,j )=1; K * represents the Gaussian kernel distance of the center position, and

[0088] After the above convolution operation, the estimated motion vector of each unit can be obtained, which can be used to replace the potential true motion vector. Then the initial motion vector m corresponding to each element is obtained i With estimated motion vector Deviation di , whose value is constrained to be between [0,1] by the following formula:

[0089]

[0090] Where dev i It is the numerical deviation of the motion vector, where β determines the width of the interaction range between the two motion vectors. 2 It can be set to 0.08 based on experience.

[0091] (2) Motion vector length ratio deviation

[0092] In order to make the difference between the mismatch and the correct match more obvious, in addition to considering the numerical deviation between each motion vector and the estimated motion vector, the present invention further considers the length and angle of the motion vector, so that it is more convenient to distinguish them using thresholds. The three deviations are normalized and summed to obtain the total deviation. The shortcomings of each deviation compensate for each other, and the precision and recall rate will be greatly improved.

[0093] Constructing length ratio deviation and angle deviation, the wrong matching motion vector has large angle and length ratio deviations, while the correct matching motion vector is basically consistent with the true motion vector.

[0094] Define the length ratio deviation of the motion vector

[0095]

[0096] in Indicates m i and The ratio of the lengths of the longest motion vector to the shortest motion vector in the (j, k)th unit. The above normalization method is a Gaussian membership function. Use as the length ratio deviation of the motion vector.

[0097] (3) Angle deviation of motion vector

[0098] Define the angular deviation of the motion vector:

[0099]

[0100] in and Indicates m i and The present invention also uses Angular deviation as a motion vector.

[0101] (4) Total motion vector deviation and quantization strategy

[0102] Will and Deviation from the previous value dev i Combined, the total deviation between the motion vector and the estimated motion vector is defined as:

[0103]

[0104] At this time, by comparing the deviation and the given threshold λ, the inner set I can be approximately detected *

[0105] I * ={(x i ,y i ):d i ≤λ,i∈N N}. (10)

[0106] from Figure 4 (a) It can be seen that in terms of total deviation, inliers and outliers (where * points represent outliers, o points represent inliers, the ordinate represents motion deviation, and the abscissa represents the index value of each feature point) are not clearly distinguishable. This is because the variation range of the deviation of each feature point is not uniform. In order to solve this problem, a quantization processing strategy is adopted.

[0107] Total deviation Quantization

[0108]

[0109] Among them, d i,j Represents x i The jth motion vector m in the neighborhood j The estimated motion vector corresponding to it The total deviation of the motion vector between s i,j The motion vector m i The motion characteristics of itself and its neighbors.

[0110] Finally, the feature point x i Itself includes all s in its neighborhood i,j Add them together to get the quantization deviation. The smaller the value, the more consistent the matching is with the motion vector consistency. At this time, the cumulative distribution of the quantized motion vector deviation of each feature point is as follows: Figure 4 As shown in (b), it can be seen that the gap between inliers and outliers becomes very obvious.

[0111]

[0112] S6: performing motion vector consistency constraints on the deviation of the motion vector, thereby optimizing the preset image feature matching model and determining hyperparameters;

[0113] S6 specifically includes:

[0114] (1) Matching model establishment and optimization

[0115] Assume that the initial matching set extracted from the two images is It contains N initial matches, x i and i is a two-dimensional column vector of the spatial position of the feature point. The initial matching set S contains many wrong matches.

[0116] In order to better preserve local features, the present invention adopts a universal mathematical model as an image feature matching model. The present invention defines I as an unknown internal point set, and its optimal solution is I * :

[0117] I * =argmin I C(I;U,S,λ) (13)

[0118] The cost C is defined as:

[0119] C(I;S,λ)=∑ i∈I ∑ j∈I (s i +λ(N-|I|)) (14)

[0120] where s i is the total deviation after quantization, and |·| is the cardinality of the set. In this cost function, the first term penalizes any match that does not maintain local consistency, the second term limits the number of outliers, and the parameter λ>0 is used to control the weight between these two terms. The goal of the present invention is to obtain the largest set of inliers by minimizing the cost function C.

[0121] In order to optimize the cost function, the initial matching set S is associated with an N×1 binary vector P, where p i ∈{0,1} indicates the i-th correspondence ( x i,y i ) is correct. Specifically, p i =1 indicates an interior point, p i =0 indicates an outlier.

[0122] By merging with p i Rearranging the common terms in the formula by the relevant terms gives the following formula:

[0123]

[0124] in

[0125] c i =s i (16)

[0126] c i We can measure the i-th match (x i ,y i ) satisfies the degree of local consistency. Obviously, a correct match will bring zero or low cost, while an incorrect match will bring high cost.

[0127] For a given initial matching set, the coordinates of the feature points are known, that is, the neighborhood structure and motion vectors between the feature points are fixed, so all the total cost values ​​can be calculated in advance. That is, the only unknown variable in equation (14) is p i The optimal solution of P that minimizes the cost function is determined by the following simple criterion:

[0128]

[0129] Therefore, the optimal inner set I * It can be determined by the following formula:

[0130] I * ={i|p i =1,i=1,···,N}. (18)

[0131] It can be seen from equation (17) that the parameter λ not only has the function of adjusting the trade-off, but also has the function of threshold. For different types of image pairs, there will be different distribution parameters, motion vector fields and costs, so they will also change.

[0132] (2) Determination of hyperparameters

[0133] So far, the present invention has transformed the feature matching problem into an optimized mathematical model. However, there are several hyperparameters, namely {n c ,n k ,λ,η}, the setting of their values ​​will seriously affect the results of the algorithm. Therefore, in order to obtain the best results and optimal parameter settings, the present invention selects 12 images from rigid deformation, rotation deformation, scale deformation and non-rigid deformation for experiments.

[0134] The present invention changes n c and n k We conducted several experiments and recorded the precision, recall rate, and F-score. Precision, recall rate, and F-score are commonly used indicators for evaluating the matching performance of algorithms.

[0135] like Figure 5 As shown, Figure 5 (a) shows that when n c =20, n k When changes, the precision, recall and F-score curves obtained by the method of the present invention; Figure 5 (d) shows that when n k =7, n c When the precision, recall rate and F-score curves of the method of the present invention are changed, it can be clearly seen from the result n c and n k In a certain range (n c ∈(10,40),n k ∈[5,9]) hardly changes and is insensitive, but beyond this range, precision, recall and F-score will drop significantly. Therefore, as long as it is within this range, the adaptive algorithm is designed to select n c and n k The optimal value of can get the best result.

[0136] In order to determine the optimal hyperparameter n in the Gaussian kernel convolution filtering process c , n k The present invention also uses the same data containing 48 pairs of images as the above experiment to conduct experiments.

[0137] First, the present invention prepares 12 groups of different (n c ,n k ) is the statistical data of the numerical deviation of the inlier and outlier motion vectors combined, where n c The value ranges from 10 to 40 in intervals of 10 (represented by four different grayscales), and n k The value ranges from 5 to 9 in intervals of 2 (three types of lines, namely, full curve, dotted line and dashed line). Figure 5 In (b), the downward trend line represents the statistics of the inliers, and the upward trend line represents the statistics of the outliers. c ,n k ) is the intersection of the inlier point and the corresponding outlier probability curve. The closer the intersection is to the lower left corner, the better the separability of the inlier layer and the outlier. Therefore, according to the experimental results, the present invention can use the initial matching set n c The relationship between the size of the set N is adaptively set to the hyperparameter n c and n k .

[0138]

[0139] Where N is the number of initial matches in the initial set of matches. [·] Elements may be rounded. odd(nc / 3) indicates the closest and not greater than n c / 3 is an odd number.

[0140] The present invention selects 10 images from the 48 test images used in the above experiment according to translation, scaling, rotation, and non-rigid transformation, and then selects another 10 remote sensing images, including all transformations. The same experiment is carried out using these 5 groups of 50 images, and the results are as follows Figure 5 (e) As shown in the results, different types of image deformations will lead to large differences in these intersections, which means that different optimal λ and η should be selected. Note that when c i =λ, p i The setting can be arbitrary.

[0141] In order to determine the best λ and η, the present invention uses the same data set (including 50 pairs of images) for experiments. First, the value of η is fixed, and different λ values ​​are used to calculate the precision, recall rate and F-score. Then the value of λ is fixed, and different η values ​​are used to calculate the precision, recall rate and F-score. The final results are shown in Figure 5 As shown in (c) and (f), when λ=0.45 and η=0.3, the present invention can obtain the highest precision, recall rate and F-score.

[0142] S7: Perform feature matching on the two images to be matched using the optimized image feature matching model to obtain an image feature matching result.

[0143] Figure 6 These are the qualitative results of the method of the present invention on 10 typical image pairs.

[0144] In order to evaluate the performance of the method of the present invention, five general image matching datasets were used in the experiment, each dataset containing 30 pairs of images and Ground Truth. These five datasets cover a wide range of content, mainly including remote sensing datasets and non-remote sensing datasets. Remote sensing datasets are divided into 720Yun, SUIRD, CIAP and SAR datasets. Non-remote sensing datasets include natural images, medical images, infrared images, etc. In order to ensure objectivity, the present invention uses the open source VLFeat toolbox to extract SIFT features from all datasets, and then constructs the initial matching set using the nearest neighbor strategy based on the similarity of the feature descriptors. The Ground Truth of all datasets is provided by manually marking each match as true or false, or by checking using the geometric transformation matrix provided by the original author.

[0145] Figure 6From top to bottom, each row contains two examples selected from five datasets: 720Yun, SUIRD, CIAP, SAR, and non-remote sensing. These image pairs have high proportion of outliers, small overlapping area, scale transformation, rotation transformation, translation transformation, and even non-rigid transformation, which brings great difficulties and challenges to the removal of false matches. From top to bottom and from left to right, the precision, recall rate and F-score of the method of the present invention are: (98.61%, 100.0%, 0.9930), (98.11%, 98.96%, 0.9854), (100.0%, 99.04%, 0.9952), (100.0%, 98.95%, 0.9947), (99.57%, 100.0%, 0.9978), (99.56%, 99.56%, 99.56%), (100.0%, 100.0%, 1), (99.49%, 99.80%, 99.64%), (98.95%, 99.15%, 0.9905), (97.84%, 98.81%, 0.9833). It can be easily found from the experimental results that the proposed method can identify most of the inliers with only a few false positives. This shows that the method is versatile and robust in processing remote sensing images with different types of image distortion and a large number of outliers. Moreover, the method has also achieved good experimental results on non-remote sensing datasets of common scenes.

[0146] Figure 6 There are two types of images in each row. The left image is the feature corresponding connection diagram, and the right image is the feature motion vector field diagram. The head and tail of each arrow on the motion vector field correspond to the spatial position of the matching feature points in the two images (the figure includes both correctly identified matches and misidentified matches, but misidentified matches are rare and almost zero). In order to make the matching in the image pair more visible, 100 matches are randomly selected in each pair of images and displayed as a straight line.

[0147] Figure 7 The experimental results of the five data sets are shown in curves. The quantitative evaluation indicators are precision, recall, F-sore and running time. Precision (P) is defined as the proportion of correct matches among all matches retained by the algorithm, recall (R) is defined as the ratio of correct matches identified by the algorithm to all correct matches, and F-score is used to evaluate the overall performance of the match. According to the number of true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN), the precision can be calculated according to the following formula:

[0148]

[0149] The recall rate is calculated as follows:

[0150]

[0151] Among them, a true positive (TP) indicates a correct match determined by the algorithm as a correct match; a true negative (TN) indicates an incorrect match determined by the algorithm as an incorrect match; a false positive (FP) indicates an incorrect match that is mistakenly determined by the algorithm to be a correct match; and a false negative (FN) indicates a correct match that is mistakenly determined by the algorithm to be a false match.

[0152] True positives (TP) and true negatives (TN) reflect the accuracy of the algorithm, while false positives (FP) and false negatives (FN) reflect the degree of error of the algorithm. F-score is a comprehensive statistic of precision and recall, and its solution formula is as follows:

[0153]

[0154] In order to make the results more convincing, the present invention will quantitatively evaluate and compare the feature matching performance of the method of the present invention, and compare it with LPM (Locality Preserving Matching, LPM), LLT (Locally Linear transform, LLT), LAF (Linear Adaptive Filtering, LAF), LMR (Learning a Two-Class Classifierfor Mismatch Removal, LMR), VFC (Vector field Consensus, VFC), RFM-SCAN and EES (Efficient Exact Search) algorithms. The codes of the compared algorithms are published from the original authors, and all parameters of the compared algorithms are set according to the original text and remain unchanged throughout the experiment.

[0155] Figure 7The results show that the method of the present invention has achieved good results on all remote sensing data sets, especially on the CIAP data set. The VFC algorithm has poor performance on some image pairs in the data set. This is because the inlier rate or number of inliers in these images is low, such as the CIAP data set, which interferes with the construction of the vector field. In SAR data sets with severe noise and affine distortion, the LMR algorithm produces many false positives and has a low recall rate. The method of the present invention and the LAF algorithm can not only recall more correct matches, but also have a higher accuracy rate than other algorithms. For the 720Yun data with non-rigid transformation and low inlier rate, the task of removing false matches is very difficult. The performance of the LLT algorithm on this data set is not ideal because the algorithm considers the linear transformation between feature points and has poor adaptability. For the SUIRD data set with large viewpoint changes, there are serious outliers, which increases the motion vector deviation. This will affect both the typical motion field of LAF and the motion consistency clustering of RFM-SCAN, thereby reducing the accuracy. Therefore, compared with other data sets, the performance of the LAF and RFM-SCAN algorithms is significantly reduced. The EES algorithm is based on the inlier model. When the image contains non-rigid deformation, low inlier ratio and external noise interference, the performance of geometric model fitting will be significantly reduced, such as the images of 720Yun, CIAP and SAR datasets. LPM and LMR algorithm based on deep learning have good performance on each dataset, but are slightly lower than the method of the present invention. In addition, the method of the present invention also has good performance on non-remote sensing datasets, indicating that the method of the present invention has certain versatility and scalability, and can be extended to process various types of images.

[0156] F-score can reflect the comprehensive performance of the algorithm. It can be seen from all remote sensing data sets that due to the combination of fixed and moving features of feature points, the method of the present invention can maintain robustness for remote sensing images of different problems and obtain the best precision and recall. However, in some cases, the indicators of the comparison algorithm may be reduced or even fail. In addition, the method of the present invention has linear complexity. From Figure 7 As can be seen from the last row, the algorithm in this study runs at a speed comparable to the LAF algorithm while ensuring comprehensive performance. It is slightly slower than the LPM algorithm, but much faster than other methods, especially when processing tens of thousands of matches.

[0157] Figure 7From left to right, LPM, LLT, LAF, LMR, VFC, RFM-SCAN, EES and the method of the present invention are quantitatively compared on five datasets: 720Yun, SUIRD, CIAP, SAR and Non-remote sensing. From top to bottom, they are the cumulative distributions of inlier rate, precision, recall rate, F-score and runtime. The coordinates (x, y) of the points on each curve represent the y value of the 30*xth image pair in each dataset (i.e., the value of precision, recall rate, F-score or runtime, in ms). The algorithm, average precision (AP), average recall rate (AR), average F-score (AF) and average runtime (runtime) represented by each curve are marked in the corresponding positions in the figure.

[0158] In some embodiments, a local consistency-based image feature matching device is also provided, comprising the following modules:

[0159] An image acquisition module, used for acquiring two images to be matched;

[0160] Feature point detection and descriptor creation module, used to detect feature points in two images through SIFT algorithm and create feature descriptors;

[0161] An initial matching set construction module, used to construct a set of initial matching sets according to the similarity of feature descriptors in two images;

[0162] A neighborhood consistency constraint module, used to delete false matches from the initial matching set through neighborhood consistency constraints to obtain a second matching set;

[0163] A motion vector deviation calculation module, used to calculate the deviation of the motion vector of each feature point through the second matching set;

[0164] A motion vector consistency constraint module, used to optimize a preset image feature matching model and determine hyperparameters by performing motion vector consistency constraints on the deviation of the motion vector;

[0165] The feature matching module is used to perform feature matching on two images to be matched through an optimized image feature matching model to obtain an image feature matching result.

[0166] The present invention uses a primary filtering strategy based on the consistency of the feature point neighborhood to remove a part of outliers with obvious errors, purify the neighborhood to obtain an approximate internal point set, and thus construct a more realistic neighborhood. The image space is gridded into multiple non-overlapping units, and an estimated motion vector is calculated for each unit through Gaussian kernel convolution operation to represent the potential motion characteristics of each unit; finally, by judging the consistency between the initial motion vector and the estimated motion vector in each unit, cunning mismatches are further eliminated to obtain a more accurate match.

[0167] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0168] The serial numbers of the embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. In a unit claim that lists several means, several of these means may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order and these words may be interpreted as identifiers.

[0169] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A local consistency-based image feature matching method, characterized in that: The local consistency includes: neighborhood consistency constraint and motion vector consistency constraint, and the image feature matching method includes the following steps: S1: Get two images to be matched; S2: Detect feature points in the two images using the SIFT algorithm and establish feature descriptors; S3: construct an initial set of matches based on the similarity of feature descriptors in the two images; S4: deleting mismatches from the initial matching set by using a neighborhood consistency constraint to obtain a second matching set; The S4 includes: S4.1: Obtain the attribute variables of each initial match in the initial match set through neighborhood consistency constraints S4.2: Set the threshold η by comparing η with Compare and filter the feature points that do not meet the neighborhood consistency constraints to obtain the second matching set U: U={(x i ,y i )∈S|Rst i >η} Among them, S represents the initial matching set, i represents the sequence number of the initial matching, (x i ,y i ) represents the i-th initial match, Rst i Represents all the neighborhoods corresponding to the i-th initial match The sum of M represents the number of neighbors taken by each initial match, j represents the sequence number of the neighborhood, that is, the jth neighborhood, K j represents the size of the jth neighborhood; S5: Calculate the deviation of the motion vector of each feature point through the second matching set; The S5 includes: S5.1: after performing vector conversion on the second matching set, spatially gridding the second matching set into a plurality of non-overlapping units, and calculating an estimated motion vector of each unit by Gaussian kernel convolution operation; S5.2: Calculate the deviation between the initial motion vector and the estimated motion vector in each unit; The S5.1 includes: S5.1.1: Convert the second matching set U into a vector set S': Among them, x i and i They represent the i-th feature point in the two images that constitute the initial match, i represents the sequence number of the feature point or the initial match, and m i =y i -x i Indicates that the i-th initial match (x i ,y i ) obtained, N represents the number of initial matches in the initial matching set; S5.1.2: Divide each dimension of the initial motion vector set S' into n c non-overlapping parts, and then we get G = n c ×n c units, so the initial motion vector set S' can be divided into G units, becoming a set Among them, n c represents the number of divisions of each dimension of the initial motion vector set S', G represents the number of divisions of the initial motion vector set S' as a whole, that is, S' is divided into G units, C j,k represents the cell in row j and column k, where j and k represent cell C respectively. j,k The row and column numbers; S5.1.3: Each unit C is obtained by Gaussian kernel convolution j,k The estimated motion vector The S5.2 includes: S5.2.1: Calculate a numerical deviation between the initial motion vector and the estimated motion vector; S5.2.2: Calculate the length ratio deviation between the initial motion vector and the estimated motion vector; S5.2.3: Calculate the angular deviation between the initial motion vector and the estimated motion vector; S5.2.4: Calculate a total motion vector deviation based on the numerical deviation, the length ratio deviation, and the angle deviation; After S5.2.4, it also includes: Performing quantization processing on the total deviation of the motion vector to obtain a quantization result; The quantization result is used to reflect the consistency between the initial motion vector and the estimated motion vector in each unit; S6: performing motion vector consistency constraints on the deviation of the motion vector, thereby optimizing the preset image feature matching model and determining hyperparameters; S7: Perform feature matching on any two images to be matched using the optimized image feature matching model to obtain an image feature matching result.

2. An image feature matching device based on local consistency, characterized in that: Includes the following modules: An image acquisition module, used for acquiring two images to be matched; Feature point detection and descriptor creation module, used to detect feature points in two images through SIFT algorithm and create feature descriptors; An initial matching set construction module, used to construct a set of initial matching sets according to the similarity of feature descriptors in two images; A neighborhood consistency constraint module, used to delete false matches from the initial matching set through neighborhood consistency constraints to obtain a second matching set; The neighborhood consistency constraint module includes: S4.1: Obtain the attribute variables of each initial match in the initial match set through neighborhood consistency constraints S4.2: Set the threshold η by comparing η with Compare and filter the feature points that do not meet the neighborhood consistency constraints to obtain the second matching set U: U={(x i ,y i )∈S|Rst i >η} Among them, S represents the initial matching set, i represents the sequence number of the initial matching, (x i ,y i ) represents the i-th initial match, Rst i Represents all the neighborhoods corresponding to the i-th initial match The sum of M represents the number of neighbors taken by each initial match, j represents the sequence number of the neighborhood, that is, the jth neighborhood, K j represents the size of the jth neighborhood; A motion vector deviation calculation module, used to calculate the deviation of the motion vector of each feature point through the second matching set; The motion vector deviation calculation module includes: S5.1: after performing vector conversion on the second matching set, spatially gridding it into a plurality of non-overlapping units, and calculating an estimated motion vector of each unit by Gaussian kernel convolution operation; S5.2: Calculate the deviation between the initial motion vector and the estimated motion vector in each unit; The S5.1 includes: S5.1.1: Convert the second matching set U into a vector set S': Among them, x i and i They represent the i-th feature point in the two images that constitute the initial match, i represents the sequence number of the feature point or the initial match, and m i =y i -x i Indicates that the i-th initial match (x i ,y i ) obtained, N represents the number of initial matches in the initial matching set; S5.1.2: Divide each dimension of the initial motion vector set S' into n c non-overlapping parts, and then we get G = n c ×n c units, so the initial motion vector set S' can be divided into G units, becoming a set Among them, n c represents the number of divisions of each dimension of the initial motion vector set S', G represents the number of divisions of the initial motion vector set S' as a whole, that is, S' is divided into G units, C j,k represents the cell in row j and column k, where j and k represent cell C respectively. j,k The row and column numbers; S5.1.3: Each unit C is obtained by Gaussian kernel convolution j,k The estimated motion vector The S5.2 includes: S5.2.1: Calculate a numerical deviation between the initial motion vector and the estimated motion vector; S5.2.2: Calculate the length ratio deviation between the initial motion vector and the estimated motion vector; S5.2.3: Calculate the angular deviation between the initial motion vector and the estimated motion vector; S5.2.4: Calculate a total motion vector deviation based on the numerical deviation, the length ratio deviation, and the angle deviation; After S5.2.4, it also includes: Performing quantization processing on the total deviation of the motion vector to obtain a quantization result; The quantization result is used to reflect the consistency between the initial motion vector and the estimated motion vector in each unit; A motion vector consistency constraint module, used to optimize a preset image feature matching model and determine hyperparameters by performing motion vector consistency constraints on the deviation of the motion vector; The feature matching module is used to perform feature matching on two images to be matched through an optimized image feature matching model to obtain an image feature matching result.

Images