Image matching method combining sparse and dense neighborhood consistency

Abstract

The application discloses an image matching method combining sparse and dense neighborhood consistency, which sequentially passes through a sparse-based attention graph neural network and a network based on dense field consistency, converts initial scores of sparse and dense matching into the same dimension, combines the initial matching scores of sparse and dense by using learnable parameters, inputs the combined matching scores into an optimal matching layer, obtains a final allocation matrix, and realizes image matching. The application adds global information to sparse matching by fusing dense matching, and dense matching also learns dense neighborhood information, avoids false matching caused by the complexity of a graph, promotes sparse and dense neighborhood consistency, and finally achieves high-accuracy matching effect.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology based on deep learning, and mainly relates to an image matching method that combines sparse and dense neighborhood consistency. Background Art

[0002] Image matching involves spatially aligning two or more images of the same object or scene acquired at different times and under different imaging conditions, or by finding corresponding patterns in the image to be matched using known patterns. With the advancement of science and technology, images have become increasingly important as a medium for information exchange. Image matching research is also a key method in image processing technology, often used in computer vision applications such as 3D reconstruction, visual positioning and navigation, and multi-target tracking.

[0003] Deep learning proposes a method that allows computers to automatically learn pattern features and integrates feature learning into the model building process, thereby reducing the incompleteness caused by manually designed features. Its powerful learning capabilities have greatly improved the accuracy of image matching, making deep learning-based image matching increasingly promising. The image matching process consists of feature point extraction, feature point description, nearest neighbor search, outlier filtering, and geometric transformation estimation. Sparse matching and dense matching algorithms are two commonly used image matching algorithms. However, sparse matching algorithms perform matching based on feature points. Because feature points are obtained as local maxima and there is no spatial connection between feature points, sparse matching algorithms lack global information. Dense matching algorithms, on the other hand, perform matching based on regional correlations. Due to the complexity of images, such as blur, repetition, or lack of texture, existing deep learning algorithms face significant challenges in processing them. Finding a method that preserves global information while avoiding blur, repetition, or lack of texture has become a pressing challenge for those skilled in the art. Summary of the Invention

[0004] The present invention addresses the problem of defects in both sparse matching and dense matching in the prior art and provides an image matching method that combines sparse and dense neighborhood consistency. The method converts the initial scores of sparse and dense matching into the same dimension after passing through a sparse attention graph neural network and a dense domain consistency network. The sparse and dense initial matching scores are combined using learnable parameters, and the combined matching scores are input into the optimal matching layer to obtain the final allocation matrix to achieve image matching. The present invention adds global information to sparse matching by fusing dense matching, and dense matching also learns dense neighborhood information, avoiding mismatches caused by the complexity of the graph, and ultimately achieving a highly accurate matching effect.

[0005] In order to achieve the above-mentioned purpose, the technical solution adopted by the present invention is: an image matching method that combines sparse and dense neighborhood consistency. The method converts the initial scores of sparse and dense matching into the same dimension after passing through a sparse attention graph neural network and a dense neighborhood consistency network in sequence. The sparse and dense initial matching scores are combined using learnable parameters, and the combined matching scores are input into the optimal matching layer to obtain the final distribution matrix to achieve image matching.

[0006] As an improvement of the present invention, the method comprises the following steps:

[0007] S1, feature extraction: After initializing and extracting features from the input image pair, the location information of feature points, descriptor information, and feature map information are obtained; the image pair is a picture of the same image taken from different perspectives; the feature extraction is performed using the superpoint algorithm;

[0008] S2, training a sparse attention graph-based neural network; encoding the position information and descriptor information obtained in step S1 and inputting them into the sparse attention graph-based neural network to obtain an initial matching score for sparse matching; learning the attention of each layer of the sparse attention graph-based neural network alternately uses self-attention and cross-attention, and calculates update information at each layer by simultaneously aggregating information from all nodes;

[0009] S3, network training based on dense neighborhood consistency; the feature map information obtained in step S1 is input into the network based on dense neighborhood consistency, and the initial matching score of dense matching is output; the similarity score of the feature map information is calculated, and the similarity score is processed by soft nearest neighbor filtering and domain consistency network to generate a four-dimensional matching score;

[0010] S4, same-dimension conversion: converting the initial matching score of the sparse matching obtained in step S2 and the initial matching score of the dense matching obtained in step S3 into the same dimension;

[0011] S5, iterative optimization: set two learnable coefficients as weights for adding the two matching scores, multiply the sparse and dense initial matching scores of the same dimension obtained after the conversion in step S4 by these two coefficients respectively, and then add them together to obtain the matching score;

[0012] S6, inputting the matching score obtained in step S5 into the optimal matching layer to obtain the final allocation matrix, thereby constructing a complete image matching model to achieve image matching.

[0013] As an improvement of the present invention, in step S2, the position of the feature point and the descriptor are coupled and encoded, and the position p is encoded by MLP. iEmbedded into high-dimensional space; After the image pair A and B are trained based on sparse attention neural network, the feature matching sub-f A 、f B , the initial matching score s of the sparse matching obtained by taking the inner product of the feature matching sub- sparse = <f A ,f B >.

[0014] As an improvement of the present invention, the sparse attention graph neural network training process in step S2 is specifically as follows:

[0015] The self-attention mechanism and the cross-attention mechanism are used alternately in different layers of the network. When the self-attention mechanism is used, the similarity between feature points in the image is learned; when the cross-attention mechanism is used, the similarity between feature points in the images is learned and converted into a probability distribution with a weight sum of 1. Finally, the weighted sum of all values ​​is performed to obtain the information transmitted between network layers; that is,

[0016] When the query for the i-th element is q i , according to the key k of this element j , retrieve their values ​​v j ; First, calculate the query q by dot product i and key k j The similarity of the key is obtained by introducing Softmax based on the similarity to obtain the weight coefficient of each key value, and the similarity is converted into a probability distribution with a weight sum of 1; finally, the weighted sum of all values ​​is obtained to obtain the information transfer m ε→i ;

[0017]

[0018] In the formula represents the representation of the i-th element in A at layer l, where [·|·] represents concatenation and ε is the set of self-attention and cross-attention.

[0019] As another improvement of the present invention, step S3 further includes:

[0020] S31: Feature map pair and The similarity score is calculated based on the feature graph information of

[0021]

[0022] Where C ijkl Represents the similarity score of the feature map pair;

[0023] S32: Similarity score C obtained in step S31 ijklPerform soft nearest neighbor filtering,

[0024]

[0025] Where C′ ijkl is the similarity score C ijkl The value after soft nearest neighbor filtering, Specific match C ijkl The ratio of the score to the best score on each pair of dimensions corresponding to images A and B;

[0026] S33: The filtered similarity score C′ ijkl Input 4D convolutional neighborhood network, learn different geometric information in the neighborhood through convolution, form consistency constraints, and output the initial matching score s with a dimension of 4 dense .

[0027] As another improvement of the present invention, the same-dimensional operation in step S4 specifically includes: selecting an area of ​​appropriate size on the original image with each feature point as the center, and after passing through the feature extraction network of step S1, the area is reduced, and the matching score of the area corresponding to its feature point is converted into the matching score of the corresponding feature point, and the initial matching score of the dense matching is converted into the same dimension as the initial matching score of the sparse matching through the coordinate index.

[0028] As another improvement of the present invention, the step S5 is specifically as follows: setting learnable coefficients w1 and w2, and the matching score s obtained by weighted summation of the learnable coefficients w1 and w2 is,

[0029] s=w1×s sparse +w2×s dense

[0030] Among them, the coefficients w1 and w2 usually start from a set of arbitrary values ​​and are iteratively updated in the direction of reducing the error as the network learns.

[0031] As a further improvement of the present invention, in step S6, the optimal matching layer performs a sinkhorn algorithm on the matching scores and outputs the optimal allocation matrix P ij By setting a matching threshold and eliminating points below the threshold, the final distribution matrix is ​​obtained, in which the i-th row and j-th column of the distribution matrix represent the matching probability between the i-th point in Figure A and the j-th point in Figure B.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] 1. The present invention improves the algorithm of sparse image matching and adds global matching information to sparse matching.

[0034] 2. The present invention combines sparse matching and dense matching algorithms, fully considering the relationship between local and global, and applying neighborhood consistency in dense branches. The neighborhood consistency of dense matching reduces mismatching caused by the complexity of the graph.

[0035] 3. The present invention sets learnable parameters and incorporates the weights of the combination of sparse matching and dense matching scores into the model training, which enhances the reliability of the matching results and has broad application prospects in the fields of 3D reconstruction, visual positioning and navigation. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a flow chart of the steps of the method of the present invention;

[0037] Figure 2 2 is a schematic diagram of image matching after using the method of the present invention in Example 2 of the present invention. DETAILED DESCRIPTION

[0038] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0039] Example 1

[0040] A joint sparse and dense neighborhood consistency image matching method, such as Figure 1 As shown, the following steps are included:

[0041] S1, feature extraction: After initializing the input images A and B and performing feature extraction, the location information of the feature points, descriptor information and feature map information are obtained;

[0042] The input image pairs, representing the same scene taken from different perspectives, are initialized and feature extracted using the superpoint algorithm. The feature extraction process is a combination of encoding and decoding. Encoding uses the same network, and feature map information is obtained through convolution and pooling. Extracting feature points and descriptor information involves two decoding processes. Feature extraction yields feature point location information, descriptor information, and feature map information. The output of the feature extraction network becomes the input to the sparse and dense neighborhood consistency networks.

[0043] S2, training a sparse attention-based neural network; encoding the position information and descriptor information obtained in step S1 and inputting them into the sparse attention-based neural network to obtain the initial matching score of the sparse matching;

[0044] The input of this step is: the location and descriptor of the feature point; the output is the initial matching score based on sparse matching:

[0045] (0) x i =d i +MLP enc (p i )

[0046] In the formula, the position of the feature point and the descriptor are coupled and encoded, and the position p is encoded using MLP. i Embedding into a high-dimensional space enables the subsequent attention mechanism to fully consider the appearance and position similarity of the features.

[0047] When the query for the i-th element is q i , according to the key k of this element j , retrieve their values ​​v j First, the query q is calculated by dot product i and key k j The similarity of the key is obtained by introducing Softmax based on the similarity to obtain the weight coefficient of each key value, and the similarity is converted into a probability distribution with a weight sum of 1. Finally, the weighted sum of all values ​​is obtained to obtain the information transfer m ε→i .

[0048]

[0049] In the formula represents the representation of the i-th element in graph A at layer l, where [·|·] denotes concatenation, and ε is the combination of self-attention and cross-attention. Each layer of the network alternates between self-attention and cross-attention. Updates are computed at each layer by simultaneously aggregating information from all nodes. Similarly, all features in graph B undergo a similar update.

[0050] After learning the attention graph network, the feature matching sub-f A 、f B :

[0051] s sparse = <f A ,f B >

[0052] Where s sparse It is the initial matching score of the sparse matching obtained by taking the inner product of the feature matcher.

[0053] The feature descriptor and keypoint position features are encoded and coupled, allowing the subsequent attention mechanism to fully consider the appearance and positional similarity of the features. The attention mechanism scans the entire image to identify and prioritize important areas. The attention mechanism in the neural network also prioritizes more critical information based on importance. The core goal is to allow the network to focus on areas where it is most needed, improving the model's computational power and training efficiency. Attention mechanisms are categorized as self-attention and cross-attention. Self-attention learns information within the image, while cross-attention transfers information between images. After training an attention graph neural network based on sparse neighborhood consistency, matching descriptors are derived, and correlation is calculated to obtain an initial matching score for the sparse portion.

[0054] S3, network training based on dense neighborhood consistency; the feature map information obtained in step S1 is input into the network based on dense neighborhood consistency, and the initial matching score of the dense matching is output;

[0055] The input of this step is: feature map pair and The output is a dense initial matching score.

[0056]

[0057] Where C ijkl Represents the similarity score of the feature map pair

[0058]

[0059] Where C′ ijkl is the similarity score C ijkl The value after soft nearest neighbor filtering, Specific match C ijkl The ratio of the score of to the best score on each pair of dimensions corresponding to images A and B is used to reduce the similarity score of non-nearest neighbor matches.

[0060] The filtered similarity score is input into the 4D convolutional neighborhood network, which learns different geometric information in the neighborhood through convolution, thereby forming consistency constraints and enhancing adaptability to complex graphs. The final output is the initial matching score s with a dimension of 4. dense .

[0061] The feature map obtained in step S1 is and Both are represented in a 4-D matching space (i, j, k, l), and their correlation scores are stored in a 4-D correlation tensor C. These matches are further processed by soft nearest neighbor filtering and a neighborhood consistency network to generate the final four-dimensional matching score.

[0062] S4, same-dimension conversion: converting the initial matching score of the sparse matching obtained in step S2 and the initial matching score of the dense matching obtained in step S3 into the same dimension;

[0063] The initial score for sparse matching is the matching score between pairs of points, a 2-dimensional tensor. Dense matching is the matching score between pairs of coordinates. Since coordinates have two dimensions, x and y, the initial score for dense matching is a 4-dimensional tensor. An appropriately sized region of the original image is selected, centered around each feature point. This region is then reduced to a single point through the feature extraction network. The matching score for this region is then converted to the matching score for the corresponding feature point. Using coordinate indexing, the initial matching score for dense matching is converted to the same dimension as the initial matching score for sparse matching.

[0064] S5, iterative optimization: set two learnable coefficients as weights for adding the two matching scores, multiply the sparse and dense initial matching scores of the same dimension obtained after the conversion in step S4 by these two coefficients respectively, and then add them together to obtain the matching score;

[0065] The learnable coefficients w1 and w2 are set, and gradient descent is turned on, usually starting from a set of arbitrary values, and then iteratively updating these values ​​in the direction of decreasing error as the network learns.

[0066] s=w1×s sparse +w2×s dense

[0067] Where s is the matching score obtained by weighted summation of learnable coefficients w1 and w2, which is the matching result that comprehensively considers the consistency of sparse and dense neighborhoods.

[0068] S6, inputting the matching score obtained in step S5 into the optimal matching layer to obtain the final allocation matrix, thereby constructing a complete image matching model to achieve image matching.

[0069] The optimal matching layer performs the sinkhorn algorithm on the matching scores and outputs the optimal allocation matrix P ij By setting a matching threshold and removing points below the threshold, we can get the final allocation matrix. The i-th row and j-th column of the allocation matrix represent the matching probability between the i-th point in image A and the j-th point in image B.

[0070] Example 2

[0071] An image matching method combining sparse and dense neighborhood consistency, comprising the following steps:

[0072] S1: Perform feature extraction on the input image pair. The input image pair passes through the feature extraction network to obtain the location information of the feature points, descriptor information, and feature map information.

[0073] The experimental dataset is Oxford and Paris R1M, which is a large dataset containing 966,221 images.

[0074] Each image is initially cropped to 640*480 pixels and passed through the feature extraction network to select 512 feature points. The resulting feature map is 1024*20*15 pixels in size, the descriptor is 256*512 pixels in size, and the feature point location is 2*512 pixels in size. Each feature point contains two-dimensional coordinates.

[0075] S2: Attention Graph Neural Network based on sparse neighborhood consistency, which uses the position information and descriptor information extracted from step S1 for encoding as input.

[0076] The result of feature point encoding is in the form of 256*512, which is input to the attention graph neural network. The attention graph neural network outputs the feature descriptor in the form of 256*512.

[0077] S3: Train a dense neighborhood consistency-based network. The dense consistency-based network uses the feature map information extracted in step S1 as input in the form of 256*20*15.

[0078] The similarity score of the feature map pair is achieved through matrix multiplication, outputting a 4D tensor of 20*15*20*15.

[0079] The dense neighborhood consistency network consists of three layers of 4D convolution and three activation layers.

[0080] The similarity score learns the information of surrounding features through 4D convolution in dense neighborhood consistency to form a consistency constraint, thereby reducing the score of non-neighbor matching and enhancing matching performance.

[0081] S4: Convert the sparse and dense matching initial scores obtained in steps S2 and S3 respectively into the same dimension.

[0082] In step S2, the initial matching score output by the sparse neighborhood consistency network is a 2-dimensional tensor, representing the similarity between each point pair; in step S3, the initial matching score output by the dense neighborhood consistency network is a 4-dimensional tensor, in which each point is represented by a 2-dimensional coordinate, thus forming a 4-dimensional tensor.

[0083] To combine the two matching scores, they need to be converted to the same dimension. On each of images A and B, a 32*32 area around the feature point is selected. After feature extraction in step 1, images A and B are transformed from their initial 640*480 size to feature maps of 20*15. Simultaneously, the 32*32 area on images A and B is transformed into a 1*1 area on the feature map, corresponding to the central point, the selected feature point. Therefore, the similarity of the feature point corresponds to the similarity of the area, combining the sparse and dense matching scores.

[0084] By scaling the feature point coordinates by the factor that reduces the size of images A and B relative to the feature map, we can use the feature points to index the dense initial matching scores to obtain the similarity matching scores for the corresponding regions in images A and B. Finally, the dense initial matching scores are converted to 512*512, the same dimensions as the sparse initial matching scores. This corresponds to the 512 feature points selected in step 1 and represents the point-to-point matching scores.

[0085] S5: Combining sparse and dense initial matching scores using learnable parameters.

[0086] Set two learnable parameters as weights for adding the two matching scores. Multiply the sparse and dense initial matching scores converted to the same dimension obtained in step 4 by these two coefficients respectively. Set gradient descent. These two learnable coefficients will be continuously iteratively updated as the network trains, moving towards reducing the error.

[0087] S6: Input the combined matching scores into the optimal matching layer to obtain the final allocation matrix.

[0088] The optimal matching layer passes the combined scores through the sinkhorn algorithm and outputs the optimal allocation matrix P ij By setting a matching threshold and removing points below the threshold, we can obtain the final distribution matrix. The i-th row and j-th column of the distribution matrix represent the matching probability between the i-th point in image A and the j-th point in image B, thus constructing a complete image matching model and achieving image matching. Figure 2 This is a schematic diagram of image matching after using the method of the present invention. The upper part is the input image, and the lower part is the matching result. From both global and local perspectives, the matching effect of combining sparse and dense neighborhood consistency is better.

[0089] In summary, the present invention is an image matching method based on joint sparse and dense neighborhood consistency. The image matching model trained using this method can run on a computer or other device. By inputting image pairs, it can output matching corresponding points. It has a wide range of applications in three-dimensional reconstruction, visual positioning and navigation, multi-target tracking and other fields.

[0090] It should be noted that the above content merely illustrates the technical idea of ​​the present invention and cannot be used to limit the scope of protection of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications all fall within the scope of protection of the claims of the present invention.

Claims

1. An image matching method combining sparse and dense neighborhood consistency, characterized by: The method converts the initial scores of sparse and dense matching into the same dimension after sequentially passing through a sparse attention-based graph neural network and a dense domain consistency-based network. The sparse and dense initial matching scores are combined using learnable parameters, and the combined matching scores are input into the optimal matching layer to obtain the final allocation matrix to achieve image matching. The method specifically includes the following steps: S1, feature extraction: After initializing and extracting features from the input image pair, the location information of feature points, descriptor information, and feature map information are obtained; the image pair is a picture of the same image taken from different perspectives, and the feature extraction is performed using the superpoint algorithm; S2, training a sparse attention graph-based neural network; encoding the position information and descriptor information obtained in step S1 and inputting them into the sparse attention graph-based neural network to obtain an initial matching score for sparse matching; learning the attention of each layer of the sparse attention graph-based neural network alternately uses self-attention and cross-attention, and calculates update information at each layer by simultaneously aggregating information from all nodes; S3, network training based on dense neighborhood consistency; the feature map information obtained in step S1 is input into the network based on dense neighborhood consistency, and the initial matching score of dense matching is output; the similarity score of the feature map information is calculated, and the similarity score is processed by soft nearest neighbor filtering and domain consistency network to generate a four-dimensional matching score; S4, same-dimension conversion: converting the initial matching score of the sparse matching obtained in step S2 and the initial matching score of the dense matching obtained in step S3 into the same dimension; S5, iterative optimization: set two learnable coefficients as weights for adding the two matching scores, multiply the sparse and dense initial matching scores of the same dimension obtained after the conversion in step S4 by these two coefficients respectively, and then add them together to obtain the matching score; S6, inputting the matching score obtained in step S5 into the optimal matching layer to obtain the final allocation matrix, thereby constructing a complete image matching model to achieve image matching.

2. The image matching method of claim 1, wherein: In step S2, the position of the feature point and the descriptor are coupled and encoded, and the position is encoded using MLP. Embedded into high-dimensional space; After the image pair A and B are trained based on sparse attention neural network, feature matching sub- 、 , the initial matching score of the sparse matching obtained by taking the inner product of the feature matching sub 3. The image matching method based on joint sparse and dense neighborhood consistency according to claim 2, characterized in that: The sparse attention graph neural network training process in step S2 is specifically as follows: the self-attention mechanism and the cross-attention mechanism are used alternately in different layers of the network. When the self-attention mechanism is used, the similarity between feature points in the image is learned; when the cross-attention mechanism is used, the similarity between feature points between images is learned and converted into a probability distribution with a weight sum of 1. Finally, all values ​​are weighted summed to obtain the information transmitted between network layers; Right now: When the query for the i-th element is , according to the key of this element , retrieve their values ; First, calculate the query by dot product and key The similarity is then calculated. Softmax is introduced based on the similarity to obtain the weight coefficient of each key value, and the similarity is converted into a probability distribution with a weight sum of 1. Finally, the weighted sum of all values ​​is obtained to obtain the information transfer ; (l+1) (l) ([ (l) || ]) In the formula (l+1) represents the representation of the i-th element in A at layer l, where [·|·] represents concatenation and Ɛ is the set of self-attention and cross-attention.

4. The image matching method of claim 2, wherein: The step S3 further comprises: S31: Feature map pair and The similarity score is calculated based on the feature graph information of ； Where, Represents the similarity score of the feature map pair; S32: Similarity score obtained in step S31 Soft nearest neighbor filtering, , ； In the formula is the similarity score The value after soft nearest neighbor filtering, Specific matches The ratio of the score to the best score on each pair of dimensions corresponding to images A and B; S33: The similarity score after filtering Input 4D convolutional neighborhood network, learn different geometric information in the neighborhood through convolution, form consistency constraints, and output the initial matching score with a dimension of 4 .

5. The image matching method of combining sparse and dense neighborhood consistency according to claim 3 or 4, characterized in that: The same-dimensional operation in step S4 specifically includes: selecting an area of ​​appropriate size on the original image with each feature point as the center, and after passing through the feature extraction network of step S1, the area is reduced, and the matching score of the area corresponding to its feature point is converted into the matching score of the corresponding feature point. Through the coordinate index, the initial matching score of the dense matching is converted into the same dimension as the initial matching score of the sparse matching.

6. The image matching method of claim 5, wherein: The step S5 is specifically as follows: setting the learnable coefficient 、 , the learnable coefficient 、 The matching score s obtained by weighted summation is, ； Among them, the coefficient 、 Starting from an arbitrary set of values, they are iteratively updated in the direction of decreasing error as the network learns.

7. The image matching method of claim 6, wherein: In step S6, the optimal matching layer performs a sinkhorn algorithm on the matching scores and outputs the optimal allocation matrix. By setting a matching threshold and eliminating points below the threshold, the final distribution matrix is ​​obtained, in which the i-th row and j-th column of the distribution matrix represent the matching probability between the i-th point in Figure A and the j-th point in Figure B.

Images