A feature matching method based on attention mechanism and neighborhood consistency

By introducing graph neural network and node shading evaluation methods in feature matching, combined with attention network, the problem of attention mechanism ignoring local information in feature matching is solved, and more stable and accurate feature matching is achieved, suitable for image matching Quasi- and three-dimensional reconstruction and other fields.

CN114758152BInactive Publication Date: 2024-11-26SOUTHEAST UNIV +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210438479.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-25
Publication Date
2024-11-26
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

When applying the attention mechanism, the feature matching method based on deep learning ignores the local information between the descriptors, resulting in a lack of neighborhood consistency and affects the matching quality.

Method used

Graph neural network and node shading are used to evaluate the neighborhood consistency of each node, and consistency information is integrated into the descriptor through a multi-layer perceptron, combining the attention network to perceive global and local information simultaneously, and improving the feature matching algorithm.

Benefits of technology

The matching quality between descriptors is improved, the stability and accuracy of feature matching are enhanced, and it is suitable for tasks such as image registration and three-dimensional reconstruction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114758152B_ABST
    Figure CN114758152B_ABST
Patent Text Reader

Abstract

The present invention provides a feature matching method based on attention mechanism and neighborhood consistency, including: using convolutional neural network to extract image key points and their descriptors; encoding key point positions and integrating position information into descriptors; using self-attention network and cross-attention network to enhance the representation ability of descriptors and find the corresponding relationship between two image descriptors; constructing a graph using k-nearest neighbor algorithm according to attention weight, using graph neural network to evaluate the neighborhood consistency of nodes, and integrating consistency information into attention aggregation; according to the enhanced descriptors, using Sinkhorn algorithm and mutual nearest neighbor screening to calculate matching results. The present invention proposes a method of integrating neighborhood consistency into the attention mechanism, dynamically evaluating the neighborhood consistency in the attention aggregation process while the attention mechanism aggregates global context information, combining this local information with global information, and jointly enhancing the representation ability of the descriptor, thereby obtaining a higher quality match.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a feature matching method and device technology based on attention mechanism and neighborhood consistency, and belongs to the field of computer vision technology based on deep learning. Background Art

[0002] Feature matching refers to finding the correct correspondence between points and features in two images, and is a key step in many 3D computer vision tasks, such as structure from motion, simultaneous localization and mapping. Getting the right match can lay a good foundation for these upstream tasks, but occlusion, blur, similar repeated textures, lighting, and perspective changes can make it extremely challenging. Therefore, how to find the information hidden in the descriptor, such as contextual information and neighborhood consistency information, to eliminate interference is one of the most important processes in feature matching.

[0003] The classic feature matching process generally consists of four parts: (1) key point detection (2) feature description (3) feature matching (4) geometric transformation estimation. Before the rise of deep learning methods, there were many detection and description methods based on manual design, which had good performance and were widely used in 3D computer vision tasks. The basic methods of feature matching and geometric transformation estimation are nearest neighbor search and random sampling consensus algorithm.

[0004] With the rapid development of deep learning, key point detection and feature description methods based on deep learning have improved the representation capabilities of key points and descriptors, enabling feature matching under challenging conditions such as drastic lighting and perspective changes.

[0005] However, it is still difficult to obtain high-quality matches only through the traditional matching method of nearest neighbor search, because it ignores the contextual information contained in the descriptor. Deep learning-based matchers can effectively solve this problem. For example, the attention network-based method can use self- and cross-attention networks to aggregate context. This is an effective feature matching structure, but it will lead to the lack of neighborhood consistency when the attention mechanism is applied to the feature matching task, because the attention network can be regarded as a graph neural network with all nodes fully connected. This means that adjacent nodes in the source graph can be mapped to very different areas in the target graph, which violates the neighborhood consistency criterion of the matching problem. Therefore, this method innovatively incorporates neighborhood consistency into attention aggregation while applying the attention mechanism. Summary of the invention

[0006] The problem solved by the present invention is that when self-attention and cross-attention networks are applied to feature matching tasks, they can perceive the global information contained in the descriptor, but they ignore the local information between descriptors. Therefore, this method uses graph neural networks and node coloring to evaluate the neighborhood consistency of each node, and uses a multi-layer perceptron to integrate the consistency information into the descriptor of each node. The combination of attention network and consistency enables this network to perceive global information and local information at the same time, further enhance the representation ability of the descriptor, and obtain better matching results.

[0007] To achieve the above object, the present invention adopts the following technical solution: a feature matching method based on attention mechanism and neighborhood consistency, which includes:

[0008] Step 1: Read the grayscale image and use a deep convolutional neural network to extract the key points p and their descriptors d.

[0009] Step 2: Take the key point p and descriptor d obtained in step 1 as input, and let the position and descriptor of the i-th key point be p and d respectively. i and d i , this method uses a multi-layer perceptron MLP pc The spatial position information of the key points is embedded into the descriptor, that is:

[0010]

[0011] Step 3: The descriptors obtained in step 2 are applied alternately in the self-attention network and the cross-attention network for 9 times. The self-attention network is used to enhance the representation ability of the descriptors, and the cross-attention network is used to find the correspondence between the descriptors of the two images. Both transfer the global information of the image. Let the input of the lth layer be x t (l) and x s (l) , both the self-attention layer and the cross-attention layer use a multi-head attention structure. The difference between the two is that the input of the former is the same descriptor, while the input of the latter is descriptors from different images. First, the input descriptor is layered and normalized, and then the linear layer is used to transform x t (l) Linear projection to K (l) , V (l) , x (l) Linear projection to Q (l) The multi-head attention is expressed as:

[0012] Attention(Q (l) , K (l) , V (l) )=Softmax(Q (l) (K(l) ) T )V (l)

[0013]

[0014]

[0015]

[0016] Among them, Q, K, and V are divided into h heads according to the number of channels. i , K i 、V i (i=0,...,h) calculate the attention respectively, then use the linear layer to merge the h heads, and then input them into the feedforward network to generate the input for the next step.

[0017] The self-attention layer in step 3 additionally incorporates information about neighborhood consistency. Now we will explain how to calculate neighborhood consistency and how to integrate it into attention aggregation. First, we use the k-nearest neighbor algorithm to construct the descriptors of the two images into two graphs based on the attention weights, named source graph G. s and the target graph G t , each node establishes edges with the 8 most similar nodes to form its "neighborhood". The edges formed in the two graphs are represented by E s (l) and E t (l) In the two constructed graphs, a random node coloring matrix R is generated as the source graph coloring matrix, and the corresponding matrix of the current key point is calculated using the inner product of the two descriptors Then The inner product of and R is used to calculate the transformed coloring matrix T, which is the coloring matrix of the target graph. In order to evaluate the neighborhood consistency of each node, information transfer is performed on the two graphs through a shared graph neural network to generate a new node coloring representation, namely

[0018]

[0019]

[0020] Neighborhood consistency can be achieved by coloring the two graphs with new nodes Then use a multilayer perceptron as a feedforward network for consistency information:

[0021]

[0022] Finally, a linear layer is used to fuse the attention information with the neighborhood consistency information, and a residual connection is used to update the descriptor.

[0023]

[0024]

[0025] Step 5. After 9 layers of attention aggregation, the inner product of the two image descriptors at the matching layer is used to calculate the final correspondence matrix S, which is used as the cost matrix of the optimal transportation problem. Then the Sinkhorn algorithm is applied iteratively to obtain the optimal allocation matrix P. Based on the optimal allocation matrix P, matches with values ​​less than the matching threshold are first excluded, and then the mutual nearest neighbor criterion is used to select the final match M;

[0026] Step 6: Build a PyTorch deep learning development environment and divide the dataset into training, validation, and test sets. During training, calculate the true match M based on the homography matrix (homography estimation task) or camera position and depth (camera pose estimation task). gt The loss function is the negative log-likelihood loss function on the optimal allocation matrix, that is,

[0027]

[0028] The attention network and the neighborhood consistency network are jointly trained under the supervision of this loss function.

[0029] Beneficial effects:

[0030] The present invention is a feature matching method based on attention mechanism and neighborhood consistency, which has the following beneficial effects:

[0031] 1. This method uses a deep convolutional network to extract sparse key points and descriptors instead of extracting dense feature maps, which effectively improves the shortcomings of the square space and time complexity of the attention mechanism.

[0032] 2. This method uses a deep attention network to strengthen the descriptor. Unlike a deep convolutional network, this network has a global receptive field and can aggregate information from all nodes, so that the descriptor can fully perceive the global context information, thereby improving the matching quality between descriptors;

[0033] 3. This method takes into account that the attention module is permutation invariant and cannot distinguish each node in space. Therefore, it is necessary to embed the position information into the descriptor. A multi-layer perceptron is used to embed the spatial position information of the key points into the initial representation of the generated features.

[0034] 4. This method improves the shortcoming of lack of local information based on the attention model, introduces the important criterion in the matching problem, the matching domain consistency criterion, and designs a graph neural network that evaluates neighborhood consistency to integrate local information into global information.

[0035] In summary, the feature matching method based on attention mechanism and neighborhood consistency improves the feature matching algorithm based on attention mechanism, and obtains a more stable and accurate feature matching model, which has broad application prospects in image registration, three-dimensional reconstruction, visual positioning and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is the overall flow chart of the present invention;

[0037] Figure 2 It is a comparison of the flow charts of conventional attention aggregation and attention aggregation incorporating neighborhood consistency;

[0038] Figure 3 It is a flow chart of the neighborhood consistency network in the method of the present invention. DETAILED DESCRIPTION

[0039] The following is a detailed description of a feature matching method based on attention mechanism and neighborhood consistency proposed by the present invention in conjunction with the accompanying drawings:

[0040] Step 1: Input a single image, perform a random homography transformation on the input image and generate a homography matrix, obtain the original image and the transformed image of the input network, and the groundtruth homography matrix used to supervise network training. Use the SuperPoint deep convolutional network to extract key points and descriptors from the two images, and obtain the input key point coordinates p and 256-dimensional descriptor d based on the attention mechanism and neighborhood consistency model. Assuming that images A and B have M and N key points respectively, the dimensions of the key points p of the two images are (M, 2) and (N, 2), respectively, and the dimensions of the descriptor d are (M, 256) and (N, 256), respectively.

[0041] The experimental dataset is R1M, which is an image retrieval dataset containing 1,001,001 high-resolution images. It covers images of more than 20 landmarks around the world, with complex perspective changes, illumination changes, blur and occlusion, etc., which is very suitable for training and evaluating feature matching networks. R1M is divided into training set, validation set and test set, containing 1,000,000, 501 and 500 images respectively.

[0042] Step 2: Figure 1 , taking the key point p and descriptor d obtained in step 1 as input, a multilayer perceptron is used to embed the spatial position information of the key point into the descriptor. The multilayer perceptron has three hidden layers, which expands the key point dimension from 2 to 32, 64, and 128 in sequence. Finally, a 256-dimensional position encoding vector is output, which is added to the descriptor to integrate the position information and generate the initial representation x of the feature. i (0) .

[0043] Step 3: Initialize the features obtained in step 2 as x i (0) The self-attention network and the cross-attention network are alternating for 9 times. The self-attention network strengthens the descriptor representation ability, and the cross-attention network finds the correspondence between the descriptors of the two pictures. Both transfer the global information of the image. Suppose the input of the l-th layer of attention network is x s (l) and x t (l) , the input of the self-attention network is the descriptor of the same image, that is, x s (l) =x t (l) , the input of the cross attention is the descriptor of the two pictures. Use three linear layers to transform x t (l) Projection to K (l) 、V (l) , x s (l) Projection to Q (l) , the feature dimension is still 256. The attention network in this method uses a multi-head attention mechanism, dividing Q, K, and V into 4 heads, each with a dimension of 64, and calculating attention in each head.

[0044] In order to incorporate neighborhood consistency information into the self-attention network, we first construct a graph using the k-nearest neighbor algorithm based on the attention weights. Each node establishes edges with the eight most similar nodes to form its "neighborhood". In the two constructed graphs, a random node coloring matrix R is generated for one of the graphs as the source graph coloring matrix. Each node random coloring vector is 64-dimensional, and the dimension of the random coloring matrix is ​​(M, 64). Figure 2 , use the inner product of the two descriptors to calculate the corresponding matrix of the current key point Its dimension is (M,N). Then The inner product of R and the transformed coloring matrix T is calculated, that is, the coloring matrix of the target graph, whose dimension is (N, 64). Information transfer is performed on the two graphs through a shared graph neural network to generate a new node coloring representation. This method uses the SplineConv network in the graph neural network library PyTorch geometric as the graph neural network for information transfer. After the coloring information is transferred, the neighborhood consistency can be obtained by the new node coloring of the two graphs. Then, a multi-layer perceptron is used as the feedforward network for consistency information, and the output dimensions of each layer are 64 and 64 respectively. Figure 3, merge the 256-dimensional attention information and the 64-dimensional neighborhood consistency information into a 320-dimensional vector, and then use a multi-layer perceptron to fuse the two. The output dimensions of each layer are 320 and 256. Finally, use a residual connection to add the updated information of this layer to the descriptor of the input layer to update the descriptor. Figure 2 A comparison between self-attention aggregation with coherence and regular self-attention aggregation is shown.

[0045] Step 4: After using global information and local information to strengthen the descriptor, multiply the two image descriptors to obtain the score matrix S, whose dimension is (M, N), and use it as the cost matrix of the optimal transportation problem. Then apply the Sinkhorn algorithm to iterate to obtain the optimal allocation matrix P, whose dimension is consistent with S. The number of iterations of the Sinkhorn algorithm in this method is 20. Based on P, first filter out matches that are less than the matching threshold, and finally use the mutual nearest neighbor criterion to select the final match M. The matching threshold in this method is 0.2;

[0046] Step 5. This method uses PyTorch for training, and sets relevant parameters based on engineering parameter setting experience. Set the batch size to 4, which means loading 4 sample images each time, set the AdamW optimization parameters as the basic iterator, set the initial learning rate to 0.0001, set the warm up epoch to 1, apply cosine learning rate decay, set the momentum factor to 0.9, set the weight decay factor to 0.0001, and train a total of 30 epochs. During training, calculate the true match M based on the groundtruth homography matrix and key points gt The condition for true matching is that the reprojection error is less than 3 pixels. The loss function is the negative log-likelihood loss function on the optimal allocation matrix, and the attention network and the neighborhood consistency network are jointly supervised by this loss function.

[0047] The above content merely illustrates the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of this technical solution belong to the technical idea proposed by the present invention and fall within the protection scope of the claims of the present invention.

Claims

1. A feature matching method based on attention mechanism and neighborhood consistency, characterized in that: The specific steps include: Step 1: Read the grayscale image and use a deep convolutional neural network to extract the key points of the image and its descriptors; Step 2: Using the key points and descriptors obtained in step 1 as input, a multi-layer perceptron is used to integrate the spatial position information of the key points into the descriptors; Step 3: The descriptors obtained in step 2 are alternately applied to the self-attention network and the cross-attention network for 9 times. The self-attention network strengthens the representation ability of the descriptors, and the cross-attention network finds the corresponding relationship between the descriptors of the two pictures. Both transmit the global information of the image. In the self-attention network, the neighborhood consistency information is integrated, and the k-nearest neighbor algorithm is used to construct a graph according to the attention weights of the two pictures. Each node establishes edges with the 8 most similar nodes to form its "neighborhood". Based on the two constructed graphs, the graph neural network is used to perform message passing to generate a new node coloring representation. The neighborhood consistency is measured by the difference between the new node coloring representations of the two graphs. The multi-layer perceptron is used to integrate the neighborhood consistency information into the self-attention aggregation, that is, to integrate the local information into the global information. Step 4: After using global information and local information to enhance the descriptor, the score matrix is ​​obtained by multiplying the two image descriptors and used as the cost matrix of the optimal transportation problem. Then, the Sinkhorn algorithm is applied iteratively to obtain the optimal allocation matrix. Based on the optimal allocation matrix, the matches with values ​​less than the matching threshold are first filtered out, and finally the mutual nearest neighbor criterion is used to select the final match. Step 5: Calculate the loss based on the output matching results and the true matching, and train the attention network and neighborhood consistency network.

2. The feature matching method based on attention mechanism and neighborhood consistency according to claim 1, characterized in that: Step 1: Different from the conventional deep attention network processing dense feature maps, the deep convolutional network is first used to extract sparse key points and descriptors, and then the attention mechanism is applied to enhance the descriptors, which improves the shortcomings of the square space and time complexity of the attention mechanism.

3. The feature matching method based on attention mechanism and neighborhood consistency according to claim 1, characterized in that: Step 2: Since the attention module is permutation invariant and cannot distinguish each node spatially, the position information is embedded into the descriptor, and a multi-layer perceptron is used to embed the spatial position information of the key points to generate the initial representation of the feature.

4. The feature matching method based on attention mechanism and neighborhood consistency according to claim 1, characterized in that: Step 3: The descriptor is strengthened by using a deep attention network. Unlike a deep convolutional network, this network has a global receptive field and aggregates information from all nodes. This feature enables the descriptor to fully perceive the global context information, thereby improving the matching quality between descriptors.

5. The feature matching method based on attention mechanism and neighborhood consistency according to claim 1, characterized in that: Step 3: While transmitting global information, the deep attention network also ignores the local information between descriptors, including the important criterion in the matching problem, the matching domain consistency criterion. To solve this problem, we first use the k-nearest neighbor algorithm to construct the descriptors of the two images into two graphs according to the attention weights, named source graph G. s and the target graph G t ,Each node in both graphs is only connected to its domain; Generate a random node coloring matrix R as the source graph coloring, and use the inner product of the two descriptors to calculate the corresponding matrix of the point Then The inner product of and R is used to calculate the transformed coloring matrix T, i.e., the target graph coloring. In order to evaluate the neighborhood consistency of each node, message passing is performed on the two graphs through a shared graph neural network to generate a new node coloring representation. The neighborhood consistency is measured by the difference between the new node coloring representations of the two graphs. Finally, a multilayer perceptron is used as the feedforward network for the consistency information.

6. The feature matching method based on attention mechanism and neighborhood consistency according to claim 1, characterized in that: Step 4: After 9 layers of attention aggregation, the inner product of the two image descriptors is calculated at the matching layer to obtain the final correspondence matrix S, which is used as the cost matrix for the optimal transportation problem; Then the Sinkhorn algorithm is applied iteratively to obtain the optimal allocation matrix P; based on P, matches smaller than the matching threshold are first excluded, and then the final match M is selected using the mutual nearest neighbor criterion.

7. The feature matching method based on attention mechanism and neighborhood consistency according to claim 1, characterized in that: Step 5: During training, the ground-truth matches M are computed from the homography matrix or camera position and depth gt ; The attention network and the neighborhood consistency network are jointly supervised and trained by the negative log-likelihood loss function on the optimal assignment matrix.