An efficient feature matching method based on parallel attention mechanism

Abstract

The application discloses a kind of efficient feature matching methods based on parallel attention mechanism, first using convolutional neural network to extract picture key point and its descriptor;Again, parallel attention layer is used to calculate self-attention and cross-attention to two groups of features, and using the self-adaptive fusion of learnable neural network self-attention network and cross-attention network, wherein self-attention network utilizes global context information to strengthen descriptor representation ability, and cross-attention network is used to find the corresponding relationship between the descriptors of two images;After descriptor is strengthened using parallel attention layer, Sinkhorn algorithm and mutual near neighbor criterion are used to calculate matching result.This method integrates self-attention and cross-attention in parallel, greatly reduces the calculation amount and parameter amount of model, improves the efficiency of model, and the self-adaptive fusion network of parallel attention layer enables the model to learn the optimal attention fusion mode, achieving win-win in performance and efficiency.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology based on deep learning, and mainly relates to an efficient feature matching method based on a parallel attention mechanism. Background Art

[0002] Vision-based artificial intelligence systems are widely used to guide machines to perceive and understand their surroundings to make better decisions, and have been playing an important role in the era of global automation and artificial intelligence. However, how to process perceived information and understand the differences or relationships between multiple visual targets is a key issue in various fields. As a key and fundamental problem in these complex tasks, image matching, also known as image correspondence, aims to identify and then correspond the same or similar structures / content in two or more images. This technology is used for high-dimensional structure recovery and information recognition and integration, such as 3D reconstruction, simultaneous localization and mapping (SLAM), image fusion, image retrieval, target recognition and tracking, and change detection, and has important theoretical significance and engineering application value.

[0003] In real-world applications, images used for matching are often taken from the same or similar scenes or objects, but captured at different times, from different perspectives, or using different imaging methods. This can introduce challenges such as perspective variations, illumination variations, scale variations, occlusions, and blur. To establish accurate and robust matching relationships, excellent feature matching techniques should extract high-quality features from both images, eliminating these interferences and finding invariances between the features. Despite the numerous outstanding feature matching techniques proposed over the past few decades, feature matching remains a challenging task, considering the three key aspects of efficiency, robustness, and accuracy.

[0004] Currently, feature matching processes can be divided into two categories: detector-based methods (sparse feature matching) and detector-free methods (dense feature matching). Specifically, detector-based methods first detect key points from the image and calculate descriptors, then match two sets of sparse features. This can be considered a two-stage matching method. Detector-free methods, on the other hand, directly extract features from all pixels in the image and perform dense matching. This can be considered a one-stage matching method.

[0005] Classic feature matching methods are generally detector-based. For detectors, several excellent hand-crafted methods, such as SIFT, SURF, BRIEF, and ORB, were first proposed and widely used in various 3D computer vision tasks. With the advent of the deep learning era, numerous convolutional neural network-based detectors, such as R2D2, SuperPoint, D2-Net, and LF-Net, have been proposed to further improve the robustness of keypoints and descriptors to changes in illumination and viewpoint. Beyond detectors, other work has focused on developing better matching algorithms—that is, how to process keypoints and descriptors extracted by detectors to achieve higher-quality matching. The most basic matcher in traditional methods is the nearest neighbor matcher, which first calculates the Euclidean distance between descriptors and then, in combination with a distance threshold, establishes correspondences between the closest descriptors. SuperGlue is the first to propose an attention-based feature matching network that uses self-attention and cross-attention to extract global contextual information for matching. OETR further restricts attention-based feature matching to common view regions by estimating overlapping regions.

[0006] Unlike SuperGlue, LoFTR applies self-attention and cross-attention directly to feature maps extracted by a convolutional neural network (CNN) and produces matches in a coarse-to-fine manner. MatchFormer builds on LoFTR by abandoning the CNN backbone and adopting a layered framework based entirely on attention to extract features, while also using the attention mechanism to find similarities. While SuperGlue and LoFTR employ a simple alternating strategy for self-attention and cross-attention, MatchFormer further proposes an interleaving strategy, focusing on self-attention in the shallow layers of the network and cross-attention in the deeper layers.

[0007] Benefiting from the global modeling capability of Transformer, attention-based networks have become the dominant method in both detector-based and detector-free networks. However, despite their excellent performance, attention-based networks often bring high training costs, large memory requirements, and high inference latency. In particular, for detector-free methods, processing dense features will exacerbate the quadratic complexity problem of the attention mechanism. Therefore, existing attention-based methods artificially arrange self-attention and cross-attention in a serial manner, such as Figure 1 As shown in (a), this makes attention-based feature matching methods inefficient, and the fixed attention arrangement limits the diversity of self-attention and cross-attention integration. Summary of the Invention

[0008] The present invention addresses the problems existing in the prior art and provides an efficient feature matching method based on a parallel attention mechanism. First, a convolutional neural network is used to extract image key points and their descriptors. Then, a parallel attention layer is used to calculate self-attention and cross-attention for two sets of features, and a learnable neural network is used to adaptively fuse the self-attention network and the cross-attention network. The self-attention network uses global context information to enhance the descriptor representation capability, and the cross-attention network is used to find the correspondence between the two image descriptors. After the descriptors are enhanced by the parallel attention layer, the Si nkhorn algorithm and the mutual nearest neighbor criterion are used to calculate the matching results. This method integrates self-attention and cross-attention in a parallel manner, significantly reducing the amount of computation and parameters of the model and improving model efficiency. At the same time, the adaptive fusion network of the parallel attention layer enables the model to learn the optimal attention fusion method, achieving a win-win situation in terms of performance and efficiency. In order to achieve the above purpose, the technical solution adopted by the present invention is: an efficient feature matching method based on a parallel attention mechanism, comprising the following steps:

[0009] S1. Input a single image, perform a random homography transformation on the input image and generate a homography matrix. Then, two images and a groundtruth homography matrix are input to the network. A deep convolutional neural network is used to extract image key points and descriptors.

[0010] S2, taking the key points and descriptors obtained in step S1 as input, using a multi-layer perceptron as a position encoder to embed the spatial position information of the key points into the descriptors;

[0011] S3, the descriptor obtained in step S2 is processed through 9 parallel attention layers, each of which includes a self-attention module and a cross-attention module.

[0012] In the self-attention module, the standard attention operation is used to process the two image descriptors separately to enhance the descriptor representation ability;

[0013] The cross attention module finds the correspondence between the descriptors of the two images according to the attention weight sharing strategy;

[0014] The network outputs of the self-attention module and the cross-attention module are connected and fused through a multi-layer perceptron. The fusion result is used to update the descriptor, and the new descriptor is used as the input of the next parallel attention layer.

[0015] S4. After using the parallel attention layer to enhance the descriptors in step S3, the score matrix is ​​obtained by dot-multiplying the two image descriptors. This score matrix is ​​used as the cost matrix of the optimal transportation problem. The Sinkhorn algorithm is applied iteratively to obtain the optimal allocation matrix. Based on the optimal allocation matrix, matches with values ​​less than the matching threshold are first filtered out, and finally the mutual nearest neighbor criterion is used to select the final match.

[0016] S5. Calculate the loss based on the output matching result of step S4 and the actual matching, train the parallel attention network and position encoder, and achieve feature matching.

[0017] As an improvement of the present invention, the multilayer perceptron in step S2 has three hidden layers, which expands the key point dimension from 2 to 32, 64, and 128 respectively, and finally outputs a 256-dimensional position encoding vector, which is added to the descriptor to achieve the integration of position information.

[0018] As an improvement of the present invention, in step S2, a multi-layer perceptron MLP is used. pc The spatial position information of the key points is embedded into the descriptor, specifically:

[0019]

[0020]

[0021] Among them, p i and d i are the position and descriptor of the i-th key point respectively; x (0) ,y (0) is the initial feature generated.

[0022] As another improvement of the present invention, when the step S3 is processed in parallel with the attention layer, the descriptor is first projected into Q, K, and V using three linear projection layers with shared weights, that is, x (l) and y (l) Projection and where x (l) and y (l) is the input of the l-th layer attention network, and then the self-attention and cross-attention are calculated in parallel.

[0023] As another improvement of the present invention, when the step S3 is processed in parallel with the attention layer, the self-attention Self and cross-attention Cross calculations are expressed as:

[0024]

[0025]

[0026]

[0027]

[0028] Where d is the number of descriptor channels, Self and Cross are the self-attention and cross-attention results respectively.

[0029] As another improvement of the present invention, when the step S3 is processed in parallel with the attention layer, in the self-attention module, Q, K, and V come from the same input, that is, or In the cross attention module, descriptors from different images are input and the attention weight sharing strategy is used to Replace with (Q x K y T ) T .

[0030] As a further improvement of the present invention, in step S4, the number of iterations of the Sinkhorn algorithm is 20, and the matching threshold is 0.2.

[0031] As a further improvement of the present invention, in the training of step S5, the loss function is a negative log-likelihood loss function on the optimal allocation matrix, that is:

[0032]

[0033] Where P is the score matrix calculated by the inner product of two groups of descriptors, M gt For true matching, the attention network and position encoder network are jointly trained with supervision from this loss function.

[0034] Compared with the existing technology, the present invention provides an efficient feature matching method based on the parallel attention mechanism, which has the following beneficial effects:

[0035] 1. This method uses a deep convolutional network to extract sparse key points and descriptors instead of extracting dense feature maps, which effectively improves the shortcoming of the attention mechanism with quadratic complexity.

[0036] 2. This method uses a deep attention network to strengthen the descriptor. Unlike the deep convolutional network, this network has a global receptive field and can aggregate information from all nodes, so that the descriptor can fully perceive the global context information, thereby improving the matching quality between descriptors.

[0037] 3. This method overcomes the shortcomings of attention-based models, which suffer from high computational complexity and low efficiency. It designs a parallel computing structure and weight-sharing strategy for self-attention and cross-attention, effectively reducing the model's computational complexity and parameter count. Furthermore, a dynamic fusion network of self-attention and cross-attention can adaptively fuse the two attention types, improving model performance.

[0038] 4、The method improves the feature matching algorithm based on attention mechanism, realizes the win-win of performance and efficiency, and has wide application prospect in image registration, three-dimensional reconstruction, visual positioning and other fields. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 is a comparison diagram of serial attention and parallel attention structure, wherein

[0040] Figure 1 (a) is a schematic diagram of serial attention structure;

[0041] Figure 1 (b) is a schematic diagram of parallel attention structure;

[0042] Figure 2 is a flow chart of parallel attention layer and position encoder. DETAILED DESCRIPTION

[0043] The present application will be further illustrated below in conjunction with the drawings and specific embodiments, and it should be understood that the following specific embodiments are only used to illustrate the present application and not to limit the scope of the present application.

[0044] Example 1

[0045] An efficient feature matching method based on parallel attention mechanism, as shown in Figure 1 (b), self-attention and cross-attention are integrated in a parallel way, which greatly reduces the calculation amount and parameter amount of the model, improves the efficiency of the model, and the adaptive fusion network of the parallel attention layer enables the model to learn the optimal attention fusion mode, improves the network performance, and specifically includes the following steps:

[0046] S1: reading the image gray diagram, using a deep convolutional neural network to extract the picture key points and descriptors; unlike the conventional deep attention network processing dense feature map, first use the deep convolutional network to extract sparse key points and descriptors, and then apply attention mechanism to strengthen the descriptors, which effectively improves the disadvantage of quadratic complexity of attention mechanism.

[0047] S2, as shown in Figure 2 , using the key points and descriptors obtained in step S1 as input, using a multilayer perceptron as a position encoder to embed the spatial position information of the key points into the descriptors;

[0048] Since the attention module has permutation invariance and cannot distinguish each node in space, it is necessary to embed the position information into the descriptor, and a multilayer perceptron is used to realize the embedding of the spatial position information of the key points to generate the initial representation of the feature.

[0049] Let the position and descriptor of the i-th key point be p i and d i , this method uses multi-layer perceptron MLP pc The spatial position information of the key points is embedded into the descriptor, namely:

[0050]

[0051]

[0052] S3. The descriptors obtained in step S2 are processed through nine parallel attention layers, leveraging global context to enhance the descriptors. In each parallel attention layer, the descriptors are first projected into Q, K, and V using a weighted linear projection layer. Self-attention and cross-attention are then computed in parallel. The self-attention network leverages global context to enhance the descriptor representation, while the cross-attention network searches for correspondences between the descriptors of the two images. Both networks transfer global information about the image. The results of the self-attention and cross-attention networks are then fused using a two-layer perceptron to update the descriptors.

[0053] Assume that the descriptors of the two images in the lth layer are x (l) and y (l) First, three linear projection layers with shared weights are used to transform x (l) and y (l) Projection and The calculation of self-attention Self and cross-attention Cross is expressed as:

[0054]

[0055]

[0056]

[0057]

[0058] Where d is the number of descriptor channels, Self and Cross are the self-attention and cross-attention results respectively. The difference between the self-attention layer and the cross-attention layer is that the input of the self-attention layer comes from the same descriptor, such as or The input of the cross attention layer comes from the descriptors of different images, and the attention weight sharing strategy is applied. Attention weight sharing means replacing the attention weight from image y to image x with the attention weight from image x to image y, that is, using (Q x K y T ) T replace

[0059] Use two multi-layer perceptrons MLP respectively x 、MLP y The self-attention and cross-attention results of the two images are fused, and finally the fusion result is used to update the descriptor as the input of the next parallel attention layer. | represents the connection operation. The formula is as follows:

[0060] x (l+1) =x (l) +MLP x ([x (l) |Self x |Cross x ])

[0061] y (l+1) =y (l) +MLP y ([y (l) |Self y |Cross y ])

[0062] Use a deep attention network to strengthen the descriptor. Unlike a deep convolutional network, this network has a global receptive field, is good at capturing long-distance dependencies, and can aggregate information from all nodes. This feature enables the descriptor to fully perceive the global context information, thereby improving the matching quality between descriptors.

[0063] The parallel attention layer computes self-attention and cross-attention synchronously and lets the network learn the best way to fuse the two attentions, rather than tuning the arrangement of the two attentions as a hyperparameter.

[0064] S4. After performing nine layers of parallel attention, the inner product of the two image descriptors at the matching layer is calculated to obtain the score matrix S, which serves as the cost matrix for the optimal transportation problem. The Sinkhorn algorithm is then applied iteratively to obtain the optimal allocation matrix P. Based on the optimal allocation matrix P, matches with values ​​less than the matching threshold are first eliminated, and then the mutual nearest neighbor criterion is used to select the final match M.

[0065] S5. Calculate the loss based on the output matching results and the true matching, and train the parallel attention network and position encoder.

[0066] Build a PyTorch deep learning development environment and divide the dataset into training, validation, and test sets. During training, calculate the true matching M based on homography (homography estimation task) or camera position and depth (camera pose estimation task). gt The loss function is the negative log-likelihood loss function on the optimal allocation matrix, that is:

[0067]

[0068] Where P is the score matrix calculated by the inner product of two groups of descriptors, M gt The attention network and the position encoder network are jointly trained under the supervision of this loss function to complete feature matching.

[0069] Example 2

[0070] Step S1: Input a single image, perform a random homography transformation on the input image, and generate a homography matrix. This yields two images and a groundtruth homography matrix for the input network. Use the SuperPoint deep convolutional network to extract keypoints p and descriptors d from the two images. Assuming images X and Y have N and M keypoints, respectively, the dimensions of the keypoint position vectors p for the two images are (M, 3) and (N, 3), respectively, and the dimensions of the descriptor d are (M, 256) and (N, 256), respectively.

[0071] The experimental dataset is R1M, an image retrieval dataset containing 1,001,001 high-resolution images. It covers images of over 20 landmarks around the world and features complex perspective variations, lighting changes, blur, and occlusion, making it ideal for training and evaluating feature matching networks. R1M is divided into training, validation, and test sets, containing 1,000,000, 501, and 500 images, respectively.

[0072] Step S2: Taking the key point p and descriptor d obtained in step S1 as input, a multi-layer perceptron is used to embed the spatial position information of the key point into the descriptor. The multi-layer perceptron has three hidden layers, which expands the key point dimension from 2 to 32, 64, and 128 in sequence. Finally, a 256-dimensional position encoding vector is output, which is added to the descriptor to achieve the integration of position information. The initial representations of the generated features of the two images are x (0) 、y (0) .

[0073] Step S3: Use 9 parallel attention layers to enhance the initial feature representation x obtained in step 2 (0) 、y (0) , the self-attention network strengthens the descriptor representation ability, and the cross-attention network finds the correspondence between the descriptors of the two images. Both transfer the global information of the image. Suppose the input of the first layer of attention network is x (l) and y (l) , first use three linear projection layers with shared weights to transform x (l) and y (l) Projection and In the self-attention module, standard attention calculation is used Among them, Q, K, and V come from the same input, that is, or In the cross-attention module, the attention weight sharing strategy is adopted, i.e., the is replaced by (Q x K y T ) T Therefore, the input of the cross-attention module is All the intermediate features have the same dimension d = 256. Finally, the outputs of the self-attention and the cross-attention are fused by a double-layer perceptron, and then updated by a residual connection to describe the sub.

[0074] In step S4, after the description is strengthened by the parallel attention layer, the score matrix S with a dimension of (M, N) is obtained by point multiplication of the two picture descriptors, which is used as the cost matrix of the optimal transport problem, and then the Sinkhorn algorithm is applied to iteratively obtain the optimal assignment matrix P, which has the same dimension as S. In the embodiment, the number of iterations of the Sinkhorn algorithm is 20. Based on P, first, the matches smaller than the matching threshold are filtered, and finally the final matches M are selected using the mutual nearest neighbor criterion. In the embodiment, the matching threshold is 0.2.

[0075] In step S5, the method is trained by using PyTorch, and the related parameters are set by referring to the experience of engineering parameter setting. The batch size is set to 8, representing that 16 sample pictures are loaded for matching each time. AdamW is set as the optimizer, the initial learning rate is set to 0.0001, the warm up epoch is set to 1, the cosine learning rate decay is applied, the momentum momentum factor is set to 0.9, the weight decay weight decay factor is set to 0.0001, and the total training epoch is 10. During the training, the real matches M gt are calculated according to the ground truth homography matrix and the key points, and if the reprojection error is less than 3 pixels, the match is considered as the ground truth match. The loss function is the negative log-likelihood loss function on the optimal assignment matrix, and the parallel attention network and the position encoder are supervised by the loss function.

[0076] In summary, the self-attention and the cross-attention are integrated in a parallel manner in the method, which realizes a win-win in performance and efficiency, and has a wide application prospect in the fields of image registration, three-dimensional reconstruction, visual positioning, etc.

[0077] It should be noted that the above content only illustrates the technical idea of the present application, and cannot be used to limit the protection scope of the present application. For ordinary skilled persons in the technical field, a number of improvements and refinements can be made without departing from the principles of the present application, and these improvements and refinements fall within the protection scope of the claims of the present application.

Claims

1. An efficient feature matching method based on parallel attention mechanism, characterized in that: The following steps are involved: S1. Input a single image, perform a random homography transformation on the input image and generate a homography matrix. Then, two images and a groundtruth homography matrix are input to the network. A deep convolutional neural network is used to extract image key points and descriptors. S2, taking the key points and descriptors obtained in step S1 as input, using a multi-layer perceptron as a position encoder to embed the spatial position information of the key points into the descriptors; S3, the descriptor obtained in step S2 is processed through 9 parallel attention layers, each of which includes a self-attention module and a cross-attention module. In the self-attention module, the standard attention operation is used to process the two image descriptors separately to enhance the descriptor representation ability; The cross attention module finds the correspondence between the descriptors of the two images according to the attention weight sharing strategy; The network outputs of the self-attention module and the cross-attention module are connected and fused through a multi-layer perceptron. The fusion result is used to update the descriptor, and the new descriptor is used as the input of the next parallel attention layer. S4. After using the parallel attention layer to enhance the descriptors in step S3, the score matrix is ​​obtained by multiplying the two image descriptors and used as the cost matrix of the optimal transportation problem. The Sinkhorn algorithm is applied iteratively to obtain the optimal allocation matrix. Based on the optimal allocation matrix, matches with values ​​less than the matching threshold are first filtered out, and finally the mutual nearest neighbor criterion is used to select the final match. S5. Calculate the loss based on the output matching result of step S4 and the actual matching, train the parallel attention network and position encoder, and achieve feature matching.

2. The efficient feature matching method based on the parallel attention mechanism according to claim 1, characterized in that: In step S2, the multilayer perceptron has three hidden layers, which expand the key point dimension from 2 to 32, 64, and 128 in sequence, and finally outputs a 256-dimensional position encoding vector, which is added to the descriptor to achieve the integration of position information.

3. The efficient feature matching method based on the parallel attention mechanism according to claim 2, characterized in that: In step S2, a multi-layer perceptron MLP is used pc The spatial position information of the key points is embedded into the descriptor, specifically: Among them, p i and d i are the position and descriptor of the i-th key point respectively; x (0) ,y (0) is the initial feature generated.

4. The efficient feature matching method based on the parallel attention mechanism according to claim 2 or 3, characterized in that: When the step S3 is processed in parallel with the attention layer, the descriptor is first projected into Q, K, and V using three linear projection layers with shared weights. (l) and y (l) Projection and where x (l) and y (l) is the input of the l-th layer attention network, and then the self-attention and cross-attention are calculated in parallel.

5. The efficient feature matching method based on the parallel attention mechanism according to claim 4, characterized in that: When the step S3 is processed in parallel with the attention layer, the calculation of self-attention Self and cross-attention Cross is expressed as: Among them, d is the number of descriptor channels; Self and Cross are the self-attention and cross-attention results respectively.

6. The efficient feature matching method based on a parallel attention mechanism according to claim 5, characterized in that: When the step S3 is processed in parallel with the attention layer, in the self-attention module, Q, K, and V come from the same input, that is, or In the cross attention module, descriptors from different images are input and the attention weight sharing strategy is used to Replace with (Q x K y T ) T .

7. The efficient feature matching method based on a parallel attention mechanism according to claim 4, characterized in that: In step S4, the number of iterations of the Sinkhorn algorithm is 20, and the matching threshold is 0.

2.

8. The efficient feature matching method based on a parallel attention mechanism according to claim 6, characterized in that: In the training of step S5, the loss function is the negative log-likelihood loss function on the optimal allocation matrix, that is: Among them, P is the score matrix calculated by the inner product of two groups of descriptors, M gt For true matching, the attention network and position encoder network are jointly trained with supervision from this loss function.

Images