Feature matching network based on high and low frequency decoupling and linear attention optimization

Abstract

The invention discloses a feature matching network based on high and low frequency decoupling and linear attention optimization. The method comprises the following steps: inputting an image pair; a down-sampling module is used for outputting feature maps of multiple scales, and the down-sampling module is used for extracting different frequency information of the image in parallel by using an independent branch network; the depth features of the small-size feature map output by the downsampling module are extracted through an optimized linear Transform module, and coarse similarity calculation is carried out on the obtained pair of depth features to output a coarse prediction result; and according to the rough prediction result, cutting out a plurality of local windows with fixed sizes on the large-size feature map output by the downsampling module, extracting depth features through a standard Transform module, and performing fine similarity calculation on the depth features of each pair of matched windows so as to output a corrected fine prediction result. According to the method, the matching precision can be remarkably improved successfully while the original calculation efficiency is kept.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a feature matching network based on high-low frequency decoupling and optimized linear attention. Background Art

[0002] Image local feature matching aims to establish accurate feature correspondences between different images and is the cornerstone of many 3D computer vision tasks, including 3D reconstruction, visual localization, simultaneous localization and map construction. However, due to factors such as scale changes, perspective diversity, lighting conditions, repetitive patterns and texture differences, different images may have significant differences in describing the same physical space. Therefore, achieving the accuracy and reliability of local feature matching remains a complex and challenging task.

[0003] The traditional image matching process is usually divided into three basic steps: feature detection, feature description, and feature matching. For feature detection and description, traditional methods mainly rely on low-level image features such as gradients and grayscale sequences. Although these methods are theoretically robust to certain forms of transformations, their effectiveness is essentially limited by researchers' prior knowledge. In recent years, with the development of computer vision and deep learning technology, researchers have achieved a series of innovative results in the field of image matching, which can be roughly divided into detector-based methods and detector-free methods. Detector-based methods are similar to traditional methods in terms of steps, and their effectiveness is highly dependent on the performance of keypoint detectors and feature descriptors. However, since the training data of detectors usually relies on manual annotations, their limitations are similar to those of traditional methods. In contrast, detector-free methods avoid independent keypoint detection and feature description stages by utilizing image context information, providing an end-to-end image matching solution. Among them, LoFTR is a detector-free image matching method based on Transformer, which captures global image context information through the Transformer mechanism and exhibits excellent matching capabilities. However, in order to relieve the computational pressure of Transformer, the coarse-to-fine two-stage network structure and linear attention mechanism designed by LoFTR in the feature extraction stage have a significant negative impact on the image matching accuracy. How to further improve the matching performance without sacrificing the original efficiency of LoFTR has become an urgent problem to be solved in the current feature matching field.

[0004] In view of this, it is necessary to design a feature matching network based on high-low frequency decoupling and optimized linear attention to overcome the above defects. Summary of the invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art.

[0006] To this end, the present invention provides a feature matching network based on high-low frequency decoupling and optimized linear attention, which can successfully achieve a significant improvement in matching accuracy while maintaining the original computational efficiency.

[0007] According to a first aspect of the present invention, a feature matching network based on high-low frequency decoupling and optimized linear attention is provided, comprising the following steps:

[0008] S1, input image pair;

[0009] S2, outputs feature maps of multiple scales through the downsampling module, which uses independent branch networks to extract different frequency information of the image in parallel;

[0010] S3: The small-size feature map output by the downsampling module is used to extract deep features through the optimized linear Transformer module, and a rough similarity calculation is performed on the obtained pair of deep features to output a rough prediction result;

[0011] S4. Based on the coarse prediction results, multiple local windows of fixed size are cropped on the large-size feature map output by the downsampling module, and deep features are extracted through the standard Transformer module. The fine similarity calculation is performed on the deep features of each pair of matching windows to output the corrected fine prediction results.

[0012] Preferably, the downsampling module adopted in S2 extracts preliminary features by splitting the channel into two parts of the same size and using parallel networks of ResNet module and VGG module respectively.

[0013] Further preferably, the ResNet module uses a residual connection, which is composed of a downsampling convolution with a step size of 2 and three 3×3 convolutions to extract low-frequency information; the VGG module is composed of two 3×3 convolutions and a maximum pooling layer with a step size of 2 to extract high-frequency information. The decoupling process when extracting preliminary features through the two is specifically expressed as follows:

[0014] Message = concat(ResNet(X low ), VGG(X high ))

[0015] Among them, Message represents the output feature, X low Represents the low-frequency channel part of the input tensor X, X high Represents the high-frequency channel part of the input tensor X. ResNet and VGG can perform different convolution operations on low-frequency and high-frequency signals respectively. Concat refers to the concatenation of different channels.

[0016] Preferably, the optimized linear Transformer module described in S3 also uses the high- and low-frequency information decoupling processing method. Specifically, during the multi-head attention operation, half of the channels corresponding to the low-frequency information are processed by 5×5 depth-separable convolution, while the other half of the channels are not processed. The formula is expressed as:

[0017] Message = concat(DWC(X low ),X high )

[0018] Among them, DWC(X low ) indicates the low frequency channel X low Perform depth-wise separable convolution.

[0019] Further preferably, the optimized linear Transformer module adopts linear attention calculation, where KV is calculated first and then Q(KV) is calculated, and the calculation formula is:

[0020]

[0021] Among them, Sim(Q,K) represents the similarity between the query vector Q and the key vector K, and Respectively represent the results of Q and K after being mapped by the kernel function;

[0022]

[0023] Among them, Message i Represents the output information of the i-th query vector, Q i represents the i-th query vector, K j represents the jth key vector, V j is the value vector associated with the jth key vector, and N represents the number of vectors.

[0024] Preferably, the kernel function in the linear Transformer module that optimizes linear attention is for:

[0025]

[0026] Among them, x is the input tensor, the function of Relu(x) is to retain positive numbers and set negative numbers to 0, and ‖·‖ represents the modulus length of the tensor.

[0027] Further preferably, in the attention calculation, a fixed noise of a channel dimension is added to each head to amplify the difference between each head, while ensuring that the overall impact on all heads is uniform. The updated kernel function It is expressed as:

[0028]

[0029] Here, noise represents the learnable noise along the channel dimension.

[0030] Preferably, after the attention calculation is completed, the calculation structure is standardized. First, a new dimension with all values ​​1 is filled for V before the attention is calculated. After the calculation is completed, all elements except the last dimension are divided by the last dimension. The calculation formula is as follows:

[0031] out(dq,N+pad))=Q(dq,N)×[K T (N, dk) × V (dv, N + pad)]

[0032]

[0033] Among them, out is the result of the normalization calculation, (·) describes the size of the tensor, dq, dk, dv represent the number of channels of the tensors Q, K, V respectively, N represents the sequence length, and pad represents the length of the padding.

[0034] Further preferably, after the attention calculation, a deep convolutional layer for V is designed as a high-frequency feature extractor, which is specifically expressed as follows:

[0035]

[0036] in, Represents the result of attention calculation, and DW(V) represents the deep convolution processing of tensor V.

[0037] Preferably, the inverted bottleneck convolution of MobileNetV2 is used as the feed-forward layer in the optimized linear Transformer module.

[0038] The beneficial effects of the present invention are as follows: in macroscopic design, the present invention decouples the network into two parts along the channel: one part adopts a simpler processing method, and the other part uses a more complex network structure to encourage the model to learn simple texture features and high-level semantic features at the same time; from another perspective, the high-low frequency decoupling increases the separation of the features of the two parts of the network, thereby enriching the diversity of feature expression; in microscopic design, it includes two key parts: one is the improved power function mapping, which is used to highlight the differences between features; the other is to apply the regularization method of multi-head attention to enhance the separation of each attention head, thereby further increasing the richness of features and improving the accuracy of the matching stage. In addition, the present invention also uses asymmetric relative position encoding of self-attention and cross-attention, inverted bottleneck convolution of the feedforward layer, fine matching standard attention and other designs to further improve the accuracy.

[0039] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0040] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0042] Figure 1 This is a structural diagram of a coarse matching module of a feature matching network based on high-low frequency decoupling and optimized linear attention of the present invention;

[0043] Figure 2 This is a structural diagram of a downsampling module based on a feature matching network of high-low frequency decoupling and optimized linear attention of the present invention;

[0044] Figure 3 This is a structural diagram of the multi-head attention part of the optimized linear Transformer module based on the feature matching network of high-low frequency decoupling and optimized linear attention of the present invention;

[0045] Figure 4 A comparison diagram of the Softmax standard attention of the feature matching network based on high-low frequency decoupling and optimized linear attention of the present invention and the linear attention described in the present application;

[0046] Figure 5 This is a structural diagram of an optimized linear Transformer module based on a feature matching network of high-low frequency decoupling and optimized linear attention of the present invention;

[0047] Figure 6 This is a comparison diagram of image matching between the feature matching network based on high-low frequency decoupling and optimized linear attention of the present invention and the existing method LoFTR;

[0048] Figure 7 This is a flowchart of the feature matching network based on high-low frequency decoupling and optimized linear attention of the present invention;

[0049] Figure 8 This is a detailed flow chart of the feature matching network based on high-low frequency decoupling and optimized linear attention of the present invention. DETAILED DESCRIPTION

[0050] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention. In the description of the present invention, it should be understood that, unless otherwise specified, the meaning of "multiple" is two or more.

[0051] See also Figures 1 to 8 , a feature matching network based on high-low frequency decoupling and optimized linear attention in a specific embodiment of the present invention comprises the following steps:

[0052] S1, input image pair;

[0053] S2, outputs feature maps of multiple scales through the downsampling module, which uses independent branch networks to extract different frequency information of the image in parallel;

[0054] S3: The small-size feature map output by the downsampling module is used to extract deep features through the optimized linear Transformer module, and a rough similarity calculation is performed on the obtained pair of deep features to output a rough prediction result;

[0055] S4. Based on the coarse prediction results, multiple local windows of fixed size are cropped on the large-size feature map output by the downsampling module, and deep features are extracted through the standard Transformer module. The fine similarity calculation is performed on the deep features of each pair of matching windows to output the corrected fine prediction results.

[0056] Specifically, the downsampling module adopted in S2 extracts preliminary features by splitting the channel into two parts of the same size and using parallel networks of ResNet modules and VGG modules respectively.

[0057] Among them, the ResNet module uses residual connection, which is composed of downsampling convolution with a step size of 2 and three 3×3 convolutions to extract low-frequency information; the VGG module is composed of two 3×3 convolutions and a maximum pooling layer with a step size of 2 to extract high-frequency information. The decoupling process of extracting preliminary features through the two is specifically expressed as follows:

[0058] Message = concat(ResNet(X low ), VGG(X high ))

[0059] Among them, Message represents the output feature, X low Represents the low-frequency channel part of the input tensor X, X highRepresents the high-frequency channel part of the input tensor X. ResNet and VGG can perform different convolution processing on low-frequency and high-frequency signals respectively. Concat refers to the splicing operation of different channels. At the same time, the optimized linear Transformer module used in S3 also uses the high- and low-frequency information decoupling processing method. Specifically, during the multi-head attention operation, half of the channels corresponding to the low-frequency information are processed by 5×5 depth-separable convolution, while the other half of the channels are not processed. The formula for this decoupling processing is expressed as:

[0060] Message = concat(DWC(X low ),X high )

[0061] Among them, DWC(X low ) indicates the low frequency channel X low Perform depth-wise separable convolution.

[0062] More specifically, the optimized linear Transformer module uses linear attention calculation, where KV is calculated first and then Q(KV), and the calculation formula is:

[0063]

[0064] Among them, Sim(Q,K) represents the similarity between the query vector Q and the key vector K, and Respectively represent the results of Q and K after being mapped by the kernel function;

[0065]

[0066] Among them, Message i Represents the output information of the i-th query vector, Q i represents the i-th query vector, K j represents the jth key vector, V j is the value vector associated with the jth key vector, and N represents the number of vectors.

[0067] The existing linear Transformer has difficulty focusing on a small area and is not suitable for ViT. Therefore, the present invention uses an improved linear attention kernel function to enhance its feature extraction effect in a small area. In addition, experiments have found that the influence of absolute position encoding will gradually weaken with the increase of network depth. For this reason, the present invention introduces relative position encoding in the self-attention calculation to enhance the position information and better distinguish the two different processing stages in the calculation. Figure 4 Comparison between the Softmax standard attention (upper side) and the linear attention (lower side) calculation of the present invention.

[0068] For kernel functions, using a power function after Relu mapping can highlight the maximum value, making the data distribution present a clustering effect, which is helpful for feature matching tasks. However, excessive suppression of small values ​​by the power function and Relu mapping will lead to feature loss, and the numerical instability caused by the power operation will also have a negative impact on the results. Therefore, it is appropriate to normalize the kernel function in the linear Transformer module to convert the kernel function f p (x) is changed to

[0069]

[0070] Among them, x is the input tensor, the function of Relu(x) is to retain positive numbers and set negative numbers to 0, and ‖·‖ represents the modulus length of the tensor.

[0071] To increase feature diversity, a fixed noise of channel dimension is added to each head in the attention calculation to amplify the difference between each head, while ensuring that the overall impact on all heads is uniform. The updated kernel function It is expressed as:

[0072]

[0073] Here, noise represents the learnable noise along the channel dimension.

[0074] After the attention calculation is completed, the calculation structure is standardized. First, a new dimension with all values ​​1 is filled for V before the attention calculation. After the calculation is completed, all elements except the last dimension are divided by the last dimension. The calculation formula is as follows:

[0075] out(dq,N+pad))=Q(dq,N)×[K T (N, dk) × V (dv, N + pad)]

[0076]

[0077] Among them, out is the result of the normalized calculation, (·) describes the size of the tensor, dq, dk, dv represent the number of channels of the tensors Q, K, and V, respectively, N represents the sequence length, and pad represents the padding length. In this way, the accuracy and convergence speed are improved while only a small amount of calculation is added.

[0078]

[0079] In order to enhance the high-frequency feature extraction capability, this paper regards the linear Transformer as a feature extractor that focuses on low-frequency information. According to the design idea of ​​high- and low-frequency decoupling, after attention calculation, a deep convolutional layer of V is designed as a high-frequency feature extractor, which is specifically expressed as follows:

[0080]

[0081] in, Represents the result of attention calculation, and DW(V) represents the deep convolution processing of tensor V.

[0082] Finally, the inverted bottleneck convolution of MobileNetV2 is used as the feed-forward layer in the optimized linear Transformer module to enhance the processing capability of local information. The overall structure of the optimized linear Transformer module is as follows: Figure 5 shown.

[0083] In order to verify the effectiveness and performance of the method proposed in the present invention, the original MegaDepth dataset was used for training in the experiment, and the data after lens distortion correction and the unprocessed original Megadepth dataset were selected for testing, and compared with different methods. The performance of the model on the dataset was evaluated by using the AUC of the posture estimation error at thresholds of 5°, 10°, and 20°. The evaluation results of different methods on the Megadepth data after lens distortion correction are shown in Table 1. The evaluation results of different methods on the unprocessed original Megadepth dataset are shown in Table 2.

[0084] It can be seen from Tables 1 and 2 below that, compared with other technologies, the present invention has higher image matching accuracy and exhibits excellent generalization performance.

[0085] Table 1: Relative pose estimation results (preprocessed Megadepth dataset)

[0086]

[0087] Table 2: Relative pose estimation results (unprocessed original Megadepth dataset)

[0088]

[0089] Therefore, the present invention decouples the network into two parts along the channel: one part adopts a simpler processing method, and the other part uses a more complex network structure to encourage the model to learn simple texture features and high-level semantic features at the same time; from another perspective, the high-low frequency decoupling increases the separation of the features of the two parts of the network, thereby enriching the diversity of feature expression; in terms of micro-design, it includes two key parts: one is an improved power function mapping, which is used to highlight the differences between features; the other is a regularization method applied to multi-head attention to enhance the separation of each attention head, thereby further increasing the richness of features and improving the accuracy of the matching stage. In addition, the present invention also uses asymmetric relative position encoding of self-attention and cross-attention, inverted bottleneck convolution of the feedforward layer, fine matching standard attention and other designs to further improve the accuracy.

[0090] Figure 6 The image matching results of the present invention and the baseline method LoFTR in different complex scenes (such as illumination changes, large angle changes, occlusion, etc.) selected from the Megadepth dataset are shown. As can be seen from the figure, the matching quality and the number of matching points of the present invention in multiple complex scenes have significant advantages, which fully demonstrates the superiority of the present invention in image matching tasks. Further, in order to evaluate the efficiency of the method of the present invention, four 3090 graphics cards were used in the experiment. The final result of the proposed method was obtained in the 20th round of training, while the result of LoFTR on the same training set was obtained in the 30th round of training. The average time for each round of training is 3 hours. Compared with LoFTR, the network convergence speed proposed in the present invention is increased by 33%, saving about 30 hours of training time. Under the premise that there is no significant change in the inference delay, the present invention has an average improvement of 1.9% in the accuracy of posture estimation compared with the results in the original LoFTR paper, and an average improvement of 5.83% compared with LoFTR on the same training set. Among them, the accuracy at the 20° threshold is improved by 3.52%, which is significantly better than the existing method, which shows the reliability of the technical solution.

[0091] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0092] The above description is based on the ideal embodiment of the present invention. Through the above description, relevant personnel can make various changes and modifications without departing from the technical concept of the present invention. The technical scope of the present invention is not limited to the content in the specification, and its technical scope must be determined according to the scope of the claims.

Claims

1. A feature matching network based on high-low frequency decoupling and optimized linear attention, characterized in that: The following steps are involved: S1, input image pair; S2, outputting feature maps of multiple scales through a downsampling module, which uses independent branch networks to extract different frequency information of the image in parallel; S3: The small-size feature map output by the downsampling module is used to extract deep features through the optimized linear Transformer module, and a rough similarity calculation is performed on the obtained pair of deep features to output a rough prediction result; S4. Based on the coarse prediction results, multiple local windows of fixed size are cropped on the large-size feature map output by the downsampling module, and deep features are extracted through the standard Transformer module. The fine similarity calculation is performed on the deep features of each pair of matching windows to output the corrected fine prediction results.

2. The feature matching network based on high-low frequency decoupling and optimized linear attention according to claim 1, characterized in that: The downsampling module adopted in S2 splits the channel into two parts of the same size and extracts preliminary features using parallel networks of ResNet modules and VGG modules respectively.

3. The feature matching network based on high-low frequency decoupling and optimized linear attention according to claim 2, characterized in that: The ResNet module uses residual connection, which is composed of downsampling convolution with a step size of 2 and three 3×3 convolutions to extract low-frequency information; the VGG module is composed of two 3×3 convolutions and a maximum pooling layer with a step size of 2 to extract high-frequency information. The decoupling process of extracting preliminary features through the two is specifically expressed as follows: Message=concat(ResNet(X low ),VGG(X high )) Among them, Message represents the output feature, X low Represents the low-frequency channel part of the input tensor X, X high Represents the high-frequency channel part of the input tensor X. ResNet and VGG can perform different convolution operations on low-frequency and high-frequency signals respectively. Concat refers to the concatenation of different channels.

4. The feature matching network based on high-low frequency decoupling and optimized linear attention according to claim 1, characterized in that: The optimized linear Transformer module used in S3 also uses the high- and low-frequency information decoupling processing method. Specifically, during the multi-head attention operation, half of the channels corresponding to the low-frequency information are processed by 5×5 depth-separable convolution, while the other half of the channels are not processed. The expression formula is: Message=concat(DWC(X low ),X high ) Among them, DWC(X low ) indicates the low frequency channel X low Perform depth-wise separable convolution.

5. The feature matching network based on high-low frequency decoupling and optimized linear attention according to claim 4, characterized in that: The optimized linear Transformer module adopts linear attention calculation, where KV is calculated first and then Q(KV) is calculated. The calculation formula is: Among them, Sim(Q,K) represents the similarity between the query vector Q and the key vector K, and Respectively represent the results of Q and K after being mapped by the kernel function; Among them, Message i Represents the output information of the i-th query vector, Q i represents the i-th query vector, K j represents the jth key vector, V j is the value vector associated with the jth key vector, and N represents the number of vectors.

6. The feature matching network based on high-low frequency decoupling and optimized linear attention according to claim 5, characterized in that: The kernel function in the Transformer module that optimizes linear attention for: Where x is the input tensor, the Relu(x) function retains positive numbers and sets negative numbers to 0, and ‖·‖ represents the modulus length of the tensor.

7. The feature matching network based on high-low frequency decoupling and optimized linear attention according to claim 6, characterized in that: In the attention calculation, a fixed noise of channel dimension is added to each head to amplify the difference between each head, while ensuring that the overall impact on all heads is uniform. The updated kernel function It is expressed as: Here, noise represents the learnable noise along the channel dimension.

8. The feature matching network based on high-low frequency decoupling and optimized linear attention according to claim 7, characterized in that: After the attention calculation is completed, the calculation structure is standardized. First, a new dimension with all values ​​1 is filled for V before the attention calculation. After the calculation is completed, all elements except the last dimension are divided by the last dimension. The calculation formula is as follows: out(dq,N+pad))=Q(dq,N)×[K T (N,dk)×V(dv,N+pad)] Among them, out is the result of the normalization calculation, (·) describes the size of the tensor, dq, dk, dv represent the number of channels of the tensors Q, K, V respectively, N represents the sequence length, and pad represents the length of the padding.

9. The feature matching network based on high-low frequency decoupling and optimized linear attention according to claim 8, characterized in that: After the attention calculation, a deep convolutional layer for V is designed as a high-frequency feature extractor, which is specifically expressed as follows: in, Represents the result of attention calculation, and DW(V) represents the deep convolution processing of tensor V.

10. The feature matching network based on high-low frequency decoupling and optimized linear attention according to claim 8, characterized in that: The inverted bottleneck convolution of MobileNetV2 is used as the feed-forward layer in the optimized linear Transformer module.

Images