Robust image matching system and method based on multi-scale attention feature fusion

Abstract

The invention relates to the technical field of computer vision and image processing, in particular to a robust image matching system and method based on multi-scale attention feature fusion, and the system comprises a feature extraction module which extracts a multi-resolution feature map containing image features from a to-be-matched image; the position coding module performs spatial position coding on the image features to improve the spatial perception capability of the model; the multi-scale attention module performs multi-scale enhancement processing on the image features, so that the model is more effectively focused on the significant part of the image; the attention module establishes a dependency relationship between image features; the coarse-level matching layer module generates a coarse-level matching result and screens out a matching pair with relatively high confidence coefficient; and the coarse-to-fine module carries out fine processing on the coarse matching result, so that more accurate feature matching is realized. According to the method, the effectiveness and reliability of the method are verified by testing the image matching performance of the to-be-tested picture under the conditions that the to-be-tested picture contains different illumination changes, extreme visual angle changes, repeated textures and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and image processing, and in particular to a robust image matching system and method based on multi-scale attention feature fusion. Background Art

[0002] Image matching is a basic and key process for many computer vision tasks (such as image stitching, 3D reconstruction, and visual positioning). Its goal is to find and match corresponding feature points in two or more images. It is an important means to achieve image information recognition and integration.

[0003] Traditional image matching algorithms involve feature detection, feature description, and feature matching, which are closely linked and have a great impact on matching performance. For example, due to changes in lighting, perspective, and motion blur, it may be difficult to extract enough high-quality key points in the feature detection stage, and it is impossible to find the correct correspondence between features, which ultimately affects the matching effect.

[0004] In recent years, with the development of deep learning technology, a large number of scholars have conducted image matching research based on deep learning methods, directly using convolutional neural networks (CNN) to extract and describe feature points from images. However, convolutional neural networks have limited receptive fields and search spaces when extracting dense features, and cannot distinguish unclear areas. It is also extremely difficult to match disordered and scattered sparse points. Therefore, there are still many deficiencies in practical applications, and the use of deep learning technology for image matching still needs further research and improvement.

[0005] In response to the above technical problems, there is an urgent need for an image matching system that can better capture image features at different resolutions, especially in complex scenes such as lighting changes, extreme perspective changes, and repeated textures, so as to improve the accuracy and robustness of image matching. Summary of the invention

[0006] In view of at least one of the above technical problems, the present invention provides a robust image matching system based on multi-scale attention feature fusion to improve the accuracy and robustness of image matching.

[0007] According to a first aspect of the present invention, there is provided a robust image matching system based on multi-scale attention feature fusion, comprising:

[0008] Includes the following modules:

[0009] Feature extraction module: used to extract feature maps of multiple resolutions containing image features from the image to be matched;

[0010] Position encoding module: used for performing spatial position encoding on the feature map;

[0011] Multi-scale attention module: used to perform convolution operation on the image features through convolution kernels of different sizes to achieve enhanced processing of multi-scale features;

[0012] Attention module: including a self-attention unit and a cross-attention unit, used to establish dependencies between the image features;

[0013] Coarse matching layer module: used to process the similarity matrix of the image features through double softmax operations, generate coarse matching results, and screen out matching pairs with higher confidence;

[0014] Coarse-to-fine module: used to refine the coarse matching results.

[0015] In some embodiments of the present invention, the feature extraction module is a ResNet and FPN architecture, comprising the following steps:

[0016] Obtain 1 / 8, 1 / 4 and 1 / 2 resolution feature maps through the ResNet and FPN architecture;

[0017] Through downsampling and convolution operations, the 1 / 4 resolution feature map is downsampled to the 1 / 8 resolution feature map, and fused with the original 1 / 8 resolution feature map, and the fused 1 / 8 resolution feature map is output.

[0018] In some embodiments of the present invention, the position encoding module is a sinusoidal position encoding module, comprising the following steps:

[0019] For each pixel point of the feature map, generate corresponding x-axis and y-axis positions respectively;

[0020] Generate a set of frequency components using an exponential function, multiply the x-axis and y-axis positions by the corresponding frequency components, and calculate their sine and cosine values;

[0021] The generated sine value and cosine value are added to the image feature positions in the feature map respectively.

[0022] In some embodiments of the invention, the multi-scale attention module comprises the following steps:

[0023] The feature map x for spatial position encoding is cloned into an initial feature map u;

[0024] Convolving the feature map x for spatial position encoding with a convolution kernel of size 5×5 to generate a preliminary feature map;

[0025] Using three different convolution kernels, namely, 1×7 and 7×1, 1×11 and 11×1, and 1×21 and 21×1, to extract features of different spans from the preliminary feature map, and adding the extracted features element by element to generate a fused feature map;

[0026] Convolving the fused feature map with a 1×1 convolution kernel to obtain an integrated feature map;

[0027] The integrated feature map is multiplied element by element with the initial feature map u, and the final result feature map is output.

[0028] In some embodiments of the present invention, when processing the image features at each position, the self-attention unit simultaneously pays attention to the image feature information at other positions in the same image to assign different attention weights to each position; the cross-attention unit is used to process the relationship between features from different images and capture the positions of similar features in two images.

[0029] In some embodiments of the present invention, the coarse matching layer module comprises the following steps:

[0030] A score matrix reflecting the similarity between different image features is calculated, and the score matrix S is:

[0031]

[0032] in, represents the feature vector of pixel i in image A, represents the feature vector of pixel j in image B, τ is a temperature parameter used to control the smoothness of the similarity distribution;

[0033] The score matrix is ​​normalized using a double softmax operator, and a matching probability is assigned to each feature. The matching probability P c for:

[0034] P c =softmax(S(i,·)) j softmax(S(·,j)) i

[0035] Among them, softmax(S(i,·)) j A vector representing the similarity scores between pixel i in image A and all pixels in image B, softmax(S(·,j)) i A vector representing the similarity scores between all pixels in image A and pixel j in image B;

[0036] Based on the matching probability, by setting the threshold θ c, select the matching pairs with higher confidence, apply the nearest neighbor criterion to filter possible outlier matches, and generate a coarse-level matching matrix, which is:

[0037]

[0038] Among them, N c is the coarse-level matching matrix, which contains multiple pairs of pixel matches MNN represents pixels It's pixels The nearest neighbor in image B, and the pixel Also pixels Nearest neighbors in image A.

[0039] In some embodiments of the present invention, the coarse-to-fine module comprises the following steps:

[0040] On the 1 / 2 resolution feature map, two local windows are cut out around each of the coarse-level matching positions to capture the target feature and the contextual information around it;

[0041] The target feature is processed by the self-attention unit and the cross-attention unit to generate a heat map, and the expected value is calculated by weighting each pixel position in the heat map to obtain the matching probability between all pixels in one window and all pixels in another window.

[0042] In some embodiments of the present invention, a total loss function is also included, and the total loss function includes a coarse-level matching loss function and a fine-level matching loss function. The coarse-level matching loss function is used to determine the matching relationship between two images at low resolution; the fine-level matching loss function is used to optimize the matching position with high certainty.

[0043] In some embodiments of the present invention, the formula of the total loss function is:

[0044] L=L c +L f

[0045] Among them, L is the total loss function, L c is the coarse-level matching loss function, L f is the fine-level matching loss function;

[0046] The coarse-level matching loss function L c for:

[0047]

[0048] in, is the number of true matching point pairs obtained based on the low-resolution feature map, Represents two pixels and Confidence score of the match between

[0049] The fine-level matching loss function L f for:

[0050]

[0051] Among them, N f is the set of matching pairs, For matching pairs, is the predicted variance, For point Its true value The error between .

[0052] According to a second aspect of the present invention, a robust image matching method based on multi-scale attention feature fusion is also provided, comprising the following steps:

[0053] Input two images to be matched into the feature extraction module, extract image features, obtain feature maps with different resolutions, and encode the feature positions in the feature maps with different resolutions to obtain encoded feature maps;

[0054] Inputting the encoded feature map into a multi-scale attention module, performing multi-scale feature extraction and fusion on the encoded feature map to obtain a fused feature map;

[0055] Inputting the fused feature map into the self-attention and cross-attention modules, establishing dependency relationships between features, and obtaining a dependency feature map;

[0056] The similarity between the dependency feature graphs is calculated to generate a coarse-level matching matrix, and matching pairs with low confidence are filtered out to finally obtain a fine-level matching matrix.

[0057] The beneficial effects of the present invention are as follows: the present invention can match images in complex scenes such as different lighting changes, perspective changes, and motion blur through an image matching system based on multi-scale attention feature fusion. Through the introduction of multi-scale feature fusion and attention mechanism, the system enhances the ability to capture features of different resolutions of the image and can better focus on the salient areas of the image; the attention mechanism further improves the dependency between internal features of the image and between different images, thereby improving the accuracy and robustness of the matching; the gradual optimization strategy from coarse matching to fine matching is adopted, so that the system can effectively filter low-confidence matching pairs, further reducing the possibility of false matching. Finally, the system significantly improves the accuracy and stability of image matching in complex scenes such as lighting changes, extreme perspective changes, and repeated textures, thereby improving the accuracy and robustness of image matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0059] Figure 1 A model diagram of a robust image matching system based on multi-scale attention feature fusion in an embodiment of the present invention;

[0060] Figure 2 is a calculation flow chart of a robust image matching system based on multi-scale attention feature fusion in an embodiment of the present invention;

[0061] Figure 3 Schematic diagram of the attention structure of a robust image matching system based on multi-scale attention feature fusion in an embodiment of the present invention;

[0062] Figure 4 It is the matching result of the MegaDepth test data set in the near view in the robust image matching system based on multi-scale attention feature fusion in the embodiment of the present invention;

[0063] Figure 5 It is the matching result of the MegaDepth test data set in the long-range view in the robust image matching system based on multi-scale attention feature fusion in the embodiment of the present invention;

[0064] Figure 6 It is the matching result of the HPatches dataset under slight viewing angle change in the robust image matching system based on multi-scale attention feature fusion in an embodiment of the present invention;

[0065] Figure 7 It is the matching result of the HPatches dataset under extreme visual changes in the robust image matching system based on multi-scale attention feature fusion in an embodiment of the present invention;

[0066] Figure 8 This is a flowchart of the steps of a robust image matching method based on multi-scale attention feature fusion in an embodiment of the present invention. DETAILED DESCRIPTION

[0067] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0068] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0069] like Figures 1 to 7 The robust image matching system based on multi-scale attention feature fusion shown in the figure includes the following modules:

[0070] Feature extraction module: used to extract feature maps of multiple resolutions containing image features from the image to be matched. It should be pointed out here that there are many ways to extract this feature, which can be a convolutional neural network, a classic image feature extraction method, an autoencoder, or other deep learning models.

[0071] Position encoding module: used to encode the spatial position of the feature map. It should be pointed out here that there are many ways of position encoding, which can be sine and cosine position encoding, learnable position encoding, or relative position encoding or coordinate embedding.

[0072] Multi-scale attention module: It is used to perform multi-scale enhancement processing on image features through convolution kernels of different sizes. It should be pointed out here that there are many forms of multi-scale attention modules, which can be changes in the size of the convolution kernel, or hollow convolution, or adaptive convolution and other forms.

[0073] Attention module: includes self-attention unit and cross-attention unit, which are used to establish dependencies between image features.

[0074] Coarse matching layer module: used to process the similarity matrix of image features through double softmax operations, generate coarse matching results, and screen out matching pairs with higher confidence.

[0075] Coarse-to-fine module: used to refine the coarse matching results. It should be pointed out here that there are many processing methods in the coarse-to-fine module, which can be nearest neighbor matching, feature similarity matrix matching, image pyramid-based matching or other processing forms.

[0076] The multiple modules in the present invention, when performing feature matching on the specific test pictures, first, extract multiple resolution feature maps containing image features from the pictures to be matched; then, embed position information for each pixel of the feature map through sine and cosine position encoding, so that the model can process spatial information at different scales, fuse its features, obtain coarse and fine granular features containing rich information, and improve the performance of the model in image understanding tasks. Then, use multi-scale convolution kernels for feature extraction, and enhance the features of the salient areas of the image through a multi-scale attention mechanism, capture the multi-scale information of the image through multi-level convolution operations, calculate the importance weight of each spatial position, so that the model can focus on the salient parts of the image more effectively. On this basis, the self-attention mechanism is used to extract the dependency between features at different positions in the same image, and the corresponding feature relationship between different images is established through cross attention to improve the feature expression ability. Finally, a preliminary matching result is generated through coarse-level matching, and the matching is further optimized through a refinement module to achieve sub-pixel accurate matching. The present invention can effectively cope with complex scenes such as illumination changes and perspective changes, and significantly improves the accuracy and robustness of image matching.

[0077] The present invention uses a robust image matching system based on multi-scale attention feature fusion, which can perform image matching processing in complex scenes such as different illumination changes, perspective changes and motion blur. Through the introduction of multi-scale feature fusion and attention mechanism, the system enhances the ability to capture features of different resolutions of the image, and can better focus on the salient areas of the image; the attention mechanism further improves the dependency between internal features of the image and between different images, thereby improving the accuracy and robustness of the matching; the gradual optimization strategy from coarse matching to fine matching is adopted, so that the system can effectively filter low-confidence matching pairs, further reducing the possibility of false matching. Finally, the system significantly improves the accuracy, stability and computational efficiency of image matching in complex scenes such as illumination changes, extreme perspective changes and repeated textures, thereby improving the accuracy and robustness of image matching.

[0078] In some embodiments of the present invention, the feature extraction module is a ResNet and FPN architecture, comprising the following steps:

[0079] The 1 / 8, 1 / 4 and 1 / 2 resolution feature maps are obtained through the ResNet and FPN architectures;

[0080] Through downsampling and convolution operations, the 1 / 4 resolution feature map is downsampled to a 1 / 8 resolution feature map, and fused with the original 1 / 8 resolution feature map to output the fused 1 / 8 resolution feature map.

[0081] By simultaneously generating feature maps with 1 / 8, 1 / 4 and 1 / 2 resolutions, and then downsampling and fusing them, the model can capture feature information at different resolutions. This multi-scale feature extraction method enables the model to focus on both global and local details at the same time, thereby enhancing the ability to handle complex scenes. The feature map (1 / 8) at a lower resolution has a larger receptive field than (1 / 4). Combining the two, global context and local details can be obtained at the same time. In visual tasks, the higher the resolution, the larger the size of the feature map, and the higher the computational overhead. Selecting a feature map with 1 / 8 resolution can achieve the best balance between computational efficiency, global information capture capability, and accuracy requirements. The feature extraction module extracts features at different scales and fuses them, making the model more stable in complex scenes such as illumination changes and perspective changes, and improving the robustness of image matching.

[0082] In some embodiments of the present invention, the position encoding module is a sinusoidal position encoding module, comprising the following steps:

[0083] For each pixel of the resolution feature map, generate the corresponding x-axis and y-axis positions respectively;

[0084] Use an exponential function to generate a set of frequency components, multiply the x-axis and y-axis positions by the corresponding frequency components, and calculate their sine and cosine values;

[0085] The generated sine and cosine values ​​are added to the image feature positions in the resolution feature map respectively.

[0086] The present invention encodes image features through sinusoidal coding, which can better perceive the spatial structure and position information of the image. The position information is directly embedded in the feature map, so that the model has position sensitivity when processing images, which helps to improve the accuracy of image matching. Sine position coding uses a fixed mathematical function to generate position information, without the need for additional parameter training, which reduces the complexity of the model, reduces the risk of overfitting, and also improves the efficiency of the encoding process. Accurate position information enhances the adaptability of the model to complex scenes during the matching process, especially in the case of changes in lighting and viewing angles, which can effectively improve the accuracy and robustness of matching.

[0087] In some embodiments of the invention, the multi-scale attention module comprises the following steps:

[0088] Clone the resolution feature map x for spatial position encoding into the initial feature map u;

[0089] Use a convolution kernel of size 5×5 to convolve the resolution feature map x for spatial position encoding to generate a preliminary feature map;

[0090] Three different convolution kernels (1×7 and 7×1, 1×11 and 11×1, and 1×21 and 21×1) are used to extract features from the preliminary feature map, and the extracted features are added element by element to generate a fused feature map;

[0091] Use a 1×1 convolution kernel to convolve the element-wise addition feature map to obtain an integrated feature map;

[0092] Multiply the integrated feature map by the initial feature map u element by element and output the final result feature map.

[0093] The multi-scale convolution strategy effectively enhances the model's perception of image features in different ranges, enabling the system to process global and local feature information at the same time; it can finely process the details and background information in the image, especially in complex scenes such as lighting changes, perspective changes, and repeated textures, and can effectively separate important feature information and enhance the robustness of matching; through element-by-element addition operations, the features extracted by different convolution kernels are effectively fused to generate a feature map with rich multi-scale information. Using 1×1 convolution to further integrate this information helps to eliminate redundancy and improve the expressiveness of feature maps. It can also highlight significant features and suppress irrelevant information, thereby further strengthening the model's focus on key features and improving the final matching accuracy and stability.

[0094] In some embodiments of the present invention, the self-attention unit, when processing the image features at each position, simultaneously pays attention to the image feature information at other positions in the same image to assign different attention weights to each position; the cross-attention unit is used to process the relationship between features from different images and capture the positions of similar features in two images. Self-attention can focus on important areas and ignore irrelevant background noise, while cross-attention can capture similar features across images, maintaining high matching accuracy even when the image appearance changes, and better understanding the overall structure of the image, especially in scenes with sparse features or complex backgrounds, to establish accurate feature correspondences, thereby improving matching accuracy.

[0095] In some embodiments of the present invention, the coarse matching layer module comprises the following steps:

[0096] Calculate the score matrix that reflects the similarity between different image features. The score matrix S is:

[0097]

[0098] in, represents the feature vector of pixel i in image A, represents the feature vector of pixel j in image B, τ is a temperature parameter used to control the smoothness of the similarity distribution;

[0099] The double softmax operator is used to normalize the score matrix and assign a matching probability to each feature. The matching probability P c for:

[0100] P c =softmax(S(i,·)) j softmax(S(·,j)) i

[0101] Among them, softmax(S(i,·)) j A vector representing the similarity scores between pixel i in image A and all pixels in image B, softmax(S(·,j)) i A vector representing the similarity scores between all pixels in image A and pixel j in image B;

[0102] Based on the matching probability, by setting the threshold θ c , select matching pairs with higher confidence, apply the nearest neighbor criterion to filter possible outlier matches, and generate a coarse-level matching matrix. The coarse-level matching matrix is:

[0103]

[0104] Among them, N c is the coarse-level matching matrix, which contains multiple pairs of pixel matches MNN represents pixels It's pixels The nearest neighbor in image B, and the pixel Also pixels Nearest neighbors in image A.

[0105] By constructing a similarity score matrix and using double Softmax operations to normalize the matrix, the similarity matching probability between image features can be quickly calculated, which reduces the complex calculation process and effectively improves the matching efficiency. It is suitable for processing large-scale image data, making the matching probability distribution of each feature point smoother and more reasonable, reducing the probability of false matching. By calculating the matching probability matrix between images and combining it with the nearest neighbor criterion for screening, outlier matching points can be effectively filtered out, thereby improving the accuracy of coarse matching results; and by introducing threshold screening, unreliable matching pairs can be effectively avoided, which improves the robustness of the system in the face of complex scenes (such as changes in lighting, changes in viewing angle, etc.), ensuring that relatively stable coarse-level matching results can be obtained under different conditions.

[0106] In some embodiments of the invention, the coarse-to-fine module comprises the following steps:

[0107] On the 1 / 2 resolution feature map, two local windows are cropped around each coarse-level matching position to capture the target feature and its surrounding context information;

[0108] The target features are processed by a self-attention unit and a cross-attention unit to generate a heat map, and the expected value is calculated by weighting each pixel position in the heat map to obtain the matching probability between all pixels in one window and all pixels in another window.

[0109] By cutting out a local window for each coarse matching position on the 1 / 2 resolution feature map, and using the self-attention and cross-attention mechanisms to process the target features and the surrounding contextual information, the feature details in the local area can be effectively captured. This method can more accurately determine the sub-pixel position of the matching point, thereby greatly improving the matching accuracy. At the same time, the enhanced processing of local information also helps to retain effective matching information and suppress noise in complex scenes (such as changes in illumination and perspective). Based on the coarse matching results, the matching position can be adjusted through further refinement processing, and the final fine-level matching results can be ensured to have a high confidence level. By weighting the expected value of each pixel position in the heat map, accurate matching at the sub-pixel level can be obtained. This step-by-step optimization strategy improves the matching robustness of the model in the face of challenges such as complex backgrounds and repeated textures.

[0110] In some embodiments of the present invention, a total loss function is also included, and the total loss function includes a coarse-level matching loss function and a fine-level matching loss function. The coarse-level matching loss function is used to determine the matching relationship under different resolution feature maps; the fine-level matching loss function is used to optimize the matching position with high certainty. Preliminary matching through the coarse-level matching loss function can provide a more accurate initial matching relationship for the model to avoid losing key matching points in the early stage; the fine-level matching loss function further refines the results by optimizing the matching position with high confidence, and is accurate to the sub-pixel level, thereby significantly improving the matching accuracy.

[0111] In some embodiments of the present invention, the formula of the total loss function is:

[0112] L=L c +L f

[0113] Among them, L is the total loss function, L c is the coarse-level matching loss function, L f is the fine-level matching loss function;

[0114] Coarse-level matching loss function L c for:

[0115]

[0116] in, is the number of true matching point pairs obtained based on the low-resolution feature map, Represents two pixels and Confidence score of the match between

[0117] Fine-level matching loss function L f for:

[0118]

[0119] Among them, N f is the set of matching pairs, For matching pairs, is the predicted variance, For point Its true value The error between .

[0120] The total loss function combines the coarse-level matching loss and the fine-level matching loss to simultaneously optimize the global matching relationship and the local matching accuracy. c Ensure that the global matching relationship is identified on the low-resolution feature map, while the fine-level matching loss L f The local accuracy of the matching points is improved through refined processing. This approach ensures a balance between global consistency and local accuracy, and improves the overall matching effect. By combining the dual optimization of coarse matching and fine matching, the total loss function makes the model more robust in different scenarios (such as changes in illumination, changes in perspective, etc.). The coarse-level loss is responsible for the overall framework, while the fine-level loss is responsible for local optimization, thereby ensuring that the model can still maintain high matching accuracy and stability in complex scenarios, thereby improving the final matching effect.

[0121] The robust image matching method based on multi-scale attention feature fusion includes the following steps:

[0122] Input two images to be matched into the feature extraction module, extract image features, obtain different resolution feature maps, and encode the feature positions in the resolution feature maps to obtain encoded feature maps;

[0123] Input the encoded feature map into the multi-scale attention module, perform multi-scale feature extraction and fusion on the encoded feature map, and obtain a fused feature map;

[0124] Input the fused feature map into the self-attention and cross-attention modules to establish the dependency relationship between features and obtain the dependency feature map;

[0125] The similarity between dependent feature maps is calculated to generate a coarse-level matching matrix, and low-confidence matching pairs are filtered out to finally obtain a fine-level matching matrix.

[0126] In order to evaluate the effectiveness and accuracy of the proposed method in different complex application scenarios, the HPatches dataset and MegaDepth dataset were selected for experimental verification, and compared with different baseline algorithms.

[0127] MegaDepth is a large-scale dataset for evaluating image matching in extreme perspective changes and repeated textures. It consists of 1 million pairs of images from 196 different outdoor scenes, each containing known pose and depth information. Next, we conduct an experimental study, in which the pose error AUC indicator is used to evaluate the performance of the algorithm, where the pose error is the maximum value of the angular error in rotation and translation, and the larger the AUC, the better the algorithm performance.

[0128] HPatches is a dataset for evaluating local descriptors. It contains image files with different illumination intensity changes and different shooting angles. In the experiment, 56 scene sequences with significant perspective changes and 52 scene sequences with significant illumination intensity changes were selected for image matching performance analysis. Next, an experimental study was carried out. The homography matrix AUC indicator was used to evaluate the performance of the algorithm. The homography under different pixel thresholds was estimated by the RANSAC method and compared with the true value. The larger the AUC value, the better the algorithm performance.

[0129] Table 1 shows the image matching performance of the proposed method on different data, as well as the comparison results with other methods. As can be seen from the table, the proposed method has good performance on several data sets, indicating that the model can effectively handle image matching tasks in complex scenes such as extreme perspective changes and illumination changes; in addition, compared with the SP+SuperGlue method based on detectors, the proposed method avoids the limitations of the feature detection stage and directly performs feature extraction and matching, so that the algorithm can still find reliable matches in low-texture areas and repeated patterns.

[0130] Table 1 AUC performance evaluation of image matching

[0131]

[0132] Figure 3 , Figure 4 The matching results of two images with extreme perspective changes and repeated textures selected from the MegaDepth test dataset are shown. Figure 3 It was shot at close range. Figure 4It was taken at a long distance. As can be seen from the figure, the proposed model can effectively handle image matching problems under extreme perspective changes and repeated textures, improving its ability in practical applications.

[0133] Figure 5 , Figure 6 The matching results of two images with different illumination intensity changes and different shooting angles selected from the HPatches dataset are shown. Figure 5 The photo was taken with a slight change in perspective. Figure 6 The images were taken under extreme visual changes. As can be seen from the figure, the matching method based on the multi-scale attention model still performs well under challenging conditions. In particular, when there are obvious changes in perspective and different lighting conditions, the model can still successfully capture the corresponding feature points in the two images. This verifies the advantages of the multi-scale attention model in image matching tasks, especially in dealing with complex scenes.

[0134] Those skilled in the art should understand that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A robust image matching system based on multi-scale attention feature fusion, characterized by: Includes the following modules: Feature extraction module: used to extract feature maps of multiple resolutions containing image features from the image to be matched; Position encoding module: used for performing spatial position encoding on the feature map; Multi-scale attention module: used to perform convolution operation on the image features through convolution kernels of different sizes to achieve enhanced processing of multi-scale features; Attention module: including a self-attention unit and a cross-attention unit, used to establish dependencies between the image features; Coarse matching layer module: used to process the similarity matrix of the image features through double softmax operations, generate coarse matching results, and screen out matching pairs with higher confidence; Coarse-to-fine module: used to refine the coarse matching results.

2. The robust image matching system based on multi-scale attention feature fusion according to claim 1, characterized in that: The feature extraction module is a ResNet and FPN architecture, comprising the following steps: Obtain 1 / 8, 1 / 4 and 1 / 2 resolution feature maps through the ResNet and FPN architecture; Through downsampling and convolution operations, the 1 / 4 resolution feature map is downsampled to the 1 / 8 resolution feature map, and fused with the original 1 / 8 resolution feature map, and the fused 1 / 8 resolution feature map is output.

3. The robust image matching system based on multi-scale attention feature fusion according to claim 1, characterized in that: The position coding module is a sinusoidal position coding module, comprising the following steps: For each pixel point of the feature map, generate corresponding x-axis and y-axis positions respectively; Generate a set of frequency components using an exponential function, multiply the x-axis and y-axis positions by the corresponding frequency components, and calculate their sine and cosine values; The generated sine value and cosine value are added to the image feature positions in the feature map respectively.

4. The robust image matching system based on multi-scale attention feature fusion according to claim 1, characterized in that: The multi-scale attention module includes the following steps: The feature map x for spatial position encoding is cloned into an initial feature map u; Convolving the feature map x for spatial position encoding with a convolution kernel of size 5×5 to generate a preliminary feature map; Using three different convolution kernels, namely, 1×7 and 7×1, 1×11 and 11×1, and 1×21 and 21×1, to extract features of different spans from the preliminary feature map, and adding the extracted features element by element to generate a fused feature map; Convolving the fused feature map with a 1×1 convolution kernel to obtain an integrated feature map; The integrated feature map is multiplied element by element with the initial feature map u, and the final result feature map is output.

5. The robust image matching system based on multi-scale attention feature fusion according to claim 2, characterized in that: When processing the image features at each position, the self-attention unit also pays attention to the image feature information at other positions in the same image to assign different attention weights to each position; the cross-attention unit is used to process the relationship between features from different images and capture the positions of similar features in two images.

6. The robust image matching system based on multi-scale attention feature fusion according to claim 1, characterized in that: The coarse matching layer module comprises the following steps: A score matrix reflecting the similarity between different image features is calculated, and the score matrix S is: in, represents the feature vector of pixel i in image A, represents the feature vector of pixel j in image B, τ is a temperature parameter used to control the smoothness of the similarity distribution; The score matrix is ​​normalized using a double softmax operator, and a matching probability is assigned to each feature. The matching probability P c for: P c =softmax(S(i,·)) j ·softmax(S(·,j)) i Among them, softmax(S(i,·)) j A vector representing the similarity scores between pixel i in image A and all pixels in image B, softmax(S(·,j)) i A vector representing the similarity scores between all pixels in image A and pixel k in image B; Based on the matching probability, by setting the threshold θ c , select the matching pairs with higher confidence, apply the nearest neighbor criterion to filter possible outlier matches, and generate a coarse-level matching matrix, which is: Among them, N c is the coarse-level matching matrix, which contains multiple pairs of pixel matches MNN represents pixels It's pixels The nearest neighbor in image B, and the pixel Also pixels Nearest neighbors in image A.

7. The robust image matching system based on multi-scale attention feature fusion according to claim 5, characterized in that: The coarse-to-fine module includes the following steps: On the 1 / 2 resolution feature map, two local windows are cut out around each of the coarse-level matching positions to capture the target feature and the contextual information around it; The target feature is processed by the self-attention unit and the cross-attention unit to generate a heat map, and the expected value is calculated by weighting each pixel position in the heat map to obtain the matching probability between all pixels in one window and all pixels in another window.

8. The robust image matching system based on multi-scale attention feature fusion according to claim 1, characterized in that: It also includes a total loss function, which includes a coarse-level matching loss function and a fine-level matching loss function. The coarse-level matching loss function is used to determine the matching relationship between two images at low image resolution; the fine-level matching loss function is used to optimize the matching position with high certainty.

9. The robust image matching system based on multi-scale attention feature fusion according to claim 8, characterized in that: The formula of the total loss function is: L=L c +L f Among them, L is the total loss function, L c is the coarse-level matching loss function, L f is the fine-level matching loss function; The coarse-level matching loss function L c for: in, is the number of true matching point pairs obtained based on the low-resolution feature map, Represents two pixels and Confidence score of the match between The fine-level matching loss function L f for: Among them, N f is the set of matching pairs, For matching pairs, is the predicted variance, For point Its true value The error between .

10. A robust image matching method based on multi-scale attention feature fusion, characterized in that: Using the robust image matching system based on multi-scale attention feature fusion as described in any one of claims 1 to 9, comprising the following steps: Input two images to be matched into the feature extraction module, extract image features, obtain feature maps with different resolutions, and encode the feature positions in the feature maps with different resolutions to obtain encoded feature maps; Inputting the encoded feature map into a multi-scale attention module, performing multi-scale feature extraction and fusion on the encoded feature map to obtain a fused feature map; Inputting the fused feature map into the self-attention and cross-attention modules, establishing dependency relationships between features, and obtaining a dependency feature map; The similarity between the dependency feature graphs is calculated to generate a coarse-level matching matrix, and matching pairs with low confidence are filtered out to finally obtain a fine-level matching matrix.

Images