Robust image matching system and method based on multi-scale attention feature fusion

Abstract

The present application relates to the technical field of computer vision and image processing, and particularly relates to a robust image matching system and method based on multi-scale attention feature fusion, which comprises: a feature extraction module extracts a plurality of resolution feature maps containing image features from a picture to be matched; a position coding module codes the spatial position of the image features to improve the spatial perception ability of the model; a multi-scale attention module performs multi-scale enhancement processing on the image features to enable the model to focus more effectively on the salient part of the image; an attention module establishes a dependency relationship between the image features; a coarse matching layer module generates a coarse matching result and filters out matching pairs with higher confidence; and a coarse-to-fine module performs fine processing on the coarse matching result to achieve more accurate feature matching. The present application verifies the effectiveness and reliability of the proposed method by testing the image matching performance of the picture to be tested under different light changes, extreme view changes and repeated textures.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and image processing, and particularly relates to a robust image matching system and method based on multi-scale attention feature fusion. BACKGROUND

[0002] Image matching, as a basic and key process of many computer vision tasks such as image stitching, three-dimensional reconstruction and visual positioning, aims to find corresponding feature points in two or more images and match them, and is an important means to realize image information recognition and integration.

[0003] Traditional image matching algorithms involve three parts of feature detection, feature description and feature matching, which are closely related to each other and have a great influence on matching performance. For example, due to the influence of light changes, view angle changes and motion blur, it is difficult to extract enough and high-quality key points in the feature detection stage, and it is difficult to find the correct correspondence between features, which ultimately affects the matching effect.

[0004] In recent years, with the development of deep learning technology, a large number of scholars have carried out image matching research based on deep learning methods, and directly use convolutional neural networks (CNN) to extract and describe feature points from images. However, the convolutional neural network has a limited receptive field and search space when extracting dense features, and cannot distinguish between unobvious regions, and it is also extremely difficult to match unordered and scattered sparse points. Therefore, there are still many deficiencies in practical applications, and further research and improvement are still needed for image matching using deep learning technology.

[0005] In view of the above technical problems, there is an urgent need for an image matching system that can better capture image features at different resolutions, especially in complex scenes such as light changes, extreme view angle changes and repeated textures, so as to improve the accuracy and robustness of image matching. SUMMARY

[0006] In view of at least one of the above technical problems, the present application provides a robust image matching system based on multi-scale attention feature fusion to improve the accuracy and robustness of image matching.

[0007] According to a first aspect of the present application, a robust image matching system based on multi-scale attention feature fusion is provided, comprising:

[0008] comprising the following modules:

[0009] a feature extraction module for extracting a plurality of resolution feature maps containing image features from the image to be matched;

[0010] a position encoding module for spatial position encoding of the feature map;

[0011] Multi-scale attention module: used for enhancing the multi-scale features by convolution operation on the image features with different size of convolution kernel;

[0012] Attention module: including self-attention unit and cross-attention unit, used for establishing the dependency between the image features;

[0013] Coarse level matching layer module: used for generating coarse level matching result by processing the similarity matrix of the image features through double softmax operation, and screening out the matching pairs with higher confidence;

[0014] Coarse to fine module: used for fine processing the coarse matching result.

[0015] In some embodiments of the present application, the feature extraction module is ResNet and FPN architecture, including the following steps:

[0016] Obtaining 1 / 8, 1 / 4 and 1 / 2 resolution feature maps through the ResNet and FPN architecture;

[0017] Downsampling and convolution operation, downsampling the 1 / 4 resolution feature map to the 1 / 8 resolution feature map, and fusing with the original 1 / 8 resolution feature map, outputting the fused 1 / 8 resolution feature map.

[0018] In some embodiments of the present application, the position encoding module is sinusoidal position encoding, including the following steps:

[0019] Generating corresponding x-axis and y-axis positions for each pixel point of the feature map;

[0020] Using an exponential function to generate a set of frequency components, multiplying the x-axis and y-axis positions by the corresponding frequency components respectively, and calculating the sine and cosine values thereof;

[0021] Adding the generated sine and cosine values to the image feature positions in the feature map respectively.

[0022] In some embodiments of the present application, the multi-scale attention module includes the following steps:

[0023] Cloning the feature map x with spatial position encoding as the initial feature map u;

[0024] Convoluting the feature map x with spatial position encoding with a convolution kernel of size 5x5 to generate a preliminary feature map;

[0025] The preliminary feature map is subjected to different span feature extraction by using three different convolution kernels of 1*7 and 7*1, 1*11 and 11*1 and 1*21 and 21*1, and the extracted features are subjected to element-by-element addition to generate a fused feature map;

[0026] The fused feature map is subjected to convolution by using a 1*1 convolution kernel to obtain an integrated feature map;

[0027] The integrated feature map is subjected to element-by-element multiplication with the initial feature map u to output a final result feature map.

[0028] In some embodiments of the present application, the self-attention unit pays attention to the image feature information of other positions in the same image when processing the image feature of each position, to assign different attention weights to the each position; the cross-attention unit is used to process the relationship between different image features to capture the positions of similar features in two images.

[0029] In some embodiments of the present application, the coarse-level matching layer module comprises the following steps:

[0030] A score matrix reflecting the similarity degree between different image features is calculated, and the score matrix S is:

[0031]

[0032] wherein, represents the feature vector of pixel i in image A, represents the feature vector of pixel j in image B, and τ is a temperature parameter for controlling the smoothness of the similarity distribution;

[0033] The score matrix is subjected to normalization processing by using a double-softmax operator, and a matching probability is assigned to each feature, and the matching probability P c is:

[0034] P c = softmax(S(i,·)) j · softmax(S(·,j)) i

[0035] wherein, softmax(S(i,·)) j represents the vector of the similarity score of pixel i in image A and all pixel points in image B, and softmax(S(·,j)) i represents the vector of the similarity score of all pixel points in image A and pixel j in image B.

[0036] Based on the matching probability, a threshold θ c, the higher confidence matching pairs are screened out, the nearest neighbor criterion is applied to filter possible outlier matches, a coarse matching matrix is generated, the coarse matching matrix is:

[0037]

[0038] Wherein, N c is a coarse matching matrix, which contains a plurality of pixel matching pairs MNN represents the pixel is the nearest neighbor of the pixel in image B, and the pixel is also the nearest neighbor of the pixel in image A.

[0039] In some embodiments of the application, the coarse-to-fine module comprises the following steps:

[0040] On the 1 / 2 resolution feature map, two local windows are cropped around each of the coarse matching positions to capture the target feature and its surrounding context information;

[0041] The target feature is processed by the self-attention unit and the cross-attention unit to generate a heat map, and the expected value is calculated by weighting each pixel position in the heat map to obtain the matching probability between all pixels in one window and all pixels in another window.

[0042] In some embodiments of the application, a total loss function is further included, the total loss function includes a coarse matching loss function and a fine matching loss function, the coarse matching loss function is used to determine the matching relationship between two images at low resolution of the image; The fine matching loss function is used to optimize the matching position with high certainty.

[0043] In some embodiments of the application, the formula of the total loss function is:

[0044] L=L c +L f

[0045] Wherein, L is the total loss function, L c is the coarse matching loss function, L f is the fine matching loss function;

[0046] The coarse matching loss function L c is:

[0047]

[0048] Wherein, is the number of true value matching point pairs calculated based on the low resolution feature map, represent the confidence score of matching between two pixel points and ;

[0049] The fine-level matching loss function L f is:

[0050]

[0051] wherein N f is a set of matching pairs, is a matching pair, is a predicted variance, is an error between a point and its true value .

[0052] According to the second aspect of the present application, a robust image matching method based on multi-scale attention feature fusion is also provided, comprising the following steps:

[0053] inputting two images to be matched into a feature extraction module, extracting image features, obtaining feature maps of different resolutions, and encoding feature positions in the feature maps of different resolutions to obtain encoded feature maps;

[0054] inputting the encoded feature maps into a multi-scale attention module, performing multi-scale feature extraction and fusion on the encoded feature maps to obtain a fused feature map;

[0055] inputting the fused feature map into a self-attention and cross-attention module to establish a dependency relationship between features to obtain a dependent feature map;

[0056] calculating the similarity between the dependent feature maps to generate a coarse-level matching matrix, filtering out low-confidence matching pairs, and finally obtaining a fine-level matching matrix.

[0057] The present application has the following advantages: the image matching system based on multi-scale attention feature fusion can perform image matching processing under different complex scenes such as illumination changes, view angle changes and motion blurs. Through the introduction of multi-scale feature fusion and attention mechanism, the system enhances the ability to capture different resolution features of images and can better focus on the salient regions of images. The attention mechanism further improves the dependency relationship between the internal features of images and between different images, thereby improving the accuracy and robustness of matching. The step-by-step optimization strategy from coarse matching to fine matching enables the system to effectively filter low-confidence matching pairs, further reducing the possibility of false matching. Ultimately, the system significantly improves the precision and stability of image matching under complex scenes such as illumination changes, extreme view angle changes and repetitive textures, thereby improving the precision and robustness of image matching. BRIEF DESCRIPTION OF DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings described below are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0059] Figure 1 The model diagram of the robust image matching system based on multi-scale attention feature fusion in the embodiments of the present application;

[0060] Figure 2 The flow chart of the robust image matching system based on multi-scale attention feature fusion in the embodiments of the present application;

[0061] Figure 3 The attention structure diagram of the robust image matching system based on multi-scale attention feature fusion in the embodiments of the present application;

[0062] Figure 4 The matching result of the MegaDepth test data set in the near view in the robust image matching system based on multi-scale attention feature fusion in the embodiments of the present application;

[0063] Figure 5 The matching result of the MegaDepth test data set in the far view in the robust image matching system based on multi-scale attention feature fusion in the embodiments of the present application;

[0064] Figure 6 The matching result of the HPatches data set under slight visual angle change in the robust image matching system based on multi-scale attention feature fusion in the embodiments of the present application;

[0065] Figure 7 The matching result of the HPatches data set under extreme visual change in the robust image matching system based on multi-scale attention feature fusion in the embodiments of the present application;

[0066] Figure 8 The step flow chart of the robust image matching method based on multi-scale attention feature fusion in the embodiments of the present application. DETAILED DESCRIPTION

[0067] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments.

[0068] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0069] As shown in the robust image matching system based on multi-scale attention feature fusion, comprising the following modules: Figures 1 to 7

[0070] Feature extraction module: for extracting multiple resolution feature maps containing image features from the picture to be matched. It should be pointed out here that there are many ways to extract features, which can be convolutional neural network, classical image feature extraction method or autoencoder or other deep learning model.

[0071] Position encoding module: for spatial position encoding of feature map. It should be pointed out here that there are many ways to encode positions, which can be sine and cosine position encoding, or learnable position encoding, or relative position encoding or coordinate embedding and other forms of position encoding.

[0072] Multi-scale attention module: for multi-scale enhancement processing of image features by different size of convolution kernel. It should be pointed out here that there are many forms of multi-scale attention module, which can be the change of convolution kernel size, or dilated convolution, or adaptive convolution and other forms.

[0073] Attention module: including self-attention unit and cross-attention unit, for establishing the dependency between image features.

[0074] Coarse level matching layer module: for processing the similarity matrix of image features by double softmax operation, generating coarse level matching result, and screening out the matching pairs with higher confidence.

[0075] Coarse to fine module: for fine processing of coarse matching result. It should be pointed out here that there are many ways to process in the coarse to fine module, which can be nearest neighbor matching, feature similarity matrix matching, image pyramid based matching or other processing forms.

[0076] ​The plurality of modules in the application, in the specific feature matching of the to-be-tested picture, first, a plurality of resolution feature maps containing image features are extracted from the to-be-matched picture; then, the position information is embedded into each pixel point of the feature map through the sine and cosine position coding, so that the model can process the spatial information under different scales, fuse the features, obtain the coarse and fine granularity features containing rich information, and improve the performance of the model in the image understanding task. Then, the multi-scale convolution kernel is used for feature extraction, and the multi-scale attention mechanism is used to enhance the features of the significant area of the image, and the multi-level convolution operation is used to capture the multi-scale information of the image, and the importance weight of each spatial position is calculated, so that the model can focus on the significant part of the image more effectively. On this basis, the self-attention mechanism is used to extract the dependency relationship between different position features in the same image, and the cross-attention is used to establish the corresponding feature relationship between different images, and the feature expression ability is improved. Finally, the coarse matching is carried out, the preliminary matching result is generated, and the fine matching module is used for further optimization, so as to realize the sub-pixel level accurate matching. The application can effectively cope with complex scenes such as illumination change, view angle change and motion blur, and significantly improves the precision and robustness of image matching.

[0077] The robust image matching system based on multi-scale attention feature fusion can perform image matching processing under different complex scenes such as illumination change, view angle change and motion blur. Through the introduction of multi-scale feature fusion and attention mechanism, the system enhances the capture ability of different resolution features of the image, and can better focus on the significant area of the image; the attention mechanism further improves the dependency relationship between the internal features of the image and between different images, thereby improving the accuracy and robustness of the matching; the step-by-step optimization strategy from coarse matching to fine matching enables the system to effectively filter low-confidence matching pairs, further reducing the possibility of false matching. Finally, the system significantly improves the precision, stability and computational efficiency of image matching under complex scenes such as illumination change, extreme view angle change and repeated texture, thereby improving the precision and robustness of image matching.

[0078] In some embodiments of the application, the feature extraction module is a ResNet and FPN architecture, including the following steps:

[0079] The 1 / 8, 1 / 4 and 1 / 2 resolution feature maps are obtained through the ResNet and FPN architecture;

[0080] The 1 / 4 resolution feature map is down-sampled to a 1 / 8 resolution feature map through down-sampling and convolution operation, and is fused with the original 1 / 8 resolution feature map, to output the fused 1 / 8 resolution feature map.

[0081] By simultaneously generating 1 / 8, 1 / 4 and 1 / 2 resolution feature maps, and then down-sampling and fusing, the model can capture feature information at different resolutions. This multi-scale feature extraction method enables the model to focus on both global and local details, thereby enhancing the processing capability of complex scenes. The feature map at a lower resolution (1 / 8) has a larger receptive field than that at a higher resolution (1 / 4). By combining the two, global context and local details can be obtained simultaneously. In visual tasks, the higher the resolution, the larger the size of the feature map, and the more the computational overhead increases. Selecting a 1 / 8 resolution feature map can achieve the best balance between computational efficiency, global information capture capability and precision requirements. The feature extraction module extracts features at different scales and fuses them, making the model more stable in complex scenes such as changes in lighting and viewing angle, and improving the robustness of image matching.

[0082] In some embodiments of the present application, the position encoding module is a sinusoidal position encoding, comprising the following steps:

[0083] For each pixel point of the resolution feature map, the corresponding x-axis and y-axis positions are generated respectively;

[0084] An exponential function is used to generate a set of frequency components, and the x-axis and y-axis positions are multiplied by the corresponding frequency components to calculate their sine and cosine values;

[0085] The generated sine and cosine values are added to the image feature positions in the resolution feature map respectively.

[0086] The present application encodes image features through sinusoidal encoding, which can better perceive the spatial structure and position information of the image. The position information is directly embedded into the feature map, making the model location-sensitive when processing images, which helps to improve the accuracy of image matching. The sinusoidal position encoding uses a fixed mathematical function to generate position information, without the need for additional parameter training, reducing the complexity of the model and the risk of overfitting, while also improving the efficiency of the encoding process. Accurate position information enhances the adaptability of the model to complex scenes during the matching process, especially in cases of changes in lighting and viewing angle, which can effectively improve the accuracy and robustness of matching.

[0087] In some embodiments of the present application, the multi-scale attention module comprises the following steps:

[0088] Clone the resolution feature map x that has been spatially position encoded as the initial feature map u;

[0089] Convolve the resolution feature map x that has been spatially position encoded using a 5x5 convolution kernel to generate a preliminary feature map;

[0090] The three different convolution kernels of 1*7 and 7*1, 1*11 and 11*1 and 1*21 and 21*1 are used to extract features from the preliminary feature map, and the extracted features are element-wise added to generate a fused feature map;

[0091] The element-wise added feature map is convolved by using the 1*1 convolution kernel to obtain an integrated feature map;

[0092] The integrated feature map is element-wise multiplied with the initial feature map u to output a final result feature map.

[0093] The strategy of multi-scale convolution effectively enhances the perception ability of the model to image features in different ranges, so that the system can process global and local feature information at the same time; can finely process the details and background information existing in the image, especially in complex scenes such as illumination change, view change and repeated texture, can effectively separate important feature information and enhance the robustness of matching; through the element-wise addition operation, the features extracted by different convolution kernels are effectively fused to generate a feature map with rich multi-scale information, and the 1*1 convolution is used to further integrate these information, which helps to eliminate redundancy, improve the expression ability of the feature map, and also can highlight the significant features and suppress the irrelevant information, thereby further strengthening the attention of the model to the key features and improving the final matching accuracy and stability.

[0094] In some embodiments of the present application, the self-attention unit pays attention to the image feature information of other positions in the same image when processing the image feature of each position, to assign different attention weights to each position; the cross-attention unit is used to process the relationship between different image features to capture the positions of similar features in the two images. The self-attention can focus on important areas and ignore irrelevant background noise, while the cross-attention can capture similar features across images, even in the case of changes in image appearance, it can maintain a high matching accuracy, and also better understand the overall structure of the image, especially in scenes with sparse features or complex background, it can establish an accurate feature correspondence relationship, thereby improving the matching accuracy.

[0095] In some embodiments of the present application, the coarse-level matching layer module includes the following steps:

[0096] A score matrix reflecting the similarity between different image features is calculated, and the score matrix S is:

[0097]

[0098] Wherein, represents the feature vector of pixel i in image A, represents the feature vector of pixel j in image B, and τ is a temperature parameter for controlling the smoothness of the similarity distribution;

[0099] The score matrix is normalized by using a double softmax operator, and a matching probability P c is assigned to each feature.

[0100] P c = softmax(S(i,·)) j · softmax(S(·,j)) i

[0101] where softmax(S(i,·)) j denotes a vector of similarity scores of pixel i in image A to all pixels in image B, and softmax(S(·,j)) i denotes a vector of similarity scores of all pixels in image A to pixel j in image B.

[0102] Based on the matching probability, a threshold θ c is set to filter out matching pairs with high confidence, and a nearest neighbor criterion is applied to filter possible outlier matches to generate a coarse matching matrix, which is:

[0103]

[0104] where N c is the coarse matching matrix, which contains multiple pairs of pixel matches MNN denotes the nearest neighbor of pixel in image B, and pixel is also the nearest neighbor of pixel in image A.

[0105] By constructing a similarity score matrix and normalizing the matrix using a double Softmax operation, the similarity matching probability between image features can be quickly calculated, the complex calculation process is reduced, the matching efficiency is effectively improved, large-scale image data can be processed, the matching probability distribution of each feature point is more smooth and reasonable, the probability of false matching is reduced, the matching probability matrix between images is calculated, and screening is performed in combination with the nearest neighbor criterion, so that outlier matching points are effectively filtered out, thereby improving the accuracy of the coarse matching result; by introducing a threshold for screening, unreliable matching pairs can be effectively avoided, the robustness of the system in the face of complex scenes (such as changes in light and changes in viewing angle) is improved, and stable coarse matching results can be obtained under different conditions.

[0106] In some embodiments of the present application, the coarse-to-fine module includes the following steps:

[0107] ​On the 1 / 2 resolution feature map, two local windows are cropped around the position of each coarse-level match to capture the target features and the surrounding context information.

[0108] The target features are processed by self-attention units and cross-attention units to generate a heatmap. The expected value is calculated by weighting the position of each pixel in the heatmap to obtain the matching probability between all pixels in one window and all pixels in another window.

[0109] By cropping local windows at each coarse-level matching location on a 1 / 2 resolution feature map and utilizing self-attention and cross-attention mechanisms to process the target features and their surrounding contextual information, feature details within the local region can be effectively captured. This method allows for more precise determination of the sub-pixel level position of the matching point, significantly improving matching accuracy. Furthermore, the enhanced processing of local information helps retain effective matching information and suppress noise in complex scenes (such as changes in illumination and viewpoint). Based on the coarse matching results, further refinement processing can adjust the matching position and ensure high confidence in the final fine-level matching results. By weighting and calculating the expected value for each pixel position in the heatmap, sub-pixel level precise matching can be obtained. This stepwise optimization strategy improves the model's robustness in the face of challenges such as complex backgrounds and repetitive textures.

[0110] In some embodiments of the present invention, a total loss function is also included. This total loss function comprises a coarse-level matching loss function and a fine-level matching loss function. The coarse-level matching loss function is used to determine the matching relationships under feature maps of different resolutions; the fine-level matching loss function is used to optimize matching positions with high determinism. Preliminary matching using the coarse-level matching loss function provides the model with relatively accurate initial matching relationships, avoiding the loss of key matching points in the initial stage. The fine-level matching loss function further refines the results by optimizing high-confidence matching positions, achieving sub-pixel accuracy, thereby significantly improving matching accuracy.

[0111] In some embodiments of the present invention, the formula for the total loss function is:

[0112] L = L c +L f

[0113] Where L is the total loss function, L c L is the coarse-level matching loss function. f This is the fine-level matching loss function;

[0114] coarse-level matching loss function L c for:

[0115]

[0116] wherein, is the number of ground truth matching point pairs obtained based on the low resolution feature map, represents the confidence score of matching between two pixel points and .

[0117] The fine-level matching loss function L f is:

[0118]

[0119] wherein, N f is a set of matching pairs, is a matching pair, is a predicted variance, is the error between the point and its ground truth .

[0120] The total loss function can optimize the global matching relationship and the local matching accuracy at the same time by combining the coarse-level matching loss and the fine-level matching loss. The coarse-level matching loss L c ensures to identify the global matching relationship on the low resolution feature map, while the fine-level matching loss L f improves the local accuracy of the matching points through fine processing. This way ensures the consideration of global consistency and local accuracy, and improves the overall matching effect. Through the dual optimization of coarse matching and fine matching, the total loss function makes the model have stronger robustness under different scenes (such as light changes, view angle changes, etc.). The coarse-level loss is responsible for the overall framework, and the fine-level loss is responsible for local optimization, so as to ensure that the model can still maintain high matching accuracy and stability in complex scenes, thereby improving the final matching effect.

[0121] The robust image matching method based on multi-scale attention feature fusion comprises the following steps:

[0122] input two images to be matched into a feature extraction module, extract image features, obtain different resolution feature maps, and encode the feature positions in the resolution feature maps to obtain encoded feature maps;

[0123] input the encoded feature maps into a multi-scale attention module to perform multi-scale feature extraction and fusion on the encoded feature maps to obtain a fused feature map;

[0124] input the fused feature map into a self-attention and cross-attention module to establish a dependency relationship between the features to obtain a dependent feature map;

[0125] The similarity between the computed dependency feature maps is calculated to generate a coarse matching matrix, and the matching pairs with low confidence are filtered out to obtain a fine matching matrix.

[0126] To evaluate the effectiveness and accuracy of the proposed method in different complex application scenarios, the HPatches dataset and the MegaDepth dataset are selected for experimental verification, and comparative analysis is conducted with different baseline algorithms.

[0127] MegaDepth is a large-scale dataset for evaluating image matching under extreme view changes and repetitive textures, consisting of 1 million pairs of images from 196 different outdoor scenes, each image containing known pose and depth information. In the following experimental study, the pose error AUC index is used to evaluate the performance of the algorithm, where the pose error is the maximum of the angular error in rotation and translation, and the larger the AUC, the better the algorithm performance.

[0128] HPatches is a dataset for evaluating local descriptors, containing image files with different changes in light intensity and different shooting angles. In the experiment, 56 scene sequences with significant view changes and 52 scene sequences with significant light intensity changes are selected for image matching performance analysis. In the following experimental study, the homography matrix AUC index is used to evaluate the performance of the algorithm, and the homography is estimated under different pixel thresholds by the RANSAC method and compared with the true value, where the larger the AUC value, the better the algorithm performance.

[0129] Table 1 shows the image matching performance of the proposed method on different data and the comparison results with other methods. From the table, it can be seen that the proposed method has good performance on several datasets, indicating that the model can effectively handle image matching tasks in complex scenarios such as extreme view changes and light changes. In addition, compared with the SP+SuperGlue method based on detector, the proposed method avoids the limitations of the feature detection stage and directly performs feature extraction and matching, making the algorithm still able to find reliable matches in low-texture areas and repetitive patterns.

[0130] Table 1 AUC performance evaluation of image matching

[0131]

[0132] Figure 3 、 Figure 4 The matching results of two images with extreme view changes and repetitive textures selected from the MegaDepth test dataset are shown, where Figure 3 is taken at close range, Figure 4are shot at a long distance. As can be seen from the figure, the proposed model can effectively handle image matching problems under extreme view angle changes and repeated texture conditions, improving the ability in practical applications.

[0133] Figure 5 、 Figure 6 The matching results of two images with different light intensity changes and different shooting angles selected from the HPatches dataset are shown, wherein Figure 5 is shot at a slight view angle change, Figure 6 is shot at an extreme visual change. As can be seen from the figure, the matching method based on the multi-scale attention model still performs well under challenging conditions. Especially in the presence of obvious view angle changes and different light conditions, the model can still successfully capture the corresponding feature points in the two images. The advantages of the multi-scale attention model in the image matching task are verified, especially in dealing with complex scenes.

[0134] Those skilled in the art should understand that the present application is not limited to the above-mentioned embodiments, and the above-mentioned embodiments and descriptions in the specification are only to illustrate the principles of the present application, and various changes and improvements can be made without departing from the spirit and scope of the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A robust image matching system based on multi-scale attention feature fusion, characterized in that, The system comprises the following modules: a feature extraction module for extracting a plurality of resolution feature maps containing image features from the images to be matched; a position encoding module for spatial position encoding of the feature maps; a multi-scale attention module for enhancing the multi-scale features by convolution operation of different size convolution kernels on the image features; an attention module comprising a self-attention unit and a cross-attention unit for establishing the dependency between the image features; a coarse-level matching layer module for processing the similarity matrix of the image features by double-softmax operation to generate coarse-level matching results and screen out matching pairs with higher confidence; a coarse-to-fine module for fine processing of the coarse matching results; the multi-scale attention module comprises the following steps: the feature map spatially positionally encoded cloning as initial feature map ; The spatial position encoded feature map is convolved with a convolution kernel of size The preliminary feature map is generated by convolution.​ use and , and as well as and Three different convolutional kernels are used to extract features across different spans from the initial feature map, and the extracted features are added element by element to generate a fused feature map. Utilizing The convolution kernel is used to convolve the fused feature map to obtain an integrated feature map. integrate the feature map with the initial feature map perform element-wise multiplication, output the final result feature map; the coarse-to-fine module comprises the following steps: two local windows are cropped around each of the coarse-level matching positions on the 1 / 2 resolution feature map to capture the target features and their surrounding context information; the target features are processed by the self-attention unit and the cross-attention unit to generate a heat map, and the expected value is calculated by weighting each pixel position in the heat map to obtain the matching probability between all pixels in one window and all pixels in another window.

2. The robust image matching system based on multi-scale attention feature fusion according to claim 1, characterized in that, The feature extraction module is a ResNet and FPN architecture, comprising the following steps: 1 / 8, 1 / 4 and 1 / 2 resolution feature maps are obtained by the ResNet and FPN architecture; the 1 / 4 resolution feature map is down-sampled to the 1 / 8 resolution feature map by down-sampling and convolution operation, and is fused with the original 1 / 8 resolution feature map to output the fused 1 / 8 resolution feature map.

3. The robust image matching system based on multi-scale attention feature fusion according to claim 1, characterized in that, The position encoding module is a sinusoidal position encoding, comprising the following steps: For each pixel point of the feature map, a corresponding axis and axis position; using an exponential function to generate a set of frequency components, multiplying the shaft position by the corresponding frequency components, respectively, to calculate the sine and cosine values thereof the shaft position and the shaft position and the generated sine and cosine values are added to the image feature positions in the feature map, respectively.

4. The robust image matching system based on multi-scale attention feature fusion according to claim 2, characterized in that, The self-attention unit pays attention to the image feature information of other positions in the same image when processing the image features of each position to assign different attention weights to each position; the cross-attention unit is used to process the relationship between different image features to capture the positions of similar features in the two images.

5. The robust image matching system based on multi-scale attention feature fusion according to claim 1, characterized in that, The coarse-level matching layer module comprises the following steps: a score matrix reflecting the similarity between different image features is calculated, and the score matrix S is: ； wherein, represents a feature vector of a pixel in image A, represents a feature vector of a pixel point in image B, is a temperature parameter for controlling the smoothness of the similarity distribution; normalizing the score matrix using a dual softmax operator and assigning a match probability to each feature, the match probability is: ； wherein, represents a pixel in image A a vector of similarity scores of all pixels in image A with all pixels in image B, represents a pixel in image A a vector of similarity scores of all pixels in image A with all pixels in image B; Based on the matching probability, by setting a threshold , screening out the matching pairs with higher confidence, applying the nearest neighbor criterion to filter possible outlier matches, generating a coarse level matching matrix, which is: ； wherein, is a coarse level matching matrix that contains pairs of pixel matches , MNN denotes a pixel is a pixel in image B, while pixel is also a pixel in image A.

6. The robust image matching system based on multi-scale attention feature fusion according to claim 1, characterized in that, It also comprises a total loss function, which comprises a coarse-level matching loss function and a fine-level matching loss function, the coarse-level matching loss function is used to determine the matching relationship between two images at low resolution of the image; the fine-level matching loss function is used to optimize the matching positions with high certainty.

7. The robust image matching system based on multi-scale attention feature fusion according to claim 6, characterized in that, The formula of the total loss function is: ； wherein L is a total loss function, is a coarse level matching loss function, is a fine level matching loss function; The coarse level matching loss function is: ； wherein, is the number of ground truth matching point pairs found based on the low resolution feature map, represents the confidence score of the matching between two pixel points and ; The fine-level matching loss function is: ； wherein, is a set of matching pairs, is a matching pair, is a variance of the prediction, is a point with an error from its true value .

8. A robust image matching method based on multi-scale attention feature fusion, characterized in that, The robust image matching system based on multi-scale attention feature fusion as claimed in any one of claims 1 to 7 comprises the following steps: input two images to be matched into the feature extraction module, extract image features, obtain feature maps of different resolutions, and encode the feature positions in the feature maps of different resolutions to obtain encoded feature maps; The encoded feature map is input into a multi-scale attention module, multi-scale feature extraction and fusion are performed on the encoded feature map, and a fused feature map is obtained; The fused feature map is input into a self-attention and cross-attention module, a dependency relationship between features is established, and a dependent feature map is obtained; Similarity between the dependent feature maps is calculated to generate a coarse-level matching matrix, and matching pairs with low confidence are filtered out, and finally a fine-level matching matrix is obtained.

Images