Multi-modal image feature matching method and device based on learnable local semantic enhancement

Abstract

The invention relates to the technical field of image feature matching, and discloses a multi-modal image feature matching device based on learnable local semantic enhancement, which comprises a data enhancement module, a feature extraction module, a local semantic feature enhancement module and a matching module, the feature extraction module is used for carrying out feature extraction on the image, the local semantic feature enhancement module is used for enhancing local semantic information of the image, and the matching module is used for carrying out coarse-level matching and refined matching on the image to obtain a final optimized image. According to the method, through extraction of a local semantic feature map and information interaction between a matching feature map and a semantic feature map, the module can deeply mine and reinforce local semantic information in an image, which is crucial to improvement of the recognition and understanding ability of a model, and through fine local feature analysis and enhancement, the recognition and understanding ability of the model is improved. The model can better identify the key elements in the image and the mutual relation of the key elements.

Description

Technical Field

[0001] The present invention relates to the technical field of image feature matching, and in particular to a method and device for multimodal image feature matching based on learnable local semantic enhancement. Background Art

[0002] Image matching is one of the key issues in the field of computer vision, which involves operations such as feature extraction, description and comparison of images to identify and establish similarities, associations or correspondences between images. With the rapid development of computer vision and the surge in image data, image matching has attracted much attention as a research focus. Not only limited to single visual data, the application scope of image matching has expanded to cover data forms of multiple modalities, which means not only matching between static images, but also including videos, three-dimensional models, infrared images, medical images (such as X-rays, CT scans, etc.), satellite images and other sensor data. These multimodal data are widely present in various real-world scenarios. For example, in the field of autonomous driving, image data needs to be fused with radar or lidar data, or in medical imaging, image data needs to be associated with medical record text data. Therefore, the processing of comprehensive multimodal data has become the key to improving image matching performance and solving matching problems in complex scenarios.

[0003] Data augmentation is a widely used method in machine learning and deep learning, which aims to increase the number and diversity of training samples by transforming and expanding existing data sets. This technology generates new data samples that simulate real-world changes by performing a series of transformation operations on the original data, such as rotation, scaling, cropping, translation, flipping, etc., as well as adding noise, blurring, color transformation, etc. Data augmentation technology can effectively alleviate the problem of data scarcity or imbalance and improve the generalization ability and robustness of the model. In fields such as computer vision, natural language processing, and speech recognition, data augmentation technology has become one of the key means to improve model performance.

[0004] There are usually two types of feature matching methods in modern technology, one is a matching method with a detector, and the other is a method without a detector:

[0005] The matching method with detector is a traditional image feature matching strategy that relies on an explicit feature point detection stage. This method first uses a specific feature detector to find significant points of interest or key points in the image, then extracts local descriptors around these points, and finally uses a matching algorithm to find the best matching pair between feature descriptors from different images. This method usually performs well when processing images with rich textures and significant feature points. However, its disadvantage is that when there are fewer significant feature points in the image, or in low-texture, blurred or noisy scenes, the detector may not be able to find stable key points, resulting in a decrease in matching results.

[0006] Detector-free matching methods are an end-to-end image feature matching strategy that avoids independent feature point detection and feature description stages. This method usually uses a deep learning model to extract dense feature maps from the image, and then uses a neural network to directly learn the correspondence between these dense descriptors. Detector-free methods achieve higher robustness in dealing with challenges such as low-texture areas and repeated patterns by leveraging the rich contextual information that is prevalent in images. However, traditional detector-free matching methods do not effectively utilize the semantic information in the image, which limits their performance in complex scenes. In addition, the disadvantage of this method is that it usually requires a large amount of data for training and consumes high computing resources. Due to its reliance on deep learning models, the matching effect may deteriorate when encountering images that are significantly different from the training data. Summary of the invention

[0007] 1. Technical issues to be resolved

[0008] In view of the deficiencies in the prior art, the present invention provides a method and device for multimodal image feature matching based on learnable local semantic enhancement, which has the advantages of improving the matching accuracy and solves the above-mentioned technical problems.

[0009] (II) Technical solution

[0010] To achieve the above object, the present invention provides the following technical solution: a method for multimodal image feature matching based on learnable local semantic enhancement, which comprises the following steps:

[0011] S1. Perform data enhancement on the input image;

[0012] S2. Perform feature extraction through CNN and transformer to obtain a feature map of one-sixteenth scale, and upsample the feature map of one-sixteenth scale to obtain feature maps of one-eighth scale, one-quarter scale, and one-half scale, and add the feature maps of one-sixteenth scale, one-eighth scale, and one-quarter scale to obtain a fused multi-scale feature map pair I. 5 ;

[0013] S3, for the fusion of multi-scale feature maps I 5 Extract local semantic features and perform feature fusion to obtain the feature map pair I that needs to be matched. Final ;

[0014] S4, to I Final Perform coarse-level matching and refined matching;

[0015] S5. Output the result of step S4.

[0016] As a preferred technical solution of the present invention, the specific process in step S2 includes:

[0017] S2.1. Perform feature extraction on the data-enhanced image to obtain a feature map at a scale of one-sixteenth. The specific steps are as follows:

[0018] I 2 =F CNN (I)

[0019]

[0020] Among them, I represents the image to be processed, I 2 and I 3 represents the feature map during processing, Represents the feature map at one-sixteenth scale, F CNN Represents all convolutional neural network layers, SelfAttention(*) represents self-attention;

[0021] S2.2. Feature map at one-sixteenth scale Upsampling is performed in sequence to obtain feature maps of one-eighth scale, one-quarter scale, and one-half scale respectively. The specific expressions are as follows:

[0022]

[0023] in, Represent the feature maps of one-eighth scale, one-quarter scale, and one-half scale respectively. Respectively represent upsampling the feature map to one-eighth scale, one-quarter scale, and one-half scale, and residual represents the residual in the feature pyramid;

[0024] S2.3. Perform self-attention and cross-attention processing on the one-eighth scale feature map to obtain the processed one-eighth scale feature map I 4 , the specific expression is as follows:

[0025]

[0026] Among them, I 4 represents the processed one-eighth scale feature map, Represents a feature map of one-eighth scale, CrossAttention represents cross attention, and SelfAttention represents self-attention;

[0027] S2.4. Fusing the feature maps of the quarter scale and the sixteenth scale into the processed feature map of the eighth scale I 4 The specific expression is as follows:

[0028]

[0029] Among them, I 5 It represents the feature map pair after fusion of multiple scales, Upsample and Downsample represent upsampling and downsampling respectively. and They represent feature maps at one-sixteenth scale and one-half scale respectively.

[0030] As a preferred technical solution of the present invention, step S3 comprises the following steps:

[0031] S3.1, after the fusion of multi-scale feature maps in step S2.4, I 5 Perform self-attention and cross-attention operations and add position encoding. The specific expression is as follows:

[0032] I 6 =CrossAttention(SelfAttention(I 5 +PositionalEncoding))

[0033] Among them, I 6 represents image pair features, CrossAttention represents cross attention, SelfAttention represents self attention, PositionalEncoding represents position encoding, I 5 Represents the feature map pair after fusion of multiple scales, and the feature map and Divided into K local semantic category vectors, 2 feature maps with K local semantic features are obtained. The specific expression is as follows:

[0034]

[0035] in, and Represents the image pair features I 6 The query image and reference image inside, and Respectively indicate the corresponding and The ExtractionModule represents the operation of extracting local semantic features from the feature map, which consists of a series of one-dimensional convolutions.

[0036] S3.2. Calculate the similarity between each cell of the local semantic feature map and its corresponding one-eighth feature map, and divide each cell of the one-eighth feature map into K categories according to the similarity. Then perform self-attention on each category for feature enhancement. The specific calculation expression is as follows:

[0037]

[0038] in, and It represents K groups of features that classify the feature map query and feature map refer. and Respectively represent taking out the i-th group of features from the K groups of features, and A represents the features after self-attention of the i-th group of features. query and A refer It is the feature map after self-attention in all categories. Similarity means calculating similarity.

[0039] S3.3. Local semantic feature map and A query and A refer The feature map performs cross attention operation, the specific expression is as follows:

[0040]

[0041] Among them, O query and O refer It represents the feature map after cross attention, Q, K, V represent query, key and value respectively;

[0042] S3.4. After the cross attention feature map O query and O refer Perform the reverse operation, the specific expression is as follows:

[0043]

[0044] Among them, CrossAttention means cross attention, O' query and O' refer Indicates the feature map O after the cross attention query and O refer Feature map after reverse operation;

[0045] S3.5. Use cross attention to feature map O' query and O' refer Processing, and finally matching the feature map I used Final , the specific expression is as follows:

[0046] I Final =Self&Cross(O' query ,O' refer )

[0047] Among them, Self represents the self-attention operation, Cross represents the cross attention, O' query and O' refer Represents the feature map pair to be processed, I Final Indicates the feature map pair that needs to be matched in the end.

[0048] As a preferred technical solution of the present invention, the specific process of step S4 is as follows:

[0049] S4.1. Coarse level matching: I Final The MNN algorithm is used for matching to obtain coarse matching results;

[0050] S4.2. Position the coarse-level matching result to a feature map of half scale The heat map corresponding to each matching point is calculated through self-attention and cross-attention operations, and the corresponding optimization points of the matching points are obtained according to the heat map. All optimization points constitute the optimized image.

[0051] The present invention also provides a device for multimodal image feature matching based on learnable local semantic enhancement, based on the above-mentioned method and device for multimodal image feature matching based on learnable local semantic enhancement, comprising a data enhancement module, a feature extraction module, a local semantic feature enhancement module and a matching module;

[0052] The data enhancement module is used to enhance the image, the feature extraction module is used to extract features from the image, the local semantic feature enhancement module is used to enhance the local semantic information of the image, and the matching module is used to perform coarse-level matching and refined matching on the image to obtain the final optimized image.

[0053] Compared with the prior art, the present invention provides a method and device for multimodal image feature matching based on learnable local semantic enhancement, which has the following beneficial effects:

[0054] 1. By combining the backbone of Attention, the present invention can more intelligently focus on key information areas when processing input data, effectively suppress irrelevant noise, and thus improve the accuracy and efficiency of feature extraction. It shows excellent performance improvement in both capturing details and understanding the overall structure.

[0055] 2. The present invention integrates feature maps of different scales. This strategy greatly enriches the hierarchy and diversity of feature expression, ensures all-round feature coverage from micro to macro, and not only retains the fine structure of the image, but also this comprehensive feature representation method greatly improves the robustness and accuracy of the model when dealing with complex and changing scenes;

[0056] 3. By extracting local semantic feature maps and matching the information interaction between feature maps and semantic feature maps, the module can deeply explore and strengthen the local semantic information in the image, which is crucial to improving the recognition and understanding capabilities of the model. Through sophisticated local feature analysis and enhancement, the model can better identify the key elements in the image and their relationships. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a schematic diagram of data enhancement of the present invention;

[0058] Figure 2 This is the overall architecture diagram of the present invention;

[0059] Figure 3 It is a schematic diagram of feature extraction of the present invention;

[0060] Figure 4 Schematic diagram of the semantic feature enhancement module of the present invention;

[0061] Figure 5 It is a schematic diagram of the process of the present invention;

[0062] Figure 6 It is a schematic diagram of the structure of the present invention;

[0063] Figure 7 The structural block diagram of the multi-modal image feature matching device of the present invention is DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] See also Figure 1-7 , a method for multimodal image feature matching based on learnable local semantic enhancement, comprising the following steps:

[0066] S1. Perform data enhancement on the input image. The specific enhancement method is as follows: Figure 1 As shown;

[0067] S2. Perform feature extraction through CNN and transformer to obtain a feature map of one-sixteenth scale, and upsample the feature map of one-sixteenth scale to obtain feature maps of one-eighth scale, one-quarter scale, and one-half scale, and add the feature maps of one-sixteenth scale, one-eighth scale, and one-quarter scale to obtain a fused multi-scale feature map pair I. 5 This step is as follows Figure 3 As shown;

[0068] S2.1. Perform feature extraction on the data-enhanced image to obtain a feature map at a scale of one-sixteenth. The specific steps are as follows:

[0069] I 2 =F CNN (I)

[0070]

[0071] Among them, I represents the image to be processed, I 2 and I 3 represents the feature map during processing, Represents the feature map at one-sixteenth scale, F CNN Represents all convolutional neural network layers, SelfAttention(*) represents self-attention;

[0072] S2.2. Feature map at one-sixteenth scale Upsampling is performed in sequence to obtain feature maps of one-eighth scale, one-quarter scale, and one-half scale respectively. The specific expressions are as follows:

[0073]

[0074] in, Represent the feature maps of one-eighth scale, one-quarter scale, and one-half scale respectively. Respectively represent upsampling the feature map to one-eighth scale, one-quarter scale, and one-half scale, and residual represents the residual in the feature pyramid;

[0075] This new backbone design that combines CNN and attention mechanism brings new ideas and methods to image matching tasks. Compared with the traditional pyramid structure CNN, this framework that combines different feature extraction methods is expected to achieve better performance in feature expression and matching results, providing more possibilities for the research and application of image matching tasks. In the next step, we will focus on how to better integrate feature information of different scales to further enhance the model's processing capabilities for image matching tasks. This step is crucial because the comprehensive use of features of different scales can enhance the model's understanding of image details and global information, and help improve the accuracy and robustness of matching;

[0076] S2.3. Perform self-attention and cross-attention processing on the one-eighth scale feature map to obtain the processed one-eighth scale feature map I 4 , the specific expression is as follows:

[0077]

[0078] Among them, I 4 represents the processed one-eighth scale feature map, Represents a feature map of one-eighth scale, CrossAttention represents cross attention, and SelfAttention represents self-attention;

[0079] Finally, we obtained feature maps of one-eighth scale after multiple optimizations and processing. These feature maps not only integrate multi-scale information, but also obtain richer and more complex representations under the action of the attention mechanism, providing a deeper and more accurate feature description for the model's image matching task. This gradual iterative optimization process is expected to improve the model's ability to abstract and express image features, thereby achieving more significant performance improvements in image matching tasks;

[0080] S2.4. Fusing the feature maps of the quarter scale and the sixteenth scale into the processed feature map of the eighth scale I 4 The specific expression is as follows:

[0081]

[0082] Among them, I 5 It represents the feature map pair after fusion of multiple scales, Upsample and Downsample represent upsampling and downsampling respectively. and They represent feature maps at one-sixteenth scale and one-half scale respectively.

[0083] The feature map at one-sixteenth scale will be upsampled to one-eighth scale. This operation aims to increase the spatial range of the feature map so that it can be aligned with the feature map at one-eighth scale for better information fusion.

[0084] The next key step is to add the downsampled and upsampled feature maps of the quarter scale and the sixteenth scale to the feature map of the eighth scale to obtain a new feature map of the eighth scale. This fusion process organically combines information of different scales and makes full use of the feature expressions at each scale, in order to improve the performance of the model for image matching tasks at different scales.

[0085] This method of fusing multi-scale information helps the model understand and process image data more comprehensively, achieving more accurate and robust results in matching tasks. By integrating feature information at different scales, it is expected to improve the model's ability to perceive image details and overall structure, thereby achieving better performance in image matching tasks.

[0086] S3, for the fusion of multi-scale feature maps I 5 Extract local semantic features and perform feature enhancement to obtain the feature map pair I that needs to be matched. Final ;

[0087] Step S3 includes the following steps:

[0088] S3.1, after the fusion of multi-scale feature maps in step S2.4, I 5 Perform self-attention and cross-attention operations and add position encoding. The specific expression is as follows:

[0089] I 6 =CrossAttention(SelfAttention(I 5 +PositionalEncoding))

[0090] Among them, I 6 represents the image pair feature, CrossAttention represents cross attention, SelfAttention represents self attention, and the attention parameters are independent of each other, that is, the parameters are not shared, PositionalEncoding represents position encoding, I 5 Represents the feature map pair after fusion of multiple scales, and the feature map and It is divided into K local semantic category vectors, and two feature maps with K local semantic features are obtained. That is, the two feature maps after the attention operation extract K semantic features respectively. The feature maps are processed by ExtractionModule to obtain two local semantic feature maps, and K representative local semantic features are extracted respectively. These features can be regarded as a high degree of summary of the image content, which can represent the most discriminative and representative local semantic feature information in each. The specific expression is as follows:

[0091]

[0092] in, and Represents the image pair features I 6 The query image and reference image inside, and Respectively indicate the corresponding and The ExtractionModule represents the operation of extracting local semantic features from the feature map, which consists of a series of one-dimensional convolutions.

[0093] S3.2, this is the appendix Figure 2 The SemFE module in Calculate the similarity between the local semantic feature map and each cell of its corresponding one-eighth feature map, and divide each cell of the one-eighth feature map into K categories according to the similarity. Then, perform self-attention on each category for feature enhancement. The specific calculation expression is as follows:

[0094]

[0095] in, and It represents K groups of features that classify the feature map query and feature map refer. and Respectively represent taking out the i-th group of features from the K groups of features, and A represents the features after self-attention of the i-th group of features. query and A refer It is the feature map after self-attention in all categories. Similarity means calculating similarity.

[0096] Through this series of complex operations, we can obtain two one-eighth scale feature maps A after feature enhancement. query and A refer, this pair of feature maps will include multi-level, multi-stage processing and interaction, so that it has a higher ability to express information and a deep understanding of image features. The purpose of this process is to further improve the adaptability and accuracy of the model for image matching tasks, and lay a more solid foundation for subsequent matching and recognition tasks;

[0097] S3.3. Local semantic feature map and A query and A refer The feature map performs cross attention operation, the specific expression is as follows:

[0098]

[0099] Among them, O query and O refer It represents the feature map after cross attention, Q, K, V represent query, key and value respectively;

[0100] S3.4. After the cross attention feature map O query and O refer Perform the reverse operation, the specific expression is as follows:

[0101]

[0102] Among them, CrossAttention means cross attention, O' query and O' refer Indicates the feature map O after the cross attention query and O refer Feature map after reverse operation;

[0103] Such a design can realize the bidirectional transmission and integration of information in another direction, ensuring that the feature map can effectively absorb and integrate important information from local semantic features;

[0104] S3.5. Use cross attention to feature map O' query and O' refer Processing, and finally matching the feature map I used Final , the specific expression is as follows:

[0105] I Final =Self&Cross(O' query ,O' refer )

[0106] Among them, Self represents the self-attention operation, Cross represents the cross attention, and the attention parameters are independent of each other. query and O' refer Represents the feature map pair to be processed, I FinalIndicates the feature map pair that needs to be matched in the end.

[0107] Finally, after this series of cross attention operations, the final two one-eighth scale feature map pairs I Final This pair of feature maps will contain feature information that has undergone multiple high-level interactions and integrations, and have a richer and more comprehensive representation capability. These final feature maps will provide key feature representations for subsequent image matching operations, providing stronger support and foundation for the matching performance and accuracy of the model;

[0108] This multi-level feature fusion and interaction based on cross-attention is crucial to the success of the image matching task. Through this meticulous and in-depth information transfer and integration between feature maps, it is expected to obtain more representative and robust feature representations, thereby improving the model's image matching and recognition effects in different scenarios. This process is a key step in the entire model design, and its results will have a profound impact on the model performance;

[0109] S4, to I Final Perform coarse-level matching and refined matching;

[0110] The specific process of step S4 is as follows:

[0111] S4.1. Coarse level matching: I Final The MNN algorithm is used for matching to obtain coarse matching results;

[0112] First, we will use the feature map to Final Perform similarity calculation. Assuming the size of the feature map is S, the size of the similarity matrix will be S*S, and the similarity matrix will be squared after calculation. Next, the nearest neighbor matching (MNN) algorithm will be used. Here, a threshold is first set to filter points, and only those points whose similarity exceeds the threshold are retained. When performing point matching, it is required that two points must have the highest similarity to each other in order to be confirmed as matching points. In this way, a series of matching points are obtained;

[0113] S4.2. Position the coarse-level matching result to a feature map of half scale The heat map corresponding to each matching point is calculated through the self-attention and cross-attention operations. The corresponding optimization points of the matching points are obtained according to the heat map, and all the optimization points constitute the optimized image.

[0114] Next, we will refine the matching points. First, we fix the points at half the scale of the feature map. In the above example, assuming that the matching point is (i, j), i is fixed on one feature map. Then, a w*w window is obtained with j as the center for the position of point j in another feature map. Within this window, self-attention and cross-attention operations are applied again. The purpose of this step is to further refine the precision and accuracy of the matching points.

[0115] After the self-attention and cross-attention operations, the heat map of the point corresponding to i will be calculated. Based on this heat map, an optimized point j will be obtained, called j'. This process is equivalent to optimizing and adjusting the matching points to make the matching more precise and accurate. The final optimized point (i, j') will represent the matching result after refined processing.

[0116] S5. Output the result of step S4.

[0117] This series of operations is to ensure the accuracy and robustness of the matching points. By combining coarse matching and refined matching, the matching points can be processed and optimized more carefully to meet the matching requirements of different scales and scenes. This meticulous and precise matching process will provide a more reliable and accurate foundation for the final image matching results and provide strong support for the matching performance of the model.

[0118] The methods for comparing the image feature matching methods in this application include HardNet, MatchosNet, TFeat, MatchNet, LoFTR and FeMIP. These methods are existing technologies and have very good feature matching capabilities. Using these methods as comparison methods can more objectively prove the image feature matching capabilities and effects of the present invention.

[0119] 1. Verify the matching effects of different methods on the dataset of infrared image and optical image matching.

[0120]

[0121] 2. Verify the matching effects of different methods in the dataset of SAR image and optical image matching.

[0122]

[0123]

[0124] It can be seen that on different multimodal data sets, the homography estimation of the present invention is better than other methods, which proves that the present invention has good accuracy and generalization when processing multimodal data;

[0125] The present invention also provides a device for multimodal image feature matching based on learnable local semantic enhancement, based on the above-mentioned method and device for multimodal image feature matching based on learnable local semantic enhancement, comprising a data enhancement module 10, a feature extraction module 20, a local semantic feature enhancement module 30, and a matching module 40;

[0126] Data enhancement module 10: This invention proposes a new multimodal image feature matching method, which innovatively combines data enhancement technology to effectively improve the quality of input images and lay a solid foundation for subsequent feature extraction and matching. Through a series of transformation operations, such as rotation, scaling, cropping, etc., data enhancement technology can generate diverse training samples, thereby enhancing the generalization ability and robustness of the model.

[0127] Feature extraction module 20: The present invention adopts an advanced feature extraction algorithm, especially combined with the backbone structure of Transformer, to efficiently capture the global and local features in the image. This structure can fully consider the correlation information between different positions in the image through the self-attention mechanism, thereby extracting richer and more accurate feature representations.

[0128] Local semantic feature enhancement module 30: The present invention innovatively proposes a new local semantic feature enhancement module. This module can perform targeted enhancement and optimization on the local semantic information in the image, further improving the accuracy and effect of feature matching. Traditional methods often ignore the rich semantic information contained in the image when processing images, while the present invention focuses on mining and utilizing these key semantic features.

[0129] Matching module 40: Coarse matching and fine matching are two important steps in the multimodal image feature matching method proposed in the present invention. In the coarse matching stage, the MNN algorithm is used to match the final features and obtain preliminary matching results on the coarse feature map. In the fine matching stage, the coarse matching results are further refined on the fine-grained feature map. The coarse matching results are positioned on one-half of the feature map and a window is divided with the matching point as the center, and the attention mechanism is applied again to refine the matching results.

[0130] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multimodal image feature matching method based on learnable local semantic enhancement. Features: The following steps are involved: S1. Perform data enhancement on the input image; S2. Perform feature extraction through CNN and transformer to obtain a feature map of one-sixteenth scale, and upsample the feature map of one-sixteenth scale to obtain feature maps of one-eighth scale, one-quarter scale, and one-half scale, and add the feature maps of one-sixteenth scale, one-eighth scale, and one-quarter scale to obtain a fused multi-scale feature map pair I. 5 ; S3, for the fusion of multi-scale feature maps I 5 Extract and enhance local semantic features, and perform feature fusion to obtain the feature map pair I that needs to be matched. Final ; S4, for I Final Perform coarse-level matching and refined matching; S5. Output the result of step S4.

2. The method for multimodal image feature matching based on learnable local semantic enhancement according to claim 1, Features: The specific process in step S2 includes: S2.

1. Extract features from the data-enhanced image to obtain a feature map at a scale of one-sixteenth. The specific steps are as follows: I 2 =F CNN (I) Among them, I represents the image to be processed, I 2 and I 3 represents the feature map during processing, Represents the feature map at one-sixteenth scale, F CNN Represents all convolutional neural network layers, SelfAttention(*) represents self-attention; S2.

2. Feature map at one-sixteenth scale Upsampling is performed in sequence to obtain feature maps of one-eighth scale, one-quarter scale, and one-half scale respectively. The specific expressions are as follows: in, Represent the feature maps of one-eighth scale, one-quarter scale, and one-half scale respectively. Respectively represent upsampling the feature map to one-eighth scale, one-quarter scale, and one-half scale, and residual represents the residual in the feature pyramid; S2.

3. Perform self-attention and cross-attention processing on the one-eighth scale feature map to obtain the processed one-eighth scale feature map I 4 , the specific expression is as follows: Among them, I 4 represents the processed one-eighth scale feature map, Represents a feature map of one-eighth scale, CrossAttention represents cross attention, and SelfAttention represents self-attention; S2.

4. Fusing the feature maps of the quarter scale and the sixteenth scale into the processed feature map of the eighth scale I 4 The specific expression is as follows: Among them, I 5 It represents the feature map pair after fusion of multiple scales, Upsample and Downsample represent upsampling and downsampling respectively. and They represent feature maps at one-sixteenth scale and one-half scale respectively.

3. The method for multimodal image feature matching based on learnable local semantic enhancement according to claim 2, Features: The step S3 comprises the following steps: S3.1, after the fusion of multi-scale feature maps in step S2.4, I 5 Perform self-attention and cross-attention operations and add position encoding. The specific expression is as follows: I 6 =CrossAttention(SelfAttention(I 5 +PositionalEncoding)) Among them, I 6 represents image pair features, CrossAttention represents cross attention, SelfAttention represents self attention, PositionalEncoding represents position encoding, I 5 Represents the feature map pair after fusion of multiple scales, and the feature map and Divided into K local semantic category vectors, 2 feature maps with K local semantic features are obtained. The specific expression is as follows: in, and Represents the image pair features I 6 The query image and reference image inside, and Respectively indicate the corresponding and The local semantic feature map of the ExtractionModule represents the operation of extracting local semantic features from the feature map; S3.

2. Calculate the similarity between each cell of the local semantic feature map and its corresponding one-eighth feature map, and divide each cell of the one-eighth feature map into K categories according to the similarity. Then perform self-attention on each category for feature enhancement. The specific calculation expression is as follows: in, and It represents K groups of features that classify the feature map query and feature map refer. and Respectively represent taking out the i-th group of features from the K groups of features, and A represents the features after self-attention of the i-th group of features. query and A refer It is the feature map after self-attention in all categories. Similarity means calculating similarity. S3.

3. Local semantic feature map and A query and A refer The feature map performs cross attention operation, the specific expression is as follows: Among them, O query and O refer It represents the feature map after cross attention, Q, K, V represent query, key and value respectively; S3.

4. After the cross attention feature map O query and O refer Perform the reverse operation, the specific expression is as follows: Among them, CrossAttention means cross attention, O' query and O' refer Indicates the feature map O after the cross attention query and O refer Feature map after reverse operation; S3.

5. Use cross attention to feature map O' query and O' refer Processing, and finally matching the feature map I used Final , the specific expression is as follows: I Final =Self&Cross(O' query ,O' refer ) Among them, Self represents the self-attention operation, Cross represents the cross attention, O' query and O' refer Represents the feature map pair to be processed, I Final Indicates the feature map pair that needs to be matched in the end.

4. The method for multimodal image feature matching based on learnable local semantic enhancement according to claim 2, Features: The specific process of step S4 is as follows: S4.

1. Coarse level matching: I Final The MNN algorithm is used for matching to obtain coarse matching results; S4.

2. Position the coarse-level matching result to a feature map of half scale The heat map corresponding to each matching point is calculated through self-attention and cross-attention operations, and the corresponding optimization points of the matching points are obtained according to the heat map. All optimization points constitute the optimized image.

5. A device for multimodal image feature matching based on learnable local semantic enhancement, based on the method and device for multimodal image feature matching based on learnable local semantic enhancement as claimed in any one of claims 1 to 4, Features: It includes a data enhancement module (10), a feature extraction module (20), a local semantic feature enhancement module (30) and a matching module (40); The data enhancement module (10) is used to enhance the image, the feature extraction module (20) is used to extract features from the image, the local semantic feature enhancement module (30) is used to enhance the local semantic information of the image, and the matching module (40) is used to perform coarse-level matching and refined matching on the image to obtain a final optimized image.

Images