Multi-modal image feature matching method and device

Abstract

The invention discloses a multi-modal image feature matching method and device, which are designed for the problem of difficult image matching in a complex scene, perform information extraction on an input image to obtain image features, and send a feature map into a rapid initialization module to generate information interaction between image pairs so as to establish effective association; the method comprises the following steps: simulating the basic logic of human observation images, constructing a mixed attention module, carrying out data enhancement on input images, purifying key points obtained after first rough matching by adopting a progressive matching strategy, carrying out information fusion and enhancement again, carrying out rough matching again, and then carrying out fine matching. Therefore, efficient and accurate image matching is realized. According to the method, feasibility verification is carried out on massive traditional images, then the traditional images are configured to multi-modal image data for training, and fine adjustment is carried out according to a test result, so that a reliable image matching result is efficiently generated under a difficult condition.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and device for multimodal image feature matching. Background Art

[0002] In recent years, many detector-free image matching frameworks have emerged, and all of them have shown good performance. The poor performance of previous matching methods in complex scenes such as object scale changes, poor textures, repeated patterns, viewpoint changes, lighting changes, and motion blur has been taken into consideration in the design of architectures and methods. Some methods and techniques have been used to effectively improve the performance of image matching methods in complex scenes. However, these methods often have the defect of sacrificing one thing for another. For example, improving efficiency will lead to a decrease in accuracy, and improving accuracy will come at the cost of high computational costs. This is a common problem with current detector-free methods.

[0003] Traditional image matching methods based on feature detectors cannot extract enough key points in complex scenes, especially in indoor environments, due to too little texture, repeated image patterns, lighting changes, etc., and even if no repeated key points are collected, it is difficult to find the correct correspondence in the image pair even if a perfect descriptor is obtained; the dense features extracted by the convolutional neural networks (CNN) used in some recently proposed detector-free methods can only be in a limited receptive field and cannot handle areas with less texture (blurred); key point matching-based schemes often fail when matching in low-texture images; the efficiency of current dense matching methods is limited by the number of matches, but the use of simplified linear attention will lead to incorrect attention diffusion areas. Summary of the invention

[0004] In view of the above problems, the present invention provides a method and device for multimodal image feature matching to solve the problem that the current feature matching of multimodal images requires a large amount of training resources, and improves the matching accuracy and efficiency.

[0005] To achieve the above object, the technical solution of the present invention is as follows:

[0006] A method for multimodal image feature matching, the specific steps are as follows:

[0007] S1, pass the multimodal image data through the feature encoder to generate two images: a coarse feature map with a resolution of 1 / 8 and a fine feature map with a resolution of 1 / 2;

[0008] The feature encoder is a convolutional neural network (CNN) with a standard convolutional architecture of feature pyramid (FPN). The convolutional neural network (CNN) has an inductive bias of translation equivariance and locality, which is suitable for local feature extraction. The downsampling introduced by the convolutional neural network (CNN) also reduces the input length, which is crucial to ensure a manageable computational cost.

[0009] S2, downsampling the 1 / 8 resolution coarse feature map to a 1 / 32 resolution feature map, and performing a position encoding operation on the 1 / 32 resolution feature map and restoring it to 1 / 8 resolution, specifically comprising the following steps:

[0010] S2.1, downsample the 1 / 8 resolution coarse feature map to 1 / 32 resolution, and add position encoding to the 1 / 32 resolution feature map;

[0011] Position encoding follows the same approach in the local image feature matching method (LoFTR), where 2D sinusoidal signals of different frequencies are used to encode position information and added to the initial features;

[0012] S2.2, perform a cross-attention operation on the 1 / 32 resolution feature map with position encoding to extract potential information in the image and perform information enhancement;

[0013] S2.3, upsampling the enhanced feature map to restore the resolution to 1 / 8;

[0014] S3, perform a mixed attention operation on the feature map restored to 1 / 8 resolution, specifically including the following steps:

[0015] S3.1. In the first stage, a mixed attention operation of the SSC type is performed, and the stacking order of the operations is self-attention-self-attention-cross attention.

[0016] S3.2, in the second stage, the SSC type mixed attention operation is performed on the SCC type mixed attention operation, and the stacking order of the SCC type mixed attention operation is self-attention-cross attention-cross attention;

[0017] The mixed stacking of self-attention and cross-attention fully considers the attention to the image information at different scales and the order of information interaction between image pairs. This design ensures that each image can fully collect its own image information while ensuring that the key information of another image can be fully obtained. In addition, the setting of residual links ensures that the information of the initial feature map will not be lost;

[0018] In order to maintain the coarse-level feature map and initial information of 1 / 8 resolution, a residual connection is set to retain it in the feature map after the mixed attention operation;

[0019] S4, using a progressive image matching strategy, matching the 1 / 8 resolution coarse feature map and the 1 / 8 resolution feature map after the mixed attention operation, and purifying the matching points to obtain key points, and obtaining the matching results through the key points, specifically including the following steps:

[0020] S4.1, the first match is performed between the coarse feature map of 1 / 8 resolution and the 1 / 8 resolution after the mixed attention operation, which is a coarse match;

[0021] S4.2. The matching points obtained after rough matching will include positive samples and negative samples. Positive samples represent matching points that are more consistent with the model parameters, while negative samples represent matching points that do not meet the model parameters. The key points are obtained by filtering out negative samples for purification.

[0022] S4.3, based on the key points obtained after the purification operation, perform sparse self-attention operation and dense cross-attention operation;

[0023] S4.4, match the key points after the sparse self-attention operation and the dense cross-attention operation again, remove the negative samples, and obtain the matching results;

[0024] The sparse self-attention is to perform self-attention on the purified key points on one of the images in the to-be-matched image pair after the aforementioned series of operations. Since the number of key points on the purified image is obviously less than the number of pixels on the original 1 / 8 resolution feature map, such sparse key points make it possible to use the standard Transformer to generate the query information vector without worrying about the huge amount of calculation;

[0025] The focused cross-attention is to further extract information from the coarse features using cross-attention after key point purification, and to associate the image pairs. For each key point in the coarse features of the image after key point purification in the image pair to be matched, the corresponding position is located in the other image of the image pair to be matched, and then a specific size of the image is cropped as a judgment of the degree of association;

[0026] S5. According to the matching results, a local window of the same size is cut from the coarse feature map of 1 / 8 resolution and the fine feature map of 1 / 2 resolution, and the matching results are refined to obtain the final fine-level matching results. The specific steps are as follows:

[0027] S5.1, obtain the matching result from the coarse-level feature map with a resolution of 1 / 8, and locate the position of the key point matching result on the fine-level feature map with a resolution of 1 / 2;

[0028] S5.2, cropping two sets of fixed-size local windows on the coarse-level feature map with 1 / 8 resolution and the fine-level feature map with 1 / 2 resolution;

[0029] S5.3, transforming the cropped features within the window to obtain two transformed local feature maps;

[0030] S5.4, performing a mixed attention operation on the two transformed local feature maps, performing vector update, refining the matching results, and obtaining the final fine-level matching results;

[0031] The refinement is to transform the cropped features in the window to obtain two transformed local feature maps, and then associate the center point of one of the image pairs to be matched with all the points of the other image in the image pair to be matched, thereby generating a heat map, which represents the matching probability. Based on the matching probability, the expectation on the probability distribution is calculated to obtain the final position, and all the matches are collected to generate the final fine-level matching. The specific steps are as follows:

[0032] S5.4.1. The local feature map obtained by cropping the coarse-level feature map with a resolution of 1 / 8 through a local window of a fixed size and the local feature map obtained by cropping the fine-level feature map with a resolution of 1 / 2 through a local window of a fixed size are taken as a set of image pairs, and the local feature map obtained by cropping the coarse-level feature map with a resolution of 1 / 8 through a local window of a fixed size is recorded as Figure A, and the local feature map obtained by cropping the fine-level feature map with a resolution of 1 / 2 through a local window of a fixed size is recorded as Figure B;

[0033] S5.4.2. Correlate the center point of graph A with all the points in graph B to generate a heat map, wherein the heat map represents the matching probability;

[0034] S5.4.3. Calculate the expectation on the probability distribution based on the matching probability, collect the expectations generated by all the matches to obtain the final fine-level match.

[0035] A multimodal image feature matching device, comprising: an image preprocessing module, a feature extraction module, a matching iteration module, and a result prediction module;

[0036] The image pair first enters the image preprocessing module to extract the potential information in the image; then the feature map will enter the feature extraction module to further extract the potential information in the image; then the image pair will enter the matching iteration module to perform coarse matching, matching point screening and obtain updated matching results through coarse matching again; finally, it enters the result prediction module, in which the final result of the coarse matching is operated on the fine-grained image to obtain the final fine-level matching result.

[0037] The image preprocessing module extracts features from the input image pairs through a feature extractor (CNN) to obtain feature maps of two different scales, 1 / 2 and 1 / 8, and then performs a fast initialization operation on the 1 / 8 feature map so that the feature map of this scale can contain more information and can also generate associations between image pairs due to the cross attention operation, and then sends them to the feature extraction module for processing;

[0038] The feature extraction module further extracts the potential information in the feature map through a specially designed hybrid attention. In the first stage, the SSC type hybrid attention operation of self-attention-self-attention-cross attention is used to extract features. In the second stage, the SCC type hybrid attention of self-attention-cross attention-cross attention is used to extract features. Through the residual connection, the final feature map also contains the original information, and then it is sent to the matching iteration module for matching operation.

[0039] After obtaining the feature map containing rich information, the matching iteration module screens the key points based on the results obtained from the initial rough matching, and then performs a second rough matching and adjusts and extracts information through sparse self-attention and dense cross-attention, so as to provide reliable key points for the prediction of the final matching result in the next stage and improve efficiency;

[0040] The result prediction module obtains key points through progressive matching and key point screening in the previous stage, and starts positioning on 1 / 2 of the fine-grained feature map based on these key points and performs the final result prediction.

[0041] Beneficial effects of the present invention

[0042] (1) In traditional methods, the feature map is usually not initialized or otherwise operated after it is obtained by the feature extractor. In the present invention, a Quick-Link initialization module is used to allow the feature maps extracted by CNN to be quickly associated with each other.

[0043] (2) The settings for self-attention and cross-attention in traditional methods are basically the same, that is, a cross-attention operation is performed after a self-attention operation. In the present invention, by simulating the behavior of humans observing image pairs from shallow to deep, an information enhancement and information interaction module is designed. The hybrid attention module composed of carefully designed self-attention and cross-attention stacks enhances the potential information of the image and generates interaction between image pairs, achieving good multi-scale information enhancement and information interaction, and finally obtaining image features containing rich image information.

[0044] (3) In the traditional method, the rough matching results are no longer screened or other operations are performed to improve the reliability of the matching results. In the present invention, the matching points obtained by the rough matching are further purified to obtain reliable key points through the key point refinement operation, and the sparse self-attention and focused standard cross-attention operations can be performed on the sparse key points. While providing reliable key points for the final fine matching and enhancing the feature map, the efficiency and accuracy can also be effectively improved.

[0045] (4) A new detector-free image matching method is proposed, and the final effect of homography estimation exceeds 90%, far exceeding other traditional methods. Compared with traditional methods, the efficiency and accuracy are improved.

[0046] The present invention trains on massive image data and performs fine-tuning on specific tasks. Efficient and accurate image matching can be achieved under difficult conditions without the need for large-scale training. The problem that feature matching of multimodal images currently requires a large amount of training resources is solved, and the accuracy and efficiency of matching are improved. The present invention changes the traditional method's single operation on the initial feature map, so that the initial feature map can contain more rich information, changes the traditional attention operation setting method, makes it closer to human behavior norms, and further improves the efficiency and accuracy of image matching through key point purification operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a flow chart of the present invention;

[0048] Figure 2 It is a schematic diagram of the overall framework of detector-free image matching of the present invention;

[0049] Figure 3 This is a schematic diagram of a quick connection initialization module of the present invention;

[0050] Figure 4 Schematic diagram of the hybrid attention module of the present invention;

[0051] Figure 5 It is a schematic diagram of a progressive matching module based on key point purification of the present invention;

[0052] Figure 6 It is a schematic diagram of purification of the key points of the present invention;

[0053] Figure 7 A schematic diagram of regional search for key point purification of the present invention;

[0054] Figure 8 Schematic diagram of sparse self-attention and focused cross-attention of the present invention;

[0055] Fig. 9It is a structural block diagram of the multimodal image feature matching device of the present invention. DETAILED DESCRIPTION

[0056] The present invention is further described in detail below in conjunction with specific embodiments.

[0057] Example 1

[0058] A method for multimodal image feature matching, the specific steps are as follows:

[0059] like Figure 1 , Figure 2 As shown, S1, the multimodal image data is passed through a feature encoder to generate two images, a coarse feature map with a resolution of 1 / 8 and a fine feature map with a resolution of 1 / 2;

[0060] The feature encoder is a convolutional neural network (CNN) with a standard convolutional architecture of feature pyramid (FPN). The convolutional neural network (CNN) has an inductive bias of translation equivariance and locality, which is suitable for local feature extraction. The downsampling introduced by the convolutional neural network (CNN) also reduces the input length, which is crucial to ensure a manageable computational cost.

[0061] S2, downsampling the 1 / 8 resolution coarse feature map to a 1 / 32 resolution feature map, and performing a position encoding operation on the 1 / 32 resolution feature map and restoring it to 1 / 8 resolution, specifically comprising the following steps:

[0062] S2.1, such as Figure 3 As shown, the coarse feature map with 1 / 8 resolution is downsampled to 1 / 32 resolution, and position encoding is added to the 1 / 32 resolution feature map;

[0063] Position encoding follows the same approach in the local image feature matching method (LoFTR), where 2D sinusoidal signals of different frequencies are used to encode position information and added to the initial features;

[0064] S2.2, perform a cross-attention operation on the 1 / 32 resolution feature map with position encoding to extract potential information in the image and perform information enhancement;

[0065] S2.3, upsampling the enhanced feature map to restore the resolution to 1 / 8;

[0066] S3, such as Figure 4 As shown, the feature map restored to 1 / 8 resolution is subjected to a mixed attention operation, which specifically includes the following steps:

[0067] S3.1. In the first stage, a mixed attention operation of the SSC type is performed, and the stacking order of the operations is self-attention-self-attention-cross attention.

[0068] S3.2, in the second stage, the SSC type mixed attention operation is performed on the SCC type mixed attention operation, and the stacking order of the SCC type mixed attention operation is self-attention-cross attention-cross attention;

[0069] The mixed stacking of self-attention and cross-attention fully considers the attention to the image information at different scales and the order of information interaction between image pairs. This design ensures that each image can fully collect its own image information while ensuring that the key information of another image can be fully obtained. In addition, the setting of residual links ensures that the information of the initial feature map will not be lost;

[0070] In order to maintain the coarse-level feature map and initial information of 1 / 8 resolution, a residual connection is set to retain it in the feature map after the mixed attention operation;

[0071] S4, such as Figure 5 , Figure 6 , Figure 7 As shown, a progressive image matching strategy is used to match the coarse feature map of 1 / 8 resolution and the feature map of 1 / 8 resolution after the mixed attention operation, and the matching points are purified to obtain key points, and the matching results are obtained through the key points, which specifically includes the following steps:

[0072] S4.1, the first match is performed between the coarse feature map of 1 / 8 resolution and the 1 / 8 resolution after the mixed attention operation, which is a coarse match;

[0073] S4.2. The matching points obtained after rough matching will include positive samples and negative samples. Positive samples represent matching points that are more consistent with the model parameters, while negative samples represent matching points that do not meet the model parameters. The key points are obtained by filtering out negative samples for purification.

[0074] S4.3, if Figure 8 As shown, based on the key points obtained after the purification operation, sparse self-attention operation and dense cross-attention operation are performed;

[0075] S4.4, match the key points after the sparse self-attention operation and the dense cross-attention operation again, remove the negative samples, and obtain the matching results;

[0076] The sparse self-attention is to perform self-attention on the purified key points on one of the images in the to-be-matched image pair after the aforementioned series of operations. Since the number of key points on the purified image is obviously less than the number of pixels on the original 1 / 8 resolution feature map, such sparse key points make it possible to use the standard Transformer to generate the query information vector without worrying about the huge amount of calculation;

[0077] The focused cross-attention is to further extract information from the coarse features using cross-attention after key point purification, and to associate the image pairs. For each key point in the coarse features of the image after key point purification in the image pair to be matched, the corresponding position is located in the other image of the image pair to be matched, and then a specific size of the image is cropped as a judgment of the degree of association;

[0078] S5. According to the matching results, a local window of the same size is cut from the coarse feature map of 1 / 8 resolution and the fine feature map of 1 / 2 resolution, and the matching results are refined to obtain the final fine-level matching results. The specific steps are as follows:

[0079] S5.1, obtain the matching result from the coarse-level feature map with a resolution of 1 / 8, and locate the position of the key point matching result on the fine-level feature map with a resolution of 1 / 2;

[0080] S5.2, cropping two sets of fixed-size local windows on the coarse-level feature map with 1 / 8 resolution and the fine-level feature map with 1 / 2 resolution;

[0081] S5.3, transforming the cropped features within the window to obtain two transformed local feature maps;

[0082] S5.4, performing a mixed attention operation on the two transformed local feature maps, performing vector update, refining the matching results, and obtaining the final fine-level matching results;

[0083] The refinement is to transform the cropped features in the window to obtain two transformed local feature maps, and then associate the center point of one of the image pairs to be matched with all the points of the other image in the image pair to be matched, thereby generating a heat map, which represents the matching probability. Based on the matching probability, the expectation on the probability distribution is calculated to obtain the final position, and all the matches are collected to generate the final fine-level matching. The specific steps are as follows:

[0084] S5.4.1. The local feature map obtained by cropping the coarse-level feature map with a resolution of 1 / 8 through a local window of a fixed size and the local feature map obtained by cropping the fine-level feature map with a resolution of 1 / 2 through a local window of a fixed size are taken as a set of image pairs, and the local feature map obtained by cropping the coarse-level feature map with a resolution of 1 / 8 through a local window of a fixed size is recorded as Figure A, and the local feature map obtained by cropping the fine-level feature map with a resolution of 1 / 2 through a local window of a fixed size is recorded as Figure B;

[0085] S5.4.2. Correlate the center point of graph A with all the points in graph B to generate a heat map, wherein the heat map represents the matching probability;

[0086] S5.4.3. Calculate the expectation on the probability distribution based on the matching probability, collect the expectations generated by all the matches to obtain the final fine-level match.

[0087] like Fig. 9 As shown, a multimodal image feature matching device includes: an image preprocessing module, a feature extraction module, a matching iteration module, and a result prediction module;

[0088] The image pair first enters the image preprocessing module to extract the potential information in the image; then the feature map will enter the feature extraction module to further extract the potential information in the image; then the image pair will enter the matching iteration module to perform coarse matching, matching point screening and obtain updated matching results through coarse matching again; finally, it enters the result prediction module, in which the final result of the coarse matching is operated on the fine-grained image to obtain the final fine-level matching result.

[0089] The image preprocessing module extracts features from the input image pairs through a feature extractor (CNN) to obtain feature maps of two different scales, 1 / 2 and 1 / 8, and then performs a fast initialization operation on the 1 / 8 feature map so that the feature map of this scale can contain more information and can also generate associations between image pairs due to the cross attention operation, and then sends them to the feature extraction module for processing;

[0090] The feature extraction module further extracts the potential information in the feature map through a specially designed hybrid attention. In the first stage, the SSC type hybrid attention operation of self-attention-self-attention-cross attention is used to extract features. In the second stage, the SCC type hybrid attention of self-attention-cross attention-cross attention is used to extract features. Through the residual connection, the final feature map also contains the original information, and then it is sent to the matching iteration module for matching operation.

[0091] After obtaining the feature map containing rich information, the matching iteration module screens the key points based on the results obtained from the initial rough matching, and then performs a second rough matching and adjusts and extracts information through sparse self-attention and dense cross-attention, so as to provide reliable key points for the prediction of the final matching result in the next stage and improve efficiency;

[0092] The result prediction module obtains key points through progressive matching and key point screening in the previous stage, and starts positioning on 1 / 2 of the fine-grained feature map based on these key points and performs the final result prediction.

[0093] In order to more clearly express the improvement in image feature matching capability and accuracy of the present application, the following provides a comparative experiment with HardNet, MatchosNet, TFeat, MatchNet, LoFTR and FeMIP under the same other objective environments, so as to verify the matching effect of the method designed by the present application;

[0094] The evaluation metric used in the present invention is homography estimation, where a reference image is paired with another image in each test sequence. The correctness identifier is calculated based on the corner error between the estimated test result and the true label. The application calculates the average reprojection error of the four corners of the image and reports the area under the cumulative curve where the corner error reaches different thresholds (@3px, @5px and @10px). In the evaluation, the higher the value of the homography estimate, the better the feature matching effect of the method.

[0095] The image feature matching method of the present invention is compared with HardNet, MatchosNet, TFeat, MatchNet, LoFTR and FeMIP. These methods are the existing technologies and have very good feature matching capabilities. Using these methods as comparison methods can more objectively prove the image feature matching capabilities and effects of the present application. The experimental results are shown in Table 1:

[0096] Table 1. Matching results of different methods on infrared image and optical image matching datasets

[0097]

[0098] Table 2 compares the matching effects of different methods under the dataset of SAR image and optical image matching. The results are as follows:

[0099] Table 2 Comparison of matching effects of different methods

[0100]

[0101] It can be seen from the experimental results in the above two tables that on different multimodal data sets, the homography estimation of the present application is superior to other methods to a certain extent, proving that the present application has good accuracy and generalization when processing multimodal data.

Claims

1. A method for multimodal image feature matching, characterized in that: The following steps are involved: S1, pass the multimodal image data through the feature encoder to generate two images: a coarse feature map with a resolution of 1 / 8 and a fine feature map with a resolution of 1 / 2; S2, down-sample the 1 / 8 resolution coarse feature map to a 1 / 32 resolution feature map, and perform position encoding on the 1 / 32 resolution feature map before restoring it to 1 / 8 resolution; S3, perform mixed attention operation on the feature map restored to 1 / 8 resolution; S4, using a progressive image matching strategy, matching the 1 / 8 resolution coarse-level feature map with the 1 / 8 resolution feature map after the mixed attention operation, and purifying the matching points to obtain key points, and obtaining the matching results through the key points; S5. According to the matching results, a local window of the same size is cropped from the coarse-level feature map with a resolution of 1 / 8 and the fine-level feature map with a resolution of 1 / 2, and the matching results are refined to obtain the final fine-level matching results.

2. The method for multimodal image feature matching according to claim 1, characterized in that: The specific steps of downsampling the 1 / 8 resolution coarse feature map to a 1 / 32 resolution feature map and restoring the 1 / 8 resolution after performing a position encoding operation on the 1 / 32 resolution feature map are as follows: S2.1, downsample the 1 / 8 resolution coarse feature map to 1 / 32 resolution, and add position encoding to the 1 / 32 resolution feature map; S2.2, perform a cross-attention operation on the 1 / 32 resolution feature map with position encoding to extract potential information in the image and perform information enhancement; S2.

3. Upsample the enhanced feature map to restore the resolution to 1 / 8.

3. The method for multimodal image feature matching according to claim 1, characterized in that: The mixed attention operation includes the following steps: S3.

1. In the first stage, a mixed attention operation of the SSC type is performed, and the stacking order of the operations is self-attention-self-attention-cross attention. S3.

2. In the second stage, the SSC type mixed attention operation is performed as the SCC type mixed attention operation. The stacking order of the SCC type mixed attention operation is self-attention-cross attention-cross attention.

4. The method for multimodal image feature matching according to claim 1, characterized in that: The specific steps of using a progressive image matching strategy to match a 1 / 8 resolution coarse feature map and a 1 / 8 resolution feature map after a mixed attention operation, purifying the matching points to obtain key points, and obtaining the matching results through the key points are as follows: S4.1, the first match is performed between the coarse feature map of 1 / 8 resolution and the 1 / 8 resolution after the mixed attention operation, which is a coarse match; S4.

2. The matching points obtained after rough matching will include positive samples and negative samples. Positive samples represent matching points that are more consistent with the model parameters, while negative samples represent matching points that do not meet the model parameters. The key points are obtained by filtering out negative samples for purification. S4.3, based on the key points obtained after the purification operation, perform sparse self-attention operation and dense cross-attention operation; S4.

4. Match the key points after sparse self-attention operation and dense cross-attention operation again, remove negative samples, and obtain the matching results.

5. The method for multimodal image feature matching according to claim 1, characterized in that: The specific steps of cutting out local windows of the same size from the coarse feature map of 1 / 8 resolution and the fine feature map of 1 / 2 resolution according to the matching results, refining the matching results, and obtaining the final matching results are as follows: S5.1, obtain the matching result from the coarse-level feature map with a resolution of 1 / 8, and locate the position of the key point matching result on the fine-level feature map with a resolution of 1 / 2; S5.2, cropping two sets of fixed-size local windows on the coarse-level feature map with 1 / 8 resolution and the fine-level feature map with 1 / 2 resolution; S5.3, transforming the cropped features within the window to obtain two transformed local feature maps; S5.

4. Perform a mixed attention operation on the two transformed local feature maps, perform vector update, refine the matching results, and obtain the final fine-level matching results.

6. The method for multimodal image feature matching according to claim 5, characterized in that: The specific steps of refining the matching results to obtain the final fine-level matching results are as follows: S5.4.

1. The local feature map obtained by cropping the coarse-level feature map with a resolution of 1 / 8 through a local window of a fixed size and the local feature map obtained by cropping the fine-level feature map with a resolution of 1 / 2 through a local window of a fixed size are taken as a set of image pairs, and the local feature map obtained by cropping the coarse-level feature map with a resolution of 1 / 8 through a local window of a fixed size is recorded as Figure A, and the local feature map obtained by cropping the fine-level feature map with a resolution of 1 / 2 through a local window of a fixed size is recorded as Figure B; S5.4.

2. Correlate the center point of graph A with all the points in graph B to generate a heat map, wherein the heat map represents the matching probability; S5.4.

3. Calculate the expectation on the probability distribution based on the matching probability, collect the expectations generated by all the matches to obtain the final fine-level match.

7. A device for multimodal image feature matching, comprising: Image preprocessing module, feature extraction module, matching iteration module, result prediction module; The image pair first enters the image preprocessing module to extract the potential information in the image; then the feature map will enter the feature extraction module to further extract the potential information in the image; then the image pair will enter the matching iteration module to perform coarse matching, matching point screening and obtain updated matching results through coarse matching again; finally, it enters the result prediction module, in which the final result of the coarse matching is operated on the fine-grained image to obtain the final fine-level matching result.

8. The multimodal image feature matching device according to claim 7, characterized in that: The image preprocessing module includes: a feature encoder and a fast initialization module, which extracts features from the input image to obtain two feature maps of different scales, namely a coarse feature map with a resolution of 1 / 8 and a fine feature map with a resolution of 1 / 2; The feature extraction module performs a self-attention-self-attention-cross attention SSC type mixed attention operation and a self-attention-cross attention-cross attention SCC type mixed attention operation, and through residual connection, the feature map after the mixed attention operation still contains the original information; The matching iteration module screens key points based on the results of the initial rough matching, and then performs a second rough matching and performs adjustment and information extraction through sparse self-attention and dense cross-attention; The result prediction module locates the key points on a fine-level feature map with a resolution of 1 / 2 and performs the final result prediction.

Images