A method and apparatus for multi-modal image feature matching
By employing a multimodal image feature matching method, which utilizes feature encoders and hybrid attention operations, the problems of high computational cost and low accuracy in detectorless image matching are solved, achieving efficient and accurate image matching results and significantly improving homography estimation performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- YUNNAN UNIV
- Filing Date
- 2024-08-22
- Publication Date
- 2026-08-04
AI Technical Summary
Existing detectorless image matching methods suffer from high computational cost and low accuracy in complex scenes. Furthermore, traditional methods perform poorly in low-texture images, and their efficiency is limited by the number of matches.
A multimodal image feature matching method is adopted, which generates feature maps of different resolutions through feature encoder, and performs progressive image matching by combining position encoding, hybrid attention and key point purification operations, including self-attention and cross-attention stacking, sparse self-attention and dense cross-attention operations, to improve the efficiency of image feature extraction and matching.
It improves the accuracy and efficiency of image matching, with homography estimation exceeding 90%, achieving efficient and accurate image matching under challenging conditions and reducing training resource requirements.
Smart Images
Figure CN119131424B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and more specifically to a method and apparatus for multimodal image feature matching. Background Technology
[0002] In recent years, many detector-free image matching frameworks have emerged, all demonstrating good performance. Their architecture and method design have taken into account the poor performance of previous matching methods in complex scenes such as object scale variations, poor textures, repetitive patterns, viewpoint changes, lighting variations, and motion blur. Furthermore, they have effectively improved the performance of image matching methods in complex scenes through various methods and techniques. However, these methods often have trade-offs; for example, improving efficiency may lead to a decrease in accuracy, while improving accuracy may come at the cost of high computational costs. This is a common problem with current detector-free methods.
[0003] Traditional image matching methods based on feature detectors often struggle to extract sufficient keypoints in complex scenes, especially indoor environments, due to insufficient texture, repetitive image patterns, and varying lighting. Even when perfect descriptors are obtained without capturing duplicate keypoints, it is difficult to find the correct correspondence between image pairs. Recently proposed detector-free methods use convolutional neural networks (CNNs) to extract dense features, but these features are limited to a finite receptive field and cannot handle areas with little or no texture. Keypoint-based matching schemes often fail when matching in low-texture images. The efficiency of current dense matching methods is limited by the number of matches, but using simplified linear attention can lead to incorrect attention diffusion areas. Summary of the Invention
[0004] To address the aforementioned problems, this invention provides a method and apparatus for multimodal image feature matching, thereby solving the problem that current multimodal image feature matching requires a large amount of training resources and improving the accuracy and efficiency of matching.
[0005] To achieve the above objectives, the technical solution of the present invention is as follows:
[0006] A method for multimodal image feature matching, the specific steps of which are as follows:
[0007] S1. The multimodal image data is processed by a feature encoder to generate two images: a coarse feature map with a resolution of 1 / 8 and a fine feature map with a resolution of 1 / 2.
[0008] The feature encoder is a convolutional neural network (CNN) with a standard convolutional architecture featuring a feature pyramid (FPN). CNNs have inductive biases with translational equivariance and locality, making them suitable for local feature extraction. The downsampling introduced by CNNs also reduces the input length, which is crucial for ensuring manageable computational costs.
[0009] S2. Downsample the coarse feature map at 1 / 8 resolution to a feature map at 1 / 32 resolution, and then restore it to 1 / 8 resolution after performing position encoding on the 1 / 32 resolution feature map. This specifically includes the following steps:
[0010] S2.1 Downsample the coarse feature map at 1 / 8 resolution to 1 / 32 resolution, and add position encoding to the 1 / 32 resolution feature map;
[0011] Location encoding follows the same approach as in the Local Image Feature Matching (LoFTR) method, where 2D sinusoidal signals of different frequencies are used to encode location information and are added to the initial features;
[0012] S2.2. Perform cross-attention operation on the 1 / 32 resolution feature map with added position encoding to extract latent information in the image and perform information enhancement.
[0013] S2.3. Upsample the enhanced feature map to restore it to 1 / 8 resolution;
[0014] S3. Perform a hybrid attention operation on the feature map restored to 1 / 8 resolution, specifically including the following steps:
[0015] S3.1 The first stage performs SSC-type hybrid attention operations, which are stacked in the order of self-attention - self-attention - cross-attention.
[0016] S3.2 In the second stage, the SSC type hybrid attention operation is transformed into an SCC type hybrid attention operation. The stacking order of the SCC type hybrid attention operation is self-attention - cross attention - cross attention.
[0017] The hybrid stacking of self-attention and cross-attention fully considers the attention given to information within the image itself at different scales, as well as the order of information interaction between image pairs. This design ensures that each image can fully collect its own image information while also fully acquiring key information from another image. Furthermore, the residual linking ensures that information from the initial feature maps is not lost.
[0018] Furthermore, in order to preserve the coarse feature map and initial information at 1 / 8 resolution, residual connections were also set up to retain it in the feature map after the hybrid attention operation;
[0019] S4. Using a progressive image matching strategy, match the coarse feature map at 1 / 8 resolution with the feature map at 1 / 8 resolution after hybrid attention operation, refine the matching points to obtain key points, and obtain the matching result through the key points. Specifically, this includes the following steps:
[0020] S4.1 Perform the first matching between the coarse feature map at 1 / 8 resolution and the 1 / 8 resolution map after the hybrid attention operation; this is a coarse matching.
[0021] S4.2 The matching points obtained after coarse matching will include positive samples and negative samples. Positive samples represent matching points that are more in line with the model parameters, while negative samples are matching points that are not in line with the model parameters. The key points are obtained by filtering out negative samples.
[0022] S4.3. Based on the key points obtained after purification, perform sparse self-attention operation and dense cross-attention operation.
[0023] S4.4. Perform matching again on the keypoints after sparse self-attention operation and dense cross-attention operation, and remove negative samples to obtain the matching results.
[0024] The sparse self-attention is to perform self-attention on the purified key points in one of the aforementioned series of operations in the image pair to be matched. Because the number of key points in the purified image is obviously less than the number of pixels in the original 1 / 8 resolution feature map, such sparse key points allow us to use the standard Transformer to generate query information vectors without worrying about huge computational costs.
[0025] The focused cross-attention technique further extracts information from coarse features after keypoint refinement and establishes connections between image pairs. For each keypoint in the coarse features of the image after keypoint refinement in the image pair to be matched, it is located in the other image of the image pair to be matched. Subsequently, a specific size of the image is cropped as a measure of the strength of the connection.
[0026] S5. Based on the matching results, crop local windows of the same size from the coarse feature map at 1 / 8 resolution and the fine feature map at 1 / 2 resolution to refine the matching results and obtain the final fine-level matching results. The specific steps are as follows:
[0027] S5.1 Obtain the matching result from the coarse feature map with a resolution of 1 / 8, and locate the position of the key point matching result on the fine feature map with a resolution of 1 / 2.
[0028] S5.2. Crop two sets of fixed-size local windows on the coarse feature map at 1 / 8 resolution and the fine feature map at 1 / 2 resolution;
[0029] S5.3 Transform the clipping features within the window to obtain two transformed local feature maps;
[0030] S5.4. Perform a hybrid attention operation on the two transformed local feature maps, update the vectors, refine the matching results, and obtain the final fine-level matching results.
[0031] The refinement process involves transforming the cropped features within the window to obtain two transformed local feature maps. Then, the center point of one of the images in the pair to be matched is associated with all points of the other image in the pair, generating a heatmap representing the matching probability. Based on this matching probability, the expected value of the probability distribution is calculated to obtain the final position. All matches are collected to generate the final refined-level match. The specific steps are as follows:
[0032] S5.4.1. Take the local feature map obtained by cropping the coarse feature map of 1 / 8 resolution through a local window of fixed size, and the local feature map obtained by cropping the fine feature map of 1 / 2 resolution through a local window of fixed size as a pair of images. The local feature map obtained by cropping the coarse feature map of 1 / 8 resolution through a local window of fixed size is denoted as Figure A, and the local feature map obtained by cropping the fine feature map of 1 / 2 resolution through a local window of fixed size is denoted as Figure B.
[0033] S5.4.2. Associating the center point of Figure A with all points in Figure B to generate a heatmap, whereby the heatmap represents the matching probability;
[0034] S5.4.3 Calculate the expectation on the probability distribution based on the matching probability, and collect the expectations generated by all the matches to obtain the final fine-level matching.
[0035] A multimodal image feature matching device includes: an image preprocessing module, a feature extraction module, a matching iteration module, and a result prediction module;
[0036] The image pairs first enter the image preprocessing module to extract potential information from the images; then the feature maps enter the feature extraction module to further extract potential information from the images; subsequently, the image pairs enter the matching iteration module to perform coarse matching, matching point filtering, and obtain updated matching results through coarse matching again; finally, they enter the result prediction module, where the final result of coarse matching is processed on the fine-grained image to obtain the final fine-level matching result.
[0037] The image preprocessing module extracts features from the input image pairs using a feature extractor (CNN) to obtain feature maps of two different scales: 1 / 2 and 1 / 8. Then, a fast initialization operation is performed on the 1 / 8 feature map to enable it to contain more information. Furthermore, due to the cross-attention operation, the image pairs can be correlated before being sent to the feature extraction module for processing.
[0038] The feature extraction module further extracts the potential information in the feature map through a specially designed hybrid attention. In the first stage, it uses the SSC type hybrid attention operation, which focuses more on itself (self-attention-self-attention-cross attention), to extract features. In the second stage, it uses the SCC type hybrid attention operation, which focuses more on the relationship between the two sides (self-attention-cross attention-cross attention), to extract features. Furthermore, through residual connections, the final feature map still contains the original information. After that, it is sent to the matching iteration module for matching operations.
[0039] After obtaining a feature map containing rich information, the matching iteration module filters key points based on the results of the initial coarse matching. Then, it performs a second coarse matching and adjusts and extracts information through sparse self-attention and dense cross-attention, providing reliable key points for the prediction of the final matching result in the next stage and improving efficiency.
[0040] The result prediction module obtains key points through the progressive matching and key point filtering in the previous stage. Based on these key points, it begins to locate the key points on the 1 / 2 fine-grained feature map and makes the final result prediction.
[0041] Beneficial effects of the present invention
[0042] (1) Traditional methods typically do not initialize or perform other operations on the feature maps after they are obtained through the feature extractor. In this invention, a Quick-Link initialization module is used to enable rapid association between the feature maps extracted by the CNN.
[0043] (2) In traditional methods, the settings for self-attention and cross-attention are basically the same, that is, a cross-attention operation is performed after a self-attention operation. In this invention, by simulating the human behavior of observing image pairs from shallow to deep, an information enhancement and information interaction module is designed. The hybrid attention module, which is composed of carefully designed self-attention and cross-attention stacks, enhances the potential information of the image and generates interaction between image pairs, thereby achieving good multi-scale information enhancement and information interaction, and finally obtaining image features containing rich image information.
[0044] (3) In traditional methods, coarse matching results are not filtered or subjected to other operations to improve their reliability. In this invention, key-point refinement is used to further refine the matching points obtained from coarse matching into reliable key points. Furthermore, sparse self-attention and standard cross-attention operations are performed on sparse key points. This provides reliable key points for the final fine matching and enhances the feature map while effectively improving efficiency and accuracy.
[0045] (4) A novel detector-free image matching method is proposed, and the final performance in homography estimation exceeds 90%, far surpassing other traditional methods. Compared with traditional methods, it improves efficiency and accuracy.
[0046] This invention trains on massive amounts of image data and fine-tunes it for specific tasks. It achieves efficient and accurate image matching under challenging conditions without requiring large-scale training. It solves the problem of current multimodal image feature matching requiring significant training resources and improves matching accuracy and efficiency. This invention changes the traditional method's single operation on the initial feature map, allowing the initial feature map to contain richer information. It also changes the traditional attention operation settings to be closer to human behavior norms, and further improves the efficiency and accuracy of image matching through keypoint refinement operations. Attached Figure Description
[0047] Figure 1 This is a flowchart of the present invention;
[0048] Figure 2 This is a schematic diagram of the overall framework for detector-free image matching in this invention;
[0049] Figure 3 This is a schematic diagram of the quick connection initialization module of the present invention;
[0050] Figure 4 This is a schematic diagram of the hybrid attention module of the present invention;
[0051] Figure 5 This is a schematic diagram of the progressive matching module based on key point purification according to the present invention;
[0052] Figure 6 This is a schematic diagram illustrating the purification process for key aspects of this invention;
[0053] Figure 7 A schematic diagram illustrating the region search for purification of key points in this invention;
[0054] Figure 8 This is a schematic diagram of sparse self-attention and focused cross-attention in this invention;
[0055] Figure 9This is a structural block diagram of the multimodal image feature matching device of the present invention. Detailed Implementation
[0056] The present invention will be further described in detail below with reference to specific embodiments.
[0057] Example 1
[0058] A method for multimodal image feature matching, the specific steps of which are as follows:
[0059] like Figure 1 , Figure 2 As shown, S1, the multimodal image data is processed by a feature encoder to generate two images: a coarse feature map with a resolution of 1 / 8 and a fine feature map with a resolution of 1 / 2.
[0060] The feature encoder is a convolutional neural network (CNN) with a standard convolutional architecture featuring a feature pyramid (FPN). CNNs have inductive biases with translational equivariance and locality, making them suitable for local feature extraction. The downsampling introduced by CNNs also reduces the input length, which is crucial for ensuring manageable computational costs.
[0061] S2. Downsample the coarse feature map at 1 / 8 resolution to a feature map at 1 / 32 resolution, and then restore it to 1 / 8 resolution after performing position encoding on the 1 / 32 resolution feature map. This specifically includes the following steps:
[0062] S2.1, such as Figure 3 As shown, the coarse feature map at 1 / 8 resolution is downsampled to 1 / 32 resolution, and positional encoding is added to the 1 / 32 resolution feature map;
[0063] Location encoding follows the same approach as in the Local Image Feature Matching (LoFTR) method, where 2D sinusoidal signals of different frequencies are used to encode location information and are added to the initial features;
[0064] S2.2. Perform cross-attention operation on the 1 / 32 resolution feature map with added position encoding to extract latent information in the image and perform information enhancement.
[0065] S2.3. Upsample the enhanced feature map to restore it to 1 / 8 resolution;
[0066] S3, such as Figure 4 As shown, the feature map restored to 1 / 8 resolution is subjected to a hybrid attention operation, which specifically includes the following steps:
[0067] S3.1 The first stage performs SSC-type hybrid attention operations, which are stacked in the order of self-attention - self-attention - cross-attention.
[0068] S3.2 In the second stage, the SSC type hybrid attention operation is transformed into an SCC type hybrid attention operation. The stacking order of the SCC type hybrid attention operation is self-attention - cross attention - cross attention.
[0069] The hybrid stacking of self-attention and cross-attention fully considers the attention given to information within the image itself at different scales, as well as the order of information interaction between image pairs. This design ensures that each image can fully collect its own image information while also fully acquiring key information from another image. Furthermore, the residual linking ensures that information from the initial feature maps is not lost.
[0070] Furthermore, in order to preserve the coarse feature map and initial information at 1 / 8 resolution, residual connections were also set up to retain it in the feature map after the hybrid attention operation;
[0071] S4, such as Figure 5 , Figure 6 , Figure 7 As shown, a progressive image matching strategy is used to match a coarse feature map at 1 / 8 resolution with a feature map at 1 / 8 resolution after a hybrid attention operation. The matched points are then refined to obtain key points, and the matching result is obtained using these key points. The specific steps include:
[0072] S4.1 Perform the first matching between the coarse feature map at 1 / 8 resolution and the 1 / 8 resolution map after the hybrid attention operation; this is a coarse matching.
[0073] S4.2 The matching points obtained after coarse matching will include positive samples and negative samples. Positive samples represent matching points that are more in line with the model parameters, while negative samples are matching points that are not in line with the model parameters. The key points are obtained by filtering out negative samples.
[0074] S4.3, such as Figure 8 As shown, based on the key points obtained after purification, sparse self-attention operation and dense cross-attention operation are performed.
[0075] S4.4. Perform matching again on the keypoints after sparse self-attention operation and dense cross-attention operation, and remove negative samples to obtain the matching results.
[0076] The sparse self-attention is to perform self-attention on the purified key points in one of the aforementioned series of operations in the image pair to be matched. Because the number of key points in the purified image is obviously less than the number of pixels in the original 1 / 8 resolution feature map, such sparse key points allow us to use the standard Transformer to generate query information vectors without worrying about huge computational costs.
[0077] The focused cross-attention technique further extracts information from coarse features after keypoint refinement and establishes connections between image pairs. For each keypoint in the coarse features of the image after keypoint refinement in the image pair to be matched, it is located in the other image of the image pair to be matched. Subsequently, a specific size of the image is cropped as a measure of the strength of the connection.
[0078] S5. Based on the matching results, crop local windows of the same size from the coarse feature map at 1 / 8 resolution and the fine feature map at 1 / 2 resolution to refine the matching results and obtain the final fine-level matching results. The specific steps are as follows:
[0079] S5.1 Obtain the matching result from the coarse feature map with a resolution of 1 / 8, and locate the position of the key point matching result on the fine feature map with a resolution of 1 / 2.
[0080] S5.2. Crop two sets of fixed-size local windows on the coarse feature map at 1 / 8 resolution and the fine feature map at 1 / 2 resolution;
[0081] S5.3 Transform the clipping features within the window to obtain two transformed local feature maps;
[0082] S5.4. Perform a hybrid attention operation on the two transformed local feature maps, update the vectors, refine the matching results, and obtain the final fine-level matching results.
[0083] The refinement process involves transforming the cropped features within the window to obtain two transformed local feature maps. Then, the center point of one of the images in the pair to be matched is associated with all points of the other image in the pair, generating a heatmap representing the matching probability. Based on this matching probability, the expected value of the probability distribution is calculated to obtain the final position. All matches are collected to generate the final refined-level match. The specific steps are as follows:
[0084] S5.4.1. Take the local feature map obtained by cropping the coarse feature map of 1 / 8 resolution through a local window of fixed size, and the local feature map obtained by cropping the fine feature map of 1 / 2 resolution through a local window of fixed size as a pair of images. The local feature map obtained by cropping the coarse feature map of 1 / 8 resolution through a local window of fixed size is denoted as Figure A, and the local feature map obtained by cropping the fine feature map of 1 / 2 resolution through a local window of fixed size is denoted as Figure B.
[0085] S5.4.2. Associating the center point of Figure A with all points in Figure B to generate a heatmap, whereby the heatmap represents the matching probability;
[0086] S5.4.3 Calculate the expectation on the probability distribution based on the matching probability, and collect the expectations generated by all the matches to obtain the final fine-level matching.
[0087] like Figure 9 As shown, a multimodal image feature matching device includes: an image preprocessing module, a feature extraction module, a matching iteration module, and a result prediction module;
[0088] The image pairs first enter the image preprocessing module to extract potential information from the images; then the feature maps enter the feature extraction module to further extract potential information from the images; subsequently, the image pairs enter the matching iteration module to perform coarse matching, matching point filtering, and obtain updated matching results through coarse matching again; finally, they enter the result prediction module, where the final result of coarse matching is processed on the fine-grained image to obtain the final fine-level matching result.
[0089] The image preprocessing module extracts features from the input image pairs using a feature extractor (CNN) to obtain feature maps of two different scales: 1 / 2 and 1 / 8. Then, a fast initialization operation is performed on the 1 / 8 feature map to enable it to contain more information. Furthermore, due to the cross-attention operation, the image pairs can be correlated before being sent to the feature extraction module for processing.
[0090] The feature extraction module further extracts the potential information in the feature map through a specially designed hybrid attention. In the first stage, it uses the SSC type hybrid attention operation, which focuses more on itself (self-attention-self-attention-cross attention), to extract features. In the second stage, it uses the SCC type hybrid attention operation, which focuses more on the relationship between the two sides (self-attention-cross attention-cross attention), to extract features. Furthermore, through residual connections, the final feature map still contains the original information. After that, it is sent to the matching iteration module for matching operations.
[0091] After obtaining a feature map containing rich information, the matching iteration module filters key points based on the results of the initial coarse matching. Then, it performs a second coarse matching and adjusts and extracts information through sparse self-attention and dense cross-attention, providing reliable key points for the prediction of the final matching result in the next stage and improving efficiency.
[0092] The result prediction module obtains key points through the progressive matching and key point filtering in the previous stage. Based on these key points, it begins to locate the key points on the 1 / 2 fine-grained feature map and makes the final result prediction.
[0093] To more clearly demonstrate the improved image feature matching capabilities and accuracy of this application, comparative experiments are provided below with HardNet, MatchosNet, TFeat, MatchNet, LoFTR, and FeMIP under the same objective environment, thereby verifying the matching effect of the method designed in this application.
[0094] The evaluation metric used in this invention is homography estimation, where one reference image is paired with another in each test sequence. The correctness identifier is calculated based on the corner error between the estimated test result and the true label. This application calculates the average reprojection error of the four corners of the image and reports the area under the cumulative curve when the corner error reaches different thresholds (@3px, @5px, and @10px). In the evaluation, a higher homography estimation value indicates better feature matching performance of the method.
[0095] The image feature matching methods compared in this invention include HardNet, MatchosNet, TFeat, MatchNet, LoFTR, and FeMIP. These methods all possess excellent feature matching capabilities compared to existing technologies. Using these methods for comparison provides a more objective demonstration of the image feature matching capabilities and effectiveness of this application. Experimental results are shown in Table 1.
[0096] Table 1 shows the matching results of different methods on a dataset of infrared and optical image matching.
[0097]
[0098] Table 2 compares the matching performance of different methods on a dataset of SAR and optical image matching. The results are as follows:
[0099] Table 2 Comparison of matching results of different methods
[0100]
[0101] The experimental results in the two tables above show that, on different multimodal datasets, the homography estimation of this application is superior to other methods to a certain extent, proving that this application has good accuracy and generalization when processing multimodal data.
Claims
1. A method for multimodal image feature matching, characterized in that: Includes the following steps: S1. The multimodal image data is processed by a feature encoder to generate two images: a coarse feature map with a resolution of 1 / 8 and a fine feature map with a resolution of 1 / 2. S2. Downsample the coarse feature map at 1 / 8 resolution to a feature map at 1 / 32 resolution, and then perform position encoding on the 1 / 32 resolution feature map to restore it to 1 / 8 resolution. The specific steps are as follows: S2.1 Downsample the coarse feature map at 1 / 8 resolution to 1 / 32 resolution, and add position encoding to the 1 / 32 resolution feature map; S2.
2. Perform cross-attention operation on the 1 / 32 resolution feature map with added position encoding to extract latent information in the image and perform information enhancement. S2.
3. Upsample the enhanced feature map to restore it to 1 / 8 resolution; S3. Perform a hybrid attention operation on the feature map restored to 1 / 8 resolution; S4. Using a progressive image matching strategy, the coarse feature map at 1 / 8 resolution and the feature map at 1 / 8 resolution after hybrid attention operation are matched, and the matching points are purified to obtain key points, and the matching results are obtained through the key points. S5. Based on the matching results, crop local windows of the same size from the coarse feature map at 1 / 8 resolution and the fine feature map at 1 / 2 resolution to refine the matching results and obtain the final fine-level matching results. The specific steps are as follows: S5.1 Obtain the matching result from the coarse feature map with a resolution of 1 / 8, and locate the position of the key point matching result on the fine feature map with a resolution of 1 / 2. S5.
2. Crop two sets of fixed-size local windows on the coarse feature map at 1 / 8 resolution and the fine feature map at 1 / 2 resolution; S5.3 Transform the clipping features within the window to obtain two transformed local feature maps; S5.
4. Perform a hybrid attention operation on the two transformed local feature maps, update the vectors, and refine the matching results to obtain the final fine-level matching result. The specific steps are as follows: S5.4.
1. Take the local feature map obtained by cropping the coarse feature map of 1 / 8 resolution through a local window of fixed size, and the local feature map obtained by cropping the fine feature map of 1 / 2 resolution through a local window of fixed size as a pair of images. The local feature map obtained by cropping the coarse feature map of 1 / 8 resolution through a local window of fixed size is denoted as Figure A, and the local feature map obtained by cropping the fine feature map of 1 / 2 resolution through a local window of fixed size is denoted as Figure B. S5.4.
2. Associating the center point of Figure A with all points in Figure B to generate a heatmap, whereby the heatmap represents the matching probability; S5.4.3 Calculate the expectation on the probability distribution based on the matching probability, and collect the expectations generated by all the matches to obtain the final fine-level matching.
2. The method for multimodal image feature matching according to claim 1, characterized in that: The hybrid attention operation includes the following steps: S3.1 The first stage performs SSC-type hybrid attention operations, which are stacked in the order of self-attention - self-attention - cross-attention. S3.
2. In the second stage, the SSC-type hybrid attention operation is transformed into an SCC-type hybrid attention operation. The stacking order of the SCC-type hybrid attention operation is self-attention - cross attention - cross attention.
3. The method for multimodal image feature matching according to claim 1, characterized in that: The specific steps of using a progressive image matching strategy to match a coarse feature map at 1 / 8 resolution with a 1 / 8 resolution feature map after hybrid attention operation, refining the matching points to obtain key points, and obtaining the matching result from the key points are as follows: S4.1 Perform the first matching between the coarse feature map at 1 / 8 resolution and the 1 / 8 resolution map after the hybrid attention operation; this is a coarse matching. S4.2 The matching points obtained after coarse matching will include positive samples and negative samples. Positive samples represent matching points that are more in line with the model parameters, while negative samples are matching points that are not in line with the model parameters. The key points are obtained by filtering out negative samples. S4.
3. Based on the key points obtained after purification, perform sparse self-attention operation and dense cross-attention operation. S4.
4. Perform matching again on the keypoints after sparse self-attention operation and dense cross-attention operation, and remove negative samples to obtain the matching results.
4. An apparatus for implementing multimodal image feature matching as described in any one of claims 1 to 3, comprising: Image preprocessing module, feature extraction module, matching iteration module, result prediction module; The image pairs first enter the image preprocessing module to extract potential information from the images; then the feature maps enter the feature extraction module to further extract potential information from the images; subsequently, the image pairs enter the matching iteration module to perform coarse matching, matching point filtering, and obtain updated matching results through coarse matching again; finally, they enter the result prediction module, where the final result of coarse matching is processed on the fine-grained image to obtain the final fine-level matching result.
5. The apparatus for multimodal image feature matching according to claim 4, characterized in that: The image preprocessing module includes a feature encoder and a fast initialization module, which extracts features from the input image to obtain two different scale feature maps: a coarse feature map at 1 / 8 resolution and a fine feature map at 1 / 2 resolution. The feature extraction module performs SSC-type hybrid attention operation (self-attention-self-attention-cross attention) and SCC-type hybrid attention operation (self-attention-cross attention-cross attention), and through residual connections, the feature map after the hybrid attention operation still contains the original information. The matching iteration module filters key points based on the results of the initial coarse matching, and then performs a second coarse matching, adjusting and extracting information through sparse self-attention and dense cross-attention. The result prediction module locates key points on a fine-level feature map at 1 / 2 resolution and then makes the final result prediction.