Multi-dimensional fusion multi-modal image matching method and device
Through the multi-dimensional fusion multi-modal image matching method, the weighted recursive feature pyramid and linear arrangement attention mechanism are used to solve the problems of feature representation limitations and context dimension information in image matching, and improve the matching accuracy and robustness. In many practical applications.
Patent Information
- Application Number
- CN202410964737.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-18
- Publication Date
- 2024-12-06
AI Technical Summary
In the existing image matching technology, the limitations of feature representation and the lack of context dimension information lead to insufficient extraction capabilities, especially in extreme cases, feature point descriptions cannot provide sufficient information, affecting the matching accuracy.
Multi-dimensional fusion multi-modal image matching method is used to extract features through weighted recursive feature pyramids, and multi-dimensional information is captured and calculated using a linear arrangement of attention mechanism in the matching stage, reducing the time complexity of attention calculation.
It improves the accuracy and robustness of image matching, can effectively capture all dimension information, improves the matching effect, and is suitable for practical applications such as image stitching, virtual reality, augmented reality, and autonomous driving.
Smart Images

Figure CN119091166A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image matching in image processing, and in particular to a method and device for multi-dimensional fusion multi-modal image matching. Background Art
[0002] Image matching is an important branch of computer vision. The goal of image matching is to determine the similarity or relative relationship between two or more images. Inventors are working hard to develop more robust and efficient image matching methods, such as introducing more robust feature representations, improving matching algorithms to cope with various changes, and optimizing computing performance to improve efficiency. With the continuous development of computer vision and deep learning, it is believed that more innovative image matching methods will emerge in the future, thereby further improving the accuracy and robustness of image matching.
[0003] There are two main problems with current image matching, namely the limitation of feature representation and the lack of contextual dimensional information: traditional hand-crafted feature descriptors or feature description methods based on deep neural networks have their own limitations. Hand-crafted feature descriptors may not be able to capture the details and changes of complex images well, and although deep neural network-based methods can learn richer feature representations, they may still fail or be unstable in extreme cases. In some challenging situations, the description of a single feature point may not provide enough information for accurate matching. At this time, utilizing the contextual information of the surrounding area may be the key to improving the matching effect. The problems mainly manifest themselves in the insufficient extraction capability of the image feature extraction stage and the failure to capture all dimensions in the attention stage. Summary of the invention
[0004] 1. Technical issues to be resolved
[0005] In view of the shortcomings of the prior art, the present invention provides a method and device for multi-dimensional fusion multi-modal image matching, which has the advantages of capturing information of all dimensions and having lower time complexity of attention calculation, thereby solving the above-mentioned technical problems.
[0006] (II) Technical solution
[0007] To achieve the above object, the present invention provides the following technical solution: a method for multi-dimensional fusion multi-modal image matching, comprising the following steps:
[0008] S1. Input multimodal image data and perform image preprocessing based on the image data, specifically including:
[0009] S1.1, complement the different input features of the multimodal dataset;
[0010] S1.2, perform feature enhancement;
[0011] S1.3, divide the data into training and validation parts;
[0012] S2, extracting features from the data preprocessed in step S1 using a weighted recursive feature pyramid;
[0013] S3, input the obtained features into the linear permutation attention mechanism, and perform attention calculation operation in the matching stage;
[0014] S4. Obtain feature matching results.
[0015] As a preferred technical solution of the present invention, step S2 comprises the following steps:
[0016] S2.1, the first layer of the weighted recursive feature pyramid does not use the recursive mechanism, and directly uses the feature pyramid to output features, which specifically includes the following five steps:
[0017] S2.1.1, the input image passes through the convolution module to obtain the feature map
[0018] S2.1.2 Input feature map Through the convolution module, we get the feature map
[0019] S2.1.3 Input feature map Through the convolution module, we get the feature map
[0020] S2.1.4. Order Input feature map Through the convolution module, and the feature Add and realize residual connection to get feature map
[0021] S2.1.5 Input feature map Through the convolution module, and the feature Add and realize residual connection to get feature map
[0022] S2.2, weighted recursive feature pyramid uses a weighted mechanism to redistribute weight coefficients from the second layer to the last layer. For the i-th layer, the following steps are specifically included:
[0023] S2.2.1. Get the fine-grained feature map obtained by recursively connecting the previous weighted recursive feature pyramid layer make
[0024] S2.2.2. Obtain the intermediate feature map obtained by recursively connecting the previous weighted recursive feature pyramid layer And the fine-grained feature map obtained by recursively connecting the previous weighted recursive feature pyramid layer with S2.2.1 Weighted and activated by the Swish function to obtain the i-th layer feature map The calculation expression is as follows:
[0025]
[0026] Among them, w1 and w2 represent weight coefficients; Represents the intermediate feature map obtained recursively; Represents the fine-grained feature map obtained by recursive connection from the previous weighted recursive feature pyramid layer; the Resize operation indicates the change of the feature map size, the Conv operation indicates the convolution layer, ∈ is used to ensure data stability;
[0027] S2.2.3. Get the coarse-grained feature map obtained by recursively connecting the previous weighted recursive feature pyramid layer and compared with the feature map obtained in step S2.2.2 Perform weighted connection and activate through Swish function to obtain feature map The calculation expression is as follows:
[0028]
[0029] Among them, w1 and w2 represent weight coefficients; Represents the recursively obtained coarse-grained feature map; Represents the feature map of the i-th layer; the Resize operation indicates changing the size of the feature map, the Conv operation indicates the convolutional layer, and ∈ is used to ensure data stability;
[0030] S2.2.4, let the i-th layer coarse-grained feature map And input the i-th layer coarse-grained feature map Through the convolution module, and the intermediate feature map of the i-th layer The residual connection is realized by adding 1×1 convolution to obtain the intermediate feature map of the i-th layer
[0031] S2.2.5. Input feature map Through the convolution module, and the feature The residual connection is realized by adding 1×1 convolution to obtain the fine-grained feature map of the i-th layer.
[0032] S2.3, perform one round of S2.2 processing, and obtain There are 3 features in total, with step sizes of 2, 4, and 8 respectively, among which, They represent the fine-grained feature map output by the second layer, the intermediate feature map output by the second layer, and the coarse-grained feature map output by the second layer respectively;
[0033] S2.4 Use as input for subsequent processing.
[0034] As a preferred technical solution of the present invention, the specific process of step S3 is as follows:
[0035] S3.1, the upper branch of the first stage is responsible for calculating the attention weights across the channel dimension C and the spatial dimension W, and the lower branch is responsible for calculating the attention weights across the channel dimension C and the spatial dimension H. The input data The upper branches are arranged as Arranged through the lower branches Then they are arranged back to their original shapes. The weights are averaged through simple operations and the results are sent to stage 2;
[0036] S3.2, the upper branch of the second stage is responsible for calculating the attention weights across the channel dimension H and the spatial dimension W, and the lower branch is responsible for calculating the attention weights across the channel dimension W and the spatial dimension H. The input data The upper branches are arranged as Arranged through the lower branches Then arrange them back to their original shapes;
[0037] S3.3. By averaging the weights, a weighted output sensor is obtained.
[0038] As a preferred technical solution of the present invention, the specific steps of step S4 are as follows:
[0039] S4.1. Use the differentiable matching layer to match the transformed features and finally obtain a confidence matrix P c , P c The matches in are selected based on a confidence threshold and a mutual nearest neighbor criterion, resulting in a coarse-grained level match prediction M c ;
[0040] S4.2. For each selected coarse prediction A local window of size w×w is cut from the fine-level feature map, and the coarse match is refined to the sub-pixel level in this local window as the final fine-grained match prediction M f .
[0041] The present invention also provides a device for multi-dimensional fusion multi-modal image matching. The above-mentioned method for multi-dimensional fusion multi-modal image matching includes a multi-modal image preprocessing module, a feature extraction module, a feature matching module, and a matching result prediction module;
[0042] The multimodal image preprocessing module is used in the multimodal image input preprocessing stage to divide the input multimodal image data into training data and test data, and send the output result to the feature extraction module;
[0043] The feature extraction module extracts features from the preprocessed image sent by the multimodal image preprocessing module to obtain coarse-grained and fine-grained features, and the result of the feature extraction module is sent to the feature matching module;
[0044] The feature matching module performs attention calculation on the coarse-grained and fine-grained features sent by the feature extraction module, performs fine-grained feature matching based on the coarse-grained feature matching, and then sends the results to the matching result prediction module;
[0045] The matching result prediction module calculates the matching pairs based on the matching results sent by the feature matching module to obtain the matching results.
[0046] Compared with the prior art, the present invention provides a method and device for multi-dimensional fusion multi-modal image matching, which has the following beneficial effects:
[0047] The present invention provides a complete and efficient multimodal image matching process. The feature extraction module introduces recursive feedback and weighted exploration optimal structure to make information acquisition more efficient. Multi-dimensional information is acquired through the attention calculation module, and more complete original information can be retained. Compared with the existing matching scheme, the matching effect is improved. The matching process can be effectively migrated to practical applications such as image stitching, virtual reality, augmented reality, autonomous driving, navigation, real-time positioning and map construction, and positioning. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 It is a flow chart of the present invention;
[0049] Figure 2 Schematic diagram of downstream applications to which the present invention can be applied;
[0050] Figure 3 It is a flow chart of the process of the present invention;
[0051] Figure 4 It is a recursive mechanism diagram of the present invention;
[0052] Figure 5 This is a diagram of the weighting mechanism of the present invention;
[0053] Figure 6 It is a weighted recursive feature pyramid feature extraction graph of the present invention;
[0054] Figure 7 Linearly arrange the attention mechanism diagram for the present invention;
[0055] Figure 8 A single module diagram for linearly arranging attention for the present invention;
[0056] Fig. 9 This is the attention mechanism diagram of the present invention;
[0057] Fig.10 It is a software module diagram of the present invention. DETAILED DESCRIPTION
[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0059] See also Figure 1 - Fig.10 , a multi-dimensional fusion multi-modal image matching method, comprising the following steps:
[0060] S1. Input multimodal image data and perform image preprocessing based on the image data, specifically including:
[0061] S1.1, complement the different input features of the multimodal dataset;
[0062] S1.2, perform feature enhancement;
[0063] S1.3, divide the data into training and validation parts;
[0064] S2, extracting features from the data preprocessed in step S1 using a weighted recursive feature pyramid;
[0065] Step S2 includes the following steps:
[0066] S2.1, the first layer of the weighted recursive feature pyramid does not use the recursive mechanism, and directly uses the feature pyramid to output features;
[0067] The specific steps of outputting features of the first layer of the weighted recursive feature pyramid in S2.1 are as follows:
[0068] S2.1.1, the input image passes through the convolution module to obtain the first layer feature map
[0069] S2.1.2 Input feature map Through the convolution module, we get the first layer feature map
[0070] S2.1.3 Input feature map Through the convolution module, we get the first layer feature map
[0071] S2.1.4. Order Input feature map Through the convolution module, and the feature Add and realize residual connection to get feature map
[0072] S2.1.5 Input feature map Through the convolution module, and the feature Add and realize residual connection to get feature map
[0073] S2.2, weighted recursive feature pyramid uses a weighting mechanism to redistribute weight coefficients from the second layer to the last layer;
[0074] In step S2.2, the weighted recursive feature pyramid from the second layer to the i-th layer in the last layer specifically includes the following steps:
[0075] S2.2.1. Get the fine-grained feature map obtained by recursively connecting the previous weighted recursive feature pyramid layer make
[0076] S2.2.2. Obtain the intermediate feature map obtained by recursively connecting the previous weighted recursive feature pyramid layer And the fine-grained feature map obtained by recursively connecting the previous weighted recursive feature pyramid layer with S2.2.1 Weighted and activated by the Swish function to obtain the i-th layer feature map The calculation expression is as follows:
[0077]
[0078] Among them, w1 and w2 represent weight coefficients; Represents the intermediate feature map obtained recursively; Represents the fine-grained feature map obtained by recursive connection from the previous weighted recursive feature pyramid layer; the Resize operation indicates the change of the feature map size, the Conv operation indicates the convolution layer, ∈ is used to ensure data stability;
[0079] S2.2.3. Get the coarse-grained feature map obtained by recursively connecting the previous weighted recursive feature pyramid layer and compared with the feature map obtained in step S2.2.2 Perform weighted connection and activate through Swish function to obtain feature map The calculation expression is as follows:
[0080]
[0081] Among them, w1 and w2 represent weight coefficients; Represents the recursively obtained coarse-grained feature map; Represents the feature map of the i-th layer; the Resize operation indicates changing the size of the feature map, the Conv operation indicates the convolutional layer, and ∈ is used to ensure data stability;
[0082] S2.2.4, let the i-th layer coarse-grained feature map And input the i-th layer coarse-grained feature map Through the convolution module, and the intermediate feature map of the i-th layer The residual connection is realized by adding 1×1 convolution to obtain the intermediate feature map of the i-th layer
[0083] S2.2.5. Input feature map Through the convolution module, and the feature The residual connection is realized by adding 1×1 convolution to obtain the fine-grained feature map of the i-th layer.
[0084] S2.3, perform one round of S2.2 processing, and obtain There are 3 features in total, with step sizes of 2, 4, and 8 respectively, among which, They represent the fine-grained feature map output by the second layer, the intermediate feature map output by the second layer, and the coarse-grained feature map output by the second layer respectively;
[0085] S2.4 Use As input for subsequent processing;
[0086] S3, input the obtained features into the linear arrangement attention mechanism for multi-dimensional fusion, and perform attention calculation operations in the matching stage;
[0087] S3.1, the first stage passes through the linear permutation attention module with unchanged height dimension and the linear permutation attention module with unchanged width dimension, and sends the result to the second stage in step S3.2;
[0088] S3.2, the second stage passes through the linear permutation attention module with unchanged channel dimension and the linear permutation attention module of the original permutation;
[0089] S3.3. Get the weighted output by averaging the weights
[0090] S4. Get feature matching results
[0091] like Figure 1 As shown, the present invention includes 4 steps.
[0092] Input multimodal image data, perform position encoding and attention calculation feature extraction on the image data, input the obtained features into the multi-dimensional feature matching module, and obtain the feature matching results.
[0093] like Figure 2 As shown, the present invention can be used in downstream tasks such as image stitching, three-dimensional reconstruction, SLAM, and autonomous driving.
[0094] like Figure 3 As shown, the overall process of the present invention has four components:
[0095] 1. In the weighted recursive feature pyramid module, coarse-level feature maps and fine-level feature maps are detected from multimodal image pairs.
[0096] 2. In the fine-grained feature matching module, the coarse feature map is reduced to a one-dimensional vector and position encoding is added. The added features are then processed by the linear permutation attention mechanism module.
[0097] 3. In the matching confidence matrix calculation module, the transformed features are matched using the differentiable matching layer, and finally a confidence matrix P is obtained. c , P c The matches in are selected based on a confidence threshold and a mutual nearest neighbor criterion, resulting in a coarse-level match prediction M c .
[0098] 4. In the fine-grained feature matching module, for each selected coarse prediction A local window of size w×w is cut from the fine-level feature map. The coarse match is refined to the sub-pixel level in this local window and processed by the linear arrangement attention mechanism module. After cross-correlation calculation, it is used as the final match prediction M f .
[0099] like Figure 4 As shown in FIG. 1 , the recursive mechanism of the present invention. This module corresponds to the design of the recursive mechanism in the weighted recursive feature pyramid, namely Figure 6 The recursive mechanism part in . The specific process is as follows:
[0100] 1. For the first layer, no recursive mechanism is used, and the feature pyramid is directly used to output the feature F i_out .
[0101] 2. For the layers from the second layer to the last layer, let Where i represents the number of layers. In the present invention, the value range of j is set to 1 to 3. When j is 1, it means that the feature map Recursively obtain fine-grained feature maps When i is 2, it indicates the variable characteristic graph Recursively obtain the intermediate feature map When j is 3, it indicates the feature map Recursively obtain the coarse-grained feature map The difference is reflected in the different information granularity and the number of channels. The value of j is positively correlated with the coarse granularity of the information and the number of channels. That is, the output of the feature pyramid is re-sent into the feature pyramid to better capture the data features.
[0102] 3. For the last layer, select features As the output of the feature pyramid.
[0103] like Figure 5 As shown, the weighting mechanism of the present invention, this module corresponds to the design of the weighting mechanism in the weighted recursive feature pyramid, that is, Figure 6 The specific process of weighting mechanism is as follows:
[0104] 1. For the first layer, no weighting mechanism is used.
[0105] 2. For the second to last layers, use the weighting mechanism to redistribute the weight coefficients, In the present invention, i represents information granularity, and the value range of i is set to 1 to 2, j represents the number of layers, and the value range of j is set to 1 to 3. According to the setting of weights, the feature pyramid can explore a better structure during the training process.
[0106] like Figure 6 As shown, the design architecture of the weighted recursive feature pyramid of the present invention has the following specific process:
[0107] 1. Input multimodal image pairs imaged in different bands, perform image preprocessing, and generate corresponding multimodal image pairs and mapping matrices.
[0108] 2. Use the fast regularization method to initialize the weights, let where w i ≥0, and set ∈=0.0001 to avoid data instability. The weights are calculated and redistributed before each weighted connection.
[0109] 3. Starting from the second weighted recursive feature pyramid layer, stage 1 obtains the fine-grained feature map obtained by recursively connecting the previous weighted recursive feature pyramid layer make
[0110] Stage 2 obtains the feature map obtained by recursively connecting the previous weighted recursive feature pyramid layer (Size is ), and the feature map obtained in stage 1 Perform weighted connection, the calculation formula is Activate through the Swish function to obtain the feature map F2.
[0111] Stage 3 obtains the feature map obtained by recursively connecting the previous weighted recursive feature pyramid layer (Size is ), and the feature map obtained in stage 2 Perform weighted connection, the calculation formula is Activate through the Swish function to obtain the feature map F3.
[0112] 4. Stage 4, order Input coarse-grained feature map (Size is ) through the convolution module, and the feature The residual connection is realized by adding 1×1 convolution to obtain the feature map (Size is ).
[0113] Stage 5, input feature map (Size is ) through the convolution module, and the feature The residual connection is realized by adding 1×1 convolution to obtain a fine-grained feature map (Size is C1), stage 4 to stage 5 is the process of feature pyramid, the scaling step of the entire feature pyramid is 2, when i takes the value of 2, we get There are 3 features in total, and their step sizes are 2, 4, and 8 respectively. As a fine-grained feature map, As an intermediate variable, it is not output. As a coarse-grained feature map, we finally select Output.
[0114] like Figure 7 As shown in the figure, the design of the linear arrangement attention mechanism uses the calculation of QKV instead of CNN to extract features, so as to be able to perform mutual attention calculations and separate the information aggregation of the HW dimension and the information aggregation of the C dimension to better extract features. The linear arrangement attention mechanism achieves the capture of multi-dimensional features.
[0115] The specific process is as follows:
[0116] 1. The upper branch of the first stage is responsible for calculating the attention mechanism weights across the channel dimension C and the spatial dimension W, and the lower branch is responsible for calculating the attention mechanism weights across the channel dimension C and the spatial dimension H. Input data Arranged through the upper branches Arranged through the lower branches Then they are arranged back to their original shapes. The weights are averaged through simple operations and the results are sent to stage 2. The formula is as follows:
[0117] χ=P(C,W,LA(P(C,W,χ)))+P(C,H,LA(P(C,W,χ)))
[0118] Among them, LA refers to LinearAttention, that is, linear attention operation, P refers to Permute, that is, permutation operation, C represents channel dimension, W represents width dimension, and H represents height dimension.
[0119] 2. The upper branch of the second stage is responsible for calculating the attention mechanism weights across the channel dimension H and the spatial dimension W, and the lower branch is responsible for calculating the attention mechanism weights across the channel dimension W and the spatial dimension H. Input data Arranged through the upper branches Arranged through the lower branches Then arrange them back to the original shape, the formula is as follows:
[0120] χ=P(H,W,LA(P(H,W,χ)))+P(W,H,LA(P(W,H,χ)))
[0121] Among them, LA refers to LinearAttention, that is, linear attention operation, P refers to Permute, that is, permutation operation, C represents channel dimension, W represents width dimension, and H represents height dimension.
[0122] 3. Finally, the weights are averaged through simple operations to obtain the weighted output.
[0123] The overall design of linear arrangement attention is very advanced. All dimensions are taken into consideration using linear arrangement attention, and the information of each dimension is integrated using weighted calculation methods. This is not available in previous image matching and provides new ideas for image matching.
[0124] In this work, linear arrangement attention is reused in the coarse matching and fine matching stages, so that the image can pay attention to the characteristics of the low-texture part that is difficult to notice, and it is expected to perform better in the part with high repetitive patterns.
[0125] like Figure 8 As shown, this is the design of the linear arrangement attention module, which corresponds to Figure 7 The four linear permutation attention modules that appear in the process correspond to the linear permutation operations in stages 1 and 2 of the linear permutation attention mechanism process.
[0126] The specific process is as follows:
[0127] 1. The input data is first permuted to transform the dimension, and then passed through the convolution layer to obtain QKV. The formula is as follows:
[0128] Q,K,V=Conv(Permute(C0,C1,X))
[0129] Among them, C0 and C1 represent two different dimensions, X represents input data, Permute represents permutation operation, Conv represents convolution operation, and Q, K, and V represent three matrices required for linear attention calculation.
[0130] 2. K is matrix multiplied with V through the φ(·)=elu(·)+1 function, and then Q is matrix multiplied with the result of the KV operation through the φ(·)=elu(·)+1 function, and the weighted average is performed with the arranged original data. The formula is as follows:
[0131] X = LinearAttention(Q,K,V)
[0132] Among them, Q, K, and V represent the three matrices required for linear attention calculation, and X represents the data obtained after attention calculation.
[0133] 3. Finally, arrange it back to the original shape and output it.
[0134] X=Permute(C0,C1,X)
[0135] Where C0 and C1 represent two different dimensions, X represents data, and Permute represents a permutation operation.
[0136] The linear arrangement module is a small plug-and-play module. The present invention places multiple linear arrangement modules in the linear arrangement attention mechanism to maximize the consideration of the dimension of the input data. It has both efficient computing speed (linear complexity) and consideration of the dimension of the input data. It is a relatively complete module design.
[0137] like Fig. 9 As shown, this is the design of the linear attention mechanism. This module corresponds to Figure 8 The linear attention calculation that appears in the linear arrangement module process corresponds to process 2 of the linear arrangement attention module.
[0138] In naive attention, matrix multiplication of Q and K is performed first.
[0139] The calculation formula of naive attention is as follows:
[0140] Attention(Q,K,V)=softmax(QK T )V.
[0141] That is, the transpose of the Q matrix and the K matrix is first matrix multiplied, then activated by the softmax function, and then matrix multiplied with the V matrix.
[0142] The linear attention first performs matrix multiplication of K with V via φ(·)=elu(·)+1, which significantly reduces the amount of computation.
[0143] The calculation formula of linear attention is as follows:
[0144]
[0145] That is, first use the φ(·) function to perform K j The activation of , then the K matrix is matrix multiplied with the V matrix, and then matrix multiplied with the Q matrix, Indicates the accumulation of operation results from 1 to i. Since the accumulation operation is performed first, the average division is required at the end to obtain V' i , where φ(·)=elu(·)+1. The elu(·) function is an activation function, and by adding 1, the φ(·) activation function is made non-negative.
[0146] The permutation module uses a naive attention mechanism, which has the problem of high computational complexity. The present invention uses a linear attention mechanism to improve the permutation module and applies it to the permutation attention mechanism. The present invention designs a linear permutation attention mechanism and uses four linear permutation modules, which has the advantages of lightweight computation and no information loss.
[0147] The present invention also provides a device for multi-dimensional fusion multi-modal image matching. The above-mentioned method for multi-dimensional fusion multi-modal image matching includes a multi-modal image preprocessing module, a feature extraction module, a feature matching module, and a matching result prediction module;
[0148] The multimodal image preprocessing module is used in the multimodal image input preprocessing stage to divide the input multimodal image data into training and test data, and send the output results to the feature extraction module;
[0149] The feature extraction module extracts features from the preprocessed image sent by the multimodal image preprocessing module to obtain coarse-grained and fine-grained features, and the result of the feature extraction module is sent to the feature matching module;
[0150] The feature matching module performs attention calculation on the coarse-grained and fine-grained features sent by the feature extraction module, performs fine-grained feature matching based on the coarse-grained feature matching, and then sends the results to the matching result prediction module;
[0151] The matching result prediction module calculates the matching pairs based on the matching results sent by the feature matching module to obtain the matching results.
[0152] The present invention designs a weighted recursive feature pyramid to solve the problem that the traditional feature pyramid has insufficient feature extraction capability and may not have the optimal structure. In the coarse-grained and fine-grained matching parts, the present invention designs a linear arrangement attention mechanism to capture information of all dimensions, while the time complexity of attention calculation is lower.
[0153] In terms of backbone network design, the present invention designs a weighted recursive feature pyramid, which solves the problems that the traditional feature pyramid has insufficient feature extraction capability and may not have an optimal structure.
[0154] In order to solve the problem of missing dimensional information in the current attention module, this paper introduces a linear arrangement attention mechanism, performs attention calculation in each dimension, and effectively separates each dimensional information, which not only reflects the modular idea in software design, but also can capture more complete input information. This makes it easier for the model to notice the hard-to-notice parts of the picture, improving the attention performance.
[0155] The evaluation index of the present invention is the one-image estimation, which is often used in computer vision and image processing to find image pairs of the same scene but different angles. The one-image estimation is used to correct the perspective offset of the image. In the test, the feature points of the image pairs are paired, and the correctness is determined by the pixel deviation between the correct result and the predicted result. The present invention sets three thresholds, which are (3 pixels, 5 pixels, and 10 pixels). In terms of data, the higher the result, the better the matching effect of the model.
[0156] The methods compared in the present invention include HardNet, MatchosNet, TFeat, MatchNet, LoFTR and FeMIP. These methods are all verified models that have achieved certain good results in the current image matching field. By comparing with these methods, the present invention can be supported by data and the effect of the present invention in the current image matching field can be verified.
[0157] 1. Verify the matching effects of different methods on the Optical-sar and RGB-NIR Scene datasets.
[0158]
[0159]
[0160] 2. Verify the matching effects of different methods on the WHU-OPT-SAR and NYU-depthV2 datasets.
[0161]
[0162] The monographic estimation index of the present invention is significantly better than the current mainstream image matching scheme, which proves the effectiveness of the present invention.
[0163] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A method for multi-dimensional fusion multi-modal image matching, characterized in that: The following steps are involved: S1. Input multimodal image data and perform image preprocessing based on the image data, specifically including: S1.1, complement the different input features of the multimodal dataset; S1.2, perform feature enhancement; S1.3, divide the data into training and validation parts; S2, extracting features from the data preprocessed in step S1 using a weighted recursive feature pyramid; S3, input the obtained features into the linear arrangement attention mechanism for multi-dimensional fusion, and perform attention calculation operations in the matching stage; S4. Obtain feature matching results.
2. The method for multi-dimensional fusion multi-modal image matching according to claim 1, characterized in that: The step S2 comprises the following steps: S2.1, the first layer of the weighted recursive feature pyramid does not use the recursive mechanism, and directly uses the feature pyramid to output features; S2.2, weighted recursive feature pyramid uses a weighting mechanism to redistribute weight coefficients from the second layer to the last layer; S2.3, perform one round of S2.2 processing, and obtain There are 3 features in total, with step sizes of 2, 4, and 8 respectively, among which, They represent the fine-grained feature map output by the second layer, the intermediate feature map output by the second layer, and the coarse-grained feature map output by the second layer respectively; S2.4 Use as input for subsequent processing.
3. The method for multi-dimensional fusion multi-modal image matching according to claim 2, characterized in that: The specific steps of outputting the first layer features of the weighted recursive feature pyramid in step S2.1 are as follows: S2.1.1, the input image passes through the convolution module to obtain the first layer feature map S2.1.2 Input feature map Through the convolution module, we get the first layer feature map S2.1.3 Input feature map Through the convolution module, we get the first layer feature map S2.1.
4. Order Input feature map Through the convolution module, and the feature Add and realize residual connection to get feature map S2.1.5 Input feature map Through the convolution module, and the feature Add and realize residual connection to get feature map 4. The method for multi-dimensional fusion multi-modal image matching according to claim 2, characterized in that: The weighted recursive feature pyramid in step S2.2 from the second layer to the i-th layer in the last layer specifically includes the following steps: S2.2.
1. Get the fine-grained feature map obtained by recursively connecting the previous weighted recursive feature pyramid layer make S2.2.
2. Obtain the intermediate feature map obtained by recursively connecting the previous weighted recursive feature pyramid layer And the fine-grained feature map obtained by recursively connecting the previous weighted recursive feature pyramid layer with S2.2.1 Weighted and activated by the Swish function to obtain the i-th layer feature map The calculation expression is as follows: Among them, w1 and w2 represent weight coefficients; Represents the intermediate feature map obtained recursively; Represents the fine-grained feature map obtained by recursive connection from the previous weighted recursive feature pyramid layer; the Resize operation indicates the change of the feature map size, the Conv operation indicates the convolution layer, ∈ is used to ensure data stability; S2.2.
3. Get the coarse-grained feature map obtained by recursively connecting the previous weighted recursive feature pyramid layer and compared with the feature map obtained in step S2.2.2 Perform weighted connection and activate through Swish function to obtain feature map The calculation expression is as follows: Among them, w1 and w2 represent weight coefficients; Represents the recursively obtained coarse-grained feature map; Represents the feature map of the i-th layer; the Resize operation indicates changing the size of the feature map, the Conv operation indicates the convolutional layer, and ∈ is used to ensure data stability; S2.2.4, let the i-th layer coarse-grained feature map And input the i-th layer coarse-grained feature map Through the convolution module, and the intermediate feature map of the i-th layer The residual connection is realized by adding 1×1 convolution to obtain the intermediate feature map of the i-th layer S2.2.
5. Input feature map Through the convolution module, and the feature The residual connection is realized by adding 1×1 convolution to obtain the fine-grained feature map of the i-th layer.
5. The method for multi-dimensional fusion multi-modal image matching according to claim 1, characterized in that: The specific process of the linear arrangement attention mechanism in step S3 is as follows: S3.1, the first stage passes through the linear permutation attention module with unchanged height dimension and the linear permutation attention module with unchanged width dimension, and sends the result to the second stage in step S3.2; S3.2, the second stage passes through the linear permutation attention module with unchanged channel dimension and the linear permutation attention module of the original permutation; S3.
3. By averaging the weights, a weighted output is obtained.
6. The method for multi-dimensional fusion multi-modal image matching according to claim 1, characterized in that: The specific steps of step S4 are as follows: S4.
1. Use the differentiable matching layer to match the transformed features and finally obtain a confidence matrix P c , P c The matches in are selected based on a confidence threshold and a mutual nearest neighbor criterion, resulting in a coarse-grained level match prediction M c ; S4.
2. For each selected coarse prediction A local window of size w×w is cut from the fine-level feature map, and the coarse match is refined to the sub-pixel level in this local window as the final fine-grained match prediction M f .
7. A device for multi-dimensional fusion multi-modal image matching, based on a method for multi-dimensional fusion multi-modal image matching according to any one of claims 1 to 6, characterized in that: It includes a multimodal image preprocessing module, a feature extraction module, a feature matching module, and a matching result prediction module; The multimodal image preprocessing module is used in the multimodal image input preprocessing stage to divide the input multimodal image data into training data and test data, and send the output result to the feature extraction module; The feature extraction module extracts features from the preprocessed image sent by the multimodal image preprocessing module to obtain coarse-grained and fine-grained features, and the result of the feature extraction module is sent to the feature matching module; The feature matching module performs attention calculation on the coarse-grained and fine-grained features sent by the feature extraction module, performs fine-grained feature matching based on the coarse-grained feature matching, and then sends the results to the matching result prediction module; The matching result prediction module calculates the matching pairs based on the matching results sent by the feature matching module to obtain the matching results.
