Multi-modal image matching method based on fusion features and attention state space

Abstract

The invention belongs to the technical field of multi-modal image matching, and particularly relates to a multi-modal image matching method based on fusion features and attention state space. The method comprises the following steps: preprocessing an obtained infrared image and a visible light image, wherein the preprocessing comprises geometric correction, noise removal and brightness normalization; performing multi-scale feature extraction on the preprocessed images to obtain multi-scale features; performing rotation scale correction on the features by using a group convolution module based on a rotation group; inputting the corrected features into an attention state space model for cross-modal feature interaction, wherein the model adopts a convolution state updating mechanism and space attention fusion; on the basis of the interacted features, a rough matching result is obtained through block similarity calculation; carrying out refined optimization on a rough matching result, carrying out local feature extraction by using a small convolution kernel, and combining an attention weighting mechanism; in the training stage, a self-adaptive uncertainty loss function supervised model based on Gaussian negative logarithm likelihood is adopted for training.

Description

Technical Field

[0001] This invention belongs to the field of multimodal image matching technology, specifically relating to a multimodal image matching method based on fusion features and attention state space. Background Technology

[0002] Among existing multimodal image matching methods, traditional matching methods based on grayscale or gradient features lack stability and accuracy in complex environments; deep learning methods have poor feature alignment capabilities in cross-modal scenarios, and the attention mechanism has high computational complexity, making them unsuitable for embedded deployment. Summary of the Invention

[0003] The purpose of this invention is to solve the problems of insufficient feature interaction capabilities and low model lightweighting in existing technologies, and to achieve efficient and accurate image matching.

[0004] This invention provides a multimodal image matching method based on fused features and attention state space, comprising the following steps:

[0005] The acquired infrared and visible light images are preprocessed, including geometric correction, noise removal, and brightness normalization.

[0006] Multi-scale feature extraction was performed on the preprocessed infrared and visible light images to obtain infrared multi-scale features and visible light multi-scale features respectively.

[0007] Rotation and scale correction are performed on the infrared multi-scale features and visible light multi-scale features respectively. The rotation and scale equivariance correction of the features are realized by using a group convolution module based on rotation group.

[0008] The corrected infrared multi-scale features and visible light multi-scale features are input into the attention state space model for cross-modal feature interaction. The attention state space model adopts a convolutional state update mechanism and spatial attention fusion.

[0009] Based on the features after feature interaction, a coarse matching result is obtained by calculating block similarity.

[0010] The coarse matching results are refined and optimized by using small convolutional kernels for local feature extraction and combining an attention weighting mechanism to optimize the matching relationship.

[0011] During the training phase, an adaptive uncertainty loss function based on Gaussian negative log-likelihood is used to supervise model training.

[0012] This invention achieves high precision and strong robustness in multimodal image matching through steps such as multi-scale feature extraction, rotation scale correction, attention state space model, and refinement optimization. At the same time, the model is lightweight and easy to deploy in embedded systems. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the method flow of an embodiment of this application.

[0014] Figure 2 This is a schematic diagram of the wavelet convolution module structure provided in the embodiments of this application.

[0015] Figure 3 This is a schematic diagram of the state space feature interaction and spatial attention fusion structure provided in the embodiments of this application.

[0016] Figure 4 This is a schematic diagram of the efficient attention state space fusion and memory interaction mechanism provided in the embodiments of this application. Detailed Implementation

[0017] As mentioned above, in visual matching navigation tasks, due to the differences in illumination response, imaging mechanisms, and texture features between different modalities, traditional matching methods based on grayscale or gradient features often struggle to guarantee stability and accuracy in complex environments. While existing deep learning methods have achieved good results within the same modality, they face insufficient feature alignment capabilities in cross-modal scenarios, are prone to errors when rotation angles are large, and have high computational complexity, making them unsuitable for deployment in embedded systems.

[0018] In this context, the problems that need to be solved are how to improve the model's feature interaction capabilities in multimodal scenarios and how to make the model lightweight for easy edge deployment.

[0019] In view of this, the present invention provides a multimodal image matching method based on fused features and attention state space. Its overall process includes steps such as image acquisition, feature extraction, feature enhancement, cross-modal feature fusion, feature matching and optimization.

[0020] This invention is primarily applied to fields requiring precise navigation at night or in inclement weather, such as drone delivery or infrastructure inspection. In a specific scenario, the drone is equipped with an inexpensive infrared camera to capture real-time thermal images, which are then matched with pre-acquired visible light satellite maps. This invention can complete image alignment in a very short time, and even if the drone tilts in strong winds, it can correct its flight path in real time to avoid collisions. The entire algorithm is lightweight and can be deployed on embedded platforms such as DJI, enabling fully autonomous flight.

[0021] Figure 1 This is a schematic diagram of the method flow of an embodiment of this application. Figure 2 This is a schematic diagram of the wavelet convolution module structure provided in the embodiments of this application.

[0022] Step 1: Preprocess the input infrared and visible light image matching pairs respectively.

[0023] First, OpenCV (an open-source computer vision library) is used to perform geometric correction on infrared and visible light images containing camera intrinsic and extrinsic parameters to remove distortion. If the infrared camera provides radiance output, the temperature value is used first, followed by linearized brightness correction. If only grayscale information is available, non-uniformity correction is performed to reduce the impact of stripes and dead pixels on the image. Then, interpolation is used to repair common bad pixels in infrared images, and median filtering is applied to visible light images to remove salt-and-pepper noise. Simultaneously, over-smoothing is avoided to prevent loss of structural information.

[0024] The images are scaled to the same 640×640 pixel resolution to facilitate processing using feature pyramids. Each image is then z-score normalized (mean minus standard deviation). The infrared and visible light images are brightness normalized for use in training the deep network.

[0025] Step 2: Extract features from the infrared image and the visible light image respectively.

[0026] First, a lightweight 3×3 small-kernel convolutional network is designed to replace the large-kernel convolutional neural network, generating three feature maps of sizes (3, 64, 240, 240), (3, 128, 120, 120), and (3, 256, 64, 64) to extract local texture and edge features. The dimensions are represented as (batch size, number of channels, height, width). The small-kernel convolutional structure significantly reduces the number of parameters while preserving spatial details, improving adaptability to edge devices.

[0027] Based on this, the extracted primary features are input into the Symlet wavelet decomposition module to perform multi-scale decomposition of the feature map, obtaining high-frequency and low-frequency components respectively. The high-frequency components are used to capture texture details and edge variation features, while the low-frequency components preserve the overall structure and brightness information of the image. Subsequently, depthwise separable convolution is used to efficiently fuse the multi-scale features, achieving joint representation of cross-frequency information and providing a more robust feature representation for subsequent feature alignment and matching.

[0028] Taking a feature map with dimensions (3, 64, 240, 240) as an example, the Symlet wavelet decomposition module decomposes it into three high-frequency components and one low-frequency component. The high-frequency components are used to capture texture details and edge variations, while the low-frequency component preserves the overall structure and brightness information of the image. After decomposition, the spatial dimensions of each component are (3, 64, 120, 120).

[0029] After decomposition, the high-frequency components are input into a depthwise separable convolutional module for feature enhancement and compression. This module extracts local spatial features through depthwise convolution and then achieves information interaction and fusion between channels through pointwise convolution, thereby improving the expressive power of high-frequency features without significantly increasing computational cost.

[0030] Subsequently, the high-frequency components processed by depthwise separable convolution are combined with their corresponding low-frequency components using wavelet reconstruction to generate a reconstructed feature map with the same spatial dimensions as the input feature map. The reconstructed output feature map has its dimensions restored to (3, 64, 240, 240).

[0031] After reconstruction, the reconstructed features of this layer are concatenated with the original input feature map along the channel dimension to obtain a fused multi-scale feature map. For example, the feature size of the first layer after concatenation is (3, 128, 240, 240), the second layer is (3, 256, 120, 120), and the third layer is (3, 512, 64, 64).

[0032] To maintain the consistency of feature map size across different levels, the concatenated multi-channel feature map is compressed using a linear dimensionality reduction module to reduce the number of channels to the same level as the original input feature map, resulting in the following representation:

[0033] ;

[0034] in, For linear mapping, B represents the batch size, i.e., the number of samples input to the model at one time; C represents the number of channels, corresponding to the number of convolutional kernels in the feature map or the channel dimension of the image; W represents the width, i.e., the number of pixels in the feature map in the horizontal direction; H represents the height, i.e., the number of pixels in the feature map in the vertical direction; X represents the original input feature map; l represents the current network layer index (e.g., layer 1, 2, 3); conv represents depthwise separable convolution operation, used for feature enhancement and compression of high-frequency components; wav represents wavelet decomposition and reconstruction operation, including decomposition into high-frequency and low-frequency components and subsequent reconstruction process; R represents the feature map obtained after wavelet reconstruction, i.e., the feature map restored to its original spatial size after high-frequency enhancement and reconstruction with low-frequency components.

[0035] Map the number of channels after splicing back The splicing here only occurs in the channel dimension, and the spatial dimensions remain unchanged.

[0036] For example, after the first layer of dimensionality reduction, the output size is restored to (3, 64, 240, 240), the second layer is (3, 128, 120, 120), and the third layer is (3, 256, 64, 64).

[0037] Through the above multi-scale feature extraction process, while maintaining the spatial resolution, the cross-frequency expression capability and structural robustness of the features are effectively improved, providing a more accurate basic feature representation for subsequent feature alignment and matching.

[0038] Step 3: Use the group convolution module to perform rotational scaling correction.

[0039] To address the rotation and scale differences in multimodal images caused by variations in shooting pose or viewpoint, this invention introduces a rotation-scale equivariance correction module based on group convolution. Group convolution is a convolution based on a rotation group; equivariance correction refers to maintaining equivariance to rotation and scale changes. The group convolution kernel is based on a rotation group... Based on this foundation, it achieves equivariant deformation responses to multiple rotational scales through parameter sharing. G represents the rotation group, a discrete rotational symmetry group upon which the group convolution module is based. Its elements correspond to a fixed set of angles used to share parameters within the convolution kernel, ensuring that features maintain an equivariant response to planar rotations of the image, thus achieving rotation invariance and scale robustness. Its operation can be represented as:

[0040] ;

[0041] in, Indicates input features, Represents the kernel convolution function. The kernel convolution matrix, This module represents the group space convolution computation, where x represents the relative position within the convolution kernel, and y represents the spatial position in the input feature map. It achieves kernel weight sharing in the rotated group space, ensuring stable response consistency of features under different rotational angles and scale changes, while maintaining the unchanged feature size after processing. This allows the feature representation to be input into the subsequent neural network, thus achieving rotation invariance and scale robustness.

[0042] Through the correction processing of this module, the feature map can still maintain good structural correspondence and matching stability in complex scenes with large viewing angle deviations (such as squinting, looking up, or rotation angles exceeding 30°), providing reliable support for subsequent multimodal feature fusion and accurate registration.

[0043] In summary, the group convolution-based correction module employed in this step achieves rotation and scale equivariance from the network structure's underlying layer by introducing prior knowledge of geometric transformations. Compared to traditional methods that rely on data augmentation, this significantly improves generalization ability. Fundamentally different from existing methods that passively rely on data augmentation or post-processing algorithms to address geometric changes, this invention actively and structurally endows the network with inherent rotation and scale equivariance. This hard-coded geometric invariance allows the model to still exhibit excellent generalization ability and stability when faced with large-angle rotations or scale scaling not fully covered in the training data, providing a novel and more reliable technical path to solve the inherent geometric inaccuracy problem in cross-modal image matching.

[0044] Step 4, Attention State Space Feature Interaction Module.

[0045] Figure 3 This is a schematic diagram of the state space feature interaction and spatial attention fusion structure provided in the embodiments of this application.

[0046] Multi-scale features from the convolutional and wavelet domains are input into the attention state space model, and information propagation and dynamic modeling between features are achieved through hidden state updates. The hidden state of each feature map layer is jointly determined by the output of the previous layer and the current input, forming a consistent representation of features in scale and space. In the specific implementation, to adapt to the spatial structural characteristics of visual features, the attention state space model adopts a convolutional hidden state update mechanism. For a feature map, the update of its hidden state can be represented as:

[0047] ;

[0048] in, This represents the hidden state passed from the previous layer. W represents the multimodal fusion features of the current input. h W x W l For learnable convolution weights, For bias terms, σ represents the convolution operation, and σ is the non-linear activation function ReLU; B represents the batch size, i.e., the number of samples input into the model at one time; C represents the number of channels, corresponding to the number of convolution kernels in the feature map or the channel dimension of the image; W represents the width, i.e., the number of pixels in the feature map in the horizontal direction; H represents the height, i.e., the number of pixels in the feature map in the vertical direction; h represents the hidden state, and l represents the network layer index.

[0049] The output has the same spatial dimensions as the input features, thus ensuring structural consistency when features are transferred across different scales.

[0050] This update approach is equivalent to introducing a spatial convolutional structure into the state-space recursive equation, enabling the hidden state to not only transmit dynamic information in the hierarchical dimension but also perceive local neighborhood features in the spatial dimension. Unlike traditional fully connected state updates, this convolutional state-space update effectively captures local contextual relationships while maintaining the model's translation invariance.

[0051] During state-space propagation, an efficient attention mechanism is incorporated, adaptively allocating attention weights based on feature saliency distribution to suppress irrelevant regions and highlight key structures. A feature pyramid structure is formed through multi-layer stacking, preserving global contextual information while possessing fine-grained local recognition capabilities, and maintaining the same feature size after processing, thus providing multi-resolution support for subsequent matching.

[0052] In summary, the attention state space feature interaction module in this step achieves a unification of long-range dependency modeling and local context awareness by integrating the sequence modeling capabilities of the attention state space model with the spatial structure requirements of visual tasks. Through the design of a convolutional state update mechanism, this invention cleverly overcomes the dual shortcomings of traditional attention state space models neglecting local context in visual applications and the enormous computational overhead of Transformer models. This design achieves a unity of long-range dependency modeling, local context awareness, and linear computational complexity, providing a feasible solution for achieving high-performance, high-precision real-time image matching on computationally limited embedded platforms—a feasibility previously unattainable by existing technologies.

[0053] Step 5: Calculate the block similarity matrix and perform coarse matching.

[0054] Figure 3 This is a schematic diagram of the state space feature interaction and spatial attention fusion structure provided in the embodiments of this application.

[0055] To improve computational efficiency and matching stability, this invention employs a block-based similarity matrix calculation strategy after the feature pyramid output. Specifically, the feature maps of the infrared and visible light images are divided into several sub-blocks, denoted as follows: and In the first In the layer feature pyramid, the block similarity matrix is ​​obtained by calculating the normalized correlation between the two.

[0056] ;

[0057] in This represents the similarity matrix of this layer; f refers to the current pyramid level; p and q are the indexes of the feature sub-blocks of the infrared image and the visible light image, respectively, used to traverse all sub-block combinations; V represents the p-th sub-block feature of the infrared image after being divided in this layer; I represents the q-th sub-block feature of the visible light image after being divided in this layer; the two are cross-correlated through normalization to obtain the similarity value at the corresponding position, forming the entire layer's similarity matrix.

[0058] The above method can significantly reduce the computational load while maintaining multi-scale consistency, obtain preliminary matching results of cross-modal features, and achieve the output of the coarse matching stage.

[0059] Step 6: Match the refinement and optimization layer.

[0060] Based on the coarse matching results, this invention further employs a refinement module combining small convolutional kernels and an attention mechanism to precisely adjust the matching relationships in local regions. Specifically, for the high-confidence candidate regions obtained in the coarse matching stage, local feature extraction is first performed using small convolutional kernels to capture subtle texture differences and geometric shifts. Let the features of the corresponding infrared and visible light regions be V and I, respectively, and the local similarity can be expressed as:

[0061] ;

[0062] in, Represents the parameters of the small convolution kernel. For convolution operations, Softmax is used to normalize the similarity score. S' represents the normalized similarity score calculated after extracting infrared and visible light features in a local region using a small convolution kernel; p and q correspond to the indices of candidate matching points in the infrared and visible light feature maps, respectively; c represents the learnable parameters of the small convolution kernel, used to extract local fine texture and geometric offset features; the two are convolved by the small convolution kernel and then passed through Softmax to obtain the similarity, and an attention weighting mechanism is further introduced for fine matching optimization.

[0063] Building upon this, an attention weighting mechanism is introduced, adaptively assigning weights based on feature saliency to enhance matching regions with high relevance within the local context. Its attention fusion form is as follows:

[0064] ;

[0065] in, This indicates an attention-weighted operation. This serves as a balancing factor. Through this mechanism, the model can suppress erroneous responses while maintaining local accuracy.

[0066] Finally, low-confidence or isolated matching points are discarded, retaining only spatially consistent and stable matching pairs. The matching results refined by this module exhibit higher alignment accuracy and stability in local regions, providing reliable support for the final geometric registration and image fusion.

[0067] Step 7: During the training phase, this invention employs an adaptive uncertainty loss function based on Gaussian Negative Log-Likelihood (NLL) for model supervision. This method not only penalizes the geometric deviation between predicted and actual matching points, but more importantly, it guides the model to learn to quantify the uncertainty of each matching prediction. Specifically, in addition to predicting the coordinates of the matching points, the model also outputs a logarithmic value of the variance in parallel. This value represents the model's "confidence" in the reliability of the current predicted coordinates. The loss function takes the form:

[0068] ;

[0069] in, p represents the coordinates of the matching points predicted by the model. gt Let ||·||² represent the true coordinates of the matching points, and ||·||² represent the square of the Euclidean distance. It is a measure of the uncertainty that the model predicts for this matching pair.

[0070] p gt L represents the true coordinates of the matching point (i.e., the coordinates of the correctly labeled corresponding point in the training samples). point The loss is calculated as the Gaussian negative log-likelihood between the model's predicted matching point coordinates and the true coordinates. This loss is used to simultaneously measure the predicted location bias and the model's uncertainty (variance) regarding the prediction, thereby guiding the network to pursue high accuracy in reliable regions and reduce penalty weights in difficult regions, thus improving overall robustness.

[0071] This loss function mechanism has the following advantages: when the model is very confident about a match ( When the value is small (i.e., the variance is small), it will penalize the bias of the predicted coordinates more severely; conversely, when the model considers a certain match to have a large uncertainty ( A larger value (i.e., a larger variance) reduces the weight of coordinate bias in the total loss, allowing the model to make larger errors in blurry or difficult regions (such as missing textures or occlusions) without compromising overall training stability. By minimizing this loss, the model not only learns how to match accurately but also when to trust its predictions, significantly improving matching robustness and reliability in complex multimodal scenarios.

[0072] Step 8: Efficient memory interaction mechanism and parallel acceleration.

[0073] Figure 4 This is a schematic diagram of the efficient attention state space fusion and memory interaction mechanism provided in the embodiments of this application.

[0074] To achieve real-time deployment of models on embedded platforms, this invention proposes an efficient memory interaction and lightweight parallel computing framework: a shared memory buffer is established within a multi-scale feature pyramid to hierarchically store intermediate features after convolution and attention processing, reducing redundant loading; feature sharing and fast access are achieved between different levels through an index mapping mechanism; a sparse access mechanism is introduced, performing read and write operations only on highly saliency feature regions; and a dual pipeline structure for computation and transmission is constructed to achieve parallel execution of data prefetching and asynchronous computation, thereby reducing memory usage and computation latency. This mechanism, while ensuring real-time performance, can reduce overall memory usage by approximately 30% to 45%, significantly improving the operating efficiency of embedded systems.

[0075] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multimodal image matching method based on fused features and attention state space, characterized in that, Includes the following steps: The acquired infrared and visible light images are preprocessed, including geometric correction, noise removal, and brightness normalization. Multi-scale feature extraction was performed on the preprocessed infrared and visible light images to obtain infrared multi-scale features and visible light multi-scale features respectively. Rotation and scale correction are performed on the infrared multi-scale features and visible light multi-scale features respectively. The rotation and scale equivariance correction of the features are realized by using a group convolution module based on rotation group. The corrected infrared multi-scale features and visible light multi-scale features are input into the attention state space model for cross-modal feature interaction. The attention state space model adopts a convolutional state update mechanism and spatial attention fusion. Based on the features after feature interaction, a coarse matching result is obtained by calculating block similarity. The coarse matching results are refined and optimized by using small convolutional kernels for local feature extraction and combining an attention weighting mechanism to optimize the matching relationship. During the training phase, an adaptive uncertainty loss function based on Gaussian negative log-likelihood is used to supervise model training.

2. The method as described in claim 1, characterized in that, The multi-scale feature extraction includes: Use a small kernel convolutional network to extract primary feature maps; Wavelet decomposition is performed on the primary features to obtain high-frequency and low-frequency components; High-frequency components are enhanced using depthwise separable convolutions; The enhanced high-frequency components and low-frequency components are reconstructed using wavelet reconstruction to obtain the reconstructed features. The reconstructed features are concatenated with the original input features through channels, and multi-scale features are obtained through linear dimensionality reduction.

3. The method as described in claim 1, characterized in that, The group convolution module is constructed based on a rotation group G, where the rotation group G represents a discrete rotation symmetric group. By sharing the weights of the convolution kernel in the rotation group space, the features maintain an equivariant response to planar rotation and scale changes of the image, and maintain structural correspondence when there is a viewpoint deviation.

4. The method as described in claim 1, characterized in that, The convolutional state update mechanism of the attention state space model updates the hidden state through convolution operations and nonlinear activation functions. The hidden state is jointly determined by the upper-level hidden state and the current input features, achieving consistent representation of features in scale and space. It also combines spatial attention mechanism to adaptively allocate weights to highlight key structures.

5. The method as described in claim 1, characterized in that, The block similarity calculation is performed at different levels of the feature pyramid. The infrared feature map and the visible light feature map are divided into multiple sub-blocks, and the normalized cross-correlation value between the sub-blocks is calculated to form a multi-scale similarity matrix, which is used for the output of the coarse matching stage.

6. The method as described in claim 1, characterized in that, The refinement and optimization step uses small convolutional kernels to extract local features, calculates local similarity scores through Softmax normalization, and adaptively allocates weights using an attention weighting mechanism to enhance highly correlated matching regions. At the same time, it removes low-confidence matching points and isolated matching points, retaining only spatially consistent and stable matching pairs.

7. The method as described in claim 1, characterized in that, The adaptive uncertainty loss function predicts the logarithm of the coordinates and variance of the matching points, while simultaneously monitoring coordinate bias and uncertainty measures. This allows the model to more strictly penalize bias in high-confidence regions and reduce the loss weight in uncertain regions, thereby improving the robustness of the matching.

8. The method as described in claim 1, characterized in that, It also includes an efficient memory interaction mechanism, which establishes a shared memory cache in the multi-scale feature pyramid to achieve hierarchical storage and feature sharing, and enables fast access to features between different levels through an index mapping mechanism, reducing repeated loading and video memory usage.

9. The method as described in claim 8, characterized in that, The efficient memory interaction mechanism also includes a sparse access mechanism and a dual pipeline structure for computation and transmission. The sparse access mechanism only performs read and write operations on highly salient feature regions, and the dual pipeline structure enables data prefetching and asynchronous computation to be executed in parallel, reducing computation latency.

10. The method as described in claim 1, characterized in that, include: The method is applied to a drone navigation system, in which the drone is equipped with an infrared camera to capture real-time images and match them with a pre-stored visible light map to achieve real-time position correction, flight path planning, or obstacle avoidance.

Images