Mamba structure-based staggered multi-modal image joint registration and fusion method

Abstract

The invention discloses a misplaced multi-modal image joint registration and fusion method based on a Mama structure, and the method comprises the steps: reading a first modal image and a second modal image which is misplaced relative to the first modal image, and converting the first modal image into a pseudo image in a second modal; sending the pseudo image and the second modal image into a differential weight driven registration network to gradually estimate a multi-scale deformation field; re-sampling the optimal deformation field output by iteration and the second modal image to obtain a registration image of the second modal; a dynamic multi-layer feature extractor constructed based on dynamic convolution and single-modal Mama extracts features of a first modal image and a registration image, and then progressive image fusion is carried out through a differential Mama fusion network; and the output fusion features are sent to a single-modal Mamba reconstruction module, and a high-quality fusion image of the first modal image and the second modal registration image is obtained. According to the method, the registration precision and the fusion quality of the dislocation image can be remarkably improved in a complex scene.

Description

Technical Field

[0001] This invention relates to the field of computer image processing technology, and in particular to a method for joint registration and fusion of misaligned multimodal images based on the Mamba structure. Background Technology

[0002] With the rapid development of computer vision technology and the increasing demands for information fusion accuracy in practical applications, image fusion, especially joint image registration and fusion techniques, has gradually become an important research direction in the field of image processing. In complex real-world environments, due to differences in sensor imaging mechanisms, changes in shooting perspective, and the influence of object motion, multimodal image pairs acquired by different sensors such as infrared, visible light, medical, and remote sensing often suffer from problems such as misalignment, scale inconsistency, or local distortion. Directly fusing misaligned image pairs will inevitably introduce blurring, artifacts, and structural distortion, thus severely affecting the fusion quality.

[0003] To address these issues, various joint registration and fusion methods based on convolutional neural networks (CNNs) have emerged in recent years. However, the inherent fixed convolutional kernels of CNNs limit the adaptability of the receptive field, making it difficult to dynamically adjust the sampling range and response intensity according to regional characteristics when processing cross-modal images with significant structural differences, thus limiting their ability to model local details. Furthermore, CNNs are inadequate in multi-scale structural alignment and geometric deformation correction, struggling to achieve hierarchical differential correction for misaligned regions, and easily mistaking inconsistent edges and textures as fusionable features, further exacerbating fusion blurring and artifacts.

[0004] To enhance long-range dependency modeling capabilities, some methods have introduced the Transformer and utilized self-attention mechanisms for global context modeling. However, the Transformer's block-based partitioning strategy disrupts the spatial continuity of images and suffers from high computational complexity and memory overhead in high-resolution scenes, making it difficult to meet the real-time and robustness requirements of practical applications. Furthermore, emphasizing only global structural information while neglecting the collaborative modeling of global semantics and local texture can lead to problems such as missing details, semantic fragmentation, and inconsistencies in local regions in the fused image.

[0005] In summary, existing joint registration and fusion methods have three main limitations: (1) Static convolution is difficult to achieve adaptive modeling of cross-modal local structural differences, resulting in solidified receptive fields; (2) Lack of differentiated, multi-level geometric correction for misaligned regions, resulting in insufficient capture of multi-scale deformations during the registration stage; (3) Disconnection between global semantics and local texture modeling, resulting in blurred details, misaligned edges, and semantic inconsistencies in the fused image, making it difficult to meet the requirements of high-quality fusion. Summary of the Invention

[0006] To address the problems of spatial misalignment and structural asymmetry in multimodal images leading to blurred fusion results and increased artifacts, this invention proposes a joint registration and fusion method for misaligned multimodal images based on the Mamba structure. This method employs weighted differential registration, dynamic multi-layer coupled feature extraction, and differential fusion coupling strategies. By introducing a global-local coupled modeling mechanism and fully utilizing the advantages of the selective state-space model Mamba in long sequence modeling, it effectively alleviates blurring and artifacts caused by structural offset, improves multi-scale deformation capture capability, and obtains high-quality fused images with consistent structure, clear texture, and semantic coherence, thereby achieving more efficient and robust multimodal image fusion. The technical solution of this invention is as follows:

[0007] A joint registration and fusion method for misaligned multimodal images based on the Mamba structure, comprising the following steps: S1. Read the first modal image and the second modal image. The second modal image is misaligned relative to the first modal image. Convert the first modal image used as a reference into a pseudo image in the second modality. S2. The misaligned image pair, composed of the pseudo image in the same mode and the misaligned second mode image, is fed into the registration network driven by the differential weight to estimate the multi-scale deformation field step by step. S3. Resample the optimal deformation field and the misaligned second modal image output by the iteration to obtain the registered second modal image; S4. A dynamic multi-layer feature extractor based on dynamic convolution and unimodal Mamba extracts features from the first modality image and the registered second modality image, respectively. S5. The features of the first modality image and the registered second modality image are progressively fused through a differential Mamba fusion network to output fused features; S6. The fusion features are fed into the single-modal Mamba reconstruction module to obtain a high-quality fusion image of the registered image pair consisting of the first modal image and the registered second modal image.

[0008] The further technical solution is that step S2 includes: S21. The misaligned image pairs are fed into the pyramid feature extraction module. The pyramid feature extraction module extracts multi-scale features of each type of image at different scales through cascaded convolution, batch normalization layers and non-linear activation functions. S22. The multi-scale features of the two types of images are sent to the multi-scale upsampling decoding module. Each decoder upsamples the multi-scale features to a uniform spatial resolution and then sends them to the next decoder. Simultaneously, the initial optical flow output from each decoder is spatially differentially weighted to obtain the coarse deformation field, including: The initial optical flow output from each decoder is upsampled to a uniform spatial resolution and stitched along the channel dimension. The resulting convolutional blocks with ReLU activation and Sigmoid activation are used to generate fusion weights. The fusion weights are then used to perform differential weighted summation on the optical flow field of each upsampled field. The summation results are then vector integrated and smoothed in multiple steps to form a coarse deformation field.

[0009] S23. The coarse deformation field features, along with the upsampled features and pseudo-image features output by the current decoder, are weighted together using channel difference to obtain feature weights. These weights are then multiplied by their respective original features to obtain the projected refined features, including: The coarse deformation field features are concatenated with the upsampled features and pseudo-image features output by the current decoder. The concatenation is then processed by a convolutional block with ReLU activation and Softmax to generate channel-level difference weights. The coarse deformation field features are multiplied by the upsampled features and pseudo-image features output by the current decoder and the corresponding channel-level difference weights. The resulting refined features are then passed through a channel attention module and sent to the next decoder.

[0010] S24. The refined features are iteratively propagated through multiple decoders (i.e., repeating S22 and S23), and the optimal deformation field is output through spatial difference weighting in the last iteration.

[0011] A further technical solution is that the method for extracting features from the first modality image and the registered second modality image in step S4 is the same, that is, the registered multimodal image is input into a multilayer dynamic feature extractor, and the features of the first modality image and the registered second modality image are extracted through independent branches respectively. Each branch includes: S41. Each type of input image is processed by a local perceptual dynamic convolutional feature extractor to extract local fine-grained features. The local perceptual dynamic convolutional feature extractor is used to adaptively adjust the receptive field and kernel weights in terms of the number of convolutional kernels, output channels, input channels and kernel space dimension to enhance the local features of the input image. S42. Extract remote dependency features and global semantic features from local fine-grained features using the dynamic state transition and selective scanning mechanism of the single-modal Mamba feature extractor, and output them.

[0012] The further technical solution is that step S5 includes: S51. The extracted first modality image features and the registered second modality image features are subjected to two channel swaps. After each channel swap, the single-modality Mamba module is used for feature enhancement. S52. Concatenate the enhanced feature pairs along the channel dimension to form a joint representation rich in cross-modal complementary information. After passing the joint representation through a normalized convolutional layer, connect it with the respective enhanced features to obtain two refined features. S53. Send the two refined features to the channel attention module to obtain shallow fusion features, thereby achieving more accurate modal weight allocation and difference compensation. In the channel attention module, channel weights for different modalities are obtained through global average pooling, multilayer perceptron, and sigmoid activation function. These weights are then multiplied and summed with their respective refined features to obtain shallow fusion features.

[0013] S54. The two refined features and the shallow fusion features are fed together as input to the multimodal fusion module. The multimodal fusion module uses N cascaded Multi-Modal Mamba modules to perform deep fusion to obtain the fusion features of the deep multimodal image.

[0014] In the multimodal fusion module, each Multi-Modal Mamba module undergoes normalization, convolution, parameter discretization, and finally a selective scanning mechanism to obtain the first fusion feature, which is then fed into the next module. The output of the last Multi-Modal Mamba module is modulated by a gate factor MLP and then added to the shallow fusion features processed by deep convolution to obtain the final fusion features.

[0015] The further technical solution is that step S6 includes: S61. The fused features are sent to the single-modal Mamba module for image reconstruction to obtain a spatial feature map; S62. The spatial feature map is sequentially passed through two layers of convolution to gradually reduce the channels and a non-linear activation function to correct the local texture. The output feature vector is then mapped to the target image channel through a convolutional layer to output a high-quality fused image.

[0016] Its further technical solution is that the implementation principle of the single-modal Mamba feature extractor mentioned in S42 and the single-modal Mamba module mentioned in S51 and S61 is the same, including: After the input features are serialized and normalized, they are fed into different branches. Relevant features are extracted in each branch using a multilayer perceptron and are denoted as the first feature and the second feature, respectively. The first feature passes through a convolutional block with a Sigmoid activation function and then enters a selective scanning mechanism to output global features. The second feature is multiplied by the global feature after passing through the Sigmoid activation function, and then added to the serialized feature after being modulated by the gate factor MLP to obtain the high-level feature and reassembled back into the spatial distribution.

[0017] The beneficial technical effects of this invention are: The aforementioned method addresses the issues of edge blurring and artifacts that easily occur after fusing misaligned image pairs through full-dimensional dynamic convolution and the differential Mamba fusion algorithm, effectively improving the quality and clarity of the fused image. By introducing dynamic convolution and differential enhancement fusion mechanisms, this invention achieves adaptive alignment and deep fusion of cross-domain features while maintaining the integrity of multimodal information. Compared to traditional static convolution or single attention mechanisms, this invention can dynamically adjust the receptive field and fusion weights according to changes in the local structure and global semantics of the image, thus possessing greater flexibility and expressive power in the extraction and reconstruction of key features such as details, texture, and gradients. In the field of image fusion, the technical solution of this invention has significant advantages: firstly, it effectively improves the spatial consistency of the fused image and reduces structural misalignment caused by registration errors; secondly, it enhances the fidelity of edges and details, avoiding blurring, ghosting, and pseudo-texture phenomena common in traditional methods; and thirdly, it achieves deep semantic complementarity of multimodal data, enabling multimodal image pairs such as infrared, visible light, medical, and remote sensing images to generate a more interpretable fusion result within a unified framework.

[0018] Compared with traditional joint image registration and fusion methods, the Mamba-based misaligned multimodal image joint registration and fusion method proposed in this invention not only significantly improves the visual quality and structural consistency of multimodal image fusion, but also provides more reliable and accurate image base data support for subsequent downstream tasks such as image segmentation and scene understanding. Attached Figure Description

[0019] Figure 1 This is a flowchart of the method for joint registration and fusion of misaligned multimodal images based on the Mamba structure provided in this application; Figure 2 This is an overall framework diagram of the misaligned multimodal image joint registration and fusion network based on the Mamba structure provided in this application; Figure 3 This is a structural diagram of the multi-scale deformation field estimation module using a registration network driven by differential weights provided in this application. Figure 4 This is a structural diagram of the progressive image fusion module using a differential Mamba fusion network provided in this application; Figure 5 This is a visualization of the registration effect of misaligned image pairs provided in this application; Figure 6 This is a visualization of the fusion effect of the registered image pairs provided in this application. Detailed Implementation

[0020] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0021] This application provides a method for joint registration and fusion of misaligned multimodal images based on the Mamba architecture. The network framework of this method is as follows: Figure 2 As shown, taking misaligned multimodal image pairs as input, it includes a misaligned image registration network ( ) and registration image fusion network ( The multimodal image pairs in this embodiment are presented in two parts: visible light image and visible light image. As the first modality image, and also the reference image required for subsequent registration, relative to Misaligned infrared images ( Using the second modality image as an example, this method will be explained and illustrated. First, the visible light image is processed through the TransNet network. Converted to pseudo-infrared image ( This reduces the intermodal distance between the two types of modes, converting it into a single-modal domain for processing. Next, a multi-scale feature extractor is used to extract pseudo-infrared images (…). ) and misaligned infrared images ( The features were then processed through multi-scale upsampling (MUD) and asymptotic difference quasi-module (PDR) to obtain the optimal deformation field. And calibrate accordingly. Generate registered infrared images ( ). The visible light image is first extracted using a Dynamic Multilayer Feature Extractor (DMFE). ) and registered infrared images ( The features are then progressively fused using the Differential Mamba Fusion Module (DPMF), and finally the fused features are input into the single-modal Mamba image reconstruction module with residuals. After decoding, a high-quality fused image is obtained.

[0022] In one embodiment, the flowchart of the joint registration and fusion method for misaligned multimodal images based on the Mamba structure is as follows: Figure 1 As shown, the specific steps include the following: S1. Read the first modal image and the second modal image, where the second modal image is misaligned relative to the first modal image. Convert the first modal image, which is used as a reference, into a pseudo-image in the second modality. To reduce the inherent modal differences between infrared and visible light images, the visible light image (… The input is a transformation network (such as TransNet), and the transformed pseudo-infrared image is obtained through style transfer of the network. The domain transformation process can be described as follows: (1) S2. The misaligned image pair, composed of the pseudo-image in the same modality and the misaligned second modality image, is fed into a registration network driven by differential weights to progressively estimate the multi-scale deformation field. This step specifically includes: S21. The misaligned image pairs are fed into the pyramid feature extraction module. The pyramid feature extraction module extracts multi-scale features of each type of image at different scales through cascaded convolution, batch normalization layers and nonlinear activation functions.

[0023] like Figure 2 As shown, the misaligned image pairs after processing by the TransNet network ( , ) sent to the pyramid feature extraction module ( This allows the network to extract image features in a multi-scale, multi-resolution space, enabling it to simultaneously possess the ability to perceive local details and understand global structures. It consists of three parts: cascaded convolutions, batch normalization layers, and non-linear activation functions. Cascaded convolutions enhance the extraction of local features, batch normalization layers buffer internal covariate shifts, and the non-linear activation function LeakyReLU preserves small slopes in the negative interval, thereby improving the feature representation of deep networks. The low-level details (texture, edges) and high-level semantics (large structure, context) of the input image are simultaneously represented as multi-scale feature maps that can be used by subsequent networks, and finally pseudo-infrared images are obtained. Multiscale features And misaligned infrared images ( Multiscale features .

[0024] S22. The multi-scale features of the two types of images are sent to the multi-scale upsampling decoding module. Each decoder upsamples the multi-scale features to a uniform spatial resolution and then sends them to the next decoder. At the same time, the initial optical flow output by each decoder is weighted by spatial difference to obtain the coarse deformation field.

[0025] After pyramid feature extraction, the multi-scale features are then sent to the multi-scale upsampling decoding module (MUD) for upsampling to unify the spatial resolution. Each decoder performs upsampling before sending the results to the next decoder, simultaneously obtaining the initial optical flow. The data is sent to the Progressive Differential Alignment (PDR) module. The PDR first upsamples the data to a uniform spatial resolution, then uses spatial difference weighting to adaptively correct multi-scale geometric biases. The specific implementation process is as follows: Figure 3 As shown, it includes the following: In this embodiment of the invention, the progressive differential alignment module (PDR) receives the initial optical flow output from each decoder. The upsampled feature vectors are concatenated along the channel dimension, after two... The convolutional blocks with ReLU activation functions yield a weighted vector of the interpolated deformable field, which is then applied at each scale. Convolutional blocks with Sigmoid activation function generate fused weights Utilizing fusion weights Differential weighted summation is performed on the optical flow field of each upsampled sample, and finally vector integration and multi-step smoothing are performed on the summation result to form a coarse deformation field. After four operations in stage S22, when k≠1, the process proceeds to stage S23; when k=1, This is the final deformation field. The formula for calculating the coarse deformation field can be expressed as: (2) S23. The coarse deformation field features, together with the upsampled features and pseudo-image features output by the current decoder, are weighted by channel difference to obtain feature weights. These weights are then multiplied by their respective original features to obtain the projected refined features, which are then sent to the next decoder.

[0026] When k≠1, the coarse deformation field characteristics obtained in the previous step will be... Together with the upsampled features output by the current decoder Pseudo-infrared image features The process involves cascading the features, followed by feature embedding through convolutional blocks with ReLU activation. Finally, the embedded features are used to generate channel-level difference weights using Softmax. And then with their respective original characteristics , , Multiplication, since different channels encode different textures and structures, emphasizes the most important features through weighting. To further enhance the network's adaptive selection of effective information and improve the representation strength of key structural regions, refined registration features are obtained. Finally, a channel attention module (CA) is introduced after weighting to obtain the refined features after projection. The CA module captures statistical information across space through global average pooling (GAP) and learns the importance distribution of each channel through a lightweight nonlinear mapping, enabling the network to dynamically focus on the structural features that contribute most to registration.

[0027] S24. The refined features are iteratively propagated through multiple decoders, and the optimal deformation field is output through spatial difference weighting in the last iteration.

[0028] If k≠1, the refined registration features The data will be input again into step S22 for registration and correction. This process is repeated four times until k=1, at which point the optimal deformation field is output. Used to convert misaligned infrared images ( To visible light images ( Registration correction.

[0029] S3. Resample the optimal deformation field output by the iteration and the misaligned second modal image to obtain the registered second modal image. In this step, the optimal deformation field... And misaligned infrared images ( Both the infrared image and the space-time vector are fed as input to the resampling module, where bilinear interpolation or a spatial transformation network is used to perform remapping, thus obtaining the registered infrared image. The registered infrared image can be represented as: (3) Where R represents resampling, This refers to the registered infrared image.

[0030] S4. A dynamic multi-layer feature extractor based on dynamic convolution and unimodal Mamba extracts features from the first modality image and the registered second modality image. In this step, the registered infrared image ( ) and raw visible light image ( As a registered image pair input, the Dynamic Multilayer Feature Extractor (DMFE) includes a locally perceptual dynamic convolutional feature extractor on each branch of the DMFE. ) and SingleMamba feature extractor ( Each independent branch extracts the registered infrared image features (second modality registered image features) and visible light image features (first modality image features). The extraction process is the same on each branch, specifically including: S41, Each type of input image is processed... Extracting low-level, local, fine-grained features. For low-level features of the input image pair, such as texture details and edge information, feature extraction is performed using a locally perceptive dynamic convolutional feature extractor. )Finish, This allows for adaptive adjustment of the receptive field and kernel weights in terms of the number of convolutional kernels, output channels, input channels, and kernel space dimensions, thereby enhancing local features of the input image, such as texture and edges.

[0031] S42, Transfer local fine-grained features through The dynamic state transition and selective scanning mechanism extracts long-range dependency features and global semantic features, which are then output as a branch of the DMFE. High-level features of the input image pairs, such as global structure and overall contour features, are extracted using the SingleMamba feature extractor (…). To complete, By leveraging Mamba's dynamic state transition and selective scanning mechanisms, efficient global context modeling is achieved. This captures long-distance dependencies and global structural information without increasing computational complexity, significantly improving the extraction capabilities of long-distance dependency features and global semantic features. Figure 2 As shown, Implement the following: After performing serialization (PatchEmbed) and normalization (LayerNorm) operations on the input features, they are fed into different branches, each of which utilizes a multilayer perceptron. Extract relevant features, and denote them as the first feature. Second feature First characteristic After passing through a convolutional block with a Sigmoid activation function, the system enters the Selective Scanning Mechanism (SSM) to output global features. Second feature After the Sigmoid activation function and combined with global features Multiplication, followed by MLP modulation by a gated factor, and then addition to the serialized features, yields high-level features. And reassembled back into the spatial distribution. The specific pseudocode for the SingleMamba algorithm implementation in this embodiment is shown in Table 1: Table 1. SingleMamba Feature Extraction Algorithm Process

[0032] in, B Indicates batch size, N Indicates the number of output feature maps. C Indicates the number of channels. D Represents the state dimension.

[0033] S5. The first modality image features and the registered second modality image features are progressively fused using a differential Mamba fusion network to output fused features. For example... Figure 4 As shown, this step specifically includes: S51. Perform two channel swaps on the first modality image features and the registered second modality image features. After each channel swap, use the single-modality Mamba module for feature enhancement.

[0034] Specifically, through Extracted visible light image features and registered infrared image features The input is fed into the shallow fusion module, which consists of two channel swapping (CE) modules and one channel attention module (CA). The purpose of channel swapping is to fully exploit the complementary information of the single-modal channels. After each complementary fusion, a SingleMamba module is added to extract the integrity information of local semantics, perform local detail restoration and context enhancement, improve the structural integrity and detail fidelity of the fused image, and obtain enhanced feature pairs. The implementation principle of SingleMamba can be found in Table 1, and will not be elaborated further here.

[0035] S52. Concatenate the enhanced feature pairs along the channel dimension to form a joint representation rich in cross-modal complementary information. After passing the joint representation through a normalized convolutional layer, connect it with the respective enhanced features to obtain two refined features.

[0036] Specifically, the enhanced features are for Cascaded along the channel dimension to form a joint representation rich in cross-modal complementary information. This operation not only preserves the semantic structure and detailed responses of each modality after reconstruction under Mamba guidance, but also explicitly constructs an alignment space for features between modalities. To further improve fusion quality and suppress redundant or conflicting information introduced by misalignment or modal heterogeneity, cascaded features... The data is fed into a normalized convolutional layer (CLN), which consists of convolutional blocks and LayerNorm. Instance normalization helps eliminate distribution shifts between different modalities caused by differences in sensor responses. Finally, to avoid over-smoothing or loss of modality-specific information during fusion, a residual connection mechanism is introduced. This connects the original shallow features of the modalities with the normalized features through skip connections, forming the final refined features. This process can also be expressed by the formula: (4) (5) S53. Send the two refined features to the channel attention module to obtain shallow fusion features, thereby achieving more accurate modal weight allocation and difference compensation.

[0037] To further enhance important information and suppress irrelevant information, the two refined features are... The data is sent to the Channel Attention (CA) module. The CA module obtains channel weights for different modalities through Global Average Pooling (GAP), Multilayer Perceptron (MLP), and Sigmoid activation function. These weights are then multiplied and summed with the refined features of each channel. The channel weights are used to adaptively calibrate the importance of each channel, strengthening the semantic and texture responses that are most discriminative for the fusion task, thus obtaining shallow fusion features. This process can be represented as: (6) (7) in, For visible light channel weights, and These represent the multilayer perceptron and the global average pooling, respectively. This indicates shallow fusion characteristics.

[0038] S54. The two refined features and the shallow fusion features are fed together as input to the multimodal fusion module. The multimodal fusion module uses N cascaded Multi-Modal Mamba modules to perform deep fusion to obtain the fusion features of the deep multimodal image.

[0039] like Figure 4 As shown, in order to further enhance cross-modal global dependency modeling and promote global-local fusion coupling, the two refined multimodal features are... and shallow fusion features Send five Multi-modalMambas together ( The module performs deep fusion. Deep fusion complements shallow fusion by modeling long-distance dependencies and enhancing global contextual semantics, jointly enhancing fusion complementarity and semantic integrity to obtain a high-quality fused image. Each The module undergoes normalization, convolution, and parameter discretization, and finally passes through a selective scanning mechanism to obtain the visible light features output by the current module. Infrared features and intermediate fusion features And send these three features into the next... This module, this process, can be expressed by the formula: (8) The last one The module's output, after being modulated by a gated MLP, is fused with the shallow features processed by deep convolution. Adding them together yields the final fusion feature. This process can be represented as: (9) MLP stands for Multilayer Perceptron Operation. This represents a depthwise convolution operation.

[0040] S6. The fusion features are fed into the single-modal Mamba reconstruction module to obtain a high-quality fused image of the registered image pair consisting of the first modality image and the registered second modality image. This step specifically includes: S61. The fused features are sent to the single-modal Mamba module for image reconstruction to obtain a spatial feature map.

[0041] The final fusion characteristics obtained through deep fusion The image is then sent to the SingleMamba module with residuals for global deep reconstruction. The SingleMamba reconstruction architecture fully leverages the computational efficiency and expressive power of the state-space model in long sequence modeling, while ensuring the stability of feature transfer through the residual mechanism, significantly improving the image reconstruction quality and robustness in complex degradation scenarios. The implementation principle of SingleMamba can be found in Table 1, and will not be elaborated further here.

[0042] S62. The spatial feature map output by the SingleMamba module is sequentially processed by two layers of convolution to gradually reduce the channels and a non-linear activation function to correct the local texture.

[0043] The spatial features are sequentially input into a local refinement subnetwork consisting of two convolutional layers. Each layer contains a 3×3 convolutional kernel to capture the local neighborhood context. The number of channels is progressively reduced layer by layer to gradually decrease computational redundancy and focus on semantically compact representation. Each convolution is followed by a LeakyReLU nonlinear activation function to introduce a local nonlinear response, enhancing the ability to model high-frequency details such as edges and textures. The above process can be represented as: (10) S63. The output feature vector is mapped to the target image channel through a convolutional layer to output a high-quality fused image.

[0044] The output feature vector is fed into a lightweight output projection layer, which uses a 1×1 convolutional kernel to precisely map the number of channels to the number of channels in the target image, outputting a fused image. For an RGB image with 3 channels, the output formula is: (11) In some embodiments, after step S6, the following steps are further included: S7. Determine if the current image pair is the last pair; if yes, end; otherwise, return to step S1, read the next misaligned image pair, and continue until the last frame is processed to complete the fusion of multiple misaligned image pairs.

[0045] The model proposed in this invention was trained on the widely used Infrared and Visible Multimodal Dataset (MSRS), which includes 1444 pairs of infrared and visible images. The model was trained using a default set of 1083 pairs, and tested on the remaining 361 pairs. To expand the validation scale, it was also tested on the shared M3FD and TNO datasets. Since misaligned image comparisons are rare in reality, this invention uses affine and elastic transformations to generate misaligned image pairs. The rotation angle of the affine transformation is set at... Range, translation pixel control within The range and elastic transformation were obtained by blurring the noise map of the two channels (range: [-1,1]) using a Gaussian filter. The misaligned images were all performed on infrared images. During the testing process, the misaligned infrared and visible light images were first aligned using a registration network, and then fused using a fusion network.

[0046] To verify the effectiveness of this invention, its advantages will be explained below from the aspects of qualitative visualization effects and quantitative analysis.

[0047] (1) Visual qualitative analysis of image registration like Figure 5 As shown, the advantages of this invention are qualitatively analyzed from the visualization effect of image registration. Before registration, the tree trunks, signal towers, buildings, and pedestrians in the infrared image are distorted and shifted relative to the visible light image. After registration, the images from the MSRS, M3FD, and TNO datasets are all corrected, and the spatial position, contour shape, and edge direction of the main structures in the scene are consistent with the visible light image. These results demonstrate that the registration network of this invention can accurately capture nonlinear deformations caused by cross-modal imaging differences, achieving consistent correction of local details and overall structure.

[0048] (2) Quantitative analysis of image registration To verify the image registration capability of this invention, Table 2 shows the quantitative image registration results of this invention on the MSRS, M3FD, and TNO datasets. Compared with the infrared images before registration, the registered images show significant improvements in mean square error (MSE), mutual information (MI), mean absolute error (MAE), and peak signal-to-noise ratio (PSNR). This indicates that the registration network of this invention can effectively correct spatial deviations between cross-modal images, improve the consistency of multimodal features, and provide a more accurate and stable alignment basis for subsequent image fusion.

[0049] Table 2. Quantitative results of image registration

[0050] (3) Image fusion visualization analysis like Figure 6As shown, the advantages of this invention can be qualitatively analyzed from the visualization effect of image fusion. Before fusion, the outlines of pedestrians in the visible light image are blurred and the edges are discontinuous, while the structures such as buildings and tree trunks in the infrared image are distorted to varying degrees. After fusion, the images in the MSRS, M3FD, and TNO datasets not only achieve accurate alignment of geometric structures, but also effectively integrate the detailed texture information of the visible light with the salient target information of the infrared image, resulting in a significant improvement in the edges, outlines, brightness, and contrast of the scene.

[0051] (4) Quantitative analysis of image fusion To verify the ability of the dynamic convolution and Mamba extraction and fusion image designed in this invention, Table 3 verifies the differential fusion capabilities of channel-switched fusion and multimodal Mamba fusion. Based on the MSRS, M3FD, and TNO datasets, this invention analyzes seven indicators of image information entropy (EN), mutual information (MI), spatial frequency (SF), average gradient (AG), visual information fidelity (VIF), quality assessment based on edge information (Qabf), and structural similarity (SSIM). Higher indicators indicate higher coverage information, sharpness, structural consistency, and fidelity. The effectiveness of the fusion method in this application is verified through quantitative analysis of the indicators.

[0052] Table 3. Quantitative Analysis of Image Fusion

[0053] In summary, this invention, through a differential calibration network, progressively estimates multi-scale deformation fields using differential spatial weighting and differential channel weighting, effectively correcting spatial misalignment and distortion in images. The invention also enhances feature representation capabilities through the construction of dynamic convolution and Mamba feature extraction modules, extracting both local fine-grained features and long-range dependent features. In the differential fusion stage, channel-swapping fusion and multimodal Mamba fusion effectively promote complementary information fusion and local-to-global fusion coupling. Through the synergistic effect of these two fusion mechanisms, the differential fusion stage significantly improves the discriminative and representational capabilities of the fused features, ultimately generating a fused image with clear details, consistent structure, and higher visual quality.

[0054] The above descriptions are merely preferred embodiments of this application, and the present invention is not limited to the above embodiments. It is understood that other improvements and variations directly derived or conceived by those skilled in the art without departing from the spirit and concept of the present invention should be considered to be included within the protection scope of the present invention.

Claims

1. A method for joint registration and fusion of misaligned multimodal images based on the Mamba structure, characterized in that, The method includes: Read the first modal image and the second modal image. The second modal image is misaligned relative to the first modal image. Convert the first modal image, which is used as a reference, into a pseudo image in the second modality. The misaligned image pair, consisting of the pseudo image in the same modality and the misaligned second modality image, is fed into a registration network driven by differential weights to gradually estimate the multi-scale deformation field. The optimal deformation field output by iteration and the misaligned second modal image are resampled to obtain the registered second modal image; A dynamic multi-layer feature extractor based on dynamic convolution and unimodal Mamba extracts features from the first modality image and the registered second modality image, respectively. Then, progressive image fusion is performed through a differential Mamba fusion network to output fused features. The fusion features are fed into the single-modal Mamba reconstruction module to obtain a high-quality fusion image of the registered image pair consisting of the first modal image and the registered second modal image.

2. The method for joint registration and fusion of misaligned multimodal images based on the Mamba structure according to claim 1, characterized in that, The misaligned image pair, composed of the pseudo-image in the same modality and the misaligned second modality image, is fed into a differential weight-driven registration network to progressively estimate the multi-scale deformation field, including: The misaligned image pairs are fed into the pyramid feature extraction module, which extracts multi-scale features of each type of image at different scales through cascaded convolution, batch normalization layers and nonlinear activation functions. The multi-scale features of the two types of images are fed into the multi-scale upsampling decoding module. Each decoder upsamples the multi-scale features to a uniform spatial resolution and then sends them to the next decoder. At the same time, the initial optical flow output by each decoder is weighted by spatial difference to obtain the coarse deformation field. The coarse deformation field features, along with the upsampled features and pseudo-image features output by the current decoder, are combined through channel differential weighting to obtain feature weights. These weights are then multiplied by their respective original features to obtain the projected refined features, which are then sent to the next decoder. The refined features are iteratively propagated through multiple decoders, and the optimal deformation field is output through spatial difference weighting in the last iteration.

3. The method for joint registration and fusion of misaligned multimodal images based on the Mamba structure according to claim 2, characterized in that, The initial optical flow output by each decoder is used to obtain a coarse deformation field through spatial difference weighting, including: The initial optical flow output of each decoder is upsampled to a uniform spatial resolution and stitched along the channel dimension. The resulting weights are generated by passing the convolutional blocks with ReLU activation function and convolutional blocks with Sigmoid activation function. The optical flow field of each upsampled sample is differentially weighted and summed using the fusion weights. The summation result is then vector-integrated and multi-step smoothed to form the coarse deformation field.

4. The method for joint registration and fusion of misaligned multimodal images based on the Mamba structure according to claim 2, characterized in that, The process of obtaining the refined features after projection includes: The coarse deformation field features are concatenated with the upsampled features and pseudo-image features output by the current decoder, and then processed through a convolutional block with ReLU activation function and Softmax to generate channel-level difference weights. The coarse deformation field features, together with the upsampled features and pseudo-image features output by the current decoder, are multiplied by the corresponding channel level difference weights, and then passed through the channel attention module to obtain the projected refined features.

5. The method for joint registration and fusion of misaligned multimodal images based on the Mamba structure according to claim 1, characterized in that, The methods for extracting features from the first modality image and the registered second modality image are the same, including: Each type of input image is processed by a local perceptual dynamic convolutional feature extractor to extract local fine-grained features. The local perceptual dynamic convolutional feature extractor is used to adaptively adjust the receptive field and kernel weights in terms of the number of convolutional kernels, output channels, input channels, and kernel space dimension to enhance the local features of the input image. The local fine-grained features are used to extract remote dependency features and global semantic features through the dynamic state transition and selective scanning mechanism of the single-modal Mamba feature extractor, and then output.

6. The method for joint registration and fusion of misaligned multimodal images based on the Mamba structure according to claim 1, characterized in that, The features of the first modality image and the registered second modality image are progressively fused using a differential Mamba fusion network, and the output fused features include: The first modality image features and the registered second modality image features are subjected to two channel swaps. After each channel swap, the single-modality Mamba module is used for feature enhancement. The enhanced feature pairs are concatenated along the channel dimension to form a joint representation rich in cross-modal complementary information. The joint representation is then passed through a normalized convolutional layer and connected with its respective enhanced features to obtain two refined features. The two refined features are sent to the channel attention module to obtain shallow fusion features, thereby achieving more accurate modal weight allocation and difference compensation. The two refined features and the shallow fusion features are fed together as input to the multimodal fusion module. The multimodal fusion module uses N cascaded Multi-Modal Mamba modules to perform deep fusion to obtain the fusion features of the deep multimodal image.

7. The method for joint registration and fusion of misaligned multimodal images based on the Mamba structure according to claim 1, characterized in that, The fused features are fed into a single-modal Mamba reconstruction module to obtain a high-quality fused image of a registered image pair consisting of the first modality image and the registered second modality image, including: The fused features are sent to the single-modal Mamba module for image reconstruction to obtain a spatial feature map; The spatial feature map is sequentially passed through two layers of convolution to gradually reduce the channels and a non-linear activation function to correct the local texture. The output feature vector is then mapped to the target image channels through a convolutional layer to output a high-quality fused image.

8. The method for joint registration and fusion of misaligned multimodal images based on the Mamba structure according to claim 6, characterized in that, In the multi-modal fusion module, each Multi-Modal Mamba module undergoes normalization, convolution, and parameter discretization. Finally, after a selective scanning mechanism, the visible light features, infrared features, and intermediate fusion features output by the current module are obtained and sent to the next module. The output of the last Multi-Modal Mamba module is modulated by a gate factor MLP and then added to the shallow fusion features processed by deep convolution to obtain the final fusion features.

9. The method for joint registration and fusion of misaligned multimodal images based on the Mamba structure according to claim 6, characterized in that, In the channel attention module, channel weights for different modalities are obtained through global average pooling, multilayer perceptron, and sigmoid activation function, and then multiplied and summed with their respective refined features to obtain the shallow fusion features.

10. The method for joint registration and fusion of misaligned multimodal images based on the Mamba structure according to any one of claims 5 to 7, characterized in that, The implementation principle of the unimodal Mamba feature extractor and the unimodal Mamba module is the same, including: After the input features are serialized and normalized, they are fed into different branches. Relevant features are extracted in each branch using a multilayer perceptron and are denoted as the first feature and the second feature, respectively. The first feature is passed through a convolutional block with a Sigmoid activation function and then enters a selective scanning mechanism to output global features; The second feature is multiplied by the global feature after passing through the Sigmoid activation function, and then added to the serialized feature after being modulated by the gate factor MLP to obtain the high-level feature and reassemble it back into the spatial distribution.

Images