A multi-modal medical image fusion method combining Mamba and frequency domain enhancement

Abstract

The application provides a kind of multi-modal medical image fusion method of combining Mamba and frequency domain enhancement, by stage-by-stage extraction modal in-context information and guide cross-modal interaction, fusion MRI and SPECT image, generate multi-scale fusion feature map.The method includes four main steps: obtaining MRI and SPECT image;Through the feature extraction and cross-modal interaction of fusion encoder structure;Encoder output feature map is input to fusion decoder, and layer-by-layer reconstruction and fusion are carried out through jump connection;The final fusion feature map is processed by output module, and the synthetic medical image is generated by using attention mechanism and channel enhancement strategy.The application can effectively enhance the synergistic expression of anatomical structure and functional metabolic information in multi-modal medical image fusion, improve the clarity and accuracy of image fusion, especially in low signal-to-noise ratio or inconsistent resolution.

Description

Technical Field

[0001] This invention belongs to the field of medical image fusion technology, specifically relating to a multimodal medical image fusion method that combines Mamba and frequency domain enhancement, suitable for integrating complementary information from different modalities of medical images to assist in clinical diagnosis and treatment planning. Background Technology

[0002] With the development of medical imaging technology, multimodal imaging techniques are playing an increasingly important role in clinical diagnosis and treatment evaluation. In particular, the combined application of magnetic resonance imaging (MRI) and single-photon emission computed tomography (SPECT) can provide anatomical and functional metabolic information respectively, offering complementary data support for comprehensive disease assessment. However, due to significant differences in imaging principles, resolution, and noise distribution among different modalities, how to fully integrate the heterogeneous information in multimodal images to obtain more discriminative fused images remains one of the key challenges in the field of multimodal medical image processing.

[0003] Traditional multimodal image fusion methods are mostly based on simple pixel-level, feature-level, or decision-level fusion strategies, lacking deep modeling of structural relationships and semantic complementarity between modalities. This often leads to structural distortion, blurred details, or missing functional information in the fused images. In recent years, deep learning methods have demonstrated strong feature extraction and representation capabilities in image fusion tasks, especially with the introduction of attention mechanisms, multi-scale fusion, and residual connections in the encoder-decoder structure, effectively improving the quality and robustness of fused images. Convolutional Neural Networks (CNNs) are widely used due to their powerful local feature extraction capabilities. Models such as TL_VGG19 achieve feature fusion through multiple convolutions. However, CNNs are limited by the size of their receptive field, making it difficult to capture long-distance dependencies and global contextual information. To address the global modeling shortcomings of CNNs, Transformers and their variants have been introduced into the field of medical image fusion. For example, FAT Fusion achieves cross-modal global feature interaction through a two-branch design, but it suffers from the loss of local details and high computational complexity.

[0004] To address the aforementioned contradictions, the Mamba architecture, based on the State Space Model (SSM), has recently achieved long sequence modeling with linear complexity, breaking through the efficiency bottleneck of the Transformer while maintaining global awareness. Furthermore, frequency domain transformation provides a new paradigm for multimodal feature interaction; by separating structural and functional information in the frequency space, it can effectively alleviate fusion conflicts caused by modal differences.

[0005] Therefore, there is an urgent need to design a multimodal medical image fusion method that combines state-space modeling capabilities with frequency domain feature interaction mechanisms to more effectively integrate the structural and functional information contained in MRI and SPECT images, while preserving key details such as image edges and textures, and improving the discriminative and diagnostic value of the fused images. Summary of the Invention

[0006] The purpose of this invention is to provide a multimodal medical image fusion method that combines Mamba structure and frequency domain enhancement strategy. It aims to fully explore the features of multimodal medical images and improve the accuracy and discrimination ability of image fusion through cross-modal feature interaction mechanism, so as to solve the problems of limited local feature capture ability of CNN, high computational complexity of Transformer and insufficient integration of multimodal information in existing methods.

[0007] The objective of this invention is achieved through the following technical solution: a multimodal medical image fusion method combining Mamba structure and frequency domain enhancement strategy, comprising the following steps: S1. Acquire MRI and SPECT images; S2. Input MRI and SPECT images into the fusion encoder structure, extract intramodal context information in stages and guide cross-modal interaction, and output fusion features and skip connection information; S3. Input the feature map output by the encoder into the fusion decoder, and combine it with skip connections to perform layer-by-layer reconstruction and fusion to generate a multi-scale fusion feature map. S4. Input the final fused feature map into the output module, and obtain the final synthetic medical image through attention mechanism and channel enhancement strategy.

[0008] Furthermore, step S1, acquiring MRI and SPECT images, includes acquiring two-dimensional MRI and two-dimensional SPECT images corresponding to the same anatomical region, respectively. The MRI image provides anatomical structural information, and the SPECT image provides functional metabolic information. This step, by strictly registering multimodal images of the same anatomical region, provides a data foundation for subsequent fusion and avoids fusion distortion caused by spatial location deviations.

[0009] Furthermore, the S2 encoder includes a two-stage fusion coding structure for extracting joint features from SPECT and MRI modalities in stages, while modeling structural dependencies between modalities and enhancing the collaborative representation of different functional and anatomical information. Compared with traditional single-stage coding, the two-stage structure can progressively improve the level of feature abstraction, achieving deep interaction between modalities while preserving detailed information.

[0010] Specifically, the first stage of the S2 encoder includes: constructing a local detail enhancement module, which captures long-distance dependencies within a modality through a Mamba-based state-space model (SSM) to enhance the perception of subtle structures such as lesion edges and organ boundaries.

[0011] Specifically, the second stage of the S2 encoder includes: constructing a frequency domain cross-modal enhancement module, mapping features to the frequency domain space through two-dimensional discrete cosine transform (DCT), and calculating the spectral attention matrix to realize intermodal feature interaction.

[0012] Furthermore, the S3 fusion decoder adopts a two-stage decoding design symmetrical to the encoder structure. Each stage sequentially includes a local detail enhancement module, a frequency domain cross-modal enhancement module, residual fusion and upsampling operations, and a skip connection enhancement and convolutional feature extraction process. By using skip connections to transfer the high-resolution features of the encoder to the decoder, the feature blurring problem of deep networks is effectively alleviated.

[0013] Furthermore, the S4 output module takes the final decoded and fused feature map as input, extracts long-range structural features within each modality through the local detail enhancement module, and inputs these features to the frequency domain cross-modal enhancement module for multimodal feature fusion. This module innovatively combines the advantages of spatial and frequency domain fusion, enabling the fused image to simultaneously possess structural clarity and functional information integrity.

[0014] Specifically, the output module includes a channel selection attention module, which generates channel attention weights (att_map) from the input SPECT image through adaptive pooling and a sigmoid activation function, and then performs channel-wise weighted fusion with the fusion features. This mechanism can adaptively highlight metabolically active regions.

[0015] The local detail enhancement module is built upon the Mamba state-space model structure. Through a combination of linear layers, one-dimensional convolutional layers, and SSM layers, it achieves a balance between local feature enhancement and global dependency modeling. Compared to the Transformer, the Mamba structure reduces the computational complexity from O(n²) to O(n). The frequency domain cross-modal enhancement module converts spatial domain features into frequency domain representations through DCT transformation, and utilizes the complementarity of different modes in frequency distribution to achieve feature enhancement.

[0016] The multimodal medical image fusion method combining Mamba and frequency domain enhancement proposed in this invention has the following advantages: This invention introduces a Local Detail Enhancement (LDE) module based on the Mamba state space model. While maintaining computational efficiency, it effectively models the structural dependencies between distant regions in MRI and SPECT images, enhances the ability to express key local features such as organ boundaries and lesion contours, and significantly improves the structural clarity and spatial consistency of the fused images.

[0017] This invention designs a frequency domain cross-modal enhancement module (FCME), which establishes a complementary modeling mechanism between modalities in the DCT frequency domain, fully explores the correspondence between structural and functional information in MRI and SPECT images, effectively alleviates fusion conflicts caused by modal differences, and improves the consistency and synergy of cross-modal fusion.

[0018] This invention employs a two-stage encoding and symmetric decoding structure, combined with a skip connection strategy, to integrate fusion features at different levels layer by layer, thereby improving the ability to restore image details and generating multi-scale fusion images with greater diagnostic value.

[0019] This invention uses a channel selection attention module (SCA) to perform channel-level reweighting on the fused feature map, which strengthens the feature representation of important regions such as lesion areas and functionally active areas, suppresses redundant background information, and effectively improves the saliency and discriminative ability of the fused image. Attached Figure Description

[0020] Figure 1 This is a flowchart of the multimodal medical image fusion method combining Mamba and frequency domain enhancement according to the present invention; Figure 2 This is a schematic diagram of the model structure of the multimodal medical image fusion method of the present invention; Figure 3 This is a schematic diagram of the Local Detail Enhancement (LDE) module in the multimodal medical image fusion method of the present invention; Figure 4 This is a schematic diagram of the frequency domain cross-modal enhancement module (FCME) of the multimodal medical image fusion method of the present invention; Figure 5 This is a schematic diagram of the Channel Selection Attention (SCA) module in the multimodal medical image fusion method of the present invention. Detailed Implementation

[0021] The technical solution of the present invention will now be described in detail through specific embodiments; Please see Figure 1-5 This invention provides a multimodal medical image fusion method combining Mamba and frequency domain enhancement, specifically including the following steps: S1. Acquire MRI and SPECT images: In this embodiment, two-dimensional MRI images and two-dimensional SPECT images corresponding to the same anatomical region are acquired. The MRI image contains strong anatomical boundary information, while the SPECT image provides functional metabolic response features. Both serve as modal input sources for subsequent image fusion.

[0022] S2. The specific steps for inputting MRI and SPECT images into the fusion encoder structure, extracting intramodal contextual information stage by stage, guiding cross-modal interaction, and outputting fusion features and skip connectivity information are as follows: (1) Constructing the Local Detail Enhancement (LDE) module: (1-1) Receiving MRI images and SPECT images First, the image is expanded into a sequence format to fit the sequence input requirements of Mamba blocks: (1-2) will , The input features are fed into two LDE modules respectively, and the input features of each modality are processed to model the contextual structural dependencies between geographically distant regions within a modality, thereby enhancing the structural awareness of features within a modality. (2) Constructing a frequency cross-modal enhancement module (FCME): Will and The input is fed into two symmetrical FCME modules, and a frequency enhancement strategy is used to achieve cross-fusion of features from different modes, mining complementary information in the frequency domain to obtain fused features. : (3) Coding feature fusion and residual connection generation: (3-1) Perform residual fusion of the fused features with the original intramodal features to form the first-stage skip connection features: (3-2) and The input features of the second-stage fusion encoder are generated through convolutional downsampling operations. and : (4) Repeat the above repeated encoding and fusion process: The second-stage fusion encoder is and Using this as input, the LDE structure modeling, FCME cross-modal frequency domain fusion, and residual enhancement strategies are repeatedly executed to finally obtain the fused coding features. and and its corresponding skip connection features and This is used in the subsequent decoder reconstruction process.

[0023] This step, by introducing Local Detail Enhancement (LDE) and Frequency Domain Cross-Modal Enhancement (FCME) modules, combined with a residual connectivity strategy and a two-stage cascaded structure design, achieves a step-by-step progression from intramodal structure modeling to intermodal feature interaction. The proposed fusion encoder not only fully exploits the complementary characteristics of MRI and SPECT images in terms of spatial structure and frequency domain information, but also effectively enhances the discriminative and expressive capabilities of the fused features, providing a high-quality, multi-scale feature foundation for subsequent decoding, reconstruction, and image synthesis.

[0024] S3. The fusion decoder adopts a two-stage decoding design symmetrical to the encoder structure. Each stage includes an LDE module, which is used to capture the contextual structural dependencies between regions that are far apart in the decoding process; an FCME module, which realizes feature interaction and supplementation between modalities; residual fusion and upsampling operations, which restore spatial resolution through transposed convolution; and skip connection enhancement and convolutional feature extraction processes, which are used to fuse high-resolution information passed from the encoding stage. The above processes output multi-scale fused feature maps in stages and input them to the output module to generate the final result.

[0025] This step, through a phased decoding modeling and skip enhancement strategy, can effectively maintain the structural continuity and modal consistency of the fused image, ensuring the final output achieves comprehensive optimization in resolution, structural clarity, and functional representation.

[0026] S4, Output module to finally decode and fuse feature maps and As input, long-range structural features within each modality are extracted using the LDE module. The data is then input into the FCME module for multimodal feature fusion to obtain the fused features. Then the input SEPECT image The input is fed into the Selective Channel Attention (SCA) module to obtain the channel attention weights att_map, and then compared with... By fusion To obtain the final MRI-SPECT fused image.

[0027] This step effectively improves the performance of the fused image in terms of boundary sharpness, functional response strength, and tissue structure consistency by integrating intramodal structure modeling, frequency domain interaction enhancement, and channel attention mechanism, ensuring that the final output image has high diagnostic reference value.

[0028] Additional notes: (1) The Local Detail Enhancement (LDE) module is built based on the Mamba state-space model and aims to enhance the perception of local features such as edges and textures, especially the subtle features of organ boundaries and lesions in medical images. It can effectively improve the expression and reconstruction of local details. By performing local enhancement and long-term dependency modeling on the input image, the LDE module can not only capture the morphological features of the target region more accurately, but also maintain the overall structural consistency of the image, avoiding information loss or distortion caused by modal differences during subsequent cross-modal fusion. Specifically, it includes: (1-1) The input feature F is processed through a linear layer to adjust its feature dimensions and extract its initial representation: Subsequently, A one-dimensional convolutional layer is input to extract local structural information, focusing on edges and textures in the image, which are crucial for the accurate localization of anatomical structures, thus obtaining local structural features. : Then, a non-linear mapping is performed using the SiLU activation function to enhance feature representation capabilities and generate locally enhanced features, which are used to strengthen the perception of local information such as edges and textures. Local enhancement features Input a State Space Model (SSM) layer to capture structural dependencies between geographically distant regions, and output a global feature tensor. : Meanwhile, original input features After processing with another linear layer and the SiLU activation function to adjust the response strength of different features, the gated feature tensor is obtained. ; (1-2) Convert the global feature tensor With gated feature tensor Element-wise multiplication preserves local details while enhancing the ability to identify lesions and organs, thereby effectively improving the accuracy of extracting important information from medical images; resulting in a feature tensor. : in, This represents element-wise multiplication; (1-3) will According to the preset number of groups Channel rearrangement is performed to obtain The calculation formula is as follows: Based on the generation channel attention weights The calculation formula is as follows: in, This indicates a global average pooling operation. Represents group convolution. This represents the Sigmoid activation function; The weights are then subjected to a channel-wise inverse rearrangement operation, which reverses the previous rearrangement steps to ensure their order matches the input. They are then multiplied element-wise with the input features to determine the contribution of different channels to the task, achieving channel-level feature reweighting. The calculation formula is as follows: (2) The SSM layer uses a zero-order preservation method to discretize the state-space equations to adapt to discrete image feature processing, which specifically includes: (2-1) Let the continuous-time state-space form be: in, For state variables, As input features, For output features, , , For the system matrix; (2-2) The zero-order preservation method is used to convert the above differential equation at time step. The system is then discretized to obtain the discrete state-space system parameters: in, For time-scale hyperparameters, Using the identity matrix, we obtain the discretized linear ordinary differential equation: (2-3) Based on the above discrete parameters, an implicit structured convolutional kernel K is constructed to obtain the serialized representation structure of the Mamba model: in, Given the input sequence, The output sequence is K, where K is the structured convolution kernel. This is a convolution operation.

[0029] (3) Frequency Domain Cross-Modal Enhancement Module (FCME) is used to guide the cross-fusion of features between the primary and secondary modalities, and to improve the modal collaborative modeling capability from the frequency domain perspective. In medical image fusion, different modalities often have different intensities in their expression of edge details and tissue morphology of the same lesion or structural region. Through frequency domain enhancement strategies, the FCME module can strengthen the cross-modal consistency modeling of structural boundaries, morphological features and lesion response regions. Frequency domain representation has the ability to focus and compress information. Especially in the case of low signal-to-noise ratio or inconsistent resolution, it can effectively improve the discriminability and robustness of image fusion, so that the fused image retains both structural clarity and functional information. Specifically, it includes: (3-1) Receive the main modal feature tensor and auxiliary modal feature tensor Channel expansion and depthwise convolution are performed on the two feature tensors respectively to obtain the query feature and key-value feature, ensuring that modal characteristics are preserved while uniformly representing the dimensionality, wherein: By performing a two-dimensional discrete cosine transform (DCT) on Q and K and mapping them to the frequency domain, the differences in frequency distribution between modes are highlighted, the boundary recognition capability is enhanced, and more discriminative information is provided for subsequent attention mechanisms. Specifically: (3-2) Calculate the frequency domain attention matrix based on the spectral attention mechanism: At the same time, through and Element-wise multiplication yields the initial frequency domain interaction features: (3-3) The attention matrix Acting on initial frequency domain interaction features Through frequency weighting, it can highlight edges, eliminate noise, enhance the expression of lesion areas or important anatomical structures, and reconstruct fusion features through inverse discrete cosine transform (IDCT). Channel fusion and frequency domain residual enhancement are performed on the reconstructed features to ensure that the final output features retain image details while improving the consistency and expression accuracy of cross-modal features, resulting in the final output features: Among them, Learnable fusion parameters This indicates element-wise multiplication.

[0030] (4) The Channel Selection Attention Module (SCA) generates channel-based attention weights to selectively focus on and enhance the expression of lesions and functionally active areas while suppressing irrelevant background information. Specifically, it includes the following steps: (4-1) For the input feature map Perform adaptive pooling to compress it into a single-dimensional channel vector: (4-2) Compressed channel features Flattened into a sequence, it is then input into a linear layer: in, For the embedded dimension; (4-3) Apply the Sigmoid activation function to obtain the channel attention weights s: The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A multimodal medical image fusion method combining Mamba and frequency domain enhancement, characterized in that, Includes the following steps: S1. Acquire MRI and SPECT images; S2. Input MRI and SPECT images into the fusion encoder structure, extract intramodal context information in stages and guide cross-modal interaction, and output fusion features and skip connection information; S3. Input the feature map output by the encoder into the fusion decoder, and combine it with skip connections to perform layer-by-layer reconstruction and fusion to generate a multi-scale fusion feature map. S4. Input the final fused feature map into the output module, and obtain the final synthetic medical image through attention mechanism and channel enhancement strategy.

2. The method according to claim 1, characterized in that, The S1 acquisition of MRI and SPECT images includes: acquiring two-dimensional MRI images and two-dimensional SPECT images corresponding to the same anatomical region, respectively. The MRI images provide anatomical structural information, and the SPECT images provide functional metabolic information for subsequent fusion processing.

3. The method according to claim 1, characterized in that, The S2 encoder includes a two-stage fusion coding structure for extracting joint features from SPECT and MRI modalities in stages, while modeling structural dependencies between modalities and enhancing the collaborative representation of different functional and anatomical information. Specifically, it includes the following steps: (1) Constructing a local detail enhancement module: The input features of each modality are processed through the local detail enhancement module to model the contextual structure dependency between regions that are far apart within the modality; (2) Construct a frequency domain cross-modal enhancement module: Achieve cross-fusion of different modal features through frequency enhancement strategies and mine complementary information in the frequency domain; (3) Coding feature fusion and residual connection generation: The fused features are residually fused with the original intramodal features to form skip connection features; (4) Repeated encoding and fusion process: The second-stage fusion encoder performs the same processing flow to obtain the fusion encoding features and their corresponding skip connection features.

4. The method according to claim 1, characterized in that, The S3 fusion decoder adopts a two-stage decoding design symmetrical to the encoder structure. Each stage includes a local detail enhancement module, a frequency domain cross-modal enhancement module, residual fusion and upsampling operations, and a skip connection enhancement and convolutional feature extraction process.

5. The method according to claim 1, characterized in that, The S4 output module takes the final decoded fusion feature map as input, extracts long-range structural features within the modality through the local detail enhancement module, inputs them to the frequency domain cross-modal enhancement module for multimodal feature fusion, and then inputs the input SPECT image to the channel selection attention module to obtain the channel attention weight att_map, and fuses it with the fusion features to obtain the final MRI-SPECT fusion image.

6. The method according to claim 2, characterized in that, The local detail enhancement module is built based on the Mamba state-space model and aims to enhance the perception of local features such as edges and textures. Specifically, it includes: (1) The input features are initially represented by linear layers and one-dimensional convolutional layers, and after being processed by the SiLU activation function, they are input into the SSM layer to capture global features; (2) Multiply the global feature tensor and the gated feature tensor element by element to obtain the fused feature tensor; (3) The fused feature tensor is rearranged through channels to generate channel attention weights and multiplied element-wise with the input features.

7. The method according to claim 5, characterized in that, The SSM layer employs a zero-order preservation method to discretize the state-space equations to adapt to discrete image feature processing, including: (1) Let the continuous-time state-space form be: in, For state variables, For input features, For output features, , , For the system matrix; (2) The zero-order preservation method is used to convert the above differential equation at time step. Discretization is then performed. (3) Based on the above discrete parameters, an implicit structured convolution kernel K is constructed to obtain the serialized representation structure of the Mamba model: 。 8. The method according to claim 2, characterized in that, The frequency domain cross-modal enhancement module is used to guide the cross-fusion of features between the primary and auxiliary modalities, improving the modal collaborative modeling capability from a frequency domain perspective, specifically including: (1) Receive the main modality feature tensor and the auxiliary modality feature tensor, and perform channel expansion and depth convolution on the two feature tensors respectively; (2) Map the features to the frequency domain space using two-dimensional discrete cosine transform and calculate the spectral attention matrix; (3) Frequency domain feature reconstruction based on attention weighting, and fused features obtained by inverse discrete cosine transform; (4) Perform channel fusion and frequency domain residual enhancement to output cross-modal fusion features.

9. The method according to claim 4, characterized in that, The channel selection attention module generates channel-based attention weights to selectively focus on and enhance the expression of lesions and functionally active areas, specifically including: (1) Perform adaptive pooling on the input feature map to compress it into a single spatial dimension channel vector; (2) Flatten the compressed channel features into a sequence form and then input them into a linear layer; (3) Apply the Sigmoid activation function to obtain the channel attention weights.

Images