Multi-modal image fusion method of coupling frequency domain Mangban

Abstract

The invention discloses a multi-modal image fusion method for coupling frequency domain Mangban. The method comprises the following steps: acquiring a multi-modal source image data set; a fusion network model comprising a generator and a discriminator is constructed, the generator obtains an initial feature map through a feature extraction module, frequency domain attention enhancement and double-path optimization processing are performed on features by using a frequency domain perception Mangbar module, cross-modal feature integration is realized through a Mangbar-based feature fusion module, and a fusion network model is constructed. Outputting a fused image through an image reconstruction module; and designing a loss function including generator loss and dual-discriminator loss to train the network. Through collaborative design of a frequency attention mechanism and a double-path structure, the frequency domain feature extraction capability and cross-modal information interaction efficiency of the model are effectively improved, and the global semantic integrity and detail richness of the fused image are remarkably enhanced.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to a multimodal image fusion method with coupled frequency domain Mamba. Background Technology

[0002] Image fusion technology integrates basic information from multiple source images to generate high-quality fused images, and is widely used in many fields such as multimodal image fusion, digital photographic image fusion, and remote sensing image fusion. Convolutional Neural Networks (CNNs), as a commonly used image fusion method, can effectively extract local features, but their limited receptive field makes it difficult to capture global information, which poses a challenge in generating high-quality fused images in multimodal image fusion. Transformers, through their self-attention mechanism, can effectively model global information, especially performing well in handling long-term dependent tasks; however, their computational complexity is high, and the computational cost increases quadratically with the length of the input sequence, resulting in slow training and inference speeds, limiting their application in large-scale image fusion tasks.

[0003] To overcome these limitations, Mamba networks, based on selective state-space models, emerged. Mamba networks can adapt to input data and effectively model global information, reducing computational cost and improving inference speed while maintaining linear complexity, demonstrating excellent performance in natural language processing and computer vision tasks. However, Mamba networks also have some shortcomings. For example, while they can capture global information, they may not fully consider the differences in frequency domain information between different modalities when processing multimodal data. Frequency domain information is crucial for image fusion because it reflects the texture and detail features of an image. Therefore, the limitations of Mamba networks in processing frequency domain information may lead to poor performance in detail preservation and contrast enhancement of the fused image. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention proposes a multimodal image fusion method using coupled frequency domain Mamba, thereby resolving the issues present in the prior art.

[0005] To achieve the above objectives, the present invention provides a multimodal image fusion method coupled with frequency domain Mamba, comprising:

[0006] Obtain a multimodal source image dataset for training;

[0007] An image fusion network model is constructed, which includes a generator and a discriminator. The generator includes a feature extraction stage, a feature interaction stage, a feature fusion stage, and an image reconstruction stage.

[0008] Construct a loss function that includes generator loss and discriminator loss, and train the image fusion network model based on the multimodal source image dataset and the loss function;

[0009] The multimodal source images to be fused are input into the trained image fusion network model, and the fused image is output in real time.

[0010] Optionally, the feature extraction stage processes the input multimodal source image through a convolutional layer to obtain preliminary features of each modality; and enhances the preliminary features through a Mamba module to obtain low-level features of each modality.

[0011] Optionally, the low-level features of each modality and the shallow fusion features obtained by the Mamba-based feature fusion module are concatenated; the concatenated feature map is downsampled to obtain the downsampled features corresponding to each modality; the downsampled features are sent to the feature interaction stage and processed by the frequency domain perception Mamba module to obtain the deep features corresponding to each modality.

[0012] The frequency domain sensing Mamba module consists of two symmetrical frequency Mamba blocks. Each frequency Mamba block sequentially performs frequency attention enhancement processing and feature optimization processing based on a dual-path structure on the input downsampled features, and outputs deep features of each modality.

[0013] Optionally, the frequency attention enhancement process includes:

[0014] The input features are subjected to average pooling and max pooling, and the results are concatenated to generate pooled fusion features.

[0015] Perform a Fast Fourier Transform on the pooling fusion features to decompose them into frequency domain amplitude information and frequency domain phase information;

[0016] A multi-head self-attention mechanism is applied to the frequency domain amplitude information and the frequency domain phase information respectively to generate enhanced amplitude information and enhanced phase information;

[0017] The enhanced amplitude information is combined with the enhanced phase information and then processed by inverse fast Fourier transform to obtain the enhanced feature map.

[0018] After performing a convolution operation on the enhanced feature map, it is multiplied element-wise with the input features to generate spatial attention-weighted features;

[0019] The spatial attention-weighted features are added to the input features via residual connections to output frequency-domain enhanced features.

[0020] Optionally, the feature optimization process based on the dual-path structure includes:

[0021] The frequency domain enhancement features are subjected to layer normalization and linear transformation to obtain transformed features. The transformed features are then segmented along the channel dimension into first path features and second path features. The first path features are processed sequentially through a convolutional layer, a 2D selective scanning mechanism, and the frequency attention enhancement module to obtain first path output features. The second path features are processed through the SiLU activation function to obtain second path output features. The first path output features and the second path output features are linearly fused to obtain interactive enhancement features. The interactive enhancement features and the frequency domain enhancement features are added together through skip connections to obtain deep features for each modality.

[0022] Optionally, the feature fusion stage uses a Mamba-based feature fusion module to perform preliminary cross-modal integration of the low-level features of each modality obtained in the feature extraction stage; the features obtained from the cross-modal integration are then concatenated with the deep features of each modality obtained in the feature interaction stage, and the concatenated features are then deeply integrated across modalities using the Mamba-based feature fusion module to generate the final fused feature map.

[0023] Optionally, the processing steps of the Mamba-based feature fusion module include:

[0024] The input features are sequentially normalized and linearly transformed. The linearly transformed features are divided into two parts along the channel dimension and input into the first branch and the second branch, respectively. The first branch extracts local spatial features through a convolutional layer and then integrates global information through 2D selective scanning. The second branch retains the nonlinear distribution of features through the SiLU activation function. The output features of the first branch and the second branch are multiplied element-wise to achieve feature interaction. The interactive features are transformed by a linear layer and then fused with the input features through residual connections.

[0025] Optionally, the image reconstruction module processes the fusion features using a Mamba module to obtain fusion features with enhanced local expressive power; it then processes the fusion features with enhanced local expressive power using an upsampling module to obtain a high-resolution feature map; finally, it performs serialization processing on the high-resolution feature map to obtain the final fusion image.

[0026] Optionally, the generator loss includes content loss and adversarial loss, wherein the content loss includes structural similarity loss, mean squared error loss, and edge information loss.

[0027] Optionally, a dual-discriminator structure is used to calculate the discriminator loss, with both discriminators using the same loss function, expressed as follows:

[0028] ;

[0029] in, It is the loss function of the first discriminator. It is the loss function of the second discriminator; Represents the source image. This is the final fused image generated by the generator, where D represents the discriminator.

[0030] Compared with the prior art, the present invention has the following advantages and technical effects:

[0031] This invention utilizes a frequency-domain-aware Mamba module to deeply extract and enhance the frequency-domain features of each modality, accurately capturing frequency-domain information. Simultaneously, it leverages a cross-modal interaction mechanism to achieve dynamic information fusion, avoiding information loss. This design not only enhances the model's understanding of single-modal features but also improves its ability to recognize and process complex scenes through intermodal synergy. Furthermore, its flexibility and adaptability allow it to adaptively optimize according to different tasks and data characteristics, resulting in outstanding performance in multimodal image processing tasks and significantly improving the global semantic integrity and detail richness of fused images. This invention also designs a carefully crafted loss function that promotes effective adversarial interaction between the generator and the adversarial unit (discriminator). This adversarial training strategy not only improves the quality of images generated by the generator but also enhances the model's understanding and fusion capabilities of multimodal image features. The generator is trained to generate more realistic and detailed images, while the discriminator is trained to more accurately distinguish between real and generated images, thereby driving overall model performance improvement.

[0032] The frequency-domain perceptual Mamba module and its accompanying adversarial training strategy of this invention demonstrate outstanding performance and innovation in the field of multimodal image fusion. Compared with existing technologies, it not only achieves breakthroughs in frequency feature extraction and cross-modal information fusion, but also further improves the quality of fused images and the robustness of the model through a carefully designed loss function and generative adversarial network (GAN) architecture. This comprehensive optimization method results in fused images that outperform traditional methods in both global semantic integrity and detail richness, providing a more efficient and accurate solution for multimodal image processing tasks, with broad application prospects and significant practical value. Attached Figure Description

[0033] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0034] Figure 1 This is an overall framework diagram of an embodiment of the present invention. Detailed Implementation

[0035] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0036] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0037] Example 1

[0038] This embodiment applies to road environment perception scenarios in intelligent transportation systems. Addressing the problem that single-modal images (such as RGB or infrared) are insufficient for accurately identifying road targets (e.g., pedestrians, vehicles, obstacles) under nighttime or low-visibility conditions, a multimodal image fusion method coupled with frequency-domain Mamba is proposed. By fusing RGB images acquired by an onboard visible light camera with thermal infrared images acquired by an infrared thermal imager, the accuracy and robustness of target detection in complex environments are significantly improved, thereby enhancing the environmental perception capabilities and safety of intelligent driving systems. The Mamba module refers to the visual state space model in "VMamba: Visual State Space Model." This model decomposes global dependencies into four linear recursions through a loop of "four-way scanning → selective state update → two-dimensional folding," and then fuses local convolutional enhancements, achieving visual modeling that "reduces linear complexity + increases the global receptive field + eliminates positional encoding."

[0039] Mamba utilizes a state-space model for global scanning and modeling, boasting high computational efficiency and strong long-sequence modeling capabilities, excelling in long text and complex sequence tasks. However, it still has shortcomings in information interaction and frequency domain processing. For example, while its low-rank attention mechanism and recursive state update method improve computational efficiency, they may result in weaker information interaction capabilities, easily losing some important information. Furthermore, in frequency domain processing tasks, Mamba's core structure—the selective state-space model—lacks a dedicated mechanism for handling frequency domain information, making it difficult to efficiently capture the frequency domain features of signals. Therefore, its performance may be unsatisfactory in scenarios requiring precise frequency domain analysis.

[0040] To address these issues, this embodiment proposes a Coupled Frequency Mamba, an improved network module that integrates frequency domain processing mechanisms with the Mamba state-space model. Building upon the linear sequence modeling capabilities of the traditional Mamba model, it introduces frequency domain awareness and cross-modal coupling mechanisms. This retains the Mamba's advantage of efficiently processing long sequences and large-scale features while overcoming its shortcomings in frequency domain feature capture and cross-modal information interaction. The core design idea of ​​the Coupled Frequency Mamba is to deeply couple frequency domain processing with the Mamba model to compensate for the shortcomings of traditional Mamba frequency domain modeling and cross-modal interaction: frequency domain feature enhancement is achieved through a frequency attention enhancement module. The method includes:

[0041] Step 1: Prepare the training dataset by selecting two different modalities of the same scene (e.g., infrared images and visible light images).

[0042] Step 2: Construct the network model, which includes a generator and a discriminator. The generator extracts and fuses features from images of different modalities to generate a high-quality fused image. The discriminator learns the feature differences between real and generated images, outputting a probability value to determine whether the input image is real or generated. The two collaborate and constrain each other during adversarial training, ultimately ensuring that the fused image generated by the generator is highly consistent with the real image in terms of detail, structure, and semantic information, thus achieving high-quality multimodal image fusion.

[0043] The generator consists of four stages: feature extraction, feature interaction, feature fusion, and feature reconstruction.

[0044] First, the feature extraction stage includes a convolutional module and a Mamba module.

[0045] In the feature extraction stage, this embodiment uses two input images. and Convolutional layers are applied separately to obtain their preliminary features. The purpose of this step is to capture basic visual features in the image, such as edges and textures. To further enhance the expressive power of these features, this embodiment also introduces a Mamba module to enhance the preliminary features. The core of the Mamba module is to utilize a state-space model to enhance the input feature map, compressing information by mapping the input sequence and using a selective scanning algorithm to accelerate processing. This design not only improves the discriminative power of the features but also provides richer information for subsequent feature fusion and reconstruction stages. After further extraction of low-level features by the Mamba module... and .

[0046] Next, the low-level features obtained in the feature extraction stage will be processed. and The cross-modal features obtained from the Mamba-based feature fusion module in the feature fusion stage are concatenated. Next, a downsampling module is used to downsample the feature map to obtain the features. and Subsequently, the downsampled features are further fed into the frequency domain-aware Mamba module in the feature interaction stage to extract more localized, deeper feature information. and This allows for a more refined representation of image features.

[0047] The feature interaction phase primarily consists of a frequency-aware Mamba module. This module comprises two frequency Mamba blocks and employs a symmetrical structural design. This design allows the model to more effectively perform feature enhancement and information fusion when processing multimodal inputs. Although both branches utilize the frequency-aware Mamba module, the interaction mechanism differs for each branch. In the infrared image branch, infrared feature information plays a dominant role, while visible light feature information serves as auxiliary; conversely, in the visible light image branch, visible light feature information is dominant, and infrared feature information is auxiliary. This design enables the model to dynamically adjust the weights of feature interactions based on the input modality, thereby achieving more effective feature extraction and information fusion.

[0048] Each frequency mamba block focuses on processing the frequency features of one modality. Through steps such as frequency domain decomposition, frequency feature extraction, frequency feature enhancement, and inverse transformation, it accurately captures and strengthens frequency information. The frequency attention enhancement module is the core module of the frequency mamba; it is responsible for deep processing of features in the frequency domain, capturing long-range dependencies of frequency information, and further enhancing frequency domain features through a multi-head self-attention mechanism. The cross-modal interaction mechanism allows for the dynamic fusion of frequency features from two modalities, achieving deep integration and optimization of information through feature interaction, collaborative enhancement, and skip connections. This design not only effectively improves the performance of multimodal image fusion but also makes the fused image more outstanding in terms of global semantics and detail representation. Its specific implementation is as follows:

[0049] The frequency attention enhancement module is the core of the Frequency Mamba. During the cross-scanning within the Mamba, to enable the model to more accurately identify and understand the spatial distribution of different frequency information in the image, frequency attention enhancement is first applied to the input 2D selective scan information. This preprocessing helps the model more effectively integrate pixel information from various directions of the image in subsequent steps. First, average pooling (AvgPool) and max pooling (MaxPool) are performed on the 2D selective scan input information to obtain two spatially reduced-dimensional pooling features. These two pooling features are then concatenated into a single feature. As shown in formula (1).

[0050] (1)

[0051] Then, the fusion features Perform a Fast Fourier Transform (FFT) to obtain the frequency domain characteristic amplitude information. and phase information Multi-head self-attention mechanisms are applied to each of them to make them pay more attention to important information in the frequency domain. and ), as shown in formula (2).

[0052] (2)

[0053] The enhanced amplitude and phase characteristics are combined to form a new frequency domain signal. The frequency domain signal is then restored to the time domain using an inverse FFT transform to obtain the enhanced feature map. As shown in formula (3).

[0054] (3)

[0055] Convolutional operations are performed on the feature map to further extract features. Then, element-wise multiplication is performed between the processed feature map and the original input feature F to implement a spatial attention mechanism. Finally, the original features and the attention-enhanced features are added together through residual connections to obtain the final output features. As shown in formula (4).

[0056] (4)

[0057] The frequency attention enhancement module extracts frequency information of features through frequency domain transformation, enhances the frequency domain features using a multi-head attention mechanism, and finally fuses the enhanced features into the original features through inverse transformation and convolution operations, so that the model can more accurately understand the spatial distribution of different frequency information in the image.

[0058] In the frequency domain sensing Mamba module, the input features are first processed separately. and Layer normalization (LN) is performed to ensure the consistency of feature distribution. Next, the normalized features are transformed through a linear layer to adapt to subsequent processing steps. Then, the transformed features are split into two parts along the channels and processed independently. The first half of the channel features undergoes spatial convolution and frequency attention enhancement to achieve local-global information mining and enhancement in the spatial-frequency domain, while the second half of the channel features undergoes SiLU activation. Finally, the two processed feature parts are enhanced through bi-branch cross-multiplication and linear mapping to achieve interactive perceptual enhancement. Skip connections are then used to integrate the enhanced features with the original two feature parts, thus obtaining the output features of the frequency domain perceptual Mamba module. Specifically, this includes:

[0059] A portion of the features are first processed for local spatial information through convolutional layers (Conv), and then enhanced for global frequency domain information through a 2D selective scanning mechanism and a frequency attention enhancement module. That is, convolution and 2D selective scanning are first performed to obtain feature F, and then this part of the frequency attention enhancement operation is performed on F to obtain the output feature. and As shown in formula (5).

[0060] (5)

[0061] Another part of the features is obtained by nonlinear transformation using the SiLU activation function. and To enhance the nonlinear expressive power of features, as shown in formula (6).

[0062] (6)

[0063] These two feature sets are then enhanced through bi-branch cross-multiplication and linear mapping to achieve interactive enhancement of the bi-branch features. Furthermore, to preserve valuable original information, the enhanced features are compared with the original features. and By integrating the data through skip connections, the output features of the frequency domain sensing module are finally obtained. and This not only enhances the expressive power of bimodal features, but also effectively maintains the integrity of each modality's information, as shown in formula (7).

[0064] (7)

[0065] in, and Corresponding to each as follows Figure 1 The output of the frequency domain sensing module of the dual-modal feature extraction branch is shown.

[0066] By employing this frequency-domain-aware Mamba module design, the model can gain a deeper understanding and extract more profound information from the input features, thereby improving its performance in multimodal image processing tasks. This feature extraction mechanism not only enhances the model's understanding of single-modal features but also improves its ability to recognize and process complex scenes through intermodal interactions.

[0067] In the feature fusion stage, this embodiment proposes a Mamba-based feature fusion module to fuse the feature information extracted from the feature stream layer by layer. The model first... and The initial integration of feature maps is achieved through a Mamba-based feature fusion module. This step combines information from different feature streams to extract and fuse preliminary cross-modal features. Next, the initially fused cross-modal features will be... Deep features obtained from frequency domain sensing Mamba modules and The deep features obtained after stitching are then further fused using a Mamba-based feature fusion module. This helps the model capture the overall structure and contextual information of the image, thereby understanding the image content at a higher level. Through this multi-layered feature integration and processing from shallow to deep layers, the feature fusion stage achieves comprehensive integration of feature information, enhances the expressive power of the features, and results in a more refined final fused feature. It can capture more comprehensive details and structural information of images.

[0068] Specifically, taking the shallow fusion stage as an example, in the Mamba-based feature fusion module, this module consists of two symmetrical processing paths. Each path includes layer normalization, linear transformation, feature segmentation, parallel branch processing, and a cross-path feature fusion unit. The input features are first normalized, and then mapped to a high-dimensional space through a linear transformation. The feature segmentation operation divides the channel dimension into two parts, which are input into the two parallel branches respectively. For branch A, the input features are first processed... and Local spatial features are extracted through convolutional layers, and then global information is integrated through 2D selective scanning. and The specific formula is shown in equation (8).

[0069] (8)

[0070] Branch B uses the SiLU activation function to preserve the nonlinear distribution of features and reduces computational overhead through skip connections, as shown in Equation (9).

[0071] (9)

[0072] Subsequently, the output features of the two branches are interacted through element-wise multiplication to enhance the semantic correlation between channels. Then, the outputs of these two processing paths are transformed through a linear layer and combined with the original input features. and Perform residual connections to obtain output features. and As shown in Equation (10). This design not only helps to form a stable gradient backpropagation channel, thereby improving the stability of training, but also improves the efficiency of training while maintaining model performance. In this way, the model can learn and fuse key information from different modalities more effectively, thereby achieving better performance in various tasks.

[0073] (10)

[0074] Final shallow fusion output features The original input channel structure is preserved, while spatial and channel enhancement information across paths is fused, as shown in Equation (11).

[0075] (11)

[0076] This embodiment proposes a feature fusion module based on Mamba, which aims to achieve efficient and stable feature enhancement and fusion through multi-path parallel processing and feature interaction enhancement, thereby improving the model's performance in complex scenarios.

[0077] Finally, image reconstruction is performed. The model first processes the feature maps using a Mamba module to enhance their local expressive power. Then, an upsampling module is used to progressively convert the low-resolution feature maps into high-resolution images. The upsampled feature maps are then processed by a sequence module to generate the final fused image. Finally, the reconstructed image is fed into a discriminator for evaluation. The discriminator learns the feature differences between the real and generated images and outputs a probability value to determine the authenticity of the input image. This process not only improves the quality of the fused image but also enhances the model's understanding and processing capabilities for images with different modalities.

[0078] The Mamba module originates from "VMamba: Visual State Space Model," and its core structure is a visual state space model. This model breaks down global dependencies into four linear recursions through a loop of "four-way scanning → selective state update → two-dimensional folding," and then integrates local convolutional enhancements to achieve visual modeling that "reduces linear complexity, increases the global receptive field, and eliminates positional encoding."

[0079] Step 3, Design the Loss Function: During GAN training, the generator and discriminator compete against each other through adversarial training. The generator attempts to generate increasingly realistic images to deceive the discriminator, while the discriminator tries to improve its ability to distinguish between real and generated images. This adversarial process drives the generator to continuously improve, eventually generating high-quality, realistic images. The generator loss and discriminator loss are key to achieving this goal, jointly defining the optimization objective of network training. The loss function mainly consists of two parts: the generator loss and the dual discriminator loss. The total loss can be defined as... As shown in formula (12).

[0080] (12)

[0081] in, It is the loss function of the generator. It is the loss function of the first discriminator. It is the loss function of the second discriminator.

[0082] Generator loss includes content loss. and combat losses .

[0083] (13)

[0084] The adversarial loss can be expressed as shown in formula (14).

[0085] (14)

[0086] in, This is the final fused image generated by the generator. and Representative discriminator, Let be the mathematical expectation.

[0087] Content loss measures the similarity between the generated image and the target image in terms of content; it includes structural similarity loss (…). Mean squared error loss ( and edge information loss () As shown in formula (15).

[0088] (15)

[0089] In content loss, structural similarity loss reflects the degree of similarity between the fused image and the source image. Structural similarity loss can be expressed as shown in Equation (16).

[0090] (16)

[0091] in, This indicates the calculation of image similarity. These are the weighting coefficients.

[0092] Mean squared error loss constrains the similarity between images at the pixel level. The mean squared error loss can be expressed as shown in Equation (17).

[0093] (17)

[0094] Edge strength loss (as shown in Equation (18)) measures the richness of texture details in an image by calculating the edge strength of the fused image.

[0095] (18)

[0096] in, This indicates the loss of edge information. and Representing the fused image I f The gradients in the x and y directions. The Euclidean norm represents the gradient, which is the edge strength of a pixel.

[0097] In this embodiment, a dual discriminator is used to calculate the loss function of the discriminator so that the fused image has a high fidelity with the source image. The two discriminators use the same loss function, which can be expressed by the formula (19).

[0098] (19)

[0099] in, Let D represent the source image and D represent the discriminator.

[0100] Step 4, Training the Network Model: The model is iteratively trained on the training set. For each batch, a loss function value is calculated, and the network weights are updated based on these values. Simultaneously, we periodically evaluate the model's performance on the validation set to monitor for overfitting and underfitting, and adjust hyperparameters such as the learning rate and batch size accordingly. After training, we perform a final evaluation on the test set, analyzing the quality of the fused images and using appropriate metrics to measure model performance. If the evaluation results are unsatisfactory, we need to go back and adjust the network architecture, loss function, or training process. Through hyperparameter tuning, we further optimize the model's performance until a satisfactory level is achieved.

[0101] Step 5, determine the fusion model: After the fusion network is trained, fix the network model parameters and determine the network model as the final fusion network model; when infrared and visible light image fusion is needed, directly input the source image into the final fusion network to obtain the fused image.

[0102] The following is a quantitative comparison of the present invention with eight other methods on the Road Scene dataset and the TNO dataset.

[0103] Step 1: Prepare the training dataset. Select two datasets from the same scene, representing two different modalities: the Road Scene dataset and the RGB-NIR dataset. Use 180 pairs of images from the Road Scene dataset as the training set, and the remaining 40 pairs for testing the model's performance. Select 40 pairs from the RGB-NIR dataset for testing the model's performance.

[0104] Step 2: Construct the network model, which includes a generator and a discriminator. The generator extracts and fuses features from images of different modalities to generate a high-quality fused image. The discriminator learns the feature differences between real and generated images, outputting a probability value to determine whether the input image is real or generated. The two collaborate and constrain each other during adversarial training, ultimately ensuring that the fused image generated by the generator is highly consistent with the real image in terms of detail, structure, and semantic information, thus achieving high-quality multimodal image fusion.

[0105] The generator comprises four stages: feature extraction, feature interaction, feature fusion, and feature reconstruction. First, feature extraction is achieved through convolutional layers to capture preliminary features of the input image. The frequency-aware Mamba module consists of two frequency Mamba blocks, each focusing on processing the frequency features of one modality. Through steps such as frequency domain decomposition, frequency feature extraction, frequency feature enhancement, and inverse transformation, frequency information is accurately captured and strengthened. The frequency attention enhancement module is its core module, responsible for deep processing of features in the frequency domain, capturing long-range dependencies of frequency information, and further enhancing frequency domain features through a multi-head self-attention mechanism. The cross-modal interaction mechanism allows for dynamic fusion of frequency features from two modalities, achieving deep integration and optimization of information through feature interaction, collaborative enhancement, and skip connections. This design not only effectively improves the performance of multimodal image fusion but also makes the fused image superior in global semantics and detail representation. Its specific implementation is as follows:

[0106] The frequency attention enhancement module is the core of the frequency domain-aware Mamba module. Before performing the cross-scan within the Mamba, to enable the model to more accurately identify and understand the spatial distribution of different frequency information in the image, frequency attention enhancement is first applied to the input 2D selective scan information. This preprocessing helps the model more effectively integrate pixel information from various directions of the image in subsequent steps. First, average pooling (AvgPool) and max pooling (MaxPool) are performed on the information after the 2D selective scan, respectively, to obtain two spatially reduced-dimensional features. Then, the two pooled features are fused into one feature. As shown in formula (1).

[0107] Then, the fusion features Performing a Fast Fourier Transform (FFT) yields the frequency domain feature amplitude and phase information as shown in the formula above. Then, a multi-head self-attention mechanism is applied to make it focus more on important information in the frequency domain. and ), as shown in formula (2).

[0108] The enhanced amplitude and phase characteristics are combined to form a new frequency domain signal. The frequency domain signal is then restored to the time domain using an inverse FFT transform to obtain the enhanced feature map. As shown in formula (3).

[0109] Convolutional operations are performed on the feature map to further extract features. Then, element-wise multiplication is performed between the processed feature map and the original input feature F to implement a spatial attention mechanism. Finally, the original features and the attention-enhanced features are added together through residual connections to obtain the final output features. As shown in formula (4).

[0110] The frequency attention enhancement module extracts frequency information of features through frequency domain transformation, enhances the frequency domain features using a multi-head attention mechanism, and finally fuses the enhanced features into the original features through inverse transformation and convolution operations, so that the model can more accurately understand the spatial distribution of different frequency information in the image.

[0111] In the frequency domain-aware Mamba module, the input features are first processed... and and Layer normalization (LN) is performed to ensure the consistency of feature distribution. Next, a linear layer transforms the LN-normalized features to obtain linearly mapped features. Then, the linearly mapped features are split into two parts and processed independently. One part of the features first undergoes local spatial information processing through a convolutional layer (Conv), and then global frequency domain information enhancement is performed through a 2D selective scanning mechanism and a frequency attention enhancement module to obtain the output features. and As shown in formula (5).

[0112] Another part of the features is obtained by nonlinear transformation using the SiLU activation function. and To enhance the expressive power of the model, as shown in formula (6).

[0113] These two feature sets are then enhanced through bi-branch cross-multiplication and linear mapping to achieve interactive enhancement of the bi-branch features. Furthermore, to preserve valuable original information, the enhanced features are compared with the original features. and By integrating the data through skip connections, the output features of the dual-modal branch frequency domain sensing Mamba module are finally obtained. and While maintaining the integrity of information, the expressive power of bimodal features is enhanced, as shown in formula (7).

[0114] By employing this frequency-domain-aware Mamba module design, the model can gain a deeper understanding and extract more profound information from the input features, thereby improving its performance in multimodal image processing tasks. This feature extraction mechanism not only enhances the model's understanding of single-modal features but also improves its ability to recognize and process complex scenes through intermodal interactions.

[0115] In the feature fusion stage, the model first performs preliminary integration of feature maps at a shallow level using a Mamba-based feature fusion module. This step combines information from different feature streams to extract and fuse preliminary cross-modal features. Following this, the preliminarily fused cross-modal features are... Deep features obtained from frequency domain sensing Mamba modules and The deep features obtained after stitching are then further fused using a Mamba-based feature fusion module. This helps the model capture the overall structure and contextual information of the image, thereby understanding the image content at a higher level. Through this multi-layered feature integration and processing from shallow to deep layers, the feature fusion stage achieves comprehensive integration of feature information, enhances the expressive power of the features, and enables the final fused features to more comprehensively capture the details and structural information of the image.

[0116] Finally, image reconstruction is performed. The model first processes the feature maps using Mamba blocks to enhance their local expressive power. Then, an upsampling module is used to progressively convert the low-resolution feature maps into high-resolution images. The upsampled feature maps are processed by a sequence module to generate the final fused image. Finally, the reconstructed image is fed into a discriminator for evaluation. The discriminator learns the feature differences between the real and generated images and outputs a probability value to determine the authenticity of the input image. This process not only improves the quality of the fused image but also enhances the model's understanding and processing capabilities for images with different modalities.

[0117] Step 3, Design the Loss Function: During GAN training, the generator and discriminator compete against each other through adversarial training. The generator attempts to generate increasingly realistic images to deceive the discriminator, while the discriminator tries to improve its ability to distinguish between real and generated images. This adversarial process drives the generator to continuously improve, eventually generating high-quality, realistic images. The generator loss and discriminator loss are key to achieving this goal, jointly defining the optimization objective of network training. The loss function mainly consists of two parts: the generator loss and the dual discriminator loss. The total loss can be defined as... As shown in formula (8).

[0118] Step 4, Training the Network Model: All experiments were performed on an NVIDIA GeForce GTX3090 GPU with a batch size of 24. During training, the generator's learning rate was set to 0.0002, and the discriminators DIS1 and DIS2 had learning rates of 0.00001. The number of training epochs was set to 3000. All images were pre-scaled to 256 × 256 before being input into the model. Six evaluation metrics were selected for assessment: EI (Edge Information) reflects the degree to which the fused image retains edge information from the input image; SF (Spatial Frequency) reflects image activity by calculating gradients in various directions, and is related to image texture and detail richness; EN (Entropy) measures the richness of information in an image, reflecting its complexity and uncertainty; MI (Mutual Information) measures the degree of information sharing between two images, reflecting their correlation and similarity; AG (Average Gradient) measures the edge strength of an image, reflecting the average degree of grayscale change between adjacent pixels; and VIFF (Visual Information Fidelity) focuses on the degree to which the fused image restores the visual information of the original image, balancing visual realism and quality, and incorporating human visual perception characteristics.

[0119] Step 5, Determine the fusion model: After the fusion network is trained, retain the weights, fix the network model parameters, and determine this network model as the final fusion network model. Test the Road Scene dataset and the RGB-NIR dataset using the trained weights respectively; simply input the results into the network to obtain the fused image.

[0120] Table 1 shows the quantitative results of each indicator on the Road Scene dataset; Table 2 shows the quantitative results of each indicator on the RGB-NIR dataset.

[0121] Table 1

[0122]

[0123] Table 2

[0124]

[0125] This embodiment proposes a frequency-domain-aware Mamba module. The frequency-domain-aware Mamba module achieves frequency processing of information through a frequency attention enhancement module. This module captures and enhances frequency-domain features in signals, thereby improving the model's ability to process frequency-domain information. This design enables the model to more accurately identify and understand the spatial distribution of different frequency information in images or sequences. Simultaneously, the frequency-domain-aware Mamba module utilizes the Mamba's dual-path structure to enhance information interaction. By dynamically adjusting the weights of feature interactions through feature fusion and interaction mechanisms, this design not only preserves important information from each modality but also further enhances the expressive power of features through cross-modal interactions. By combining frequency processing and dual-path interaction, the frequency-domain-aware Mamba module significantly improves the performance of the Mamba in information interaction and frequency-domain processing, enabling it to better handle complex multimodal data and frequency-domain tasks.

[0126] This embodiment achieves information interaction through a cross-modal interaction mechanism, utilizing a frequency attention enhancement module to process frequency information. The cross-modal interaction mechanism dynamically adjusts the weights of different modal features, enhancing feature representation capabilities and avoiding information loss. The frequency attention enhancement module performs frequency domain decomposition, feature extraction, and enhancement on the input image, highlighting key frequency components. This invention also includes input, processing, and output modules, suitable for multimodal image fusion tasks. Furthermore, a loss function incorporating generator loss and dual discriminator loss is designed. The generator loss consists of content loss (including structural similarity loss, mean squared error loss, and edge information loss) and adversarial loss. The dual discriminator loss is used to improve the fidelity between the fused image and the source image, thereby driving the generator to produce high-quality images.

[0127] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A multimodal image fusion method using coupled frequency domain Mamba, characterized in that, Includes the following steps: Obtain a multimodal source image dataset for training; An image fusion network model is constructed, which includes a generator and a discriminator. The generator includes a feature extraction stage, a feature interaction stage, a feature fusion stage, and an image reconstruction stage. Construct a loss function that includes generator loss and discriminator loss, and train the image fusion network model based on the multimodal source image dataset and the loss function; The multimodal source images to be fused are input into the trained image fusion network model, and the fused image is output in real time.

2. The multimodal image fusion method with coupled frequency domain Mamba as described in claim 1, characterized in that, The feature extraction stage processes the input multimodal source image through convolutional layers to obtain preliminary features for each modality; the preliminary features are then enhanced by the Mamba module to obtain low-level features for each modality.

3. The multimodal image fusion method with coupled frequency domain Mamba as described in claim 2, characterized in that, The low-level features of each modality and the cross-modal features obtained through the Mamba-based feature fusion module are concatenated; the concatenated feature map is downsampled to obtain the downsampled features corresponding to each modality. The downsampled features are sent to the feature interaction stage and processed by the frequency domain sensing Mamba module to obtain the deep features corresponding to each mode. The frequency domain sensing Mamba module consists of two symmetrical frequency Mamba blocks. Each frequency Mamba block sequentially performs frequency attention enhancement processing and feature optimization processing based on a dual-path structure on the input downsampled features, and outputs deep features of each modality.

4. The multimodal image fusion method with coupled frequency domain Mamba as described in claim 3, characterized in that, The process of frequency attention enhancement processing includes: The input features are subjected to average pooling and max pooling, and the results are concatenated to generate pooled fusion features. Perform a Fast Fourier Transform on the pooling fusion features to decompose them into frequency domain amplitude information and frequency domain phase information; A multi-head self-attention mechanism is applied to the frequency domain amplitude information and the frequency domain phase information respectively to generate enhanced amplitude information and enhanced phase information; The enhanced amplitude information is combined with the enhanced phase information and then processed by inverse fast Fourier transform to obtain the enhanced feature map. After performing a convolution operation on the enhanced feature map, it is multiplied element-wise with the input features to generate spatial attention-weighted features; The spatial attention-weighted features are added to the input features through a residual connection to output frequency domain enhanced features.

5. The multimodal image fusion method with coupled frequency domain Mamba as described in claim 3, characterized in that, The feature optimization process based on the dual-path structure includes: The frequency domain enhancement features are subjected to layer normalization and linear transformation to obtain transformed features. The transformed features are then segmented along the channel dimension into first path features and second path features. The first path features are processed sequentially through a convolutional layer, a 2D selective scanning mechanism, and the frequency attention enhancement module to obtain first path output features. The second path features are processed through the SiLU activation function to obtain second path output features. The first path output features and the second path output features are linearly fused to obtain interactive enhancement features. The interactive enhancement features and the frequency domain enhancement features are added together through skip connections to obtain deep features for each modality.

6. The multimodal image fusion method with coupled frequency domain Mamba as described in claim 5, characterized in that, The feature fusion stage uses a Mamba-based feature fusion module to perform preliminary cross-modal integration of the low-level features of each modality obtained in the feature extraction stage; the features obtained from the cross-modal integration are then concatenated with the deep features of each modality obtained in the feature interaction stage, and the concatenated features are then subjected to in-depth cross-modal integration using the Mamba-based feature fusion module to generate the final fused feature map.

7. The multimodal image fusion method with coupled frequency domain Mamba as described in claim 6, characterized in that, The processing steps of the Mamba-based feature fusion module include: The input features are sequentially normalized and linearly transformed. The linearly transformed features are divided into two parts along the channel dimension and input into the first branch and the second branch, respectively. The first branch extracts local spatial features through a convolutional layer and then integrates global information through 2D selective scanning. The second branch retains the nonlinear distribution of features through the SiLU activation function. The output features of the first branch and the second branch are multiplied element-wise to achieve feature interaction. The interactive features are transformed by a linear layer and then fused with the input features through residual connections.

8. The multimodal image fusion method with coupled frequency domain Mamba as described in claim 6, characterized in that, The image reconstruction module processes the fused feature map using the Mamba module to obtain fused features with enhanced local expressive power; it then processes the fused features with enhanced local expressive power using the upsampling module to obtain a high-resolution feature map; finally, it performs serialization processing on the high-resolution feature map to obtain the final fused image.

9. The multimodal image fusion method with coupled frequency domain Mamba as described in claim 1, characterized in that, The generator loss includes content loss and adversarial loss, and the content loss includes structural similarity loss, mean squared error loss, and edge information loss.

10. The multimodal image fusion method of coupled frequency domain Mamba according to claim 1, characterized in that, The discriminator loss is calculated using a dual-discriminator structure, with both discriminators using the same loss function, expressed as follows: ； in, It is the loss function of the first discriminator. It is the loss function of the second discriminator; Represents the source image. This is the final fused image generated by the generator, where D represents the discriminator.

Images