Image fusion method and device based on mamba and convolutional neural network

Abstract

This invention provides an image fusion method and device based on Mamba and convolutional neural networks, comprising: acquiring multiple pairs of infrared and visible light images respectively using an infrared imaging device and a visible light imaging device in the same scene, preprocessing the multiple pairs of images, and constructing a model training set and a test set using the preprocessed image pairs; constructing an infrared and visible light image fusion model MCDFuse using Mamba and convolutional neural networks; training the infrared and visible light image fusion model MCDFuse using the training set, wherein the input of the infrared and visible light image fusion model MCDFuse is the preprocessed infrared and visible light image pairs, and the output is the fused image; and performing image fusion testing on the test set using the trained infrared and visible light image fusion model MCDFuse.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image fusion method and device based on Mamba and convolutional neural networks. Background Technology

[0002] With the development of multi-source sensing technology, infrared and visible light image fusion technology is becoming increasingly important in fields such as nighttime surveillance, military reconnaissance, target detection, autonomous driving, and disaster monitoring. It overcomes the limitations of single-modality imaging by combining the thermal radiation penetration capability of infrared images with the texture and color information of visible light images. Among existing deep learning-based fusion methods, CNNs (Convolutional Neural Networks) struggle to model global dependencies due to their limited receptive field; Transformers, while capable of capturing long-distance relationships, suffer from high computational complexity; autoencoders are insufficient for capturing fine-grained information in complex scenes; generative adversarial networks (GANs) suffer from mode collapse; diffusion models are complex to train; transfer learning is affected by domain differences; and contrastive learning easily introduces redundant information. Mamba, as an emerging sequence modeling architecture, leverages its linear complexity and ability to align heterogeneous data for image fusion. Examples include FusionMamba, which enhances feature extraction through dynamic differential convolution, and MambaDFuse, which employs a two-stage strategy to fuse multimodal information. However, existing Mamba methods suffer from insufficient detail and texture enhancement, poor input channel adaptability, and feature redundancy, making it difficult to balance spatial information preservation and global dependency modeling in complex scenes. Therefore, more efficient fusion methods are urgently needed to improve performance. Summary of the Invention

[0003] To address the aforementioned problems in the existing technology, embodiments of the present invention provide an image fusion method and device based on Mamba and convolutional neural networks.

[0004] In a first aspect, embodiments of the present invention provide an image fusion method based on Mamba and convolutional neural networks, comprising: acquiring multiple pairs of infrared and visible light images respectively using an infrared imaging device and a visible light imaging device in the same scene, preprocessing the multiple pairs of image images, and constructing a model training set and a test set using the preprocessed image pairs; constructing an infrared and visible light image fusion model MCDFuse using Mamba and convolutional neural networks; training the infrared and visible light image fusion model MCDFuse using the training set, wherein the input of the infrared and visible light image fusion model MCDFuse is the preprocessed infrared and visible light image pairs, and the output is the fused image; and performing image fusion testing on the test set using the trained infrared and visible light image fusion model MCDFuse.

[0005] Based on the above method embodiments, the image fusion method based on Mamba and convolutional neural networks provided in this embodiment of the invention includes preprocessing the multiple pairs of images, including: adjusting the infrared image channel to a single channel and the visible light image to a three-channel channel; uniformly adjusting the images to a predetermined resolution; rotating, cropping, and flipping the images; and modifying the image names to ensure that the order corresponds.

[0006] Based on the above method embodiments, the image fusion method based on Mamba and convolutional neural networks provided in this embodiment of the invention, wherein the infrared and visible light image fusion model MCDFuse is constructed using Mamba and convolutional neural networks, includes: constructing a model composed of an encoder and a decoder, wherein the encoder includes an image reconstruction part, a visible light feature extraction part, and an infrared feature extraction part; the image reconstruction part adopts a dynamic feature activation module (DFA) to activate image channel features through a Mamba dynamic scanning mechanism; a dual-branch structure is used to process the feature map, the global feature extraction part adopts a hierarchical feature extraction module (HFE) to construct a global feature map based on the next-generation convolutional neural network ConvNeXt, and the local feature enhancement part uses a dual-branch detail enhancement module (DDE) to extract detail features using reversible residual blocks and depth-separating convolution; the decoder has the same structure as the image reconstruction part of the encoder.

[0007] Based on the above method embodiments, the image fusion method based on Mamba and convolutional neural networks provided in this embodiment of the invention, wherein the image reconstruction part adopts a Dynamic Feature Activation (DFA) module, which activates image channel features through the Mamba dynamic scanning mechanism, including:

[0008] X l+1,vi =M l (X vi )

[0009] X l+1,ir =M l (X ir )

[0010] Among them, X l+1,vi The visible light image feature map for layer l+1; M l For Mamba operation; X vi Visible light image; X l+1,ir For the feature map of the l+1 layer infrared image; X ir For infrared images; the state-space model includes:

[0011]

[0012] SSM(X) = f out (X1⊙X2)

[0013] in, The symbol for function combination is denoted by ; X1 is the result obtained by performing the corresponding function transformation on the eigenvector X; LN is the normalization function; f SS2D σ is the scanning function; σ is the SiLU activation function; f DW For grouped convolution operations; f proj f represents a linear transformation; gate This represents gated convolution, and ⊙ is the symbol for the Hadamard product; f out X1 is the output result; X2 is the result obtained by performing another corresponding function transformation on the feature vector X; SSM is the state-space model.

[0014] Based on the above method embodiments, the image fusion method based on Mamba and convolutional neural networks provided in this embodiment of the invention uses a hierarchical feature extraction module (HFE) for the global feature extraction part and constructs a global feature map based on the next-generation convolutional neural network (ConvNeXt). The method includes: constructing a hierarchical feature extraction module (HFE) to extract and output visible light global features; using a hierarchical progressive next-generation convolutional neural network (ConvNeXt) as a basic module to output infrared global features; integrating the visible light global features and infrared global features according to the linear combination of channels and the gating mechanism to obtain integrated global features; and outputting the integrated global features at the top layer of the network.

[0015] Based on the above method embodiments, the image fusion method based on Mamba and convolutional neural networks provided in this embodiment of the invention, wherein the local feature enhancement part extracts detailed features through a dual-branch detail enhancement module (DDE) using reversible residual blocks and depthwise separable convolution, includes: constructing a dual-branch detail enhancement module (DDE) for extracting local features, using reversible residual blocks as the basic module, dividing the input features into a first channel and a second channel according to channels, and performing a reversible transformation through an affine coupling layer:

[0016]

[0017] in, This is the first channel; For the second channel; ⊙ is the symbol for the Hadamard product; exp is the symbol for the exponential function; T φ T is the first inverse residual mapping domain; ρ For the second inverse residual mapping domain; T η is the third inverse residual mapping domain; l is the local feature extracted by function f; Concat is the feature concatenation function; [] is the closed interval symbol; C is the number of channels; X is the feature map; For output features.

[0018] Based on the above method embodiments, the image fusion method based on Mamba and convolutional neural network provided in this embodiment of the invention includes the following: the decoder is trained in two stages to generate the fused image. The first stage realizes single-modal reconstruction, reconstructing the original infrared light input image and the original visible light input image. The second stage focuses on multimodal fusion optimization. The encoder input includes cross-modal and multi-frequency features, and the Mamba is used as the basic unit of the decoder.

[0019] Secondly, embodiments of the present invention provide an image fusion device based on Mamba and convolutional neural networks, comprising: a first main module, used to acquire multiple pairs of infrared and visible light images respectively through an infrared imaging device and a visible light imaging device in the same scene, and to preprocess the multiple pairs of image images, and to construct a model training set and a test set using the preprocessed image pairs; a second main module, used to construct an infrared and visible light image fusion model MCDFuse using Mamba and convolutional neural networks; a third main module, used to train the infrared and visible light image fusion model MCDFuse using the training set, wherein the input of the infrared and visible light image fusion model MCDFuse is the preprocessed infrared and visible light image pairs, and the output is the fused image; and a fourth main module, used to perform image fusion testing on the test set using the trained infrared and visible light image fusion model MCDFuse.

[0020] Thirdly, embodiments of the present invention provide an electronic device, comprising:

[0021] At least one processor, at least one memory, and a communication interface; wherein,

[0022] The processor, memory, and communication interface communicate with each other;

[0023] The memory stores program instructions that can be executed by the processor, which calls the program instructions to execute the image fusion method based on Mamba and convolutional neural networks provided by any of the various implementations of the first aspect.

[0024] Fourthly, embodiments of the present invention provide a non-transitory computer-readable storage medium storing computer instructions that cause a computer to execute an image fusion method based on Mamba and convolutional neural networks provided by any of the various implementations of the first aspect.

[0025] The image fusion method and device based on Mamba and convolutional neural networks provided in this invention enhances Mamba feature representation through dynamic feature activation module and channel attention mechanism, and improves global receptive field and detail enhancement capability by combining ConvNeXt (next-generation convolutional neural network) and inverse residual structure. It solves the problems of insufficient long-distance dependency modeling and computational complexity of existing methods. This method is significantly superior to related technologies in terms of fusion accuracy, detail preservation and inference speed, and can improve the performance of downstream target detection. It is suitable for infrared and visible light image fusion in complex scenes. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a schematic diagram of the image fusion method based on Mamba and convolutional neural networks provided in an embodiment of the present invention;

[0028] Figure 2 A schematic diagram of the image fusion device based on Mamba and convolutional neural networks provided in an embodiment of the present invention;

[0029] Figure 3 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention;

[0030] Figure 4 This is a schematic diagram of the overall architecture of MCDFuse provided in an embodiment of the present invention;

[0031] Figure 5 This is a schematic diagram of the State Space Model (SSM) architecture provided in an embodiment of the present invention.

[0032] Figure 6 This is a schematic diagram illustrating the dynamic feature activation principle provided in an embodiment of the present invention;

[0033] Figure 7 This is a schematic diagram of the channel attention mechanism module architecture provided in an embodiment of the present invention;

[0034] Figure 8 This is a schematic diagram of the hierarchical feature extraction module architecture provided in an embodiment of the present invention;

[0035] Figure 9 This is a schematic diagram of the dual-branch detail enhancement module architecture provided in an embodiment of the present invention;

[0036] Figure 10A schematic diagram of sample image pairs for a dataset provided in an embodiment of the present invention;

[0037] Figure 11 This is a schematic diagram of the qualitative comparison results on the UAVScene dataset provided in an embodiment of the present invention;

[0038] Figure 12 This is a schematic diagram illustrating the qualitative comparison of the marine environment in the CompScene dataset provided in an embodiment of the present invention.

[0039] Figure 13 This is a schematic diagram illustrating qualitative comparisons in other complex environments of the CompScene dataset provided in an embodiment of the present invention.

[0040] Figure 14 This is a schematic diagram illustrating a qualitative comparison between the present invention and existing infrared and visible light image fusion methods in downstream target detection of UAV datasets, as provided in an embodiment of the invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. In addition, the technical features of the various embodiments or individual embodiments provided by the present invention can be arbitrarily combined to form feasible technical solutions. Such combinations are not constrained by the order of steps and / or structural composition patterns, but must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention. If there are step numbers in the following embodiments, they are only set for ease of explanation and the order between steps is not limited. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0042] This invention provides an image fusion method based on Mamba and convolutional neural networks, see [link to relevant documentation]. Figure 1The method includes: acquiring multiple pairs of infrared and visible light images respectively using an infrared imaging device and a visible light imaging device in the same scene, preprocessing the multiple pairs of images, and constructing a training set and a test set for the model using the preprocessed image pairs; constructing an infrared and visible light image fusion model MCDFuse using a Mamba and a convolutional neural network; training the infrared and visible light image fusion model MCDFuse using the training set, wherein the input of the infrared and visible light image fusion model MCDFuse is the preprocessed infrared and visible light image pairs, and the output is the fused image; and performing image fusion testing on the test set using the trained infrared and visible light image fusion model MCDFuse.

[0043] In the same scene, multiple pairs of infrared and visible light images were acquired using infrared and visible light imaging devices. The images were preprocessed, and the preprocessed image pairs were used to construct a dataset for model training and testing. Each image pair contained one infrared image and one visible light image, and spatial alignment was ensured through registration techniques. An infrared and visible light image fusion model, MCDFuse, was constructed using Mamba and CNN (Convolutional Neural Network). The MCDFuse model was trained using training samples from the constructed dataset. The input to the model was the preprocessed infrared and visible light image pairs, and the output was the fused image of the input image pairs. The trained MCDFuse model was then used to test the infrared and visible light image fusion on test samples from the constructed dataset.

[0044] Based on the above method embodiments, as an optional embodiment, the image fusion method based on Mamba and convolutional neural networks provided in this embodiment of the invention, wherein the multiple pairs of images are preprocessed, including: adjusting the infrared image channel to a single channel and the visible light image to a three-channel channel; uniformly adjusting the images to a predetermined resolution; rotating, cropping and flipping the images; and modifying the image names to ensure that the order corresponds.

[0045] Adjust the mid-infrared image channel of the image pair to a single channel and the visible light image channel to a three-channel channel; uniformly adjust the mid-infrared and visible light images of the image pair to a specified resolution to ensure input consistency; apply data augmentation strategies such as rotation, cropping, and flipping to the mid-infrared and visible light images of the image pair; modify the names of the mid-infrared and visible light images of the image pair to arrange them in order, with each pair of names corresponding one-to-one.

[0046] Based on the above method embodiments, as an optional embodiment, the image fusion method based on Mamba and convolutional neural networks provided in this embodiment of the invention, wherein the infrared and visible light image fusion model MCDFuse is constructed using Mamba and convolutional neural networks, includes: constructing a model composed of an encoder and a decoder, wherein the encoder includes an image reconstruction part, a visible light feature extraction part, and an infrared feature extraction part; the image reconstruction part adopts a dynamic feature activation module (DFA) to activate image channel features through a Mamba dynamic scanning mechanism; a dual-branch structure is used to process the feature map, the global feature extraction part adopts a hierarchical feature extraction module (HFE) to construct a global feature map based on the next-generation convolutional neural network ConvNeXt, and the local feature enhancement part uses a dual-branch detail enhancement module (DDE) to extract detail features using reversible residual blocks and depth-separating convolution; the decoder has the same structure as the image reconstruction part of the encoder.

[0047] Construct an MCDFuse model consisting of an encoder and a decoder, with the overall architecture as follows: Figure 4 As shown, the encoder comprises three parts: high-quality image reconstruction, visible light feature extraction, and infrared feature extraction. The core of the high-quality image reconstruction part is the DFA (Dynamic Feature Activation Module), which aims to activate the latent features of each channel of the image through Mamba's dynamic scanning mechanism to obtain high-quality feature maps for different modalities. When processing the feature maps, a dual-branch structure of global feature extraction and local feature enhancement is used to process the feature maps of both visible light and infrared images simultaneously. The global feature extraction part uses the HFE (Hierarchical Feature Extraction Module), constructing global feature maps for infrared and visible light based on the ConvNeXt network. The local feature enhancement part is implemented through the DDE (Dual-Branch Detail Enhancement Module), employing inverse residual blocks of a reversible neural network for dual-path reversible transformation and depthwise separable convolution to efficiently extract and enhance detailed features. The decoder design is consistent with the high-quality image reconstruction part of the encoder to ensure consistency of cross-modal data features.

[0048] Based on the above method embodiments, as an optional embodiment, the image fusion method based on Mamba and convolutional neural networks provided in this embodiment of the invention, wherein the image reconstruction part adopts a Dynamic Feature Activation (DFA) module, which activates image channel features through the Mamba dynamic scanning mechanism, including:

[0049] X l+1,vi =M l (X vi )

[0050] X l+1,ir =M l (X ir )

[0051] Among them, Xl+1,vi The visible light image feature map for layer l+1; M l For Mamba operation; X vi Visible light image; X l+1,ir For the feature map of the l+1 layer infrared image; X ir For infrared images; the state-space model includes:

[0052]

[0053] SSM(X) = f out (X1⊙X2)

[0054] in, The symbol for function combination is denoted by ; X1 is the result obtained by performing the corresponding function transformation on the eigenvector X; LN is the normalization function; f SS2D σ is the scanning function; σ is the SiLU activation function; f DW For grouped convolution operations; f proj f represents a linear transformation; gate This represents gated convolution, and ⊙ is the symbol for the Hadamard product; f out X1 is the output result; X2 is the result obtained by performing another corresponding function transformation on the feature vector X; SSM is the state-space model.

[0055] The core module of the encoder is the dynamic feature activation module. Based on this module, features from both infrared and visible light source images can be obtained. First, the input infrared image is defined as... Visible light image is Here, H and W represent the height and width of the source image, respectively. Infrared is a single-channel input, and visible light is a three-channel input. After passing through the dynamic feature activation module, the output feature maps of visible light and infrared are shown as X. l+1,vi and X l+1,ir As shown. Compared to the Transformer-based image reconstruction module design, although both the attention mechanism and SSM can simulate global dependencies, Mamba's SSM mechanism processes each channel independently through depthwise convolution, reducing computation and memory requirements. It models the spatiotemporal relationships of the image through the state transition matrix, eliminating the need to calculate a global attention matrix for each pixel as in the Transformer, thus reducing memory usage.

[0056] The detailed process of SSM is as follows: Figure 5As shown, SSM uses depthwise convolution to independently convolve each input channel with its corresponding kernel, preserving spatial structure while enhancing local representation. This channel operation also minimizes inter-channel interference, making it well-suited for multimodal image fusion. SSM is specifically shown as SSM(X). Without improvements to the original SSM model, only suboptimal performance can be achieved. This is because when using a four-direction unfolding strategy, the flattened sequence treats each pixel's four neighbors as its four closest neighbors. These neighbors may be directly adjacent in the 2D feature map (e.g., vertically, horizontally, and vertically), but in the 1D sequence, the correlation of some pixels is affected by the scanning strategy and they are no longer directly adjacent. The upper left corner of the feature map represents diagonally adjacent pixel blocks, whose correlation is ignored as the sequence unfolds. The lower right corner represents superpixel blocks; for a given superpixel block, the channels of its core pixel block and diagonally adjacent pixel blocks still have the potential for further learning.

[0057] This increased spatial distance causes the model to reduce the number of originally diagonally adjacent pixels during the learning process due to scanning, thus affecting the model's performance. The principle is as follows: Figure 6 As shown, this invention adds convolutional layers and a channel attention mechanism to optimize this process. First, convolutional layers, involving dimensionality reduction and expansion, compress and restore the number of channels in the feature map. Then, the channel attention mechanism weights each channel of the feature map, dynamically adjusting the importance of each channel to enhance the representational power of important channels and suppress redundant and irrelevant channel information, thereby improving the model's ability to express key features and its discriminative performance. The structure of the channel attention mechanism module is shown below. Figure 7 As shown. Where (B, C, H, W) represent batch size, number of channels, image height, and width, respectively, the core process of the dynamic feature activation module can be expressed by the following formula:

[0058] Y l =SSM(f proj (X l ))+ω1·X l

[0059] X l+1 =CA(Conv(f) proj (Y l )))+ω2·Y l

[0060] Among them, residual weight The CA function, used to control the weighting of input features in skip connections, represents the channel attention mechanism. After the cascading of four modules, high-quality reconstructed infrared and visible light feature maps are finally obtained.

[0061] Based on the above method embodiments, as an optional embodiment, the image fusion method based on Mamba and convolutional neural networks provided in this embodiment of the invention uses a hierarchical feature extraction module (HFE) for the global feature extraction part, and constructs a global feature map based on the next-generation convolutional neural network (ConvNeXt). The method includes: constructing a hierarchical feature extraction module (HFE) to extract and output visible light global features; using a hierarchical progressive next-generation convolutional neural network (ConvNeXt) as a basic module to output infrared global features; integrating the visible light global features and infrared global features according to the linear combination of channels and the gating mechanism to obtain integrated global features; and outputting the integrated global features at the top layer of the network.

[0062] The hierarchical feature extraction (HFE) module extracts global features from the reconstructed infrared and visible light images, thereby capturing the macroscopic structure and semantic information of the images to ensure scene consistency. This invention uses ConvNeXt, a hierarchical progressive feature learning module, as the basic module of HFE. The output infrared global features are represented as follows: The global features of visible light are represented as For details, see the structure below. Figure 8 As shown, each layer of the network progressively transforms image features into a more abstract, global semantic representation. Through linear combination of channels and gating mechanisms, it implicitly integrates global information, ultimately outputting global features at the top layer.

[0063] Based on the above method embodiments, as an optional embodiment, the image fusion method based on Mamba and convolutional neural networks provided in this embodiment of the invention, wherein the local feature enhancement part extracts detailed features through a dual-branch detail enhancement module (DDE) using reversible residual blocks and depthwise separating convolution, includes: constructing a dual-branch detail enhancement module (DDE) for extracting local features, using reversible residual blocks as the basic module, dividing the input features into a first channel and a second channel according to channels, and performing a reversible transformation through an affine coupling layer:

[0064]

[0065] in, This is the first channel; For the second channel; ⊙ is the symbol for the Hadamard product; exp is the symbol for the exponential function; T φ T is the first inverse residual mapping domain; ρ For the second inverse residual mapping domain; T η is the third inverse residual mapping domain; l is the local feature extracted by function f; Concat is the feature concatenation function; [] is the closed interval symbol; C is the number of channels; X is the feature map; For output features.

[0066] The dual-branch detail enhancement module (DDE) extracts local features from the reconstructed infrared and visible light data for detail enhancement. The output infrared local features are... Local features of visible light are To compensate for the loss of detail caused by Mamba's scanning mechanism and CNN's nonlinear operations, this paper uses reversible residual blocks as the basic module of DDE. Inspired by the reversible architectures of NICE and RealNVP, an affine coupling layer is constructed through affine transformation (scaling + offset) to ensure lossless preservation of image detail. IRB is used to balance computational efficiency and feature extraction capability. See [link to detailed structure] Figure 9 As shown, through multi-layer reversible transformation, the input infrared and visible light features can be nonlinearly enhanced without losing the original information.

[0067] Based on the above method embodiments, as an optional embodiment, the image fusion method based on Mamba and convolutional neural network provided in this embodiment of the invention includes the following: the decoder is trained in two stages to generate the fused image. The first stage realizes single-modal reconstruction, reconstructing the original infrared light input image and the original visible light input image. The second stage focuses on multimodal fusion optimization. The encoder input includes cross-modal and multi-frequency features, and the Mamba is used as the basic unit of the decoder.

[0068] To make the computation and inference process more efficient, the decoder employs a two-stage training process to generate the fused image. The first stage performs single-modal reconstruction, reconstructing the original input images in infrared and visible light. The second stage focuses on multimodal fusion optimization. The encoder input contains cross-modal and multi-frequency features. The decoder structure of this invention is consistent with the design of DFA, that is, using Mamba as the basic unit of the decoder.

[0069] The first stage of training utilizes self-supervised learning to ensure the model effectively separates global and local features, and the decoder accurately reconstructs the original image. The reconstructed image from visible light is represented as follows: Infrared reconstructed image The two-stage training generates the final multimodal fused image through cross-modal feature fusion and optimizes the fusion quality. Based on the global features generated by the hierarchical feature extraction module (HFE) and the local features generated by the DDE module, global feature fusion and local feature fusion of infrared and visible light can be performed, where global feature fusion is represented as G(X) = G vi ⊕G ir Local feature fusion is represented as L(X) = L vi ⊕L ir The final infrared and visible light fused image is represented as F(X)=Concat{G(X,L(X)}, where the Concat{·} function represents the concatenation of two feature maps in the channel dimension.

[0070] The model training process is divided into two stages: the initial training stage and the fine-tuning stage. The loss functions in the initial training stage include image reconstruction loss, feature decomposition loss, and gradient loss, while the loss functions in the fine-tuning stage include fusion loss and feature decomposition loss.

[0071] In the initial training phase, the model first processes visible light and infrared images through an encoder, and then decodes them to generate reconstructed images. The loss function in this phase includes the following components:

[0072] Image reconstruction loss: The differences between the reconstructed image and the source image are divided into structural differences and pixel-level differences, where pixel-level loss is expressed as:

[0073]

[0074] Structural loss is expressed as:

[0075]

[0076] Here, the SSIM(·,·) function represents the structural similarity exponential function.

[0077] Feature decomposition loss: The features decomposed by the model include global and local features. Global features contain modality-shared information and are therefore highly correlated, while local features map modality-specific information and are therefore less correlated. By maximizing the ratio of local to global correlation, the model can be encouraged to separate modality-shared and modality-specific features. However, the correlation coefficient only measures linear correlation and cannot capture differences at the distribution level. Furthermore, to avoid excessively high correlation of local features due to gradient problems or data noise during training, this invention redefines the feature decomposition loss by combining information bottleneck constraints and adversarial constraints, expressed as follows:

[0078]

[0079] The CC(·,·) function represents the global correlation coefficient of the spatial dimension.

[0080]

[0081] Where μ X ,μ Y σ represents the mean of the feature. X ,σ Y The standard deviation of the feature is represented. KL divergence, which implements the information bottleneck constraint, is expressed as:

[0082]

[0083] P(G vi ) = Softmax(Gvi )

[0084] P(G ir ) = Softmax ir )

[0085] Where i represents the spatial location or channel index of the feature map. The adversarial constraint is expressed as:

[0086] L adv =max(0,γ-CC(L) vi ,L ir ))

[0087] Here, γ is a threshold used to limit the maximum correlation of local features and avoid spurious associations between modalities.

[0088] Gradient loss: To ensure that the edge information of the reconstructed image is consistent with the input, and to prevent blurring or oversmoothing.

[0089]

[0090] in This is the Sobel gradient operator.

[0091] Therefore, the total loss in the first stage can be expressed as: The formula is shown below:

[0092]

[0093] α1, α2, and α3 are the loss weights.

[0094] In the fine-tuning phase, the models are jointly trained to generate high-quality multimodal fused images. The loss function in this phase includes fusion loss and feature decomposition loss. The feature decomposition loss is the same as in the first phase, used to constrain the rationality of feature decomposition. The fusion loss is a composite loss function, including the image's L1 loss, gradient loss (calculated via Sobel convolution), and similarity loss between the two modalities (visible light and infrared) and the generated fused image. The image fusion quality is optimized by calculating the gradients of the original image and the original image, expressed as:

[0095]

[0096] Where F is the fused image, the first part, intensity loss, ensures that the pixel values ​​of the fused image are close to the maximum values ​​of the two inputs, and the second part, gradient loss, ensures that edge information is maximized. Therefore, the total two-stage loss can be expressed as: The formula is shown below:

[0097]

[0098] Where β is the loss weight.

[0099] The image fusion method based on Mamba and convolutional neural networks provided in this invention enhances Mamba feature representation through dynamic feature activation module and channel attention mechanism, and improves global receptive field and detail enhancement capability by combining ConvNeXt (next-generation convolutional neural network) and inverse residual structure. It solves the problems of insufficient long-distance dependency modeling and computational complexity of existing methods. This method is significantly superior to related technologies in terms of fusion accuracy, detail preservation and inference speed, and can improve the performance of downstream target detection. It is suitable for infrared and visible light image fusion in complex scenes.

[0100] The superiority of this invention can be demonstrated through experiments. First, the dataset and implementation details are introduced, followed by a description of the evaluation metrics used for quantitative assessment. Then, a comparative analysis between this invention and state-of-the-art methods is provided. Next, ablation studies are conducted, and finally, the applicability of this invention to downstream tasks is evaluated.

[0101] This invention uses three datasets: UAVScene, TNO, and CompScene. UAVScene consists of infrared and visible light image pairs captured from the perspective of a drone, including images from the registered public datasets DroneVehicle and VTUAV. The TNO image fusion dataset is a publicly available benchmark multimodal image dataset released by the Dutch TNO (Telecommunications Research and Development Institute). The CompScene dataset comprises 257 challenging infrared and visible light image pairs under complex environments (rain, fog, motion blur, and low light), drawn from four publicly available benchmark datasets: MSRS, M3FD, RoadScene, and VAIS. Some example image pairs from these datasets are shown below. Figure 10 As shown, there are four datasets, each containing five pairs of infrared and visible light images. The left side of each dataset shows the visible light image, and the right side shows the infrared image. This invention uses the PyTorch framework and is trained on a single NVIDIA GeForce RTX 3090 GPU. Training samples are randomly cropped into 128×128 blocks and normalized to the [0,1] range. The Adam optimizer uses a batch size of 4. There are a total of 120 training epochs, divided into two phases of 40 and 80 epochs. The initial learning rate is set to 1×10⁻⁶. -4 The feature dimension decays by a factor of 0.5 every 20 training epochs. Both the DFA and HFE modules have a feature dimension of 64. The HFE module consists of 3, 3, 27, and 3 ConvNeXt blocks. The decoder and encoder are structurally symmetrical.

[0102] This invention compares the proposed method with 12 current state-of-the-art image fusion methods, including DDFM, FusionGAN, SDNet, FusionMamba, DCEvo, DIDFuse, EMMA, SimpleFusion, SwinFusion, CDDFuse, DenseFuse, and U2Fuse.

[0103] To quantitatively evaluate all methods, this invention employs seven widely used evaluation metrics: information entropy (EN), spatial frequency (SF), average gradient (AG), peak signal-to-noise ratio (PSNR), visual information fidelity (VIF), and edge-based similarity metric (Q). AB / F The mean squared error (MSE) is also mentioned. Specifically, EN reflects the amount of information contained in an image by measuring the complexity of its gray-level distribution; SF measures the sharpness of image details and textures by evaluating the frequency of image intensity changes; AG represents edge sharpness by calculating the average magnitude of image gray-level changes; PSNR evaluates image fidelity by quantifying the pixel-level differences between the fused image and the source image; VIF reflects the perceptual quality by evaluating the amount of visual information retained in the fused image; Q... AB / F Structure preservation capability is measured by comparing the edge similarity between the fused image and the source image; MSE evaluates the reconstruction accuracy of the image by calculating the squared value of the pixel-level error. It should be noted that, except for MSE, the higher the value of all other evaluation metrics, the better the fusion performance. Figure 11 This paper displays the partial fusion effects of 13 image fusion algorithms on the UAVScene dataset. The green boxes represent the source images in visible light and infrared, while the red and blue boxes represent marked and magnified portions of different locations within the images, facilitating comparison. The UAVScene dataset consists of images captured by drones, exhibiting significant multi-view and multi-scale characteristics. It typically features complex backgrounds and diverse terrain features, with single frames often containing moving objects (such as vehicles and crowds) and constantly changing backgrounds, presenting highly dynamic and complex content. This invention representatively selected three common object categories—"vehicles," "houses," and "people"—for comparison and analysis. Clearly, among these fourteen methods, this invention better integrates the texture details in visible light images and the thermal radiation information in infrared images. When visible light is affected by illumination, resulting in inconsistent image brightness and shadow distribution, this invention better utilizes infrared images for regional compensation. After fusion, the contrast between bright and dark areas of vehicles and road monitoring facilities is coordinated, and the visual consistency is the best. When infrared light is affected by outdoor high temperatures, resulting in large areas of white spots and obscuring structural details, this invention better preserves the structural details of windows and the outlines of human movements, visually avoiding "whitening" or "overflowing" phenomena, which is more conducive to subsequent identification or posture estimation of people.

[0104] To further verify the feasibility of the algorithm in complex scenarios, such as Figure 12 and Figure 13 As shown, each pair of images is labeled with the environmental challenges it faces, among which... Figure 12 Primarily focused on the marine environment. Figure 13 Primarily targeting land-based environments, this invention, when used to target naval vessels, addresses situations where high-temperature areas of the ship obscure key details in visible light images due to excessive infrared radiation. It not only clearly displays the ship's model and porthole structure but also prevents overexposure of hot areas in the fused image. For land-based environments, this invention offers better infrared penetration and visible light structural details in rainy or foggy weather, providing richer cloud contour information. In low-light environments, it maintains good target recognition, with clearer and more discernible fused images of vehicle logos. High-brightness areas in the fused image are more effectively restored, and the edges of roadside signs are more complete. In motion-blurred environments, this invention provides more accurate target localization, and blurred edges can be reconstructed more clearly.

[0105] Table 1 shows the quantitative comparison results of MCDFuse and 12 advanced algorithms on the UAVScene dataset. The present invention achieves the best and second best results in all metrics. For example, the SF value is improved by 8.86% compared to MCDFuse and the PSNR value is improved by 3.67% compared to DCEvo. This is because the model of the present invention enhances detail extraction and enriches the expression of the fused image while flexibly expanding the receptive field.

[0106] Table 1: Qualitative comparison results of the UAVScene dataset. The best results are highlighted in bold, and the next best are highlighted in underline.

[0107]

[0108] On the TNO dataset, as shown in Table 2, the present invention achieves the best performance in all metrics except PSNR. In particular, the VIF value is improved by 13.58% compared with the suboptimal method DCEvo. This is because the model of the present invention can adaptively enhance dynamic features and dynamically adjust the weights of each modality during the fusion process, thereby optimizing the final fused image and making it clearer and more realistic in visual effect.

[0109] Table 2: Qualitative comparison results of the TNO dataset. The best results are highlighted in bold, and the next best are highlighted in underline.

[0110]

[0111] On the CompScene dataset, as shown in Table 3, this invention achieves optimal performance on all metrics except VIF, particularly improving the AG value by 5.49% compared to the suboptimal EMMA method. Since complex scenes present greater challenges and require models to reconstruct more details, the high-quality reconstruction part of this invention uses Mamba's state space to model important information in the image. This more effectively captures and represents low-frequency and high-frequency information in the image and selectively fuses them, thus preserving more structured information during image fusion and aiding in a better understanding of image information in complex environments.

[0112] Table 3: Qualitative comparison results of the CompScene dataset. The best results are highlighted in bold, and the next best are highlighted in underline.

[0113]

[0114] Table 4 presents a comparison of the model inference efficiency of the methods that performed well in qualitative analysis. This invention randomly selected 10 images from the CompScene dataset and measured the average fusion processing time for each image. It should be noted that the model inference time here does not include I / O operations. The results show that the inference time of this invention is the shortest. This improvement can be attributed to the use of Mamba, which models sequence dependencies through efficient state transitions and parameter sharing, reducing the quadratic complexity of the self-attention mechanism to O(n^2). 2 The inference efficiency is significantly improved by replacing the linear complexity mechanism O(n).

[0115] Table 4: Comparison Results of Model Inference Efficiency

[0116]

[0117] This invention employs ablation experiments to demonstrate the effectiveness of the eigenvalue decomposition loss. Using the same seven metrics as the quantitative experiments, this invention divides the eigenvalue decomposition loss formula into three parts: correlation constraint CC-Loss, information bottleneck constraint KL-Loss, and adversarial loss Adv-Loss. On the UAVScene dataset, 300 pairs of images were selected for testing, with different combinations of loss functions used for each training iteration. The experimental results are shown in Table 5.

[0118] Table 5: Ablation experiments of the eigenvalue decomposition loss on the UAVScene dataset, with the optimal value indicated in bold.

[0119]

[0120] This invention achieves near-optimal results across all seven metrics. While AG performance is better when KL-Loss is added alone, and PSNR performance is better when Adv-Loss is added alone, this invention only achieves suboptimal results in these two metrics. However, considering the overall effect, the optimal metric results can only be obtained when all three constraints are met simultaneously. Therefore, the ablation results in Table 5 demonstrate the effectiveness and rationality of the eigenvalue decomposition loss function design in this invention.

[0121] Table 6: YOLOv11 performance on the DroneVehicle dataset, AP@0.5–0.95 (%), with the highest value bolded and the second highest value bolded and underlined.

[0122]

[0123] This invention uses the DroneVehicle dataset for downstream target detection experiments. A total of 1200 pairs of visible light and infrared images were selected for training, and 300 pairs were used for testing. All image pairs underwent normalization preprocessing to ensure data consistency and quality. After completing the multimodal fusion of visible light and infrared images, the labels corresponding to the targets were redefined to accurately reflect the semantic content of the fusion results. YOLO11 was used for target detection on the fused images, and the outputs of various fusion methods were compared and analyzed. To supplement the qualitative analysis, this paper performs a quantitative evaluation based on the mean average accuracy (mAP) metric. Detection performance was evaluated by calculating mAP within a range of Intersection over Union (IoU) thresholds from 0.5 to 0.95 with a step size of 0.05. As shown in Table 6, the proposed method achieved the highest mAP in all settings, demonstrating its effectiveness in UAV target detection tasks.

[0124] like Figure 14 As shown, this invention outperforms 12 current state-of-the-art methods in vehicle detection tasks. Even in complex environments such as densely parked parking lots, and despite local occlusion issues like tree branches, the method still demonstrates excellent detection performance. Throughout vehicle movement, the model consistently and accurately locates vehicle features. Compared to other methods, this invention exhibits stronger robustness to environmental complexity, with virtually no false positives or false negatives. Experimental results show that MCDFuse effectively captures key target features and extracts core information, demonstrating good reliability and practicality in downstream target detection tasks.

[0125] The implementation of the various embodiments of the present invention is based on programmed processing (i.e., software) through a device with processor functionality. Therefore, in practical engineering, the technical solutions and functions of the various embodiments of the present invention can be encapsulated into various modules. Based on this reality, and building upon the above embodiments, the embodiments of the present invention provide an image fusion device based on Mamba and convolutional neural networks, which is used to execute the image fusion method based on Mamba and convolutional neural networks in the above method embodiments. See also... Figure 2 The device includes: a first main module for acquiring multiple pairs of infrared and visible light images respectively using infrared imaging and visible light imaging devices in the same scene, preprocessing the multiple pairs of images, and constructing a training set and a test set for the model using the preprocessed image pairs; a second main module for constructing an infrared and visible light image fusion model MCDFuse using Mamba and convolutional neural networks; a third main module for training the infrared and visible light image fusion model MCDFuse using the training set, wherein the input of the infrared and visible light image fusion model MCDFuse is the preprocessed infrared and visible light image pairs, and the output is the fused image; and a fourth main module for performing image fusion testing on the test set using the trained infrared and visible light image fusion model MCDFuse.

[0126] The image fusion device based on Mamba and convolutional neural networks provided in this embodiment of the invention employs... Figure 2 Several modules in the method enhance Mamba feature representation through dynamic feature activation module and channel attention mechanism. Combined with ConvNeXt (next-generation convolutional neural network) and inverse residual structure, it improves global receptive field and detail enhancement capability. It solves the problems of insufficient long-distance dependency modeling and computational complexity of existing methods. This method is significantly better than related technologies in terms of fusion accuracy, detail preservation and inference speed. It can improve the downstream target detection performance and is suitable for infrared and visible light image fusion in complex scenes.

[0127] It should be noted that the apparatus in the device embodiments provided by the present invention can be used not only to implement the methods in the above method embodiments, but also to implement the methods in other method embodiments provided by the present invention. The difference lies only in setting corresponding functional modules (i.e., software). Its principle is basically the same as that of the above device embodiments provided by the present invention. As long as those skilled in the art, based on the above device embodiments and referring to the specific technical solutions in other method embodiments, obtain corresponding technical means and technical solutions composed of these technical means by combining technical features, and under the premise of ensuring the practicality of the technical solution, they can improve the apparatus in the above device embodiments to obtain corresponding device-type embodiments (i.e., software) for implementing the methods in other method-type embodiments. For example:

[0128] Based on the above-described device embodiments, as an optional embodiment, the image fusion device based on Mamba and convolutional neural networks provided in this embodiment of the invention further includes: a first submodule, used to implement and preprocess the multiple pairs of image pairs, including: adjusting the infrared image channel to a single channel and the visible light image to a three-channel channel; uniformly adjusting the images to a predetermined resolution; rotating, cropping, and flipping the images; and modifying the image names to ensure sequential correspondence.

[0129] Based on the above-described device embodiments, as an optional embodiment, the image fusion device based on Mamba and convolutional neural networks provided in this embodiment of the invention further includes: a second submodule, used to implement the infrared and visible light image fusion model MCDFuse constructed using Mamba and convolutional neural networks, including: constructing a model composed of an encoder and a decoder, wherein the encoder includes an image reconstruction part, a visible light feature extraction part, and an infrared feature extraction part; the image reconstruction part uses a dynamic feature activation module (DFA) to activate image channel features through a Mamba dynamic scanning mechanism; a dual-branch structure is used to process the feature map, the global feature extraction part uses a hierarchical feature extraction module (HFE) to construct a global feature map based on the next-generation convolutional neural network ConvNeXt, and the local feature enhancement part uses a dual-branch detail enhancement module (DDE) to extract detail features using reversible residual blocks and depth-separating convolution; the decoder has the same structure as the image reconstruction part of the encoder.

[0130] Based on the above-described device embodiments, as an optional embodiment, the image fusion device based on Mamba and convolutional neural networks provided in this embodiment of the invention further includes: a third sub-module, used to implement the image reconstruction part using a Dynamic Feature Activation (DFA) module, which activates image channel features through a Mamba dynamic scanning mechanism, including:

[0131] X l+1,vi =M l (X vi )

[0132] X l+1,ir =M l (X ir )

[0133] Among them, X l+1,vi The visible light image feature map for layer l+1; M l For Mamba operation; X vi Visible light image; X l+1,ir For the feature map of the l+1 layer infrared image; X ir For infrared images; the state-space model includes:

[0134]

[0135] SSM(X) = f out (X1⊙X2)

[0136] in, The symbol for function combination is denoted by ; X1 is the result obtained by performing the corresponding function transformation on the eigenvector X; LN is the normalization function; f SS2D σ is the scanning function; σ is the SiLU activation function; f DW For grouped convolution operations; f proj f represents a linear transformation; gate This represents gated convolution, and ⊙ is the symbol for the Hadamard product; f out X1 is the output result; X2 is the result obtained by performing another corresponding function transformation on the feature vector X; SSM is the state-space model.

[0137] Based on the above-described device embodiments, as an optional embodiment, the image fusion device based on Mamba and convolutional neural networks provided in this embodiment of the invention further includes: a fourth sub-module, used to implement the global feature extraction part using a hierarchical feature extraction module HFE, and to construct a global feature map based on a next-generation convolutional neural network ConvNeXt, including: constructing a hierarchical feature extraction module HFE to extract and output visible light global features, using a hierarchical progressive next-generation convolutional neural network ConvNeXt as a basic module to output infrared global features, integrating the visible light global features and infrared global features according to the linear combination and gating mechanism between channels to obtain integrated global features, and outputting the integrated global features at the top layer of the network.

[0138] Based on the above-described device embodiments, as an optional embodiment, the image fusion device based on Mamba and convolutional neural networks provided in this embodiment of the invention further includes: a fifth sub-module, used to implement the local feature enhancement part by using a dual-branch detail enhancement module (DDE) to extract detail features using reversible residual blocks and depthwise separation convolution, including: constructing a dual-branch detail enhancement module (DDE) for extracting local features, using reversible residual blocks as the basic module, dividing the input features into a first channel and a second channel according to channels, and performing a reversible transformation through an affine coupling layer:

[0139]

[0140] in, This is the first channel; For the second channel; ⊙ is the symbol for the Hadamard product; exp is the symbol for the exponential function; T φ T is the first inverse residual mapping domain; ρ For the second inverse residual mapping domain; T ηis the third inverse residual mapping domain; l is the local feature extracted by function f; Concat is the feature concatenation function; [] is the closed interval symbol; C is the number of channels; X is the feature map; For output features.

[0141] Based on the above device embodiments, as an optional embodiment, the image fusion device based on Mamba and convolutional neural network provided in this embodiment of the invention further includes: a sixth sub-module, used to implement the construction of the decoder including: the decoder adopts two-stage training to generate the fused image, the first stage realizes single-modal reconstruction, reconstructs the original infrared light input image and the original visible light input image, and the second stage focuses on multimodal fusion optimization; the encoder input includes cross-modal and multi-frequency features, and uses Mamba as the basic unit of the decoder.

[0142] The method in this embodiment of the invention is implemented using an electronic device; therefore, it is necessary to introduce the relevant electronic device. For this purpose, this embodiment of the invention provides an electronic device, such as... Figure 3 As shown, the electronic device includes at least one processor, a communications interface, at least one memory, and a communications bus, wherein the at least one processor, the communications interface, and the at least one memory communicate with each other via the communications bus. The at least one processor can invoke logical instructions stored in the at least one memory to execute all or part of the steps of the methods provided in the foregoing method embodiments.

[0143] Furthermore, when the logical instructions in at least one of the aforementioned memories can be implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various method embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0144] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0145] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0146] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Based on this understanding, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, or sometimes in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0147] It should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Any expressions such as "predetermined threshold," "preset threshold," etc., without specifying a particular value, can be determined by those skilled in the art through simple experimentation or appropriate adjustments.

[0148] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image fusion method based on Mamba and convolutional neural networks, characterized in that, include: In the same scene, multiple pairs of infrared and visible light images are acquired by infrared imaging devices and visible light imaging devices, respectively. The multiple pairs of images are preprocessed, and the preprocessed image pairs are used to construct a model training set and a test set. The infrared and visible light image fusion model MCDFuse is constructed using Mamba and convolutional neural networks, including: constructing a model consisting of an encoder and a decoder, wherein the encoder includes an image reconstruction part, a visible light feature extraction part and an infrared feature extraction part; The image reconstruction part employs a Dynamic Feature Activation (DFA) module, which activates image channel features through a Mamba dynamic scanning mechanism. A dual-branch structure is used to process the feature map. The global feature extraction part uses a Hierarchical Feature Extraction (HFE) module, which constructs a global feature map based on the next-generation convolutional neural network ConvNeXt. The local feature enhancement part uses a dual-branch Detail Enhancement (DDE) module, employing reversible residual blocks and depthwise separable convolution to extract detail features. The decoder has the same structure as the encoder's image reconstruction part. The infrared and visible light image fusion model MCDFuse is trained using the training set. The input of the infrared and visible light image fusion model MCDFuse is a preprocessed pair of infrared and visible light images, and the output is the fused image. The trained infrared and visible light image fusion model MCDFuse is then used to perform image fusion tests on the test set.

2. The image fusion method based on Mamba and convolutional neural networks according to claim 1, characterized in that, The preprocessing of the multiple pairs of images includes: adjusting the infrared image channel to a single channel and the visible light image to a three-channel channel; uniformly adjusting the images to a predetermined resolution; rotating, cropping, and flipping the images; and modifying the image names to ensure that the order corresponds.

3. The image fusion method based on Mamba and convolutional neural networks according to claim 2, characterized in that, The image reconstruction section employs a Dynamic Feature Activation (DFA) module, which activates image channel features through a Mamba dynamic scanning mechanism, including: ； ； in, for Layer visible light image feature map; For Mamba computation; Visible light image; for Layered infrared image feature map; For infrared images; the state-space model includes: ； ； ； in, This is a function composition symbol; The result is obtained by performing the corresponding function transformation on the feature vector X; LN is the normalization function; For scanning functions; The SiLU activation function; This is a grouped convolution operation; Represents a linear transformation; ⊙ represents gated convolution, and ⊙ is the symbol for Hadamard product. For the output result; The result is obtained by performing another corresponding function transformation on the feature vector X; SSM is the state-space model.

4. The image fusion method based on Mamba and convolutional neural networks according to claim 3, characterized in that, The global feature extraction part adopts a hierarchical feature extraction module (HFE) and constructs a global feature map based on the next-generation convolutional neural network (ConvNeXt). It includes: constructing a hierarchical feature extraction module (HFE) to extract and output visible light global features; using a hierarchical progressive next-generation convolutional neural network (ConvNeXt) as a basic module to output infrared global features; integrating the visible light global features and infrared global features according to the linear combination between channels and the gating mechanism to obtain integrated global features; and outputting the integrated global features at the top layer of the network.

5. The image fusion method based on Mamba and convolutional neural networks according to claim 4, characterized in that, The local feature enhancement part extracts detailed features through a dual-branch detail enhancement module (DDE) using reversible residual blocks and depthwise separable convolution. This includes: constructing a dual-branch detail enhancement module (DDE) for local feature extraction, using reversible residual blocks as the basic module, dividing the input features into a first channel and a second channel, and performing a reversible transformation through an affine coupling layer. ； ； ； ； ； ； in, This is the first channel; This is the second channel; The symbol for the Hadamard product is exp; the symbol for the exponential function is exp. This is the first inverse residual mapping domain; For the second inverse residual mapping domain; For the third inverse residual mapping domain; represents the local features extracted by function f; Concat is the feature concatenation function; [] represents the closed interval symbol; C is the number of channels; X is the feature map; For output features.

6. The image fusion method based on Mamba and convolutional neural networks according to claim 5, characterized in that, The construction of the decoder includes: the decoder uses two-stage training to generate the fused image. The first stage realizes single-modal reconstruction, reconstructing the original infrared light input image and the original visible light input image. The second stage focuses on multimodal fusion optimization. The encoder input contains cross-modal and multi-frequency features, and the Mamba is used as the basic unit of the decoder.

7. An image fusion device based on Mamba and convolutional neural networks, characterized in that, include: The first main module is used to acquire multiple pairs of infrared and visible light images respectively through infrared imaging devices and visible light imaging devices in the same scene, and to preprocess the multiple pairs of image pairs, and to use the preprocessed image pairs to construct a model training set and a test set. The second main module is used to implement the MCDFuse infrared and visible light image fusion model constructed using Mamba and convolutional neural networks. This includes: constructing a model consisting of an encoder and a decoder; the encoder includes an image reconstruction part, a visible light feature extraction part, and an infrared feature extraction part; the image reconstruction part uses a Dynamic Feature Activation (DFA) module to activate image channel features through a Mamba dynamic scanning mechanism; a dual-branch structure is used to process the feature map, with the global feature extraction part using a Hierarchical Feature Extraction (HFE) module to construct the global feature map based on the next-generation convolutional neural network ConvNeXt; and the local feature enhancement part using a dual-branch detail enhancement (DDE) module to extract detail features using reversible residual blocks and depthwise segregating convolution; the decoder has the same structure as the image reconstruction part of the encoder. The third main module is used to train the infrared and visible light image fusion model MCDFuse using the training set. The input of the infrared and visible light image fusion model MCDFuse is a preprocessed pair of infrared and visible light images, and the output is the fused image. The fourth main module is used to perform image fusion testing on the test set using the trained infrared and visible light image fusion model MCDFuse.

8. An electronic device, characterized in that, include: At least one processor, at least one memory, and a communication interface; wherein, The processor, memory, and communication interface communicate with each other; The memory stores program instructions that can be executed by the processor, which invokes the program instructions to perform the method described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions that cause the computer to perform the method described in any one of claims 1 to 6.

Images