Double-branch infrared and visible light image fusion method based on Mama and CNN

Abstract

The invention discloses an infrared and visible light image fusion method based on Mama and CNN double branches, and relates to the field of image fusion. The method comprises a double-branch encoder unit, a fusion unit and an image reconstruction unit, the double-branch feature encoding unit comprises a Mama branch and a CNN branch, global features of infrared and visible light images are extracted through the Mama branch, local features of the infrared and visible light images are extracted through the CNN branch, the global features and the local features are spliced in a channel dimension, the spliced features are input into the fusion unit, and the fusion unit is used for fusion of the infrared and visible light images. The fusion unit carries out fusion processing on the spliced features through a channel capable of learning the weight and a space attention module, and finally the reconstruction unit generates a fusion image. According to the method, the Mama modeling global semantic capability and the CNN local texture extraction advantage can be fully combined, multi-dimensional information complementary fusion is achieved, the contrast ratio, marginal definition and target discernibility of the fused image are effectively improved, and the method is suitable for scenes such as night vision enhancement, security monitoring and intelligent driving.

Description

Technical Field

[0001] The present invention relates to the field of multimodal image processing, and in particular to a method for fusing infrared and visible light images by combining Mamba state space modeling with a convolutional neural network (CNN). The method is suitable for various practical application scenarios such as nighttime monitoring, autonomous driving, and military reconnaissance. Background Art

[0002] Infrared and visible light image fusion technology aims to combine homologous image information from different imaging devices to produce a fused image with richer information and enhanced visual performance. Infrared images have excellent penetrating power, enabling them to capture thermal radiation information from objects in complex environments such as low illumination and inclement weather, and possess strong environmental adaptability. However, infrared images have limitations in spatial resolution, texture representation, and noise immunity, often manifesting as blurry images, high noise levels, and a loss of detail. In contrast, visible light images offer higher resolution and rich texture detail, accurately depicting the structure and appearance of objects. However, they are susceptible to changes in ambient lighting, particularly at night or in strong backlight conditions, where image quality can significantly degrade. Therefore, fusing the perceptual capabilities of infrared images with the detailed representation of visible light images can complement each other, effectively improving overall image quality and scene understanding. The fused images not only possess enhanced visual perception capabilities but also provide more reliable input support for downstream computer vision tasks such as multimodal saliency detection, object detection, and semantic segmentation, significantly enhancing the robustness and accuracy of models in complex environments. Especially in key tasks such as nighttime monitoring, low-light pedestrian detection, and vehicle recognition, the fusion of infrared and visible light images has shown broad application prospects and has gradually become an important research direction in the field of multimodal fusion.

[0003] In recent years, the development of deep learning technology has provided new solutions for image fusion. In particular, image fusion methods based on deep learning models such as autoencoders (AEs), convolutional neural networks (CNNs), Transformers, and Mamba have demonstrated outstanding performance and application prospects. DenseFuse is a classic AE-based fusion method. This method pre-trains the autoencoder on the MS-COCO dataset and designs two fusion strategies: additive averaging and maximum selection to adapt to the feature requirements of different scenarios. Liu et al. proposed the first CNN-based image fusion method. They used a Siamese CNN to generate a weight map from the source image, decomposed the source image using a Laplacian pyramid, then applied a Gaussian pyramid decomposition to the weight map, and finally performed fusion using a multi-scale approach. To overcome the insufficient global feature extraction of CNN-based methods, Ma et al. introduced cross-domain distance learning and an attention-guided module to fully integrate complementary information and global interactions. The attention mechanism of the Transformer inevitably incurs quadratic computational complexity. MambaDfuse is a two-stage multimodal image fusion model based on Mamba. By designing a bipolar feature extractor and a two-stage feature fusion module, it combines the strengths of convolutional neural networks and Mamba to effectively extract and fuse multimodal features. Although it combines convolution with Mamba, the overall feature extractor design still focuses more on extracting high-level semantic features, resulting in some low-level details being weakened or ignored during the fusion process.

[0004] Therefore, in order to address the problems of existing image fusion methods, such as high computational complexity, insufficient detail preservation, and weak global modeling capabilities, the present invention proposes a dual-branch infrared and visible light image fusion method based on Mamba and CNN. This method designs a dual-branch encoder, uses the Mamba module to extract global thermal information features for infrared and visible light images, and uses the CNN module to extract local texture details, thereby achieving collaborative feature modeling of multimodal images. At the same time, a spatial and channel attention fusion strategy with learnable weights is introduced in the fusion stage to effectively enhance the feature expression of key areas and improve the accuracy and robustness of feature fusion. In addition, to ensure the quality and expressiveness of the fused image, this paper also proposes a decoder for reconstructing the original image. This method has the advantages of low computational overhead, strong adaptability, and simple deployment. It is particularly suitable for visual perception systems with limited resources or high real-time requirements, and has broad application potential in practical scenarios such as intelligent transportation and security monitoring. Summary of the Invention

[0005] To address the above problems, the present invention proposes a dual-branch image fusion method based on Mamba and CNN to achieve efficient extraction and precise fusion of key information in multimodal images. The method mainly includes: proposing a dual-branch encoder structure based on Mamba and CNN, using the Mamba module to extract global thermal radiation features, and adopting the CNN module to obtain local detail features; on this basis, designing a channel and spatial attention fusion strategy with learnable weights, through which fine-grained weighted fusion of multimodal features is performed, and the contribution of different modal features is adaptively adjusted to highlight the structure and texture expression of the fused image, thereby generating fused images with high definition and strong expressiveness in diverse environments, and improving the perceptual performance and robustness of downstream visual tasks.

[0006] The specific contents are as follows:

[0007] (1) A dual-branch encoder structure based on Mamba and CNN is proposed: In view of the complementarity of feature representation between infrared images and visible light images, a dual-branch encoder architecture consisting of Mamba and CNN branches is designed to extract global and local features. After the two branches extract features in parallel, the global and local features extracted by the two branches are fused through channel-dimensional splicing operations to preserve the complementary information in infrared and visible light images.

[0008] (2) A dual-branch channel and attention fusion strategy with learnable weights (LSCM) is proposed: To achieve fine-grained fusion of infrared and visible light images at the feature level, a fusion module consisting of channel attention and spatial attention is designed, and learnable modal weights are introduced to achieve adaptive adjustment of inter-modal information contribution. In the feature fusion stage, this strategy performs weighted fusion of the features output by the dual-branch encoder to highlight key area information, suppress redundant content, and improve the structural clarity and detail expression of the fused image.

[0009] (3) A decoder is designed to reconstruct the original image: the decoder decodes the fused features through a series of convolutional layers to obtain the reconstructed image I F We use a double convolution (DoubleConv) module, where each DoubleConv includes two convolution (Conv) and activation function (ReLU) operations. After passing through two layers of decoding networks, the final reconstructed image is the same size as the original input image. To ensure the quality and accuracy of the output image, we use a 1x1 convolution kernel in the last layer of decoding to match the number of channels of the input image. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a schematic diagram of the overall framework based on the dual-branch fusion of Mamba and CNN in the present invention;

[0011] Figure 2 Schematic diagram of the structure of the dual-branch channel and attention fusion strategy with learnable weights of the present invention;

[0012] Figure 3 Schematic diagram of the decoder structure of the present invention; DETAILED DESCRIPTION

[0013] The present invention is a dual-branch infrared and visible light image fusion method based on Mamba and CNN. The following describes the specific implementation of the present invention, combined with a typical image fusion experiment. Those skilled in the art should understand that these embodiments are intended only to illustrate the technical principles of the present invention and are not intended to limit its scope of application.

[0014] This paper verifies its performance on a public image fusion dataset. The experimental platform uses Python, the deep learning framework PyTorch, and the experimental equipment is an NVIDIA GeForce RTX 4060 Laptop GPU. Model training and inference are both completed on this platform. The specific implementation steps are as follows:

[0015] Step 1: Initialize the network model. The CNN branch aims to model local features, and the convolution kernel size in its convolutional layers, dense blocks, and residual blocks is set to 3. The Mamba branch aims to model global features, and its state space model dimension is 8.

[0016] Step 2: Infrared and visible light image feature extraction. In view of the complementarity of infrared images and visible light images in feature representation, a Mamba and CNN dual-branch encoder is provided in a specific embodiment to extract local and global features. The dual-branch encoder structure of the present invention is mainly divided into three stages: local feature extraction, global feature extraction and feature fusion. Figure 1 As shown in the figure, the CNN branch in the dual-branch encoder realizes local detail feature extraction. The CNN branch extracts features of the image through convolutional layers, dense blocks and residual blocks to obtain I local_IR :

[0017] I local_IR =Dense(Conv(I IR ))+Resnet(Conv(I IR )) (1)

[0018] Among them, Conv represents the convolution layer, which is used to perform preliminary feature extraction. Dense and Resnet represent the dense block and residual block in the CNN branch respectively. The dense block is used to enhance feature flow and capture local information, while the residual block is used to improve deep learning capabilities. VISSince the structure of the encoder is the same, I can be obtained from formula (1) local_VIS .

[0019] For the Mamba branch in the encoder, we use 8 Mamba blocks to extract global features and get I global_IR :

[0020] I global_IR =PU(M8(…M1(PE(F(I IR ))))) (2)

[0021] Among them, F represents the division of the image into patches, PE represents the Patches Embedded operation, which embeds the image into a high-dimensional space, M represents the Mamba block, which aims to extract global features and capture long-range dependencies, and PU represents the Patches UnEmbedded operation, which restores spatial information. Similarly, we can obtain the global feature I of the visible light image extracted by the Mamba branch from formula (2): gobal_VIS

[0022] Finally, the local features output by the CNN branch and the global features output by the Mamba branch are concatenated in the channel dimension to obtain the encoder output r1:

[0023] r1=Concat(I local_IR ,I global_IR ) (3)

[0024] Here, Concat represents channel concatenation, which can preserve the features of the original image to the greatest extent. Similarly, the feature r2 of the visible light image can be obtained by formula (3). r1 and r2 are used as inputs of the invention content (2).

[0025] Step 3: Fusion of infrared and visible light features: The infrared image features and visible light image features extracted in the specific implementation plan (1) are fused using a dual-branch channel and attention fusion strategy (LSCM) with learnable weights. This fusion strategy is mainly divided into three stages: channel attention calculation, spatial attention enhancement, and modality weight adaptive fusion. Figure 2 As shown, first, channel attention is applied to the input feature map Perform global average pooling and maximum pooling operations to obtain channel information. The pooling result passes through a shared fully connected layer and a Sigmoid activation function to generate a channel weight map, and obtain CAM (r1):

[0026] CAM(r1)=σ(MLP(MaxPool(r1))+MLP(MaxPool(r1))) (4)

[0027] Among them, σ(·) is the sigmoid activation function, AvgPool(x) and MaxPool(x) represent the average pooling and maximum pooling of the input feature map respectively. Then the channel attention map After multiplying the input feature r1 channel by channel, the feature map r1′ is obtained:

[0028] r1′=CAM(r1)⊙r1 (5)

[0029] where ⊙ represents element-wise multiplication.

[0030] After channel enhancement, spatial attention obtains spatial importance through convolution operations. It dynamically adjusts the feature values ​​of each spatial position by learning the spatial dimensions of the image. Specifically, the feature map of each channel is globally average pooled and max pooled to generate the spatial attention map SAM(r1) of each spatial position:

[0031] SAM(r1)=σ(Conv 7×7 ([AvgPool(r1);MaxPool(r1)])) (6)

[0032] Among them, σ(·) is the sigmoid activation function, Conv 7×7 For the convolution operation with a kernel size of 7×7, “;” represents the concatenation in the channel dimension. Multiply element-wise with r1′ to get the enhanced feature map r1″:

[0033] r1″=SAM(r1)⊙r1′ (7)

[0034] Here, ⊙ represents element-wise multiplication. Similarly, r2″ can be obtained from equations (4), (5), (6), and (7).

[0035] In order to further improve the flexibility and adaptability of fusion, learnable infrared and visible light modal weight parameters are designed. During the training process, the network automatically learns the modal weight coefficients according to the content of the input image, uses Softmax(·) normalization to ensure that the sum of the two is 1, and finally combines the feature maps after attention enhancement in a weighted summation manner to generate the final fusion feature r f :

[0036] r f =w IR ×r1″+w VIS ×r2″ (8)

[0037] Among them, w IR and w VIS are the weights learned during training.

[0038] In order to maintain a high resolution of the fused image and further improve the continuity and smoothness of the fused features, a layer of two-dimensional convolution is performed after fusion for feature refinement. The output of this convolution will serve as the input for feature reconstruction in step four.

[0039] Step 4: Feature reconstruction. The output of feature fusion is further used as the input of feature reconstruction, and decoded through a series of convolutional layers to finally obtain the reconstructed image I F The invention uses a double convolution (DoubleConv) module, each DoubleConv includes two convolution (Conv) and activation function (ReLU) operations, such as Figure 3 After two layers of decoding networks, a reconstructed image with the same size as the original input image is finally generated. To ensure the quality and accuracy of the output image, we use a 1x1 convolution kernel in the last layer of decoding to match the number of channels of the input image.

[0040] Step 5: Construct an infrared and visible light image fusion dataset. The experiment selected the publicly available MSRS infrared-visible light image fusion dataset, covering typical daytime and nighttime scenes, as the validation set. The training data consisted of 1083 pairs of 480×640 image pairs from MSRS. During preprocessing, the training samples were randomly cropped to 64×64 pixels. All images were registered image pairs.

[0041] Step 6: Network training. 365 pairs of registered datasets from the MSRS dataset were selected as the test set, and the batch size was set to 64. The Adam optimizer was used for model training, with 40 training rounds and an initial learning rate of 0.001. In the first half of the training, the learning rate remained unchanged at the initial value, and in the second half of the training, the learning rate was gradually reduced linearly. A composite loss function was used to optimize the model parameters. The composite function consists of an auxiliary strength loss and a gradient loss, with t1 and t2 set to 7 and 50. Auxiliary strength loss:

[0042]

[0043] Among them, H and W represent the height and width of the image respectively, max() represents the maximum selection of the element, and I f represents the fused image, and ‖·‖1 represents the 1-norm.

[0044] Gradient loss:

[0045]

[0046] in, Represents the gradient operator, which represents the texture information of the image. Here we use Sobel to calculate the gradient, and |·| represents the absolute operation.

[0047] Finally, the loss function of the fusion network is the weighted sum of the auxiliary strength loss and the gradient loss, and the contribution of each loss term is balanced by the adjustable weight coefficients t1 and t2 to obtain the final loss

[0048]

Claims

1. A dual-branch infrared and visible light image fusion method based on Mamba and CNN, characterized by: The method comprises a dual-branch feature encoding unit, a fusion unit and an image reconstruction unit. The dual-branch feature encoding unit comprises a Mamba branch and a CNN branch. The global features of the infrared and visible light images are extracted by the Mamba module, and the local features are extracted by the CNN module. The global features and local features of the infrared and visible light images are spliced ​​in the channel dimension and input into the fusion unit. The fusion unit fuses the spliced ​​features through a channel and spatial attention module with learnable weights. The fused feature information is input into the image reconstruction unit for reconstruction, and the final fused image is output.

2. The dual-branch infrared and visible light image fusion network based on Mamba and CNN according to claim 1 is characterized in that: The CNN branch in the dual-branch feature encoding unit includes a convolutional layer, a dense block and a residual block. The infrared image I IR With visible light image I VIS The initial features are extracted through the convolution layer, and the features are processed by dense blocks to obtain texture enhancement features. The feature expression ability is then improved through the residual block, and the local texture features I are finally output. local_IR with I local_VIS The Mamba module in the dual-branch feature encoding unit includes multiple Mamba blocks, I IR with I VIS Global features are extracted through state space modeling. The specific process is as follows: Divide the infrared image into image blocks; Mapping image blocks to high-dimensional space through Patch Embedded operation; Use multi-layer Mamba blocks to extract global features layer by layer; The spatial information is restored through the Patch Unembedded operation, and the global features of infrared and visible light are finally output. gobal_IR with I gobal_VIS ; Use channel concatenation (Concat) to combine local features I local_IR , I local_VIS With global feature I gobal_IR , I gobal_VIS Combine them to get r1 and r2 respectively.

3. The dual-branch infrared and visible light image fusion network based on Mamba and CNN according to claim 1 is characterized in that: The fusion unit includes a channel attention module, a spatial attention module and a learnable weight module. The specific fusion process is as follows: The channel attention module obtains channel information through global average pooling and maximum pooling operations, and combines the fully connected layer with the Sigmoid activation function to generate channel weight maps CAM(r1) and CAM(r2). The channel weight maps are multiplied by the input features channel by channel to obtain enhanced channel features r1′ and r2′; The spatial attention module performs global average pooling and maximum pooling operations on the feature map of each channel, and generates spatial weight maps SAM(r1) and SAM(r2) through 7×7 convolution. The spatial weight map is multiplied element-by-element with the channel-enhanced features to obtain fused features r1″ and r2″. The learnable weight module performs Softmax(·) normalization to ensure that the sum of the two is 1. The feature maps after attention enhancement are combined in a weighted summation manner to generate the final fused feature r f .

4. The dual-branch infrared and visible light image fusion network based on Mamba and CNN according to claim 3 is characterized in that: The calculation formula of the channel weight map generated by the channel attention module is: CAM(r1)=σ(MLP(AvgPool(r1))+MLP(MaxPool(r1))) Where σ(·) is the sigmoid activation function, AvgPool(x) and MaxPool(x) represent average pooling and maximum pooling of the input feature map, respectively. The channel weight map is multiplied by the input feature channel by channel to obtain the enhanced channel feature r1′: r1′=CAM(r1)⊙r1 where ⊙ represents element-wise multiplication.

5. The dual-branch infrared and visible light image fusion network based on Mamba and CNN according to claim 3 is characterized in that: The calculation formula for the spatial weight map generated by the spatial attention module is: SAM(r1)=σ(Conv 7×7 ([AvgPool(r1);MaxPool(r1)])) Among them, σ(·) is the sigmoid activation function, Conv 7×7 The convolution operation with a kernel size of 7×7 is used. ";" represents the concatenation in the channel dimension. The spatial weight map is multiplied element-by-element with the channel-enhanced features to obtain the fused features: r1″=SAM(r1)⊙r1′ where ⊙ represents element-wise multiplication.

6. The dual-branch infrared and visible light image fusion network based on Mamba and CNN according to claim 3, characterized in that: The learnable weight fusion feature is: r f =w IR ×r1″+w VIS ×r2″ Among them, w IR and w VIS are the weights learned during training.

Images