A svmamba and dual attention guided infrared and visible image fusion method

Abstract

This invention discloses a method for fusing infrared and visible light images using SVMamba and dual attention guidance, belonging to the fields of artificial intelligence and image processing technology. The method first uses a saliency extraction network to generate a saliency mask from the infrared image and obtains an initial saliency-guided fusion map through adaptive weighting. Then, the infrared and visible light images are input into a saliency-guided SVMamba feature extraction module to extract multi-scale features. The extracted infrared and visible light features are then fed into a dual attention interaction aggregation module, where channel attention and spatial attention are used to model the infrared and visible light features respectively, and the resulting features are collaboratively fused to generate a fused feature. Finally, a decoder is used to reconstruct the fused feature to generate a fused image. This invention improves the content perception capability of key image regions and the efficiency of cross-modal feature interaction, generating a fused image with prominent targets, clear details, and consistent structure.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence and image processing technology, specifically relating to an SVMamba-guided infrared and visible light image fusion method. Background Technology

[0002] In complex environments, such as nighttime, low-light, backlight, and scenes involving smoke, rain, or snow, relying solely on a single sensor often fails to yield clear, stable, and sufficiently discriminative visual information. Visible light images contain rich texture details and structural information, but important content is easily lost in insufficient light. Infrared images, on the other hand, can stably highlight target areas, especially heat sources such as pedestrians and vehicles, through thermal radiation imaging. However, infrared images often lack texture and structural details, resulting in poor image naturalness.

[0003] Therefore, this paper proposes a deep learning-based method for the fusion of infrared and visible light images. The infrared and visible light image fusion technology integrates the complementary information of the two spectra to generate a fused image that combines the thermal radiation information of the infrared image and the texture details of the visible light image, thereby achieving a more complete presentation of image information.

[0004] In specific application scenarios, particularly in security monitoring, military reconnaissance, and autonomous driving, critical target areas often have higher mission importance. Therefore, accurate identification and effective enhancement of these areas are crucial for obtaining practically valuable fusion results. Furthermore, existing fusion methods have limited capabilities in multi-scale, cross-modal feature fusion, easily weakening or omitting key information at different scales, resulting in insufficient integration of infrared salient targets with visible light textures. Summary of the Invention

[0005] Based on the technical problems existing in the background technology, this invention proposes an infrared and visible light image fusion method guided by SVMamba and dual attention. The method achieves infrared and visible light image fusion through a saliency mask-guided SVMamba model and a dual attention interaction mechanism, so as to improve the ability of the fused image to express salient targets and structural details in specific application scenarios.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A method for fusing infrared and visible light images using SVMamba and dual attention guidance includes the following steps:

[0008] S1: Acquire the visible light image to be fused. With infrared images ;

[0009] S2: Transfer the infrared image The input saliency extraction network generates a saliency mask, and an initial saliency-guided fusion graph SGFM is obtained based on an adaptive weighting strategy; the adaptive weighting strategy operates as follows:

[0010]

[0011] Where: M is the saliency mask extracted from the infrared image using U²-Net, Infrared image, Visible light image, Element-wise multiplication;

[0012] S3: For visible light images respectively Infrared images The initial fused image SGFM is then used for multi-level image patch embedding operations to uniformly map it to feature spaces of different scales, thereby obtaining multi-scale visible light features. Infrared features and saliency features ;

[0013] S4: Visible light characteristics infrared features The saliency vision module SVMamba is input in parallel at various scales, and the saliency features SF n Guided by this method, multi-scale feature extraction is performed on infrared and visible light images to obtain visible light extracted features. Features extracted with infrared ;

[0014] The SVMamba operation of the saliency vision module is as follows: , i∈{ir,vi}

[0015] in, For SVMamba module operations, Infrared or visible light characteristics, Significance feature;

[0016] S5: Extracting visible light features obtained from SVMamba at various scales Features extracted with infrared The input is the dual attention interaction aggregation module DAIAM, which uses channel attention and spatial attention to interactively model cross-modal features and obtain the corresponding fused features. ;

[0017] The DAIAM module operates as follows:

[0018] in, For DAIAM module operation, Features are extracted for visible light. Features extracted using infrared technology;

[0019] S6: The fusion features obtained in step S5 are... Inject the backbone coding feature stream of the corresponding scale to supplement and enhance multimodal features of the same scale;

[0020] S7: The 4-level fusion features obtained in step S5 are... Input a pre-built feature decoding module and combine it with cross-modal fusion features at various scales to reconstruct the fused image;

[0021] S8: Optimize the fused image by taking the fused image obtained in step S7 and the original visible light image and infrared image obtained in S1 as inputs through a pre-constructed loss function;

[0022] S9: Set the number of iterations for steps S1 to S8, and output the image after iterative processing as a fused infrared and visible light image.

[0023] Furthermore, step S4 is specifically implemented as follows:

[0024] S41: In the saliency vision module SVMamba, the saliency feature SFn first passes through a lightweight mapping module consisting of convolution and a sigmoid function to generate a saliency guiding weight Wn. This weight is used to adjust the feature response of different regions position by position, so that salient regions receive higher attention while background regions are suppressed. Its expression is:

[0025]

[0026] in, 1 This represents a convolution operation with a kernel size of 1. 2 ReLU represents a convolution operation with a kernel size of 3, and ReLU represents a non-linear activation function. The normalized activation function;

[0027] S42: Combine the significance guiding weight Wn obtained in step S41 with the infrared / visible light features (i∈{ir,vi}) shallow features LFn are obtained by multiplying the local features obtained by depthwise separable convolution DWConv pixel by pixel; its expression is:

[0028]

[0029] in, This represents a depthwise separable convolution, where Wn represents the saliency-guided weights. Indicates linear projection. Indicates normalization;

[0030] S43: The shallow feature LFn obtained in step S42 is modeled in two-dimensional selective state space using SS2D to form a global receptive field; and then multiplied again with the saliency weight Wn to generate the deep feature DFn; its expression is:

[0031]

[0032] Where Wn represents the significance-guided weight, and LFn represents the shallow features. Indicates a 2D selective scanning module. Represents a smooth activation function;

[0033] S44: Subsequent operations normalize and remap the deep features DFn using normalized LN and linear projection, and then combine them with the input features. The residuals are summed to obtain the final output features. The output features The specific expression is as follows:

[0034]

[0035] in, Indicates linear projection. Indicates normalization, Indicates deep features, This indicates infrared / visible light encoded features.

[0036] Furthermore, the specific method of step S5 is as follows:

[0037] S51: The dual-attention interaction aggregation module DAIAM first uses spatial attention weights... With channel attention weights The spatial attention weights respectively enhance the response capability of structural and texture information in visible light features and salient targets in infrared features; With channel attention weights The specific expression is:

[0038]

[0039]

[0040] in, Indicates average pooling. This indicates max pooling. This indicates a splicing operation at the channel level. This indicates a convolution operation with a kernel size of 7. This represents a convolution operation with a kernel size of 1. ReLU represents the normalized activation function, while ReLU represents the nonlinear activation function.

[0041] S52: Perform pixel-by-pixel addition and pixel-by-pixel difference operations on the enhanced visible light and infrared features to obtain the common structural features and differential features of the shared regions of the two modes, and form more discriminative cross-modal complementary features through channel stitching. Its expression is:

[0042]

[0043] in, This indicates a splicing operation at the channel level. Represents spatial attention weights. Indicates channel attention weights. This represents the absolute value operation;

[0044] S53: Apply the cross-modal complementary features obtained in step S52 The input convolutional gated units are used for nonlinear fusion to obtain the final fused features. Its expression is:

[0045]

[0046] in, This represents a convolution operation with a kernel size of 1. This indicates a pixel-by-pixel multiplication operation. This is the normalized activation function.

[0047] Furthermore, step S8 specifically involves:

[0048] The fused image obtained in step S7 is optimized end-to-end using a pre-constructed joint loss function, which consists of pixel intensity loss, gradient loss, and structural consistency loss. The expression for the loss function is as follows:

[0049]

[0050] in, Represents the joint loss function; and This is a hyperparameter used to balance the proportions of various losses;

[0051] and These represent the pixel intensity loss function, gradient loss function, and structural consistency loss function, respectively.

[0052] The pixel intensity loss The expression used to maintain the consistency of the fused image with the source image in terms of brightness is:

[0053]

[0054] in, Describing the L1 norm, This represents the operation of taking the maximum value at each pixel position from the two input images. Indicates a fused image. These represent infrared images and visible light images, respectively.

[0055] The gradient loss The expression used to ensure that the fused image retains edge and texture details from the source image is:

[0056]

[0057] in, Describing the L1 norm, This represents the operation of taking the maximum value at each pixel position from the two input images. Represents the gradient operator;

[0058] The structural consistency loss Images are fused using multi-scale structural similarity measurement. The degree of structural similarity between the image and the source image is expressed as:

[0059]

[0060] Among them, MS-SSIM This represents a multi-scale structural similarity measure, used to comprehensively assess the structural consistency between images across multiple spatial scales. These represent infrared images and visible light images, respectively.

[0061] An infrared and visible light image fusion system guided by SVMamba and dual attention is disclosed. The image fusion network of the system includes a saliency information extraction module, an SVMamba module, and a dual attention interaction aggregation module (DAIAM). The saliency information extraction module is used to extract the saliency information of features, the SVMamba module is used to capture long-range dependencies and enhance saliency information, and the dual attention interaction aggregation module (DAIAM) is used to combine cross-modal infrared features and visible light features to comprehensively utilize multi-scale features.

[0062] A storage medium for an SVMamba and dual-attention guided infrared and visible light image fusion method, the storage medium having a system of the SVMamba and dual-attention guided infrared and visible light image fusion method, the storage medium having a processor and a memory communicatively connected to the processor; the number of the processor and the memory is not less than one;

[0063] The memory stores instructions that can be executed by the processor, the instructions being used by the processor to execute the SVMamba and dual attention-guided infrared and visible light image fusion method.

[0064] Compared with the prior art, this application has the following beneficial effects:

[0065] This invention designs a novel infrared and visible light image fusion network, SVMA-Net, which guides and enhances the content representation ability of key regions through mask extraction branches and saliency guidance mechanisms. It also dynamically adjusts the feature contribution of channel and spatial dimensions by combining a dual attention interaction strategy, thereby obtaining a fused image with prominent targets and rich details while maintaining computational efficiency.

[0066] This invention addresses the task requirement of higher recognition of key target regions in specific application scenarios by designing a saliency vision module (SVMamba), which combines a saliency mask with a state space model to dynamically enhance the contribution weight of salient targets and strengthen the structural and semantic information of the fused image.

[0067] This invention designs a multi-scale dual attention interaction aggregation module that combines channel attention and spatial attention at different scales. By enhancing the deep interaction between infrared saliency and visible light texture, it improves the detail fidelity and structural consistency of the fused image.

[0068] Compared with existing technologies, this invention demonstrates superior performance across multiple evaluation metrics, effectively capturing significant targets and revealing clear texture details. Ablation experiments further validate the effectiveness of the proposed module. Attached Figure Description

[0069] Figure 1 This is a schematic diagram of the overall network framework structure in an embodiment of the present invention;

[0070] Figure 2 This is a schematic diagram of the SVMamba module in an embodiment of the present invention;

[0071] Figure 3 This is a schematic diagram of the dual attention interaction aggregation module in an embodiment of the present invention. Detailed Implementation

[0072] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Unless otherwise specified, the experimental methods described in the following embodiments are conventional methods.

[0073] This embodiment presents a method for fusing infrared and visible light images using SVMamba and dual attention guidance. The method first uses a saliency extraction network to generate a saliency mask from the infrared image and obtains an initial saliency-guided fusion map through adaptive weighting. Then, the infrared and visible light images are input into a saliency-guided SVMamba feature extraction module to extract multi-scale features. The extracted infrared and visible light features are then fed into a dual attention interaction aggregation module, where channel attention and spatial attention are used to model the infrared and visible light features respectively, and the resulting features are collaboratively fused to generate a fused feature. Finally, a decoder is used to reconstruct the fused feature to generate a fused image. This invention improves the content perception capability of key image regions and the efficiency of cross-modal feature interaction, generating a fused image with prominent targets, clear details, and consistent structure.

[0074] This embodiment proposes a deep feature fusion module in the infrared-visible light image fusion scheme, and designs a novel attention combination mechanism to extract key information from multi-scale feature maps. Based on this, a multi-scale feature interaction module is also proposed, integrating complementary information and details from multiple different scales to create a multi-level feature map, enabling interactive utilization between multi-scale features and thus more fully expressing the features.

[0075] This embodiment proposes an SVMamba module that introduces saliency information to focus on modeling key regions, thereby enhancing the ability of the feature map to express important targets and structural information. Based on this, a multi-scale dual-attention interactive aggregation module is designed. Through the synergistic effect of channel attention and spatial attention, it achieves effective complementarity and interactive fusion between infrared saliency features and visible light texture details, thus fully integrating semantic and detail information from different levels of features.

[0076] The method in this embodiment is as follows:

[0077] Step 1: The infrared image is input into the saliency information extraction branch to generate a saliency mask, and then fused with the visible light image through an adaptive weighting method to obtain a saliency-guided fusion map, which is used to provide global saliency constraints for subsequent feature extraction;

[0078] Step 2: Input the infrared image and the visible light image into the dual-channel encoder structure respectively, extract features through the saliency visual Mamba (SVMamba) module, selectively enhance key regions under the action of the saliency guidance mechanism, and obtain multi-scale feature representations containing local details and global context information.

[0079] Step 3: Input infrared features and visible light features at different scales into the dual attention interaction aggregation module. Through the collaborative modeling of channel attention and spatial attention, the two modalities are effectively fused at the level of shared structural information and complementary difference information to obtain the fused feature representation.

[0080] Step 4: Input the fused multi-scale features into the decoder. Through layer-by-layer feature reconstruction and upsampling operations, the spatial resolution is gradually restored, and cross-modal features in skip connections are fused to generate a fused image with consistent structure and clear details.

[0081] Step 5: Iterate according to steps 1 to 4 until stopping, and output the final fused image.

[0082] In this invention, the iterative training of the fusion network will not be described in detail. Instead, existing datasets can be combined to introduce neural networks for effective training, and the trained network can be output for use.

[0083] The specific method is as follows:

[0084] Includes the following steps:

[0085] S1: Acquire the visible light image to be fused. With infrared images ;

[0086] S2: Transfer the infrared image The input saliency extraction network generates a saliency mask, and an initial saliency-guided fusion graph SGFM is obtained based on an adaptive weighting strategy; the adaptive weighting strategy operates as follows:

[0087]

[0088] Where: M is the saliency mask extracted from the infrared image using U²-Net, Infrared image, Visible light image, Element-wise multiplication;

[0089] S3: For visible light images respectively Infrared images The initial fused image is subjected to multi-level patch embedding operations using SGFM to uniformly map it to feature spaces of different scales, thereby obtaining multi-scale visible light features. Infrared features and saliency features ;

[0090] S4: Visible light characteristics infrared features The saliency vision module SVMamba is input in parallel at various scales, and the saliency features SF n Guided by this method, multi-scale feature extraction is performed on infrared and visible light images to obtain visible light extracted features. Features extracted with infrared ;

[0091] The SVMamba operation of the saliency vision module is as follows: , i∈{ir,vi}

[0092] in, For SVMamba module operations, Infrared or visible light characteristics, Significance feature;

[0093] The specific method of step S4 is as follows: S41: In the saliency vision module SVMamba, the saliency feature SFn first passes through a lightweight mapping module composed of convolution and the sigmoid function to generate a saliency guiding weight Wn, which is used to adjust the feature response of different regions position by position, so that the salient regions receive higher attention while the background regions are suppressed; its expression is:

[0094]

[0095] in, 1 This represents a convolution operation with a kernel size of 1. 2 ReLU represents a convolution operation with a kernel size of 3, and ReLU represents a non-linear activation function. The normalized activation function;

[0096] S42: Combine the significance guiding weight Wn obtained in step S41 with the infrared / visible light features (i∈{ir,vi}) shallow features LFn are obtained by multiplying the local features obtained by depthwise separable convolution DWConv pixel by pixel; its expression is:

[0097]

[0098] in, This represents a depthwise separable convolution, where Wn represents the saliency-guided weights. Indicates linear projection. Indicates normalization;

[0099] S43: The shallow feature LFn obtained in step S42 is modeled in two-dimensional selective state space using SS2D to form a global receptive field; and then multiplied again with the saliency weight Wn to generate the deep feature DFn; its expression is:

[0100]

[0101] Where Wn represents the significance-guided weight, and LFn represents the shallow features. Indicates a 2D selective scanning module. Represents a smooth activation function;

[0102] S44: Subsequent operations normalize and remap the deep features DFn using normalized LN and linear projection, and then combine them with the input features. The residuals are summed to obtain the final output features. The output features The specific expression is as follows:

[0103]

[0104] in, Indicates linear projection. Indicates normalization, Indicates deep features, Indicating infrared / visible light encoded features

[0105] S5: Extracting visible light features obtained from SVMamba at various scales Features extracted with infrared The input is the dual attention interaction aggregation module DAIAM, which uses channel attention and spatial attention to interactively model cross-modal features and obtain the corresponding fused features. ;

[0106] The DAIAM module operates as follows:

[0107] in, For DAIAM module operation, Features are extracted for visible light. Features extracted using infrared technology;

[0108] The specific method for step S5 is as follows:

[0109] S51: The dual-attention interaction aggregation module DAIAM first uses spatial attention weights... With channel attention weights The spatial attention weights respectively enhance the response capability of structural and texture information in visible light features and salient targets in infrared features; With channel attention weights The specific expression is:

[0110]

[0111]

[0112] in, Indicates average pooling. This indicates max pooling. This indicates a splicing operation at the channel level. This indicates a convolution operation with a kernel size of 7. This represents a convolution operation with a kernel size of 1. ReLU represents the normalized activation function, while ReLU represents the nonlinear activation function.

[0113] S52: Perform pixel-by-pixel addition and pixel-by-pixel difference operations on the enhanced visible light and infrared features to obtain the common structural features and differential features of the shared regions of the two modes, and form more discriminative cross-modal complementary features through channel stitching. Its expression is:

[0114]

[0115] in, This indicates a splicing operation at the channel level. Represents spatial attention weights. Indicates channel attention weights. This represents the absolute value operation;

[0116] S53: Apply the cross-modal complementary features obtained in step S52 The input convolutional gated units are used for nonlinear fusion to obtain the final fused features. Its expression is:

[0117]

[0118] in, This represents a convolution operation with a kernel size of 1. This indicates a pixel-by-pixel multiplication operation. This is the normalized activation function.

[0119] S6: The fusion features obtained in step S5 are... Inject the backbone coding feature stream of the corresponding scale to supplement and enhance multimodal features of the same scale;

[0120] S7: The 4-level fusion features obtained in step S5 are... Input a pre-built feature decoding module and combine it with cross-modal fusion features at various scales to reconstruct the fused image;

[0121] S8: Optimize the fused image by taking the fused image obtained in step S7 and the original visible light image and infrared image obtained in S1 as inputs through a pre-constructed loss function;

[0122] S9: Set the number of iterations for steps S1 to S8, and output the image after iterative processing as a fused infrared and visible light image.

[0123] The fused image obtained in step S7 is optimized end-to-end using a pre-constructed joint loss function, which consists of pixel intensity loss, gradient loss, and structural consistency loss. The expression for the loss function is as follows:

[0124]

[0125] in, Represents the joint loss function; and This is a hyperparameter used to balance the proportions of various losses;

[0126] and These represent the pixel intensity loss function, gradient loss function, and structural consistency loss function, respectively.

[0127] The pixel intensity loss The expression used to maintain the consistency of the fused image with the source image in terms of brightness is:

[0128]

[0129] in, Describing the L1 norm, This represents the operation of taking the maximum value at each pixel position from the two input images. Indicates a fused image. These represent infrared images and visible light images, respectively.

[0130] The gradient loss The expression used to ensure that the fused image retains edge and texture details from the source image is:

[0131]

[0132] in, Describing the L1 norm, This represents the operation of taking the maximum value at each pixel position from the two input images. Represents the gradient operator;

[0133] The structural consistency loss Images are fused using multi-scale structural similarity measurement. The degree of structural similarity between the image and the source image is expressed as:

[0134]

[0135] Among them, MS-SSIM This represents a multi-scale structural similarity measure, used to comprehensively assess the structural consistency between images across multiple spatial scales. These represent infrared images and visible light images, respectively.

[0136] The specific details of each of the above steps are as follows:

[0137] The overall network structure is as follows Figure 1 As shown, it mainly consists of three parts: the saliency information extraction branch, the SVMamba module (Salient Visual Mamba, SVMamba), and the Dual-Attention Interactive Aggregation Module (DAIAM).

[0138] The key to this method is to introduce saliency information to enhance the structural features in the fused image and further highlight the saliency target.

[0139] In this invention, the image fusion network consists of a saliency information extraction module, an SVMamba module, and a fusion module, which work together to generate the final fused image. Specifically, the saliency information extraction module extracts the saliency information of features, the SVMamba module captures long-range dependencies and enhances the saliency information, and the fusion module is responsible for further optimizing and integrating the features.

[0140] In Step 1, the saliency information extraction branch generates an infrared saliency mask using U²-Net. A preliminary saliency-guided fusion map (SGFM) is obtained through a designed adaptive weighting strategy (Mask-Guided Fusion). Subsequently, the SGFM is projected onto feature spaces of different dimensions using multi-level patch embedding to construct hierarchical saliency features, which serve as global guiding signals for subsequent feature modulation in each layer.

[0141] The adaptive weighting strategy works as follows:

[0142]

[0143] Where: M is the saliency mask extracted from the infrared image using U²-Net, Infrared image, Visible light image, This is element-wise multiplication.

[0144] like Figure 2 As shown, in SVMamba, the salient feature SFn first passes through a lightweight mapping module composed of convolution and sigmoid to generate saliency-guided weights. These weights are used to adjust the feature responses of different regions positionally, giving higher attention to salient regions while suppressing background regions. Through the dual-input structure design of "backbone features + saliency guidance," SVMamba inherits the global modeling advantage of linear complexity of Mamba while significantly enhancing the model's ability to respond differently to infrared salient targets, visible light textures, and local structures. This provides more discriminative and semantically consistent multi-scale feature representations for subsequent cross-modal fusion.

[0145] The SVMamba module operates as follows: , i∈{ir,vi}

[0146] in, For SVMamba module operations, Visible light characteristics or infrared characteristics, These are features extracted using visible light or infrared light.

[0147] To effectively combine cross-modal infrared and visible light features and comprehensively utilize multi-scale features, this application designs a multi-scale dual-attention interactive aggregation module, such as... Figure 3 As shown.

[0148] The DAIAM module first constructs spatial attention weights for visible light. Infrared channel attention weights ,in Used to highlight structural and texture information in visible light features. Used to enhance the responsiveness of infrared signatures to prominent targets. and The specific expression is as follows:

[0149]

[0150]

[0151] In the formula: Indicates average pooling. This indicates max pooling. This indicates a splicing operation at the channel level. This represents a 7×7 convolution, used to expand the receptive field and extract spatial context information. This represents a 1×1 convolution, used for channel compression.

[0152] Building upon this, DAIAM enhances the common representation of the two modalities in congruent regions through pixel-by-pixel addition and explicitly extracts the difference information between the two modalities through interpolation operations, thereby simultaneously modeling "shared structure" and "modal complementarity." This design effectively captures the difference information between visible light texture and infrared thermal radiation, forming more discriminative cross-modal complementary features at the feature level. Furthermore, the feature response capability is further enhanced through convolutional gating units to obtain the final fused features. The specific operation is shown in the following formula:

[0153]

[0154]

[0155] In the formula: This represents the absolute value operation. This indicates features extracted using visible light. Indicates infrared extraction features, This indicates a splicing operation at the channel level. Represents a 1×1 convolution. This is the normalized activation function.

[0156] To improve the quality and semantic richness of the fused images, this application employs a joint loss function composed of a fusion loss function and a segmentation loss function to train the network. The joint loss function consists of pixel intensity loss, gradient loss, and structural consistency loss. The expression for the loss function is as follows:

[0157]

[0158] in, Represents the joint loss function; and This is a hyperparameter used to balance the proportions of various losses;

[0159] and These represent the pixel intensity loss function, gradient loss function, and structural consistency loss function, respectively.

[0160] Pixel intensity loss The expression used to maintain the consistency of the fused image with the source image in terms of brightness is:

[0161]

[0162] in, Describing the L1 norm, This represents the operation of taking the maximum value at each pixel position from the two input images. Indicates a fused image. These represent infrared images and visible light images, respectively.

[0163] Gradient loss The expression used to ensure that the fused image retains edge and texture details from the source image is:

[0164]

[0165] in, Describing the L1 norm, This represents the operation of taking the maximum value at each pixel position from the two input images. Represents the gradient operator;

[0166] Structural consistency loss Images are fused using multi-scale structural similarity measurement. The degree of structural similarity between the image and the source image is expressed as:

[0167]

[0168] Among them, MS-SSIM This represents a multi-scale structural similarity measure, used to comprehensively assess the structural consistency between images across multiple spatial scales. These represent infrared images and visible light images, respectively.

[0169] To better illustrate the solution and practical effects of this application, specific embodiments are provided to explain the implementation of the invention. Training and testing of this invention are conducted on public datasets to ensure the reliability and fairness of the experimental results. Specifically, three datasets were used: the MSRS dataset, the LLVIP dataset, and the TNO dataset. Training was performed on the MSRS dataset. The MSRS dataset is specifically designed for spectral road scenes and contains 1444 pairs of infrared and visible light images, covering various types of target objects such as cars, pedestrians, and bicycles. To verify the model's generalization ability, testing was conducted on the LLVIP and TNO datasets. The LLVIP test set contains 44 pairs of infrared and visible light images in low-light environments to test the model's stability in low-light scenes; the TNO test set contains 40 pairs of typical outdoor scene images and is one of the most commonly used datasets for IVIF tasks.

[0170] The training process uses the Adam optimizer to train the fusion framework, and the learning rate is adjusted in segments according to the training phases: the learning rate for the first 50 epochs is 1×10. −3 The learning rate for epochs 50–80 is 1×10⁻⁶. −4 The learning rate is 1×10 for 80–100 epochs. −5 The value was set to 0.001, the batch size was 4, and 100 complete iterations were performed to update the framework parameters. The experiment was conducted using the PyTorch framework, with the computing environment configured as Ubuntu 22.04, a GTX 4090 GPU, an Intel(R) Xeon(R) Silver 4310, and 30 GBRAM. The parameters of the fused graph on the LLVIP dataset are shown in Table 1.

[0171] Table 1. Mean values ​​of various metrics for different methods on the LLVIP dataset.

[0172] Method Year EN SF AG SD CC SCD DenseFuse 2018 6.872 14.454 4.386 37.021 0.787 1.594 U2Fusion 2020 6.219 14.237 4.323 33.159 0.765 1.248 RFN-Nest 2021 6.947 8.508 3.275 40.278 0.780 ]]> ​ 1.616 ]]> ​ SwinFusion 2022 6.802 16.003 4.907 39.000 0.746 1.385 Tardal 2022 6.114 9.010 2.756 26.075 0.720 0.783 YDTR 2022 6.094 10.798 2.873 23.918 0.763 0.732 SOSMaskFuse 2023 6.658 14.661 4.306 35.492 0.743 1.255 CDDFuse 2023 6.923 16.630 4.762 40.881 0.756 1.564 CrossFuse 2024 6.432 13.529 4.039 26.847 0.770 1.063 ITFuse 2024 6.330 6.011 2.233 26.784 0.770 0.952 FusionMamba 2024 6.906 17.053 4.719 41.639 0.748 1.518 DCEvoFuse 2025 6.927 17.237 ]]> ​ 5.027 40.606 0.742 1.446 SFMFusion 2025 7.002 ]]> ​ 17.402 5.097 ]]> ​ 43.212 ]]> ​ 0.735 1.487 OURS 2025 7.139 16.392 5.577 46.925 0.762 1.756 ]]> ​

[0173] Table 1 shows the average values ​​of various evaluation metrics, with the best and second-best results indicated by bold and underline, respectively. According to the data in Table 1, this method performs optimally in EN, AG, SD, and SCD metrics. EN describes the richness of information contained in the image. AG is an important indicator of edge sharpness and detail sharpness. SD represents the overall distribution characteristics of image brightness, indirectly reflecting the contrast levels and the distinctness of light and dark relationships. SCD reflects the comprehensive preservation of the differences between the source images by the fused image. This indicates that the fused image generated by this algorithm not only has richer information content but also possesses a higher gradient structure, clearer brightness levels, and can more effectively integrate complementary information from the source images.

[0174] To further illustrate the fusion performance of this method, experiments were conducted on 40 pairs of images from the TNO dataset in this embodiment. The parameters of the fused image on the TNO dataset are shown in Table 2.

[0175] Table 2. Mean values ​​of various metrics for different methods on the LLVIP dataset.

[0176] Method Year EN SF AG SD CC SCD DenseFuse 2018 6.872 12.627 4.746 36.428 0.529 1.814 ]]> ​ U2Fusion 2020 6.978 15.277 6.176 ]]> ​ 37.536 0.519 ]]> ​ 1.786 RFN-Nest 2021 7.035 7.959 3.573 38.822 0.517 1.789 SwinFusion 2022 6.893 14.203 5.387 39.396 0.473 1.720 Tardal 2022 6.842 11.717 3.985 43.991 0.437 1.535 YDTR 2022 6.414 10.774 3.709 27.856 0.490 1.559 SOSMaskFuse 2023 7.085 13.951 5.077 46.146 0.426 1.510 CDDFuse 2023 7.103 ]]> ​ 16.955 ]]> ​ 6.111 45.554 ]]> ​ 0.483 1.769 CrossFuse 2024 6.948 13.690 4.998 41.200 0.406 1.372 ITFuse 2024 6.039 5.134 2.155 21.583 0.515 1.454 FusionMamba 2024 6.766 15.934 5.457 42.546 0.470 1.711 DCEvoFuse 2025 6.931 15.324 5.461 39.775 0.468 1.678 SFMFusion 2025 6.823 15.964 5.662 40.665 0.465 1.666 OURS 2025 7.193 17.348 6.765 44.046 0.507 1.830

[0177] On the TNO dataset, this invention achieves state-of-the-art results in EN, SF, AG, and SCD. The leading performance in SF and AG indicates that this method is superior in detail texture reconstruction and edge structure preservation. The highest SCD value demonstrates that cross-modal complementary features are effectively preserved, and the fused image exhibits excellent performance in structural consistency and differential integration. Although not absolutely optimal in CC and SD metrics, the overall metrics remain among the top, indicating that the algorithm achieves a good balance in maintaining brightness levels and modal correlation.

[0178] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for fusing infrared and visible light images using SVMamba and dual attention guidance, characterized in that, Includes the following steps: S1: Obtain the visible light image to be fused. With infrared images ; S2: Transfer the infrared image The input saliency extraction network generates a saliency mask, and an initial saliency-guided fusion graph SGFM is obtained based on an adaptive weighting strategy; the adaptive weighting strategy operates as follows: Where: M is the saliency mask extracted from the infrared image using U²-Net, Infrared image, Visible light image, Element-wise multiplication; S3: For visible light images respectively Infrared images The initial fused image (SGFM) undergoes multi-level patch embedding operations to uniformly map it to feature spaces of different scales, thereby obtaining multi-scale visible light features. Infrared features and saliency features ; S4: Visible light characteristics infrared features The saliency vision module SVMamba is input in parallel at various scales, and the saliency features SF n Guided by this method, multi-scale feature extraction is performed on infrared and visible light images to obtain visible light extracted features. Features extracted with infrared ; The SVMamba operation of the saliency vision module is as follows: , i∈{ir,vi} in, For SVMamba module operations, Infrared or visible light characteristics, Significant features; S5: Extracting visible light features obtained from SVMamba at various scales Features extracted with infrared The input is the dual attention interaction aggregation module DAIAM, which uses channel attention and spatial attention to interactively model cross-modal features and obtain the corresponding fused features. ; The DAIAM module operates as follows: in, For DAIAM module operation, Features are extracted for visible light. Features extracted using infrared technology; S6: The fusion features obtained in step S5 are... Inject the backbone coding feature stream of the corresponding scale to supplement and enhance multimodal features of the same scale; S7: The 4-level fusion features obtained in step S5 are... Input a pre-built feature decoding module and combine it with cross-modal fusion features at various scales to reconstruct the fused image; S8: Optimize the fused image by taking the fused image obtained in step S7 and the original visible light image and infrared image obtained in S1 as inputs through a pre-constructed loss function; S9: Set the number of iterations for steps S1 to S8, and output the image after iterative processing as a fused infrared and visible light image.

2. The SVMamba and dual attention-guided infrared and visible light image fusion method according to claim 1, characterized in that, The specific method of step S4 is as follows: S41: In the saliency vision module SVMamba, the saliency feature SFn first passes through a lightweight mapping module composed of convolution and the sigmoid function to generate saliency guiding weights Wn, which are used to adjust the feature response of different regions position by position, so that the saliency region receives more attention while the background region is suppressed. Its expression is: in, 1 This represents a convolution operation with a kernel size of 1. 2 ReLU represents a convolution operation with a kernel size of 3, and ReLU represents a non-linear activation function. The normalized activation function; S42: Combine the significance guiding weight Wn obtained in step S41 with the infrared / visible light features (i∈{ir,vi}) shallow features LFn are obtained by multiplying the local features obtained by depthwise separable convolution DWConv pixel by pixel; its expression is: in, This represents a depthwise separable convolution, where Wn represents the saliency-guided weights. Indicates linear projection. Indicates normalization; S43: The shallow feature LFn obtained in step S42 is modeled in two-dimensional selective state space using SS2D to form a global receptive field; and then multiplied again with the saliency weight Wn to generate the deep feature DFn; its expression is: Where Wn represents the significance-guided weight, and LFn represents the shallow features. Indicates a 2D selective scanning module. Represents a smooth activation function; S44: Subsequent operations normalize and remap the deep features DFn using normalized LN and linear projection, and then combine them with the input features. The residuals are summed to obtain the final output features. The output features The specific expression is as follows: in, Indicates linear projection. Indicates normalization, Indicates deep features, This indicates infrared / visible light encoded features.

3. The SVMamba and dual attention-guided infrared and visible light image fusion method according to claim 1, characterized in that, The specific method for step S5 is as follows: S51: The dual-attention interaction aggregation module DAIAM first uses spatial attention weights... With channel attention weights This enhances the response capability of salient targets in visible light features and infrared features, respectively; the spatial attention weights With channel attention weights The specific expression is: in, Indicates average pooling. This indicates max pooling. This indicates a splicing operation at the channel level. This indicates a convolution operation with a kernel size of 7. This represents a convolution operation with a kernel size of 1. ReLU represents the normalized activation function, while ReLU represents the nonlinear activation function. S52: Perform pixel-by-pixel addition and pixel-by-pixel difference operations on the enhanced visible light and infrared features to obtain the common structural features and differential features of the shared regions of the two modes, and form more discriminative cross-modal complementary features through channel stitching. Its expression is: in, This indicates a splicing operation at the channel level. Represents spatial attention weights. Indicates channel attention weights. This represents the absolute value operation; S53: Apply the cross-modal complementary features obtained in step S52 The input convolutional gated units are used for nonlinear fusion to obtain the final fused features. Its expression is: in, This represents a convolution operation with a kernel size of 1. This indicates a pixel-by-pixel multiplication operation. This is the normalized activation function.

4. The SVMamba and dual attention-guided infrared and visible light image fusion method according to claim 1, characterized in that, Specifically, step S8 is as follows: The fused image obtained in step S7 is optimized end-to-end using a pre-constructed joint loss function, which consists of pixel intensity loss, gradient loss, and structural consistency loss. The expression for the loss function is as follows: in, Represents the joint loss function; and This is a hyperparameter used to balance the proportions of various losses; and These represent the pixel intensity loss function, gradient loss function, and structural consistency loss function, respectively. The pixel intensity loss The expression used to maintain the consistency of the fused image with the source image in terms of brightness is: in, Describing the L1 norm, This represents the operation of taking the maximum value at each pixel position from the two input images. Indicates a fused image. These represent infrared images and visible light images, respectively. The gradient loss The expression used to ensure that the fused image retains edge and texture details from the source image is: in, Describing the L1 norm, This represents the operation of taking the maximum value at each pixel position from the two input images. Represents the gradient operator; The structural consistency loss Images are fused using multi-scale structural similarity measurement. The degree of structural similarity between the image and the source image is expressed as: Among them, MS-SSIM This represents a multi-scale structural similarity measure, used to comprehensively assess the structural consistency between images across multiple spatial scales. These represent infrared images and visible light images, respectively.

5. A system for fusing infrared and visible light images using SVMamba and dual attention guidance, characterized in that: The system's image fusion network includes a saliency information extraction module, an SVMamba module, and a dual attention interaction aggregation module (DAIAM). The saliency information extraction module is used to extract the saliency information of features, the SVMamba module is used to capture long-range dependencies and enhance saliency information, and the dual attention interaction aggregation module (DAIAM) is used to combine cross-modal infrared features and visible light features to comprehensively utilize multi-scale features.

Images