A method for fusion of infrared and visible light images using Mamba

Abstract

The present invention relates to the field of image processing technology and provides a method for fusing infrared and visible light images using Mamba. The method comprises: vi and infrared image I ir Input the shallow feature extractor to perform shallow feature extraction, and obtain the shallow visible light image feature #imgabs0# and the shallow infrared image feature #imgabs1#; respectively input the shallow visible light image feature #imgabs2# and the shallow infrared image feature #imgabs3# into their respective corresponding deep feature extractors for deep feature extraction, and obtain the deep visible light image feature #imgabs4# and the deep infrared image feature #imgabs5#; perform splicing operation on the deep visible light image feature #imgabs6# and the deep infrared image feature #imgabs7# and input them into the feature decoder to obtain the fused image I F The proposed method demonstrates strong generalization capability and can adapt to infrared and visible image fusion tasks of various resolutions without additional fine-tuning.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method for fusing infrared and visible light images using Mamba. Background Art

[0002] Image fusion technology, as a strategy that brings together multiple information sources to enhance image clarity, enrich information levels, and promote deeper image analysis and operations, has made significant progress in many scientific and technological fields such as computer vision, medical imaging, and remote sensing. It is particularly worth mentioning that by fusing the characteristics of infrared and visible light images, the synergy of cross-spectral information is achieved, which greatly improves the ability to detect and recognize objects and enhances the overall quality of the image. This provides a more acute observation for monitoring activities in low light and even night conditions, ensuring high accuracy of target tracking. Therefore, this technology has shown great application potential and strategic value in terms of concealment and accuracy in reconnaissance operations, intelligent deployment of security monitoring systems, high-sensitivity monitoring of the ecological environment, and many other potential areas.

[0003] At present, image fusion techniques are mainly divided into two categories: traditional methods and deep learning-driven methods. Traditional fusion techniques face problems such as information loss, unsatisfactory fusion effects, dependence on artificial feature selection, and high computational costs. In contrast, deep learning-based methods can directly extract feature representations from raw data and more effectively capture the complex structure inside the image. With their excellent generalization ability, these models can adapt to various scenes, lighting conditions, and target types, thereby improving the flexibility and robustness of fusion strategies. Using deep learning architectures, the associations and weight distributions between different image frequency bands can be adaptively learned to achieve dynamic image fusion, thereby improving the consistency and quality of the fusion output. Because of this, image fusion strategies based on deep learning have gradually become a research hotspot. These strategies can be roughly divided into: convolutional neural networks (CNNs), autoencoders (AEs), generative adversarial networks (GANs), diffusion models, and Transformer-based methods. However, each method has its inherent challenges. First, CNN-based methods sacrifice details due to the limitations of semantic understanding and fixed filter sizes; the simple form of AE limits its ability to express complex features and is highly dependent on training data. Secondly, although GAN can generate high-quality images, its training process is often unstable and has poor operability in the latent space. The challenges faced by the diffusion model are mainly high computational costs and long training cycles. Transformer is limited by high computational complexity and memory consumption, and is not as efficient as CNN in processing image position information. Summary of the invention

[0004] The present invention proposes a method for fusing infrared and visible light images using Mamba, so as to solve or partially solve the above problems.

[0005] The present invention provides a method for fusing infrared and visible light images using Mamba, the method comprising:

[0006] Get the visible light image I to be fused vi and infrared image I ir ;

[0007] The visible light images I vi and infrared image I ir Input shallow feature extractor to extract shallow features and obtain shallow visible light image features and shallow infrared image features ;

[0008] The shallow visible light image features are respectively And the shallow infrared image features Input the corresponding deep feature extractor to extract deep features and obtain deep visible light image features and deep infrared image features ,The deep feature extractor includes multiple levels of parallel visual Mamba modules;

[0009] The deep visible light image feature And the deep infrared image features Perform a splicing operation, and input the obtained splicing image features into the feature decoder for feature reconstruction to obtain the fused image I F ;in,

[0010] The visible light images I vi and infrared image I ir Input shallow feature extractor to extract shallow features and obtain shallow visible light image features and shallow infrared image features ,include:

[0011] The visible light images I vi and the infrared image I ir Input the convolution module in each feature extraction path to obtain the first-level visible light image features and first-level infrared image features :

[0012]

[0013] in, Represents a convolution module, whose input channel is 1 and output channel is 8;

[0014] The first-level visible light image features are respectively And the first-level infrared image features Input the first Transformer module in each feature extraction path, wherein a differential amplification fusion feature extraction module is provided between the two first Transformer modules to fuse the infrared image features into the visible light image features to obtain the secondary visible light image features , the visible light image features are integrated into the infrared image features to obtain the secondary infrared image features :

[0015]

[0016] in, represents the first Transformer module, which has 8 channels. Represents nonlinear activation function GeLU, DAF 2 E represents the differential amplification fusion feature extraction module;

[0017] The secondary visible light image features are respectively And the secondary infrared image features Input the reflection convolution module in each feature extraction path to obtain the three-level visible light image features and three-level infrared image features :

[0018]

[0019] in, Represents a reflection-convolution module with 8 input channels and 16 output channels;

[0020] The three-level visible light image features are respectively And the three-level infrared image features Input the second Transformer module in each feature extraction path, wherein a differential amplification fusion feature extraction module is provided between the two second Transformer modules to fuse the infrared image features into the visible light image features to obtain the four-level visible light image features , the visible light image features are integrated into the infrared image features to obtain the four-level infrared image features , taking the four-level visible light image features as shallow visible light image features , taking the four-level infrared image features as shallow infrared image features ;

[0021]

[0022] in, Represents the second Transformer module, which has 16 channels.

[0023] Furthermore, the first Transformer module and the second Transformer module are Transformer modules with the same structure but different numbers of channels, and the Transformer module is composed of a cascade of one or more multiple deconvolution head transposed attention mechanism modules and a gated deconvolution feedforward network module, wherein:

[0024] The operation of the multi-deconvolution head transposed attention mechanism module is expressed as:

[0025]

[0026] in, Transpose the input of the attention mechanism module for the multiple deconvolution heads, Transpose the output of the attention mechanism module for the multiple deconvolution heads, represents 1×1 point-wise convolution, is the query vector, is the key vector, is a value vector, represents a learnable scaling parameter;

[0027] The operation of the gated deconvolution feedforward network module is expressed as:

[0028]

[0029] in, represents the input of the gated deconvolution feedforward network module, represents the output of the gated deconvolution feedforward network module, represents 1×1 point-wise convolution, (·) represents 3×3 depthwise convolution, l=0,1,2, represents the normalization operation, Represents element-wise multiplication.

[0030] Further, the shallow visible light image features are respectively And the shallow infrared image features Input the corresponding deep feature extractor to extract deep features and obtain deep visible light image features and deep infrared image features ,include:

[0031] The shallow visible light image features are respectively And the shallow infrared image features Input the first parallel vision Mamba modules in the respective feature extraction paths, wherein a channel differential amplification fusion feature extraction module is provided between the two first parallel vision Mamba modules to fuse the channel features of the infrared image into the visible light image features to obtain the five-level visible light image features , the channel features of the visible light image are integrated into the infrared image features to obtain the five-level infrared image features :

[0032]

[0033] in, represents the channel differential amplification fusion feature extraction module, represents the first parallel vision Mamba module, which has 16 input channels and 32 output channels;

[0034] The five-level visible light image features are respectively And the five-level infrared image features The second parallel vision Mamba modules in the respective feature extraction paths are input, wherein a position differential amplification fusion feature extraction module is provided between the two second parallel vision Mamba modules to fuse the position features of the infrared image into the visible light image features to obtain the six-level visible light image features. , the position features of the visible light image are integrated into the infrared image features to obtain the six-level infrared image features :

[0035]

[0036] in, represents the position differential amplification fusion feature extraction module, represents the second parallel vision Mamba module, which has 32 input channels and 64 output channels;

[0037] The six-level visible light image features are respectively And the six-level infrared image features Input the third parallel vision Mamba modules in the respective feature extraction paths, wherein a corner differential amplification fusion feature extraction module is provided between the two third parallel vision Mamba modules to fuse the corner features of the infrared image into the visible light image features to obtain the seven-level visible light image features , the corner features of the visible light image are integrated into the infrared image features to obtain the seven-level infrared image features :

[0038]

[0039] in, represents the corner differential amplification fusion feature extraction module, represents the third parallel vision Mamba module, which has 64 input channels and 128 output channels;

[0040] Among them, the first parallel vision Mamba module, the second parallel vision Mamba module and the third parallel vision Mamba module are parallel vision Mamba modules with the same structure but different numbers of channels.

[0041] Furthermore, the parallel vision Mamba module is used to complete the following operations:

[0042]

[0043]

[0044]

[0045] in, represents the Split operation, n represents the level of PViM, n∈[1,3], represents the input features, represents the 1 / N features that are divided, Represents Mamba operation, represents the adjustment factor, represents the concatenation operation, Represents a projection operation.

[0046] Furthermore, the differential amplification fusion feature extraction module includes a cascaded feature fusion submodule and a differential amplification submodule, wherein:

[0047] The feature fusion submodule is used to complete the following operations:

[0048]

[0049]

[0050] in, represents element addition operation, represents element-wise multiplication, represents the concatenation operation; represents the intermediate features output by the parallel vision Mamba module during the k-th level visible light image feature extraction process, represents the intermediate features output by the parallel vision Mamba module during the k-th level infrared image feature extraction process; or, represents the intermediate features output by the Transformer module during the feature extraction of the k-th level visible light image, The intermediate features output by the Transformer module during the feature extraction of the k-th level infrared image; represents the intermediate features of the k-th level visible light image after feature fusion, Represents the intermediate features of the k-th level infrared image after feature fusion;

[0051] The differential amplifier module is used to complete the following operations:

[0052]

[0053]

[0054] in, represents the intermediate features of the k-th level visible light image after differential amplification, Represents the intermediate features of the k-th level infrared image after differential amplification.

[0055] Furthermore, the channel differential amplification fusion feature extraction module includes a cascaded feature fusion submodule, a differential amplification submodule and a channel weighting module, and the channel weighting module is used to complete the following operations:

[0056]

[0057] in, represents the intermediate features of the fifth-level visible light image after channel weighting, represents the intermediate features of the fifth-level infrared image after channel weighting, represents the Sigmoid activation function, where it compresses each element to (0, 1), H and W represent the height and width of the feature respectively, i represents the row index in the feature, and j represents the column index in the feature;

[0058] The five-level visible light image features are expressed as:

[0059]

[0060] The five-level infrared image features are expressed as:

[0061] .

[0062] Furthermore, the position differential amplification fusion feature extraction module includes a cascaded feature fusion submodule, a differential amplification submodule and a position weighting module, and the position weighting module is used to complete the following operations:

[0063]

[0064] in, represents the intermediate features of the sixth-level visible light image after position weighting, represents the intermediate features of the sixth-level infrared image after position weighting, represents the Sigmoid activation function, where it compresses each element to (0, 1), W represents the width of the feature, i represents the row index in the feature, and j represents the column index in the feature;

[0065] The six-level visible light image feature is expressed as:

[0066]

[0067] The six-level infrared image feature is expressed as:

[0068] .

[0069] Furthermore, the corner differential amplification fusion feature extraction module includes a cascaded feature fusion submodule, a differential amplification submodule and a corner weighting module, and the corner weighting module is used to complete the following operations:

[0070]

[0071] in, represents the intermediate features of the seventh-level visible light image after corner weighting, represents the intermediate features of the seventh-level infrared image after corner weighting, b and c are the indexes of samples and channels respectively, i represents the row index in the feature, j represents the column index in the feature, rows and cols are the row and column positions of the corners respectively;

[0072] The seven-level visible light image feature is expressed as:

[0073]

[0074] The seven-level infrared image feature is expressed as:

[0075] .

[0076] Furthermore, the secondary visible light image feature is expressed as:

[0077]

[0078] The secondary infrared image feature is expressed as:

[0079]

[0080] The four-level visible light image feature is expressed as:

[0081]

[0082] The four-level infrared image feature is expressed as:

[0083] .

[0084] The present application provides a method for fusion of infrared and visible light images using Mamba, which constructs a hybrid feature encoder that combines the local feature extraction advantages of convolutional neural networks (CNNs) with the global attention mechanism of Transformer to achieve effective capture of shallow features. On this basis, a set of deep feature encoders is designed, which further enhance the feature representation capabilities based on the parallel vision Mamba architecture. Comprehensive experimental results from a variety of public data sets show that the present invention is significantly superior to a variety of existing advanced methods in both qualitative and quantitative evaluation. In addition, the present invention demonstrates strong generalization capabilities and can adapt to infrared and visible light image fusion tasks of various resolutions without additional fine-tuning.

[0085] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only used for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. In the accompanying drawings:

[0087] Figure 1 This is a flow chart of a method for fusion of infrared and visible light images using Mamba according to an embodiment of the present invention;

[0088] Figure 2 This is a diagram of the infrared and visible light image fusion network structure using Mamba according to an embodiment of the present invention;

[0089] Figure 3 This is a structural diagram of a Transformer module according to an embodiment of the present invention;

[0090] Figure 4 A schematic diagram of a parallel vision Mamba module according to an embodiment of the present invention;

[0091] Figure 5 Schematic diagram of a differential amplification fusion feature extraction module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0092] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0093] Those skilled in the art will understand that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined.

[0094] In order to solve the problems of high computing resource requirements and large training data requirements in the prior art, the present invention proposes a new fusion algorithm. This module innovatively enhances and refines the complementary information in the two modalities from three key dimensions: channel characteristics, spatial layout, and image edge information, further improving the richness and accuracy of features. The following is a detailed description of an infrared and visible light image fusion method using Mamba in an embodiment of the present invention in conjunction with the accompanying drawings.

[0095] like Figure 1 As shown, the infrared and visible light image fusion method using Mamba includes the following steps:

[0096] S1. Obtain the visible light image I to be fused vi and infrared image I ir ;

[0097] S2, respectively, the visible light image I vi and infrared image I ir Input shallow feature extractor to extract shallow features and obtain shallow visible light image features and shallow infrared image features ;

[0098] In an embodiment of the present invention, the shallow feature extractor includes a cascaded Transformer module and a reflection-convolution module, wherein the Transformer block uses attention mechanism and feature encoding technology to perform high-level feature encoding and cross-modal information fusion. The reflection convolution block optimizes the initial features through adaptive preprocessing and convolution feature extraction, and effectively integrates the features of the two modalities.

[0099] S3, respectively, the shallow visible light image features And the shallow infrared image features Input the corresponding deep feature extractor to extract deep features and obtain deep visible light image features and deep infrared image features ,The deep feature extractor includes multiple levels of parallel visual Mamba modules;

[0100] The embodiment of the present invention relies on the parallel visual Mamba architecture to further enhance the capability of feature representation.

[0101] S4, the deep visible light image features And the deep infrared image features Perform a splicing operation, and input the obtained splicing image features into the feature decoder for feature reconstruction to obtain the fused image I F .

[0102] In an embodiment of the present invention, the designed feature decoder integrates three layers of parallel visual Mamba modules, as well as a reflective convolution block and a set of Transformer blocks, which work together to ultimately generate a high-quality fused image.

[0103] Figure 2 The infrared and visible light image fusion network structure diagram using Mamba in an embodiment of the present invention is shown. Figure 2 It can be seen that in the embodiment of the present invention, the encoder part includes a shallow feature extractor composed of a reflection-convolution block + Transformer, and a three-layer cascaded parallel visual Mamba module.

[0104] By the attached Figure 2 It can be seen that in step S2, the visible light image I vi and infrared image I ir Input shallow feature extractor to extract shallow features and obtain shallow visible light image features and shallow infrared image features include:

[0105] S21, respectively, the visible light image I vi and the infrared image I ir Input the convolution module in each feature extraction path to obtain the first-level visible light image features and first-level infrared image features :

[0106] (1)

[0107] in, Represents a convolution module, whose input channel is 1 and output channel is 8;

[0108] S22, respectively, the first-level visible light image features And the first-level infrared image features Input the first Transformer module in each feature extraction path, wherein a differential amplification fusion feature extraction module is provided between the two first Transformer modules to fuse the infrared image features into the visible light image features to obtain the secondary visible light image features , the visible light image features are integrated into the infrared image features to obtain the secondary infrared image features :

[0109] (2)

[0110] in, represents the first Transformer module, which has 8 channels. Represents nonlinear activation function GeLU, DAF 2 E represents the differential amplification fusion feature extraction module;

[0111] S23, respectively, the secondary visible light image features And the secondary infrared image features Input the reflection convolution module in each feature extraction path to obtain the three-level visible light image features and three-level infrared image features :

[0112] (3)

[0113] in, Represents a reflection-convolution module with 8 input channels and 16 output channels;

[0114] S24, respectively, the three-level visible light image features And the three-level infrared image features Input the second Transformer module in each feature extraction path, wherein a differential amplification fusion feature extraction module is provided between the two second Transformer modules to fuse the infrared image features into the visible light image features to obtain the four-level visible light image features , the visible light image features are integrated into the infrared image features to obtain the four-level infrared image features , taking the four-level visible light image features as shallow visible light image features , taking the four-level infrared image features as shallow infrared image features ;

[0115] (4)

[0116] in, Represents the second Transformer module, which has 16 channels.

[0117] Further, in step S3, the shallow visible light image features are respectively And the shallow infrared image features Input the corresponding deep feature extractor to extract deep features and obtain deep visible light image features and deep infrared image features ,include:

[0118] S31, respectively, the shallow visible light image features And the shallow infrared image features Input the first parallel vision Mamba modules in the respective feature extraction paths, wherein a channel differential amplification fusion feature extraction module is provided between the two first parallel vision Mamba modules to fuse the channel features of the infrared image into the visible light image features to obtain the five-level visible light image features , the channel features of the visible light image are integrated into the infrared image features to obtain the five-level infrared image features :

[0119] (5)

[0120] in, represents the channel differential amplification fusion feature extraction module, represents the first parallel vision Mamba module, which has 16 input channels and 32 output channels;

[0121] S32, respectively, the five-level visible light image features And the five-level infrared image features The second parallel vision Mamba modules in the respective feature extraction paths are input, wherein a position differential amplification fusion feature extraction module is provided between the two second parallel vision Mamba modules to fuse the position features of the infrared image into the visible light image features to obtain the six-level visible light image features. , the position features of the visible light image are integrated into the infrared image features to obtain the six-level infrared image features :

[0122] (6)

[0123] in, represents the position differential amplification fusion feature extraction module, represents the second parallel vision Mamba module, which has 32 input channels and 64 output channels;

[0124] S33, respectively, the six-level visible light image features And the six-level infrared image features Input the third parallel vision Mamba modules in the respective feature extraction paths, wherein a corner differential amplification fusion feature extraction module is provided between the two third parallel vision Mamba modules to fuse the corner features of the infrared image into the visible light image features to obtain the seven-level visible light image features , the corner features of the visible light image are integrated into the infrared image features to obtain the seven-level infrared image features :

[0125] (7)

[0126] in, represents the corner differential amplification fusion feature extraction module, Represents a third parallel vision Mamba module, which has 64 input channels and 128 output channels. It can be understood that the first parallel vision Mamba module, the second parallel vision Mamba module and the third parallel vision Mamba module are parallel vision Mamba modules with the same structure but different numbers of channels.

[0127] Further, Figure 3 The structure diagram of the Transformer module of the embodiment of the present invention is shown. In the embodiment of the present invention, the first Transformer module and the second Transformer module are Transformer modules with the same structure but different numbers of channels. The Transformer block promotes the effective fusion and enhancement of cross-modal features through the attention mechanism and feature encoding capability. Figure 3 It can be seen that the Transformer module of the embodiment of the present invention is composed of a cascaded combination of one or more multiple deconvolution head transposed attention mechanism modules and a gated deconvolution feedforward network module.

[0128] The multi-deconvolution head transposed attention mechanism is a technique that aims to capture multi-scale features through multiple deconvolution heads and use the transposed attention mechanism to strengthen the representation of important features. The transposed attention mechanism allows the model to focus on different parts of the input features, thereby enhancing or suppressing the features of certain specific areas. Figure 3 As shown in (a), the operation of the multi-deconvolution head transposed attention mechanism module is expressed as:

[0129] (8)

[0130] in, Transpose the input of the attention mechanism module for the multiple deconvolution heads, Transpose the output of the attention mechanism module for the multiple deconvolution heads, represents 1×1 point-wise convolution, is the query vector, is the key vector, is a value vector, represents a learnable scaling parameter.

[0131] Here, , , They are all feature vectors extracted from pictures. is the query vector, which indicates the interest or focus of the current processing location on other locations. . is a key vector, a vector used to match the query vector. The key vector provides some features or information about the input position. . is a value vector, which is the information that we actually want to pay attention to. It contains the actual content information of the input position. Display pictures After reshaping the tensor, H×W represents the spatial dimension, C represents the number of channels, The role of is to apply softmax(·) control In addition, Figure 3 The R in the figure indicates tensor reshaping.

[0132] Furthermore, the gated deconvolution feedforward network is a neural network architecture that combines deconvolution operations with a gating mechanism. The deconvolution operation is used to restore the spatial resolution of the image, while the gating mechanism allows the network to dynamically adjust the importance of each channel, thereby effectively fusing information from different layers. In image fusion tasks, the gated deconvolution feedforward network can help retain the key features of the original image while reducing noise and redundant information. Figure 3 As shown in (b), the operation of the gated deconvolution feedforward network module is expressed as:

[0133] (9)

[0134] in, represents the input of the gated deconvolution feedforward network module, represents the output of the gated deconvolution feedforward network module, represents 1×1 point-wise convolution, (·) represents 3×3 depthwise convolution, l=0,1,2, represents the normalization operation, Represents element-wise multiplication.

[0135] Figure 4FIG. 1 shows a schematic diagram of a parallel vision Mamba module in an embodiment of the present invention, wherein: Figure 4 (a) is a structural diagram of the parallel vision Mamba module of an embodiment of the present invention, Figure 4 (b) is the specific structure diagram of the Mamba module, consisting of Figure 4 It can be seen that after shallow extraction, the four-level visible light image features and four-level infrared image features First, it passes through the normalization layer, and then is divided into features with each channel number C / N. Each independent feature is sent to the Mamba model for processing, and its output is integrated with the adjustment factor via the residual connection. Specifically, the parallel vision Mamba module is used to complete the following operations:

[0136] (10)

[0137] (11)

[0138] (12)

[0139] in, represents the Split operation, n represents the level of PViM, n∈[1,3], represents the input features, represents the 1 / N features that are divided, Represents Mamba operation, represents the adjustment factor, represents the concatenation operation, Represents a projection operation, and N can be 8.

[0140] Figure 5 FIG. 1 shows a differential amplification fusion feature extraction module of an embodiment of the present invention, wherein it should be noted that the differential amplification fusion feature extraction module of the embodiment of the present invention is the same as the first two structures of the specific structures of the subsequent channel differential amplification fusion feature extraction module, the position differential amplification fusion feature extraction module and the corner differential amplification fusion feature extraction module. Therefore, Figure 5 It can be seen that the differential amplification fusion feature extraction module includes a feature fusion submodule and a differential amplification submodule, wherein:

[0141] The feature fusion submodule is used to complete the following operations:

[0142] (13)

[0143] (14)

[0144] in, represents element addition operation, represents element-wise multiplication, represents the concatenation operation; represents the intermediate features output by the parallel vision Mamba module during the k-th level visible light image feature extraction process, represents the intermediate features output by the parallel vision Mamba module during the k-th level infrared image feature extraction process; or, represents the intermediate features output by the Transformer module during the feature extraction of the k-th level visible light image, The intermediate features output by the Transformer module during the feature extraction of the k-th level infrared image; represents the intermediate features of the k-th level visible light image after feature fusion, Represents the intermediate features of the k-th level infrared image after feature fusion;

[0145] Furthermore, the fused features are differentially amplified, and the role of differential amplification is to enhance the contrast of complementary features between the two images, reduce redundant information, thereby optimizing information extraction and generating a higher quality fused image. The differential amplification submodule is used to complete the following operations:

[0146] (15)

[0147] (16)

[0148] in, represents the intermediate features of the k-th level visible light image after differential amplification, Represents the intermediate features of the k-th level infrared image after differential amplification.

[0149] It should be noted that in the shallow feature extraction stage, the differential amplification fusion feature extraction module does not integrate the weighting strategy. Therefore, the differential amplification fusion feature extraction module directly adds the output features of the Transformer module to obtain the image features of the current level output. Specifically, based on the attached Figure 1 It can be seen that the secondary visible light image feature is expressed as:

[0150] (17)

[0151] The secondary infrared image feature is expressed as:

[0152] (18)

[0153] The four-level visible light image feature is expressed as:

[0154] (19)

[0155] The four-level infrared image feature is expressed as:

[0156] (20)

[0157] Furthermore, in the deep feature extraction process, the differential amplification fusion feature extraction module applied to three consecutive parallel visual Mamba layers introduces weighted components of channels, positions and corners respectively.

[0158] As can be seen from the accompanying drawings, the channel differential amplification fusion feature extraction module includes a cascaded feature fusion submodule and a differential amplification submodule, and a channel weighting module is cascaded after the differential amplification submodule. Figure 5 (b) is used to complete the following operations:

[0159] (twenty one)

[0160] in, represents the intermediate features of the fifth-level visible light image after channel weighting, Represents the intermediate features of the fifth-level infrared image after channel weighting; represents the Sigmoid activation function, where it compresses each element to (0, 1), H and W represent the height and width of the feature respectively, i represents the row index in the feature, and j represents the column index in the feature;

[0161] Furthermore, based on the Figure 1 It can be seen that the five-level visible light image features can finally be expressed as:

[0162] (twenty two)

[0163] The five-level infrared image features can finally be expressed as:

[0164] (twenty three)

[0165] Furthermore, the position differential amplification fusion feature extraction module includes a cascaded feature fusion submodule and a differential amplification submodule, and a position weighting module is cascaded after the differential amplification submodule. Figure 5 (c) is used to complete the following operations:

[0166] (twenty four)

[0167] in, represents the intermediate features of the sixth-level visible light image after position weighting, represents the intermediate features of the sixth-level infrared image after position weighting;

[0168] Furthermore, based on the Figure 1 It can be seen that the six-level visible light image features can finally be expressed as:

[0169] (25)

[0170] The six-level infrared image features can finally be expressed as:

[0171] (26)

[0172] Furthermore, the corner differential amplification fusion feature extraction module includes not only a cascaded feature fusion submodule and a differential amplification submodule, but also a cascaded corner weighting module after the differential amplification submodule, and the corner weighting module is used to complete the following operations:

[0173] (27)

[0174] in, represents the intermediate features of the seventh-level visible light image after corner weighting, represents the intermediate features of the seventh-level infrared image after corner weighting, b and c are the indexes of samples and channels respectively, i represents the row index in the feature, j represents the column index in the feature, rows and cols are the row and column positions of the corners respectively;

[0175] Furthermore, the seven-level visible light image features can be finally expressed as:

[0176] (28)

[0177] Furthermore, based on the Figure 1 It can be seen that the seven-level infrared image features can finally be expressed as:

[0178] (29)

[0179] In addition, it should be noted that the loss function of the present invention is defined as:

[0180] (30)

[0181] in, Indicates the strength loss, represents the gradient loss, and Represent the weights of the two loss functions respectively.

[0182] The representation is as follows:

[0183] (31)

[0184] The definition is as follows:

[0185] (32)

[0186] In a specific embodiment of the present invention, a comprehensive dataset containing 1083 MSRS images, 300 M3FD images, and 300 LLVIP images is used to train the above framework. These images are cut into sub-images of size 64×64 pixels as training samples, and sliding window sampling is performed with a step size of 64 pixels to enrich the training diversity. During the training process, 30 epochs are set, of which the first 8 epochs use a fixed initial learning rate, and the learning rate is halved for each subsequent epoch. This strategy is used to gradually fine-tune the model parameters. This setting is based on preliminary parameter tuning experiments. In order to fully verify the accuracy and effectiveness of the proposed framework, model inference and performance evaluation are implemented on the LLVIP dataset. The network is implemented on an NVIDIA™ (NVIDIA®) GPU (GTX 4090) using PyTorch as the programming environment.

[0187] Table 1 systematically summarizes and shows the quantitative evaluation results of the ten compared algorithms and this method on six general evaluation indicators. The higher the value of all indicators, the better the performance. For each evaluation indicator, the average score is calculated separately using 50 test instances in the LLVIP dataset. The best result is highlighted in bold, the second best result is underlined, and the third best performance is italicized to enhance the visual comparison effect. The analysis results show that this method has reached the highest level in 7 evaluation indicators.

[0188] Table 1. Average index values ​​obtained by existing fusion methods and the present invention on the LLVIP dataset

[0189] EN MI SF AG SD VIF Qabf MS_SSIM PPTFusion 6.269±0.620 2.311±0.432 8.298±4.836 2.381±1.589 30.236±11.764 0.734±0.082 0.41±0.104 0.966±0.067 RFN-Nest 6.637±0.503 2.497±0.399 7.079±3.385 2.284±1.328 38.905±10.188 0.718±0.124 0.402±0.069 0.987±0.059 SwinFuse 3.815±1.538 1.438±0.572 11.875±8.454 2.605±2.658 34.139±21.557 0.390±0.136 0.298±0.160 0.518±0.184 YDTR 6.527±0.481 3.091±0.550 8.260±5.814 2.099±1.711 29.940±9.919 0.642±0.052 0.306±0.083 0.929±0.070 DATFuse 6.987±0.267 <![CDATA[ 3.935±0.386 ]]> 10.152±5.673 2.705±1.853 36.971±7.256 0.777±0.130 0.472±0.095 0.979±0.069 IRFS <![CDATA[ 7.026±0.371 ]]> 2.619±0.431 <![CDATA[ 12.893±7.277 ]]> <![CDATA[ 3.417±2.288 ]]> <![CDATA[ 40.801±11.524 ]]> <![CDATA[ 0.828±0.141 ]]> <![CDATA[ 0.574±0.072 ]]> 1.073±0.057 LRRNet 6.348±0.521 2.356±0.374 10.565±5.719 2.865±1.940 28.931±10.737 0.605±0.100 0.477±0.111 0.800±0.116 SemLA 6.849±0.401 2.405±0.516 9.631±5.129 2.730±1.759 36.571±10.218 0.648±0.132 0.448±0.123 0.890±0.163 Ours 7.264±0.298 4.008±0.745 14.415±8.421 4.158±2.819 46.234±8.841 1.003±0.103 0.713±0.064 <![CDATA[ 1.032±0.056 ]]>

[0190] For the method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0191] The embodiment of the present application provides a method and device for fusion of infrared and visible light images using Mamba. The method constructs a hybrid feature encoder, which combines the local feature extraction advantages of convolutional neural network (CNN) and the global attention mechanism of Transformer to achieve effective capture of shallow features. On this basis, a set of deep feature encoders is designed, relying on the parallel visual Mamba architecture to further enhance the ability of feature representation. In addition, a differential amplification fusion feature extraction module is proposed. The uniqueness of this module is that it can systematically enhance and refine the complementary information between the two image modalities from three dimensions-channel, spatial position and image edge, ensuring the comprehensiveness and accuracy of the fusion features. This design not only deepens the understanding of subtle differences in complex scenes, but also significantly improves the model's recognition and distinction capabilities when facing diverse targets. Comprehensive experimental results of a variety of public data sets show that the present invention is significantly superior to a variety of existing advanced methods in both qualitative and quantitative evaluation. In addition, the present invention exhibits a strong generalization ability and can adapt to infrared and visible light image fusion tasks of various resolutions without additional fine-tuning.

[0192] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, any one of the claimed embodiments may be used in any combination.

[0193] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for fusion of infrared and visible light images using Mamba, characterized in that: The method comprises: Get the visible light image I to be fused vi and infrared image I ir ; The visible light image I vi and infrared image I ir Input shallow feature extractor to extract shallow features and obtain shallow visible light image features and shallow infrared image features The shallow visible light image features are respectively And the shallow infrared image features Input the corresponding deep feature extractor to extract deep features and obtain deep visible light image features and deep infrared image features The deep feature extractor includes multiple levels of parallel visual Mamba modules; The deep visible light image feature And the deep infrared image features Perform a splicing operation, and input the obtained splicing image features into the feature decoder for feature reconstruction to obtain the fused image I F ; Wherein, the visible light image I vi and infrared image I ir Input shallow feature extractor to extract shallow features and obtain shallow visible light image features and shallow infrared image features include: The visible light image I vi and the infrared image I ir Input the convolution module in each feature extraction path to obtain the first-level visible light image features and first-level infrared image features in, Represents a convolution module, whose input channel is 1 and output channel is 8; The first-level visible light image features are respectively And the first-level infrared image features Input the first Transformer module in each feature extraction path, wherein a differential amplification fusion feature extraction module is provided between the two first Transformer modules to fuse the infrared image features into the visible light image features to obtain the secondary visible light image features Infrared image features are integrated with visible light image features to obtain secondary infrared image features. Among them, Tran 8 (·) represents the first Transformer module, which has 8 channels. Represents nonlinear activation function GeLU, DAF 2 E represents the differential amplification fusion feature extraction module; The secondary visible light image features are respectively And the secondary infrared image features Input the reflection convolution module in each feature extraction path to obtain the three-level visible light image features and three-level infrared image features in, Represents a reflection-convolution module with 8 input channels and 16 output channels; The three-level visible light image features are respectively And the three-level infrared image features Input the second Transformer module in each feature extraction path, wherein a differential amplification fusion feature extraction module is provided between the two second Transformer modules to fuse the infrared image features into the visible light image features to obtain the four-level visible light image features Infrared image features are integrated with visible light image features to obtain four-level infrared image features. The four-level visible light image features are used as shallow visible light image features The four-level infrared image features are used as shallow infrared image features Among them, Tran 16 (·) indicates the second Transformer module, which has 16 channels; The shallow visible light image features are respectively And the shallow infrared image features Input the corresponding deep feature extractor to extract deep features and obtain deep visible light image features and deep infrared image features include: The shallow visible light image features are respectively And the shallow infrared image features Input the first parallel vision Mamba modules in the respective feature extraction paths, wherein a channel differential amplification fusion feature extraction module is provided between the two first parallel vision Mamba modules to fuse the channel features of the infrared image into the visible light image features to obtain the five-level visible light image features The channel features of the visible light image are integrated into the infrared image features to obtain the five-level infrared image features. Among them, DAF 2 ECh represents the channel differential amplification fusion feature extraction module. represents the first parallel vision Mamba module, which has 16 input channels and 32 output channels; The five-level visible light image features are respectively And the five-level infrared image features The second parallel vision Mamba modules in the respective feature extraction paths are input, wherein a position differential amplification fusion feature extraction module is provided between the two second parallel vision Mamba modules to fuse the position features of the infrared image into the visible light image features to obtain the six-level visible light image features. The position features of the visible light image are integrated into the infrared image features to obtain the six-level infrared image features. Among them, DAF 2 EPo represents the position difference amplification fusion feature extraction module. represents the second parallel vision Mamba module, which has 32 input channels and 64 output channels; The six-level visible light image features are respectively And the six-level infrared image features Input the third parallel vision Mamba modules in the respective feature extraction paths, wherein a corner differential amplification fusion feature extraction module is provided between the two third parallel vision Mamba modules to fuse the corner features of the infrared image into the visible light image features to obtain the seven-level visible light image features The corner features of the visible light image are integrated into the infrared image features to obtain seven levels of infrared image features. Among them, DAF 2 ECo represents the corner differential amplification fusion feature extraction module. represents the third parallel vision Mamba module, which has 64 input channels and 128 output channels; Among them, the first parallel vision Mamba module, the second parallel vision Mamba module and the third parallel vision Mamba module are parallel vision Mamba modules with the same structure but different numbers of channels.

2. The method according to claim 1, characterized in that The first Transformer module and the second Transformer module are Transformer modules with the same structure but different numbers of channels, and the Transformer module is composed of a cascade of one or more multiple deconvolution head transposed attention mechanism modules and a gated deconvolution feedforward network module, wherein: The operation of the multi-deconvolution head transposed attention mechanism module is expressed as: Among them, Fx is the input of the multi-deconvolution head transposed attention mechanism module, is the output of the multi-deconvolution head transposed attention mechanism module, W1(·) represents 1×1 point-wise convolution, is the query vector, is the key vector, is a value vector, α represents a learnable scaling parameter; The operation of the gated deconvolution feedforward network module is expressed as: Among them, Fy represents the input of the gated deconvolution feedforward network module, represents the output of the gated deconvolution feedforward network module, represents 1×1 point-wise convolution, represents a 3×3 depthwise convolution, l = 0, 1, 2, LN(·) represents a normalization operation, and ⊙ represents element-wise multiplication.

3. The method according to claim 1, characterized in that The parallel vision Mamba module is used to complete the following operations: Where Sp[·] represents the Split operation, n represents the level of PViM, n∈[1,3], represents the input features, represents the divided 1 / N feature, Mamba(·) represents the Mamba operation, θ represents the adjustment factor, Concat(·) represents the concatenation operation, and Pro(·) represents the projection operation.

4. The method according to claim 1, characterized in that: The differential amplification fusion feature extraction module includes a cascaded feature fusion submodule and a differential amplification submodule, wherein: The feature fusion submodule is used to complete the following operations: in, represents the element addition operation, represents element multiplication operation, Concat(·) represents concatenation operation; represents the intermediate features output by the parallel vision Mamba module during the k-th level visible light image feature extraction process, represents the intermediate features output by the parallel vision Mamba module during the k-th level infrared image feature extraction process; or, represents the intermediate features output by the Transformer module during the feature extraction of the k-th level visible light image, The intermediate features output by the Transformer module during the feature extraction process of the k-th level infrared image; represents the intermediate features of the k-th level visible light image after feature fusion, Represents the intermediate features of the k-th level infrared image after feature fusion; The differential amplifier module is used to complete the following operations: in, represents the intermediate features of the k-th level visible light image after differential amplification, Represents the intermediate features of the k-th level infrared image after differential amplification.

5. The method according to claim 4, characterized in that The channel differential amplification fusion feature extraction module includes a cascaded feature fusion submodule, a differential amplification submodule and a channel weighting module, and the channel weighting module is used to complete the following operations: in, represents the intermediate features of the fifth-level visible light image after channel weighting, represents the intermediate features of the fifth-level infrared image after channel weighting, represents the Sigmoid activation function, where it compresses each element to (0,1), H and W represent the height and width of the feature respectively, i represents the row index in the feature, and j represents the column index in the feature; The five-level visible light image features are expressed as: The five-level infrared image features are expressed as:

6. The method according to claim 4, characterized in that The position differential amplification fusion feature extraction module includes a cascaded feature fusion submodule, a differential amplification submodule and a position weighting module, and the position weighting module is used to complete the following operations: in, represents the intermediate features of the sixth-level visible light image after position weighting, represents the intermediate features of the sixth-level infrared image after position weighting, represents the Sigmoid activation function, where it compresses each element to (0,1), W represents the width of the feature, i represents the row index in the feature, and j represents the column index in the feature; The six-level visible light image feature is expressed as: The six-level infrared image feature is expressed as:

7. The method according to claim 4, characterized in that The corner differential amplification fusion feature extraction module includes a cascaded feature fusion submodule, a differential amplification submodule and a corner weighting module, and the corner weighting module is used to complete the following operations: in, represents the intermediate features of the seventh-level visible light image after corner weighting, represents the intermediate features of the seventh-level infrared image after corner weighting, b and c are the indexes of the sample and channel respectively, i represents the row index in the feature, j represents the column index in the feature, rows and cols are the row and column positions of the corners respectively; The seven-level visible light image feature is expressed as: The seven-level infrared image feature is expressed as:

8. The method according to claim 4, characterized in that The secondary visible light image feature is expressed as: The secondary infrared image feature is expressed as: The four-level visible light image feature is expressed as: The four-level infrared image feature is expressed as:

Images