An infrared dual-band image fusion method based on self-attention and spatial-frequency domain cross fusion

Abstract

This invention discloses a dual-band infrared image fusion method based on self-attention and space-frequency domain cross-fusion, belonging to the field of infrared image fusion technology. This method is primarily based on the highly efficient selective state-space model Mamba, combined with the classic convolutional neural network and self-attention mechanism. Through the interaction of space-frequency domain features, the feature extraction, feature fusion, and image reconstruction components of the fusion model are constructed. This method effectively enhances the information of small and weak targets in the fused image while suppressing background clutter, achieving dual-band infrared image fusion. The specific implementation method of the invention is further described with reference to the accompanying drawings.

Description

Technical Field

[0001] The present invention belongs to the technical field of image fusion in digital image processing, and relates to an infrared dual-band image fusion method based on self-attention and space-frequency domain cross fusion. Background Art

[0002] The widespread adoption of various camouflage and stealth technologies has greatly enhanced the concealment capabilities of detected targets, posing unprecedented challenges to traditional target detection and recognition technologies, particularly in the areas of target detection and anti-interference recognition within infrared thermal imaging seekers. To address these challenges, dual-band infrared image fusion technology has emerged as an effective solution. By fusing infrared images from different bands, richer target feature information can be obtained, enabling more effective identification of camouflaged and stealthed targets and enhancing the adaptability of infrared guidance systems in complex environments.

[0003] In recent years, deep learning-based image fusion methods have achieved widespread application and development in the field of image fusion due to their powerful feature extraction capabilities. Compared to traditional methods, deep learning-based fusion methods can extract features from different modalities through different network branches, better capturing feature differences between modalities and effectively improving feature expression. However, current deep learning fusion methods still face challenges in the task of infrared dual-band image fusion. As an autoregressive task, the quality of the fused image depends not only on the local context of the image but also on the long-range dependencies between pixels. However, image fusion methods based on convolutional neural networks are limited by their local receptive fields and cannot capture the global context of the image. Image fusion methods based on visual Transformers can effectively model long-range dependencies in images through a self-attention mechanism, but they suffer from high computational complexity and slow inference speed. Recently, a new state-space model, the Linear-Time Sequence Modeling with Selective State Spaces (Mamba), has attracted widespread attention in the field of image processing due to its efficient performance in modeling long-range dependencies. Summary of the Invention

[0004] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide an infrared dual-band image fusion method based on self-attention and spatial-frequency domain cross-fusion to solve the problems that the existing fusion methods are difficult to capture the global context information of the image, or have high complexity and slow inference speed.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] An infrared dual-band image fusion method based on self-attention and spatial-frequency domain cross fusion includes the following steps:

[0007] S1, acquire medium-wave and long-wave images;

[0008] S2, the long-wave image and the medium-wave image are respectively processed by image block embedding and several levels of CSA-Mamba to obtain long-wave image features and medium-wave image features respectively; the CSA-Mamba includes a Conv-Mamba module and a CSA module connected in series, the input of the Conv-Mamba module is the input of the CSA-Mamba, the input and output of the Conv-Mamba are added together to serve as the input of the CSA module, and the output of the CSA module is added together with the input of the CSA module to serve as the output of the CSA-Mamba;

[0009] S3, the long-wave image features and the medium-wave image features are inputted into a feature fusion module, which is composed of several levels of space-frequency domain cross fusion modules. After the long-wave image features and the medium-wave image features are fused several times, the fused features are outputted.

[0010] In S4, the fused features are processed by several levels of CSA-Mamba modules and then embedded through inverse blocks to obtain a reconstructed image, which is the fused image.

[0011] A further improvement of the present invention is that

[0012] Preferably, the processing process of the Conv-Mamba module in the CSA-Mamba in S2 and S5 is as follows: the image is split into two parts by channel, and input into the convolution branch and the Mamba branch respectively; the output features of the two branches are channel-joined and channel-shuffled, and the extracted features are output;

[0013] Preferably, the convolution branch consists of N linear deformable convolution modules, and the deformable convolution module includes deformable convolution, batch normalization and ReLU activation function.

[0014] Preferably, the Mamba branch includes two information streams, one of which passes through layer normalization, linear projection, depth convolution, SiLU activation function and SS2D module in sequence; the other information stream linearly projects the output of layer normalization, and the result of linear projection is processed by SiLu activation function. After the activation function processing result and the output of the SS2D module are element-by-element multiplied, the linear projection obtains the output of the Mamba branch.

[0015] Preferably, in S2, the input features enter the CSA module, and for each attention head, a query vector, a key vector and a value vector are generated by linear projection; each attention head generates a corresponding attention matrix through the query vector, the key vector and the value vector, and the attention matrix of each head is spliced ​​along the channel dimension. The splicing result is restored to the dimension by linear projection, and the restored dimension feature is multiplied element-by-element with the input feature after passing the Sigmoid activation function to obtain the output feature of the CSA module.

[0016] Preferably, the input features enter the CSA module, and before linear projection, they are first subjected to an average pooling operation to reduce the feature dimension, and layer normalization is performed, and the features after layer normalization are linearly projected.

[0017] Preferably, in S3, in the space-frequency domain cross-fusion module, the medium wave image features and the long wave image features are respectively processed by the Mamba block to obtain the medium wave image process feature one and the long wave image process feature two, respectively; the medium wave image features and the long wave image features are processed by the frequency domain fusion module to obtain the frequency domain fusion features, the frequency domain fusion features are element-by-element subtracted from the medium wave image features and the long wave image features to obtain the medium wave difference features and the long wave difference features, the medium wave difference features and the long wave difference features are respectively processed by the Mamba block to obtain the medium wave image process feature two and the medium wave image process feature three, the medium wave image process feature two, the long wave image process feature one and the medium wave image features are added to obtain the medium wave mixed modal features; the long wave image process feature two, the medium wave image process feature one and the long wave image features are added to obtain the long wave mixed modal features, and the medium wave mixed modal features and the long wave mixed modal features are spliced ​​to obtain the fusion features.

[0018] Preferably, the medium-wave difference feature and the long-wave difference feature are respectively processed by the Mamba block, and learnable parameters are introduced as weight coefficients of the difference features.

[0019] Preferably, in the frequency domain fusion module of the medium-wave image features and the long-wave image features, the amplitude component and the phase component are first obtained by two-dimensional fast Fourier transform, and the amplitude component of the medium-wave image features and the amplitude component of the long-wave image features are spliced ​​along the channel dimension to obtain the amplitude splicing feature; the phase component of the long-wave image features and the phase component of the long-wave image features are spliced ​​to obtain the phase splicing feature; the amplitude splicing feature and the phase splicing feature are respectively integrated through convolution operation and ReLU activation function, and then processed by 2D-IFFT. After channel shuffling, the frequency domain fusion feature is obtained.

[0020] Preferably, in S4, the medium-wave mixed modal features and the long-wave mixed modal features are concatenated and then linearly projected to output fusion features.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] The present invention discloses an infrared dual-band image fusion method based on self-attention and space-frequency domain cross fusion. Based on the existing image fusion methods and the designed infrared dual-band image fusion network, this method effectively enhances the information of weak targets in the fused image, while suppressing background clutter interference, and ultimately achieves the fusion purpose. This method suppresses background clutter interference while enhancing weak targets, improves the dual-band image fusion effect, and promotes the performance of the fused image in downstream tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is the CSA-Mamba module structure diagram;

[0024] Figure 2 The overall architecture of the fusion network is based on self-attention and spatial-frequency domain cross fusion;

[0025] Figure 3 This is the structure diagram of the frequency domain fusion module;

[0026] Figure 4 This is the structure diagram of the space-frequency domain feature cross fusion module;

[0027] Figure 5 This is the infrared dual-band image fusion result based on self-attention and spatial-frequency domain cross fusion. DETAILED DESCRIPTION

[0028] Hereinafter, the terms "first," "second," "third," and "fourth" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the technical features indicated. Thus, a feature identified as "first," "second," "third," or "fourth" may explicitly or implicitly include one or more of such features.

[0029] The co-shooting method provided in the embodiments of the present application can be applied to terminal devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). The embodiments of the present application do not impose any restrictions on the specific types of terminal devices.

[0030] It should be noted that the terms "first," "second," and the like in the description and drawings of the present invention are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatus.

[0031] Current deep learning techniques still face significant challenges in the specific task of infrared dual-band image fusion. As a complex autoregressive task, image fusion relies not only on local image details but also on long-range dependencies between pixels. Fusion methods based on convolutional neural networks are limited by their local perception range and struggle to capture the global context of the image. While fusion methods based on visual Transformers can model long-range dependencies through self-attention mechanisms, they suffer from high computational complexity and low inference efficiency.

[0032] The present invention discloses an infrared dual-band image fusion method based on self-attention and space-frequency domain cross fusion. The method is mainly based on the efficient selective state space model Mamba, combined with the classic convolutional neural network and self-attention mechanism, and through the space-frequency domain feature interaction, the feature extraction part, feature fusion part, and image reconstruction part of the fusion model are constructed respectively. The present invention effectively enhances the weak target information in the fused image, while suppressing background clutter interference, and realizes infrared dual-band image fusion. The specific implementation method of the present invention is further described in conjunction with the accompanying drawings, see Figure 1 , the method specifically comprises the following steps:

[0033] S1, obtain infrared dual-band image data, the infrared dual-band image data includes a medium wave image and long-wave images .

[0034] In S2, both the long-wave and medium-wave images are input into the feature extraction module. Each long-wave and medium-wave image undergoes a patch embedding operation in turn, and then passes through N cascaded CSA-Mamba modules to construct the feature extraction part of the fusion model. The number of CSA-Mamba modules is determined by the final fusion index.

[0035] Among them, image block embedding is a technology that divides an image into small blocks and embeds them into a certain vector space, which is used for image processing and feature extraction in deep learning.

[0036] CSA-Mamba is a feature extraction module consisting of a Conv-Mamba module and a CSA (Channel Self-Attention) module. The input of Conv-Mamba is the input of the entire CSA-Mamba. The output of the Conv-Mamba module is connected to the input features via a residual connection to preserve the original features. The output and input of the Conv-Mamba module are then summed and used as the input of the CSA module. The output and input of the CSA module are also connected to the input features via a residual connection to preserve the original features. The output and input of the CSA module are then summed and used as the final output of the CSA-Mamba module. The formula is as follows:

[0037] (1)

[0038] (1) The Conv-Mamba module, such as Figure 2 As shown on the left side of , it includes the following: Let the input of the module be , the input features are first divided into two parts by channel splitting operation, and input into the convolution branch and Mamba branch respectively. The process is shown in the following formula:

[0039] (2)

[0040] The convolution branch consists of N linear deformable convolution modules, following the structure of classic CNNs. Each module includes linear deformable convolution (LDConv), batch normalization (BN), and ReLU activation function. The features extracted by the linear deformable convolution module are restored through 1×1 convolution. The process is shown in the following formula:

[0041] (3)

[0042] in, Indicates cascading modules.

[0043] The Mamba branch follows the standard Mamba module architecture. One information flow goes through layer normalization, linear projection, depthwise convolution, SiLU activation function, and SS2D module. The other information flow linearly projects the output of the layer normalization result, activates it through SiLU activation function, and directly multiplies it element-wise with the output of the SS2D module. The output of the Mamba branch is obtained after linear projection. The process is shown in the following formula:

[0044] (4)

[0045] in, Representation layer normalization operation, Represents a linear projection.

[0046] SiLU, or Sigmoid Linear Unit, is an activation function that is used to introduce nonlinear factors in neural networks to help them learn complex patterns.

[0047] The SS2D module is a 2D selective scanning module (2D-Selective-Scan). This module is used to process visual signals (such as images) and extract effective features. It achieves efficient processing and feature extraction of visual signals through a four-directional scanning strategy and a selective scanning spatial state sequence model.

[0048] Finally, the output features of the two branches are concatenated (Concat) and shuffled (Shuffle) to obtain the output of the Conv-Mamba module, which is the process of feature extraction. The process is shown in the following formula:

[0049] (5)

[0050] in, Represents a channel splicing operation.

[0051] The output features of Conv-Mamba are added to the input features and used as the input of the subsequent CSA module.

[0052] (2) The channel self-attention (CSA) module, such as Figure 2 As shown on the right side of , it includes the following: the input features are first average pooled to reduce the dimension of the features to reduce the computational cost, and then layer normalization is performed:

[0053] (6)

[0054] in, Represents an average pooling operation.

[0055] After normalization, the query vector (Query), key vector (Key), and value vector (Value) are generated for multiple attention heads through linear projection. Each attention head calculates the attention matrix based on its own Q, K, and V. The process is shown in the following formula:

[0056] (7)

[0057] (8)

[0058] in, is the number of attention heads of MHSA, is the scaling factor, , that is, calculating self-attention along the channel dimension.

[0059] The channel dimensions of the attention matrix of each head are then concatenated and linearly projected to restore the dimension. The process is shown in the following formula:

[0060] (9)

[0061] Finally, the feature is activated by the Sigmoid function and then multiplied element-by-element with the input feature to obtain the output feature of the attention module, which is the extracted feature 2, as shown in the following formula:

[0062] (10)

[0063] In the above process, both the medium-wave image and the long-wave image are sequentially subjected to the image block embedding operation and N cascaded CSA-Mamba modules to extract their respective features and obtain the long-wave image features and medium-wave image features.

[0064] S3, see Figure 1 The long-wavelength and medium-wavelength image features are fed into a fusion module for feature fusion. This fusion module consists of M cascaded SFCIM modules to construct the feature fusion portion of the fusion model. After M iterations, the modal mixture features are channel-concatenated and linearly projected to obtain the final output of the feature fusion portion.

[0065] (11)

[0066] See also Figure 3Each SFCIM is a spatial-frequency cross-fusion module (SFCIM). The input medium-wave image features and long-wave image features need to be processed separately by the Mamba block and simultaneously processed by a frequency domain fusion module (FIM). The medium-wave image features are differentiated from the frequency domain fusion features output by the frequency domain fusion module (FIM), and the long-wave image features are differentiated from the frequency domain fusion features. Specifically, it includes the following three parts:

[0067] (1) The medium-wave image features and long-wave image features are respectively passed through the Mamba block to capture the long-distance dependencies between features:

[0068] (12)

[0069] (13)

[0070] The specific process of the Mamba block has been introduced in step 2 and will not be repeated here.

[0071] (2) Frequency Integration Module (FIM), such as Figure 4 As shown, it includes the following: the output of the feature extraction module, that is, the extracted medium wave image features and long-wave image features , the amplitude component (Amplitude) and phase component (Phase) are obtained through two-dimensional fast Fourier transform (2D-Fast Fourier Transform, 2D-FFT):

[0072] (14)

[0073] (15)

[0074] in, represents the Fourier transform operation, 、 represent the amplitude component and phase component respectively.

[0075] For an image , the mathematical representation of its Fourier transform is as follows:

[0076] (16)

[0077] Where, 、 are coordinates in Fourier space.

[0078] Its amplitude component and phase components The mathematical representation of is as follows:

[0079] (17)

[0080] (18)

[0081] in, 、 are the real and imaginary parts of the frequency components of the image, respectively.

[0082] Then, the amplitude components of the medium-wave and long-wave image features are spliced ​​along the channel dimension, and the phase components of the medium-wave and long-wave image features are spliced. Then, a 1×1 convolution operation and a ReLU activation function are performed to integrate the corresponding amplitude and phase components:

[0083] (19)

[0084] (20)

[0085] Finally, the 2D-IFFT operation is used to transform the fused amplitude and frequency components back to the spatial domain, and the channel shuffling operation is performed to fully mix the frequency domain features of different bands to obtain the frequency domain fusion feature, which is the output of FIM:

[0086] (twenty one)

[0087] (3) The frequency domain fusion features output by the frequency domain fusion module FIM are subtracted element by element from the medium wave image features and the long wave image features to obtain the difference features:

[0088] (twenty two)

[0089] (twenty three)

[0090] The difference features are also passed through the Mamba block to capture the long-range dependencies between features:

[0091] (twenty four)

[0092] (25)

[0093] Finally, introduce learnable parameters and , as the weight coefficient of the difference feature, the weighted difference feature is added to the original input feature to obtain the enhanced feature. In order to further utilize the complementary characteristics of the medium-wave image and long-wave image features, the enhanced feature is merged with the feature processed by the Mamba block in the other modality to obtain the mixed modal feature:

[0094] (26)

[0095] (27)

[0096] Get the fusion features output by the feature fusion module.

[0097] In step S4, the fused features are input into the image reconstruction module. This module processes the features using N CSA-Mamba modules and an inverse patch embedding module to obtain and output the reconstructed image. In this process, the fused features are first processed by a cascade of N CSA-Mamba modules and then undergo an inverse patch embedding operation to construct the image reconstruction portion of the fusion model.

[0098] The above-mentioned model design method for the entire model establishment includes the following steps, which mainly include four parts: the first part is to produce a fusion image data set; the second part is to construct the feature extraction part; the third part is to construct the feature fusion part; and the fourth part is to construct the overall fusion model and test the fusion model.

[0099] During the test, the above step S1 is to obtain infrared dual-band image data and construct a fusion image dataset ,in represents a medium wave image, Represents a long-wavelength image. After construction, the trained model is obtained by training and testing the training and test sets.

[0100] Implementation Effect

[0101] The experimental results are as follows Figure 5 As shown in the figure, there are three groups of infrared dual-band images under complex backgrounds. The first group is a bright cloud background, the second group is a sea and sky background, and the third group is a sea, mountain and sky background. The red boxed area in the figure is the location of the weak target, and the fusion result represents the image fusion result based on the above image fusion mode.

[0102] This paper proposes an innovative infrared dual-band image fusion technique based on the highly efficient selective state-space model Mamba, integrating convolutional neural networks with a self-attention mechanism. Through a carefully designed spatial-frequency domain feature interaction strategy, fusion models for three key components, namely feature extraction, feature fusion, and image reconstruction, are constructed. This method not only significantly improves the clarity and recognition of small and dim targets in the fused image, but also effectively reduces background clutter interference, successfully achieving high-quality fusion of infrared dual-band images.

[0103] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An infrared dual-band image fusion method based on self-attention and spatial-frequency domain cross fusion, characterized in that: The following steps are involved: S1, acquire medium-wave and long-wave images; S2, the long-wave image and the medium-wave image are respectively processed by image block embedding and several levels of CSA-Mamba to obtain long-wave image features and medium-wave image features respectively; the CSA-Mamba includes a Conv-Mamba module and a CSA module connected in series, the input of the Conv-Mamba module is the input of the CSA-Mamba, the input and output of the Conv-Mamba are added together to serve as the input of the CSA module, and the output of the CSA module is added together with the input of the CSA module to serve as the output of the CSA-Mamba; S3, the long-wave image features and the medium-wave image features are inputted into a feature fusion module, which is composed of several levels of space-frequency domain cross fusion modules. After the long-wave image features and the medium-wave image features are fused several times, the fused features are outputted. In S3, in the space-frequency domain cross-fusion module, the medium wave image feature and the long wave image feature are respectively processed by the Mamba block to obtain the medium wave image process feature one and the long wave image process feature two, respectively; the medium wave image feature and the long wave image feature are processed by the frequency domain fusion module to obtain the frequency domain fusion feature, the frequency domain fusion feature is subtracted element-by-element from the medium wave image feature and the long wave image feature to obtain the medium wave difference feature and the long wave difference feature, the medium wave difference feature and the long wave difference feature are respectively processed by the Mamba block to obtain the medium wave image process feature two and the medium wave image process feature three, the medium wave image process feature two, the long wave image process feature one and the medium wave image feature are added to obtain the medium wave mixed modal feature; the long wave image process feature two, the medium wave image process feature one and the long wave image feature are added to obtain the long wave mixed modal feature, and the medium wave mixed modal feature and the long wave mixed modal feature are spliced ​​to obtain the fusion feature; In the frequency domain fusion module, the medium-wave image features and long-wave image features are firstly obtained by two-dimensional fast Fourier transform to obtain the amplitude component and phase component. The amplitude component of the medium-wave image features and the amplitude component of the long-wave image features are spliced ​​along the channel dimension to obtain the amplitude splicing feature; the phase component of the long-wave image features and the phase component of the long-wave image features are spliced ​​to obtain the phase splicing feature; the amplitude splicing feature and the phase splicing feature are respectively integrated through convolution operation and ReLU activation function, and then processed by 2D-IFFT. After channel shuffling, the frequency domain fusion feature is obtained. In S4, the fused features are processed by several levels of CSA-Mamba modules and then embedded through inverse blocks to obtain a reconstructed image, which is the fused image.

2. The infrared dual-band image fusion method based on self-attention and spatial-frequency domain cross fusion according to claim 1 is characterized in that: The processing process of the Conv-Mamba module in CSA-Mamba in S2 and S5 is as follows: the image is split into two parts by channel, which are input into the convolution branch and the Mamba branch respectively. The output features of the two branches are channel-wise spliced ​​and shuffled, and the extracted features are output.

3. The infrared dual-band image fusion method based on self-attention and spatial-frequency domain cross fusion according to claim 2 is characterized in that: The convolution branch consists of N linear deformable convolution modules, and the deformable convolution module includes deformable convolution, batch normalization and ReLU activation function.

4. The infrared dual-band image fusion method based on self-attention and spatial-frequency domain cross fusion according to claim 2, characterized in that: The Mamba branch includes two information streams, one of which passes through layer normalization, linear projection, depth convolution, SiLU activation function and SS2D module in sequence; the other information stream linearly projects the output of layer normalization, and the result of linear projection is processed by SiLu activation function. After the activation function processing result and the output of SS2D module are element-wise multiplied, the linear projection is used to obtain the output of the Mamba branch.

5. The infrared dual-band image fusion method based on self-attention and spatial-frequency domain cross fusion according to claim 1, characterized in that: In S2, the input features enter the CSA module. For each attention head, a query vector, a key vector, and a value vector are generated through linear projection. Each attention head generates a corresponding attention matrix through the query vector, key vector, and value vector. The attention matrices of each head are spliced ​​along the channel dimension, and the spliced ​​results are restored to dimension through linear projection. The features of the restored dimension are passed through the Sigmoid activation function and then multiplied element-by-element with the input features to obtain the output features of the CSA module.

6. The infrared dual-band image fusion method based on self-attention and spatial-frequency domain cross fusion according to claim 1, characterized in that: The input features enter the CSA module. Before linear projection, they are first subjected to average pooling operation to reduce the feature dimension and layer normalization. The features after layer normalization are then linearly projected.

7. The infrared dual-band image fusion method based on self-attention and spatial-frequency domain cross fusion according to claim 1, characterized in that: The medium-wave difference features and long-wave difference features are processed by the Mamba block respectively, and learnable parameters are introduced as the weight coefficients of the difference features.

8. The infrared dual-band image fusion method based on self-attention and spatial-frequency domain cross fusion according to claim 1, characterized in that: In S4, the medium-wave mixed modal features and the long-wave mixed modal features are concatenated and linearly projected to output the fusion features.

Images