Infrared dual-band image fusion method based on self-attention and space-frequency domain cross fusion

Abstract

The invention discloses an infrared dual-band image fusion method based on self-attention and space-frequency domain cross fusion, and belongs to the technical field of infrared image fusion. The method is mainly based on an efficient selective state space model Mamba, combines a classical convolutional neural network and a self-attention mechanism, and constructs a feature extraction part, a feature fusion part and an image reconstruction part of a fusion model through space-frequency domain feature interaction. According to the invention, weak and small target information in the fused image is effectively enhanced, background clutter interference is suppressed, and infrared dual-band image fusion is realized. The specific implementation method of the invention is further described in combination with an attached drawing.

Description

Technical Field

[0001] The invention belongs to the technical field of image fusion in digital image processing, and relates to an infrared dual-band image fusion method based on self-attention and space-frequency domain cross fusion. Background Art

[0002] With the widespread application of various camouflage and stealth technologies, the concealment ability of the detected objects has been greatly enhanced, making the traditional target detection and recognition technology face unprecedented challenges, especially in the target detection and anti-interference recognition of infrared thermal imaging seekers. In order to meet these severe challenges, the use of infrared dual-band image fusion technology has become an effective solution. By fusing infrared images of different bands, more abundant target feature information can be obtained, so as to more effectively identify camouflaged and stealth targets and enhance the adaptability of infrared guidance systems in complex environments.

[0003] In recent years, image fusion methods based on deep learning have been widely used and developed in the field of image fusion due to their powerful feature extraction capabilities. Compared with traditional methods, fusion methods based on deep learning can achieve feature extraction under different modalities through different network branches, better capture feature differences under different modalities, and effectively improve feature expression capabilities. However, current deep learning fusion methods still have problems that need to be improved in the infrared dual-band image fusion task: as an autoregressive task, the quality of the fused image depends not only on the local context information of the image, but also on the long-distance dependencies between pixels. However, image fusion methods based on convolutional neural networks are limited by the local receptive field and cannot capture the global context information of the image. Image fusion methods based on visual Transformers can effectively model long-distance dependencies in images through self-attention mechanisms, but have the disadvantages of high computational complexity and slow inference speed. Recently, a new state space model, the efficient selective state space model Mamba (Linear-Time Sequence Modeling with Selective State Spaces), has attracted widespread attention in the field of image processing due to its efficient performance in modeling long-distance dependencies. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and to provide an infrared dual-band image fusion method based on self-attention and space-frequency domain cross-fusion to solve the problem that the existing fusion methods are difficult to capture the global context information of the image, or have high complexity and slow inference speed.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions: An infrared dual-band image fusion method based on self-attention and space-frequency domain cross fusion includes the following steps: S1, acquiring medium-wave and long-wave images; S2, the long-wave image and the medium-wave image are respectively processed by image block embedding and several levels of CSA-Mamba in sequence to obtain long-wave image features and medium-wave image features respectively; the CSA-Mamba comprises a series-connected Conv-Mamba module and a CSA module, the input of the Conv-Mamba module is the input of the CSA-Mamba, the input and output of the Conv-Mamba are added together as the input of the CSA module, and the output of the CSA module is added together with the input of the CSA module as the output of the CSA-Mamba; S3, the long-wave image features and the medium-wave image features are inputted into a feature fusion module, the feature fusion module is composed of several levels of space-frequency domain cross fusion modules, and after the long-wave image features and the medium-wave image features are fused several times, the fusion features are outputted; S4, after the fused features are processed by several levels of CSA-Mamba modules, they are embedded through inverse blocks to obtain a reconstructed image, which is a fused image.

[0006] A further improvement of the present invention is that Preferably, the processing process of the Conv-Mamba module in CSA-Mamba in S2 and S5 is as follows: the image is split into two parts by channels, which are input into the convolution branch and the Mamba branch respectively, and the output features of the two branches are channel-joined and channel-shuffled, and the extracted features are output; Preferably, the convolution branch consists of N linear deformable convolution modules, and the deformable convolution module includes deformable convolution, batch normalization and ReLU activation function.

[0007] Preferably, the Mamba branch includes two information streams, one of which passes through layer normalization, linear projection, depth convolution, SiLU activation function and SS2D module in sequence; the other information stream linearly projects the output of layer normalization, and the result of linear projection is processed by SiLu activation function. After the activation function processing result and the output of SS2D module are element-by-element multiplied, the output of Mamba branch is obtained by linear projection.

[0008] Preferably, in S2, the input feature enters the CSA module, and for each attention head, a query vector, a key vector and a value vector are generated by linear projection; each attention head generates a corresponding attention matrix through the query vector, the key vector and the value vector, and the attention matrix of each head is spliced ​​along the channel dimension, and the splicing result is restored to a dimension by linear projection, and the restored dimension feature is multiplied element-by-element with the input feature after passing the Sigmoid activation function to obtain the output feature of the CSA module.

[0009] Preferably, the input features enter the CSA module, and before linear projection, they are first subjected to an average pooling operation to reduce the feature dimension, and layer normalization is performed, and the features after layer normalization are linearly projected.

[0010] Preferably, in S3, in the space-frequency domain cross-fusion module, the medium wave image feature and the long wave image feature are respectively processed by the Mamba block to obtain the medium wave image process feature one and the long wave image process feature two; the medium wave image feature and the long wave image feature are processed by the frequency domain fusion module to obtain the frequency domain fusion feature, the frequency domain fusion feature is subtracted element by element from the medium wave image feature and the long wave image feature to obtain the medium wave difference feature and the long wave difference feature, the medium wave difference feature and the long wave difference feature are respectively processed by the Mamba block to obtain the medium wave image process feature two and the medium wave image process feature three, the medium wave image process feature two, the long wave image process feature one and the medium wave image feature are added to obtain the medium wave mixed modal feature; the long wave image process feature two, the medium wave image process feature one and the long wave image feature are added to obtain the long wave mixed modal feature, and the medium wave mixed modal feature and the long wave mixed modal feature are concatenated to obtain the fusion feature.

[0011] Preferably, the medium-wave difference feature and the long-wave difference feature are respectively processed by the Mamba block, and learnable parameters are introduced as weight coefficients of the difference features.

[0012] Preferably, in the frequency domain fusion module of the medium wave image features and the long wave image features, the amplitude component and the phase component are first obtained by two-dimensional fast Fourier transform, and the amplitude component of the medium wave image features and the amplitude component of the long wave image features are spliced ​​along the channel dimension to obtain an amplitude splicing feature; the phase component of the long wave image features and the phase component of the long wave image features are spliced ​​to obtain a phase splicing feature; the amplitude splicing features and the phase splicing features are respectively integrated by convolution operation and ReLU activation function, and then processed by 2D-IFFT, and after channel shuffling, the frequency domain fusion features are obtained.

[0013] Preferably, in S4, the medium-wave mixed modal features and the long-wave mixed modal features are concatenated and then linearly projected to output fusion features.

[0014] Compared with the prior art, the present invention has the following beneficial effects: The present invention discloses an infrared dual-band image fusion method based on self-attention and space-frequency domain cross fusion. The method is based on the existing image fusion method and the designed infrared dual-band image fusion network, which effectively enhances the weak target information in the fused image and suppresses the background clutter interference, thereby finally achieving the fusion purpose. The method suppresses the background clutter interference while enhancing the weak target, improves the dual-band image fusion effect, and promotes the performance of the fused image in downstream tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is the CSA-Mamba module structure diagram; Figure 2 The overall architecture of the fusion network based on self-attention and spatial-frequency domain cross-fusion; Figure 3 This is the structure diagram of the frequency domain fusion module; Figure 4 It is the structure diagram of the space-frequency domain feature cross fusion module; Figure 5 This is the infrared dual-band image fusion result based on self-attention and space-frequency domain cross fusion. DETAILED DESCRIPTION

[0016] In the following, the terms "first", "second", "third", and "fourth" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, a feature defined as "first", "second", "third", and "fourth" may explicitly or implicitly include one or more of the features.

[0017] The co-shooting method provided in the embodiment of the present application can be applied to terminal devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), etc. The embodiment of the present application does not impose any restrictions on the specific type of the terminal device.

[0018] It should be noted that the terms "first", "second", etc. in the specification and drawings of the present invention are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0019] In the specific task of infrared dual-band image fusion, current deep learning technology still faces significant challenges: as a complex autoregressive task, the fusion effect of image fusion depends not only on the local detail information of the image, but also on the long-distance dependency between pixels. The fusion method based on convolutional neural network is limited by its local perception range and it is difficult to grasp the global context information of the image; while the fusion method based on visual Transformer can model long-distance dependency through the self-attention mechanism, it has the problems of high computational complexity and low reasoning efficiency.

[0020] The present invention discloses an infrared dual-band image fusion method based on self-attention and space-frequency domain cross-fusion. The method is mainly based on the efficient selective state space model Mamba, combined with the classic convolutional neural network and self-attention mechanism, and through the space-frequency domain feature interaction, the feature extraction part, feature fusion part, and image reconstruction part of the fusion model are constructed respectively. The present invention effectively enhances the weak target information in the fused image, while suppressing the background clutter interference, and realizes the infrared dual-band image fusion. The specific implementation method of the present invention is further described in conjunction with the accompanying drawings, see Figure 1 , the method specifically comprises the following steps: S1, obtaining infrared dual-band image data, the infrared dual-band image data includes a medium-wave image and long wave images .

[0021] S2, both the long-wave image and the medium-wave image are input into the feature extraction module. Each long-wave image and the medium-wave image are sequentially subjected to a patch embedding operation and then passed through N cascaded CSA-Mamba modules to construct the feature extraction part of the fusion model. The number of CSA-Mamba settings is determined according to the final fusion index.

[0022] Among them, image block embedding is a technology that divides an image into small blocks and embeds them into a certain vector space, which is used for image processing and feature extraction in deep learning.

[0023] CSA-Mamba is a feature extraction module, which consists of a Conv-Mamba module and a CSA (Channel Self-Attention) module. The input of Conv-Mamba is the input of the entire CSA-Mamba. The output and input features of the Conv-Mamba module are connected through residual connections to retain the original features. The output and input of Conv-Mamba are added together as the input of the CSA module. The output and input features of the CSA module are also connected through residual connections to retain the original features. The output and input of CSA are added together as the final output of CSA-Mamba; the formula is as follows: (1) (1) The Conv-Mamba module, such as Figure 2 As shown on the left side of , it includes the following contents: Assume that the input of the module is , the input features are first divided into two parts by channel splitting (Channel Split) operation, and are input into the convolution branch and Mamba branch respectively. The process is shown in the following formula: (2) The convolution branch consists of N linear deformable convolution modules, following the composition structure of classic CNNs. Each module includes linear deformable convolution (LDConv), batch normalization (BN), and ReLU activation function. The features extracted by the linear deformable convolution module are restored through 1×1 convolution. The process is shown in the following formula: (3) in, Indicates cascading modules.

[0024] The Mamba branch follows the standard Mamba module architecture. One information flow goes through layer normalization, linear projection, depth convolution, SiLU activation function and SS2D module. The other information flow linearly projects the output of the layer normalization result, and then uses SiLU activation function to directly multiply the output of the SS2D module element by element. After linear projection, the output of the Mamba branch is obtained. The process is shown in the following formula: (4) in, Representation layer normalization operation, Represents a linear projection.

[0025] SiLU, or Sigmoid Linear Unit, is an activation function that is used to introduce nonlinear factors in neural networks to help the network learn complex patterns.

[0026] The SS2D module is a 2D selective scanning module (2D-Selective-Scan). This module is used to process visual signals (such as images) and extract effective features. It realizes efficient processing and feature extraction of visual signals through a four-way scanning strategy and a selective scanning space state sequence model.

[0027] Finally, the output features of the two branches are concatenated (Channel Concatenation, Concat) and shuffled (Channel Shuffle, Shuffle) to obtain the output of the Conv-Mamba module, which is the process of feature extraction. The process is shown in the following formula: (5) in, Represents a channel concatenation operation.

[0028] The output features of Conv-Mamba are added to the input features and used as the input of the subsequent CSA module.

[0029] (2) The channel self-attention (CSA) module, such as Figure 2 As shown on the right side of , it includes the following: the input features are first subjected to average pooling operation to reduce the dimension of features to reduce the computational cost, and then layer normalization is performed: (6) in, Represents an average pooling operation.

[0030] After normalization, for multiple attention heads, query vectors (Query), key vectors (Key) and value vectors (Value) are generated through linear projection. Each attention head calculates the attention matrix based on its own Q, K, and V. The process is shown in the following formula: (7) (8) in, is the number of attention heads of MHSA, is the scaling factor, , that is, calculating self-attention along the channel dimension.

[0031] Then the channel dimensions of the attention matrix of each head are concatenated and linearly projected to restore the dimension, as shown in the following formula: (9) Finally, the feature is multiplied element-by-element with the input feature after the Sigmoid activation function to obtain the output feature of the attention module, which is the extracted feature 2, as shown in the following formula: (10) In the above process, both the medium-wave image and the long-wave image are successively subjected to the image block embedding operation and N cascaded CSA-Mamba modules to extract their respective features and obtain the long-wave image features and the medium-wave image features.

[0032] S3, see Figure 1 , the long-wave image features and medium-wave image features are input into the fusion module for feature fusion. The fusion module is composed of M SFCIM modules in cascade to construct the feature fusion part of the fusion model. After M iterations, the modal mixture features are channel spliced, and the final output of the feature fusion part is obtained by linear projection.

[0033] (11) See also Figure 3 Each SFCIM is a spatial-frequency cross-fusion module (Spatial-Frequency CrossIntegration Module, SFCIM). The input medium-wave image features and long-wave image features need to be processed by Mamba blocks respectively, and at the same time, they are processed together by a frequency domain fusion module FIM. The medium-wave image features and the frequency domain fusion features output by the frequency domain fusion module FIM are differentiated, and the long-wave image features and the frequency domain fusion features are differentiated. Specifically, it includes the following three parts: (1) The medium-wave image features and long-wave image features are respectively passed through the Mamba block to capture the long-distance dependencies between features: (12) (13) The specific process of the Mamba block has been introduced in step 2 and will not be repeated here.

[0034] (2) Frequency Integration Module (FIM), such as Figure 4 As shown, it includes the following contents: the output of the feature extraction module, that is, the extracted medium wave image features and long-wave image features , respectively, through the two-dimensional fast Fourier transform (2D-Fast Fourier Transform, 2D-FFT) to obtain the amplitude component (Amplitude) and the phase component (Phase): (14) (15) in, represents the Fourier transform operation, , represent the amplitude component and the phase component respectively.

[0035] For an image , the mathematical representation of its Fourier transform is as follows: (16) In the formula, , are coordinates in Fourier space.

[0036] Its amplitude component and phase component The mathematical representation of is as follows: (17) (18) in, , are the real and imaginary parts of the frequency components of the image, respectively.

[0037] Then, the amplitude components of the medium-wave and long-wave image features are concatenated along the channel dimension, and the phase components of the medium-wave and long-wave image features are concatenated, and then a 1×1 convolution operation and a ReLU activation function are performed to integrate the corresponding amplitude and phase components: (19) (20) Finally, the 2D-IFFT operation is used to transform the fused amplitude component and frequency component back to the spatial domain, and the channel shuffle operation is performed to fully mix the frequency domain features of different bands to obtain the frequency domain fusion feature, which is the output of FIM: (twenty one) (3) The frequency domain fusion features output by the frequency domain fusion module FIM are subtracted element by element from the medium wave image features and the long wave image features to obtain the difference features: (twenty two) (twenty three) The difference features are also passed through the Mamba block to capture the long-range dependencies between features: (twenty four) (25) Finally, we introduce learnable parameters and , as the weight coefficient of the difference feature, the weighted difference feature is added to the original input feature to obtain the enhanced feature. In order to further utilize the complementary characteristics of the medium-wave image and the long-wave image features, the enhanced feature is merged with the feature processed by the Mamba block in the other modality to obtain the mixed modality feature: (26) (27) Get the fusion features output by the feature fusion module.

[0038] S4, the fusion features are input into the image reconstruction module, which is processed by N CSA-Mamba modules and an inverse block embedding module to obtain the reconstructed image and output it. In this process, the fusion features are first processed by cascading N CSA-Mamba modules, and then by the inverse block embedding (Un Patch Embedding) operation to construct the image reconstruction part of the fusion model.

[0039] The above-mentioned model design method for the entire model building includes the following steps, which mainly include four parts: the first part is to make a fused image data set; the second part is to build a feature extraction part; the third part is to build a feature fusion part; and the fourth part is to build an overall fusion model and test the fusion model.

[0040] During the test, S1 in the above step is to obtain infrared dual-band image data and construct a fused image data set ,in represents a medium wave image, Represents a long-wave image. After construction, training and testing are performed through the training set and the test set to obtain the trained model.

[0041] Implementation Effect The experimental results are as follows Figure 5 As shown, there are three groups of infrared dual-band images under complex backgrounds. The first group is a bright cloud background, the second group is a sea and sky background, and the third group is a sea, mountain and sky background. The red boxed areas in the figure are the locations of weak targets, and the fusion results represent the image fusion results based on the above image fusion mode.

[0042] This paper proposes an innovative infrared dual-band image fusion technology, which is based on the efficient selective state space model Mamba and integrates convolutional neural network and self-attention mechanism. Through the carefully designed spatial-frequency domain feature interaction strategy, the fusion models of three key parts, feature extraction, feature fusion and image reconstruction, are constructed respectively. This method not only significantly improves the clarity and recognition of weak targets in the fused image, but also effectively reduces the interference of background clutter, and successfully realizes the high-quality fusion of infrared dual-band images.

[0043] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An infrared dual-band image fusion method based on self-attention and space-frequency domain cross fusion, characterized in that: The following steps are involved: S1, acquiring medium-wave and long-wave images; S2, the long-wave image and the medium-wave image are respectively processed by image block embedding and several levels of CSA-Mamba in sequence to obtain long-wave image features and medium-wave image features respectively; the CSA-Mamba comprises a series-connected Conv-Mamba module and a CSA module, the input of the Conv-Mamba module is the input of the CSA-Mamba, the input and output of the Conv-Mamba are added together as the input of the CSA module, and the output of the CSA module is added together with the input of the CSA module as the output of the CSA-Mamba; S3, the long-wave image features and the medium-wave image features are inputted into a feature fusion module, the feature fusion module is composed of several levels of space-frequency domain cross fusion modules, and after the long-wave image features and the medium-wave image features are fused several times, the fusion features are outputted; S4, after the fused features are processed by several levels of CSA-Mamba modules, they are embedded through inverse blocks to obtain a reconstructed image, which is a fused image.

2. The infrared dual-band image fusion method based on self-attention and space-frequency domain cross fusion according to claim 1 is characterized in that: The processing process of the Conv-Mamba module in CSA-Mamba in S2 and S5 is as follows: the image is split into two parts by channel, which are input into the convolution branch and the Mamba branch respectively. After the output features of the two branches are channel-joined and channel-shuffled, the extracted features are output.

3. The infrared dual-band image fusion method based on self-attention and space-frequency domain cross fusion according to claim 2 is characterized in that: The convolution branch consists of N linear deformable convolution modules, and the deformable convolution module includes deformable convolution, batch normalization and ReLU activation function.

4. The infrared dual-band image fusion method based on self-attention and space-frequency domain cross fusion according to claim 2 is characterized in that: The Mamba branch includes two information flows, one of which passes through layer normalization, linear projection, depth convolution, SiLU activation function and SS2D module in sequence; the other information flow linearly projects the output of layer normalization, and the result of linear projection is processed by SiLu activation function. After the activation function processing result and the output of SS2D module are element-by-element multiplied, the linear projection obtains the output of Mamba branch.

5. The infrared dual-band image fusion method based on self-attention and space-frequency domain cross fusion according to claim 1, characterized in that: In S2, the input features enter the CSA module, and for each attention head, a query vector, a key vector, and a value vector are generated through linear projection; each attention head generates a corresponding attention matrix through the query vector, the key vector, and the value vector, and the attention matrix of each head is spliced ​​along the channel dimension. The splicing result is restored to the dimension through linear projection, and the restored dimension features are multiplied element-by-element with the input features after the Sigmoid activation function to obtain the output features of the CSA module.

6. The infrared dual-band image fusion method based on self-attention and space-frequency domain cross fusion according to claim 1, characterized in that: The input features enter the CSA module. Before linear projection, they are first subjected to average pooling operation for feature dimensionality reduction and layer normalization. The features after layer normalization are then linearly projected.

7. The infrared dual-band image fusion method based on self-attention and space-frequency domain cross fusion according to claim 1, characterized in that: In S3, in the space-frequency domain cross-fusion module, the medium wave image feature and the long wave image feature are processed by the Mamba block respectively to obtain the medium wave image process feature one and the long wave image process feature two respectively; the medium wave image feature and the long wave image feature are processed by the frequency domain fusion module to obtain the frequency domain fusion feature, the frequency domain fusion feature is subtracted element by element from the medium wave image feature and the long wave image feature to obtain the medium wave difference feature and the long wave difference feature, the medium wave difference feature and the long wave difference feature are processed by the Mamba block respectively to obtain the medium wave image process feature two and the medium wave image process feature three respectively, the medium wave image process feature two, the long wave image process feature one and the medium wave image feature are added to obtain the medium wave mixed modal feature; the long wave image process feature two, the medium wave image process feature one and the long wave image feature are added to obtain the long wave mixed modal feature, and the medium wave mixed modal feature and the long wave mixed modal feature are concatenated to obtain the fusion feature.

8. The infrared dual-band image fusion method based on self-attention and space-frequency domain cross fusion according to claim 7, characterized in that: The medium-wave difference features and long-wave difference features are processed by the Mamba block respectively, and learnable parameters are introduced as weight coefficients of the difference features.

9. The infrared dual-band image fusion method based on self-attention and space-frequency domain cross fusion according to claim 7, characterized in that: In the frequency domain fusion module of the medium-wave image features and the long-wave image features, the amplitude component and the phase component are first obtained through two-dimensional fast Fourier transform, and the amplitude component of the medium-wave image features and the amplitude component of the long-wave image features are spliced ​​along the channel dimension to obtain the amplitude splicing feature; the phase component of the long-wave image features and the phase component of the long-wave image features are spliced ​​to obtain the phase splicing feature; the amplitude splicing feature and the phase splicing feature are respectively integrated through convolution operation and ReLU activation function, and then processed by 2D-IFFT. After channel shuffling, the frequency domain fusion feature is obtained.

10. The infrared dual-band image fusion method based on self-attention and space-frequency domain cross fusion according to claim 7, characterized in that: In S4, the medium-wave mixed modal features and the long-wave mixed modal features are concatenated and linearly projected to output fusion features.

Images