Infrared and visible light image fusion method based on attention and Transform

Abstract

The invention provides an infrared and visible light image fusion method based on attention and Transform, and belongs to the field of image fusion, and the method comprises the steps: carrying out the element-by-element splicing of an infrared image and a visible light image, and obtaining a dual-channel feature; extracting shallow layer features by using a shallow layer feature extraction module; complementary attribute extraction of important features is carried out through a double-attention residual module containing residual learning and two attention mechanisms; paying attention to remote context features by means of a Transform module; for the output of the Transform module, inputting the output into a second double-attention residual module to extract high-level semantic features; carrying out dimension reduction processing through an image reconstruction module to obtain a fusion result; and finally, calculating fusion loss, training the proposed model in an unsupervised manner, and realizing end-to-end generation of a high-quality fusion image with source image complementary information.

Description

Technical Field

[0001] The present invention belongs to the technical field of image fusion, and in particular relates to an infrared and visible light image fusion method based on attention and Transformer. Background Art

[0002] Infrared (IR) and visible light (VIS) image fusion is a method of fusing infrared images with visible light images to obtain a comprehensive image that is consistent with the source image in terms of details and thermal targets. In terms of imaging principle, infrared sensors can reflect the radiation characteristics of foreground targets through thermal radiation imaging, but generally lack structural content and texture details. Visible light sensors can describe the background details of the scene through light reflection, but are less affected by changes in lighting and weather conditions. Therefore, infrared and VIS image fusion can make full use of target radiation information (i.e., corresponding infrared image) and texture information (i.e., corresponding VIS image) to synthesize a synthetic image with prominent foreground target perception and rich background detail representation for subsequent image processing or decision-making. At present, infrared and visible light image fusion has become an important and valuable image processing technology, widely used in many fields such as target detection and pedestrian re-identification.

[0003] Infrared and visible light image fusion technology is mainly divided into traditional methods and deep learning methods. Traditional methods can be divided into five categories according to the adopted theory: multi-scale transformation-based methods, sparse representation-based methods, subspace decomposition-based methods, hybrid-based methods, and optimization model-based methods. These methods usually perform activity level measurements in the spatial domain or transform domain (converting the image to the transform domain through some mathematical transformation) and manually design fusion rules to achieve image fusion. Since these traditional fusion methods rely on manual production, they are time-consuming and have limited fusion performance for various scenes.

[0004] Methods based on deep learning can be divided into three categories: methods based on autoencoders (AE), methods based on generative adversarial networks (GAN), and methods based on convolutional neural networks (CNN). Most of the existing fusion methods based on deep learning use convolutional neural networks (CNN) for feature extraction. Convolution operations are good at capturing local features, but cannot represent the long-term dependencies of source images, resulting in insufficient extraction of some important global context information. Summary of the invention

[0005] In response to the above-mentioned technical problems, the present invention provides an infrared and visible light image fusion method based on attention and Transformer to solve the problems existing in the above-mentioned prior art.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] A method for fusion of infrared and visible light images based on attention and Transformer, comprising the following steps:

[0008] Step S1, splicing the infrared image IR and the visible light image VIS element by element to obtain dual-channel data {IR, VIS};

[0009] Step S2, input {IR, VIS} into a shallow feature extraction module (SFEM) to extract shallow features;

[0010] Step S3, inputting the shallow features extracted in the shallow feature extraction module (SFEM) into a dual attention residual module (DARM) including residual learning and two attention mechanisms to extract complementary attributes of important features;

[0011] Step S4: Input the features extracted by the dual attention residual module (DARM) into the Transformer module (TRM) to extract long-range context features;

[0012] Step S5, input the features extracted by the Transformer module (TRM) into the second dual attention residual module (DARM) to extract high-level semantic features;

[0013] Step S6, input the features extracted by the second dual attention residual module (DARM) into the image reconstruction module for dimensionality reduction processing to obtain a fusion result;

[0014] Step S7: Calculate the fusion loss and optimize the network.

[0015] Compared with the prior art, the present invention has the following beneficial effects:

[0016] 1. The present invention designs a dual attention residual module (DARM) and a Transformer module (TRM), which can not only extract local useful features of the input image well, but also fully construct global context dependencies. Specifically, this method can focus on more important complementary information and obtain a fusion result with more texture details.

[0017] 2. The present invention designs a loss function composed of pixel loss, gradient loss and structural loss to train the proposed fusion model in an unsupervised manner, which can automatically generate information fused images with significant targets and clear environments without the need to manually design activity level measurement and fusion strategies. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The present invention is further described in detail below in conjunction with the accompanying drawings;

[0019] Figure 1 It is a schematic diagram of the process structure of the present invention;

[0020] Figure 2 It is a block diagram of the overall network structure of the present invention;

[0021] Figure 3 It is a structural diagram of a shallow feature extraction module (SFEM) and an image reconstruction module (IRM) of the present invention;

[0022] Figure 4 It is a structural diagram of the Transformer module (TRM) of the present invention;

[0023] Figure 5 It is a structural diagram of a dual attention residual module (DARM) of the present invention;

[0024] Figure 6 This is a structural diagram of the channel attention block (CAB) of the present invention; DETAILED DESCRIPTION

[0025] Refer to the attached Figure 1-6The infrared and visible light image fusion method based on attention and Transformer is shown. The present invention designs an infrared and visible light image fusion method based on attention and Transformer. The method is based on an end-to-end infrared and visible light image fusion model of attention and Transformer. In the image fusion process, the infrared image IR and the visible light image VIS are spliced ​​element by element to obtain dual-channel data {IR, VIS}, and a shallow feature extraction module (SFEM) is used to extract shallow features. A dual attention residual module (DARM) is used to extract complementary attributes of important features. A transformer module (TRM) is used to extract remote context features. A second dual attention residual module (DARM) is used to extract high-level semantic features. Finally, an image reconstruction module (IRM) is used to perform dimensionality reduction processing to obtain a fusion result. The specific process includes:

[0026] Step S1: infrared image I IR and visible light image I VIS Perform element-by-element splicing to obtain a dual-channel map {I IR ,I VIS}.

[0027] Step S2: IR ,I VIS} Input to the shallow feature extraction module to extract shallow features;

[0028] Among them, the shallow feature extraction module is composed of convolutional layer, batch normalization and ReLU. The operation of the shallow feature extraction module can be expressed as in Represents the features after convolution filtering; denote the 5×5 convolution of input channel 2 and output channel C, respectively. The padding size of the replication mode is 2. BN(·) and Re(·) denote batch normalization and ReLU, respectively.

[0029] Step S3: Output of shallow feature extraction module Enter the Dual Attention Residual Module (DARM) to focus on important features for complementary attribute extraction;

[0030] The dual-attention residual module combines residual learning and two attention mechanisms (channel attention and spatial attention) into a feature-preserving module, which has three main blocks: residual block (RB), channel attention block (CAB), and spatial attention block (SAB);

[0031] like Figure 5 As shown, the DARM input Input into RB to obtain the texture features of visible light images and the thermal distribution of infrared images; CAB and SAB process the output of RB in parallel In order to more fully capture important features;

[0032] There are two multiplication operations in DARM. One is the output of CAB and multiplied, the other is the output of SAB and Multiplication can be expressed as:

[0033]

[0034] Among them, F MC and F MS are the results of these two multiplications. MC 、F MS and A residual connection is constructed between them to fully extract the complementary information that may be lost during the convolution operation. The residual connection can be expressed as:

[0035]

[0036] Among them, F A Represents the result of addition; output feature map of DARM It is generated by the ReLU activation function;

[0037] The architecture of the residual block RB, channel attention block CAB, and spatial attention block SAB is shown below:

[0038] RB consists of two convolutional layers and can be expressed as:

[0039]

[0040] in, represents a 3×3 convolution with both input and output channels of C;

[0041] like Figure 6 As shown, the specific implementation of CAB is: given a dimension R C×H×W We first use the convolutional layer to generate three components P(f), Q(f), H(f), and transform P(f), Q(f) from R C×H×W Refactoring to R C×HW , we get M P , M Q ; Then for M P and M Q The transpose of is used for matrix multiplication, and then the attention feature map M∈R is calculated through the softmax layer. C×C ; Then, perform a matrix multiplication between H(f) and the transpose of M, and finally reshape the result and add it back to the source image H f ;

[0042] SAB consists of two convolutional layers, a ReLU layer and a sigmoid layer, which are used to highlight the features in the spatial space; the feature extraction process of SAB can be described as:

[0043]

[0044] in, Represents a 3×3 convolution with input channel C and output channel C / R. It indicates that the input channel and output channel are convolved with C / R and 1 respectively by 3×3; Sig(·) is the sigmoid activation function.

[0045] Step S4: The output of the double attention residual module Input Transformer module (TRM) to extract long-range context features and fully extract global complementary information;

[0046] The structure of TRM is as follows Figure 4 As shown, TRM consists of four layers and two addition operations; the first layer of TRM is LN, followed by MSA and addition operations, expressed as:

[0047]

[0048] Among them, F SA is the result of the addition operation, is the input of TRM; LN(·) and MSA(·) represent layer normalization and multi-head self-attention respectively; first, SA Perform LN processing, then perform multi-layer perception (MLP), and then SA Another addition is constructed between the output of MLP, which can be expressed as:

[0049]

[0050] in is the output of TRM;

[0051] The implementation details of TRM are as follows: Divide into non-overlapping M×M windows and generate Features, among which is the total number of windows. All these windows calculate the standard self-attention in several windows. For the window feature The Q, K, and V corresponding to the query index, key index, and value index can be calculated as follows:

[0052] Q=F W ·P Q ,

[0053] K=FW ·P K ,

[0054] V=F W ·P V ,

[0055] Where P Q , P K , and P V is the projection matrix. The attention metric within the window is calculated by the self-attention scheme and is expressed as:

[0056]

[0057] Where S(·) is the Softmax function. d represents the dimension, and B represents the learnable relative position encoding. The MLP has two LN layers with a GELU nonlinearity between them to facilitate further feature transformation.

[0058] Step S5: Transformer module (TRM) output The input is fed into the second Dual Attention Residual Module (DARM) to extract high-level semantic features.

[0059] Step S6: Input the output of the second dual attention residual module (DARM) to the image reconstruction module (IRM) for dimensionality reduction to obtain the fusion result I F ;

[0060] IRM consists of a convolutional layer and a tanh activation function and can be expressed as:

[0061]

[0062] in Represents 1×1 convolution, the input channel is C, and the output channel is 1. represents the input feature of the convolution, and Tanh(·) represents the tanh activation function.

[0063] Step S7, calculate the fusion loss and optimize the network parameters; the fusion loss L is calculated by the pixel loss L P , gradient loss L G and structural loss L S Composition, defined as:

[0064] L=L P +λ·L G +γ·L S

[0065] Where λ and γ are weight factors that control each contribution;

[0066] Pixel loss L P Defined as:

[0067] Among them, ||·|| F represents the matrix Frobenius norm; H and W represent the height and width of the source image respectively;

[0068] α is a weight parameter that controls the balance between the two items;

[0069] Gradient loss L G Defined as:

[0070]

[0071] Among them, ||·||2 represents the matrix l2-norm norm; represents the gradient operation; max{,} represents the maximum selection;

[0072] Structural similarity loss L S Defined as:

[0073] L S =1-SSIM(I F ,max{I IR ,I VIS})

[0074] Among them, SSIM(·) represents the structural similarity measure.

[0075] The above description is only a preferred embodiment of the present invention, and therefore cannot be used to limit the scope of implementation of the present invention. That is, equivalent changes and modifications made according to the scope of the patent application of the present invention and the contents of the specification should still fall within the scope covered by the patent of the present invention.

Claims

1. A method for fusion of infrared and visible light images based on attention and Transformer, characterized in that: The following steps are involved: Step S1, concatenating the paired infrared image IR and visible light image VIS element by element to obtain dual-channel data {IR, VIS}; Step S2, input {IR, VIS} into a shallow feature extraction module (SFEM) to extract shallow features; Step S3, inputting the shallow features extracted in the shallow feature extraction module into a dual attention residual module (DARM) including residual learning and two attention mechanisms to extract complementary attributes of important features; Step S4: Input the features extracted by the dual attention residual module into the Transformer module (TRM) to extract long-range context features; Step S5, input the features extracted by the Transformer module (TRM) into the second dual attention residual module (DARM) to extract high-level semantic features; Step S6, input the features extracted by the second dual attention residual module (DARM) into the image reconstruction module (IRM) for dimensionality reduction processing to obtain a fusion result; Step S7: Calculate the fusion loss and optimize the network.

2. The infrared and visible light image fusion method based on attention and Transformer according to claim 1, characterized in that: In step S2, the shallow feature extraction module (SFEM) is composed of a convolutional layer, batch normalization and ReLU. The operation of the shallow feature extraction module can be expressed as: in Represents the features after convolution filtering; denote 5×5 convolutions of input channel 2 and output channel C, respectively; the padding size of the replication mode is 2; BN(·) and Re(·) are batch normalization and ReLU, respectively.

3. The infrared and visible light image fusion method based on attention and Transformer according to claim 1, characterized in that: In step S3, the dual attention residual module (DARM) consists of residual learning and two attention mechanisms. The module has three main blocks: residual block (RB), channel attention block (CAB) and spatial attention block (SAB).

4. The method for fusion of infrared and visible light images based on attention and Transformer according to claim 3, characterized in that: The specific implementation of step S3 is: Step S31: Input of the dual attention residual module (DARM) Input into the residual block (RB) to obtain the texture features of the visible light image and the thermal distribution of the infrared image; Step S32: The channel attention block (CAB) and the spatial attention block (SAB) process the output of the residual block (RB) in parallel.

5. The method for fusion of infrared and visible light images based on attention and Transformer according to claim 4, characterized in that: The specific implementation of the residual block (RB) in step S31 is: in represents a 3×3 convolution with both input and output channels of C; It represents the input of the Dual Attention Residual Module (DARM).

6. The method for fusion of infrared and visible light images based on attention and Transformer according to claim 4, characterized in that: The specific implementation of the channel attention block (CAB) in step S32 is: Given a dimension R C×H×W We first use the convolutional layer to generate three components P(f), Q(f), H(f), and reconstruct P(f), Q(f) from R C×H×W Refactoring to R C×HW , we get M P , M Q ; Then for M P and M Q The transpose of is used for matrix multiplication, and then the attention feature map M∈R is calculated through the softmax layer. C×C ; Then, perform a matrix multiplication between H(f) and the transpose of M, and finally reshape the result and add it back to the source image H f .

7. The method for fusion of infrared and visible light images based on attention and Transformer according to claim 4, characterized in that: The specific implementation of the spatial attention block (SAB) in step S32 is: Among them, Conv3 C,C / R (·) represents a 3×3 convolution of input channel C and output channel C / R; It indicates that the input channel and output channel are convolved with C / R and 1 respectively by 3×3; Sig(·) is the sigmoid activation function.

8. The method for fusion of infrared and visible light images based on attention and Transformer according to claim 1, characterized in that: In step S4, the specific implementation of the Transformer module (TRM) is as follows: the TRM consists of four layers and two addition operations; the first layer of the TRM is LN, followed by MSA and addition operations, which is expressed as: Among them, F SA is the result of the addition operation, is the input of TRM; LN(·) and MSA(·) represent layer normalization and multi-head self-attention respectively; first, SA Perform LN processing, then perform multi-layer perception (MLP), and then SA Another addition is constructed between the output of MLP, which can be expressed as: in is the output of TRM.

9. The method for fusion of infrared and visible light images based on attention and Transformer according to claim 1, characterized in that: In step S6, the specific implementation of the image reconstruction module (IRM) is as follows: in Represents 1×1 convolution, the input channel is C, and the output channel is 1; represents the input feature of the convolution, and Tanh(·) represents the tanh activation function.

10. The method for fusion of infrared and visible light images based on attention and Transformer according to claim 1, characterized in that: In step S7, the fusion loss is calculated and the network parameters are optimized, including: the fusion loss L is calculated by the pixel loss L P , gradient loss L G and structural loss L S Composition, defined as: L=L P +λ·L G +γ·L S Where λ and γ are weight factors that control each contribution; Pixel loss L P Defined as: Among them, ||·|| F represents the Frobenius norm of the matrix; H and W represent the height and width of the source image respectively; α is a weight parameter that controls the balance of the two items; Gradient loss L G Defined as: Among them, ‖·‖2 represents the matrix l2-norm norm; represents the gradient operation; max{,} represents the maximum selection; Structural similarity loss L S Defined as: L S =1-SSIM(I F ,max{I IR ,I VIS }) Among them, SSIM(·) represents the structural similarity measure.

Images