An infrared and visible light image fusion method combining a transformer and a CNN dual encoder

Abstract

The application discloses an infrared and visible light image fusion method combining a double encoder of a Transformer and a CNN, constructs an infrared and visible light image fusion model combining the double encoder of the Transformer and the CNN based on a U-Net framework, wherein the fusion network based on the U-Net framework is composed of a double encoder and a decoder, and the model is trained by using a training set; infrared images and visible light images are input into the infrared and visible light image fusion model combining the double encoder of the Transformer and the CNN which has been trained, the double encoder is used to extract infrared features and visible light features of the infrared images and the visible light images pre-fused, and feature representation is performed in a top-down manner; the decoder is used to map the feature representation to an original resolution, image features are gradually fused in a bottom-up manner, and a fusion image is obtained. The application effectively combines local information and global information, improves the quality of the fusion image, and reduces the calculation complexity caused by a single Transformer structure.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an infrared and visible light image fusion method combining a dual encoder of a Transformer and a CNN. BACKGROUND

[0002] Existing image fusion methods mainly include two types: one is a traditional fusion method, and the other is a deep learning-based image fusion method. Traditional algorithms mainly include sparse representation, multi-scale transformation, subspace and hybrid fusion methods, etc. These algorithms usually decompose images into different hierarchical features or establish sparse matrices according to image features, and then perform image fusion through the decomposition results combined with the designed fusion strategy. Although it can preserve part of the details of the source image and highlight the target to a certain extent, the modal difference between the source images makes the hand-designed traditional method more and more complex, and it is difficult to further improve the fusion performance. At the same time, the calculation complexity is high, the time-consuming is long, and the spatial consistency in the fusion process is easily ignored, and the generalization performance of the traditional model is poor.

[0003] The existing deep learning-based image fusion method has better fusion performance, and the network model has stronger feature extraction capability. The main fusion models at present are divided into CNN (Convolution Neural Network) and GAN (Generative Adversarial Network). The image fusion method based on CNN and GAN extracts features of infrared images and visible light images through convolution layers, which preserves the infrared pixel amplitude and visible light detail information to a certain extent. However, the features extracted by the convolution kernel only have strong regional correlation at the current position, and the extraction capability of long-distance context information is not considered, and the detail texture information of the fused image is not fully extracted, which cannot enhance the information fusion from a global perspective. In view of the problems existing in the existing infrared and image fusion method, many scholars use the Transformer to realize the long-distance dependence of information, which mainly benefits from its self-attention mechanism and global feature extraction capability. However, the use of Transformer to extract features requires a large amount of computing resources. SUMMARY

[0004] The technical problem to be solved by the present application is to provide an infrared and visible light image fusion method combining a dual encoder of a Transformer and a CNN, which combines CNN and Transformer to form a parallel dual encoder structure, can focus on local context information and global information, and can train the network model with high efficiency and calculation advantage to improve the quality of the fused image.

[0005] The present application adopts the following technical solutions to solve the above technical problems:

[0006] The application provides an infrared and visible light image fusion method combining a Transformer and a CNN dual encoder.

[0007] S1. Based on a U-Net framework, an infrared and visible light image fusion model combining a Transformer and a CNN dual encoder is constructed, wherein the model is composed of a dual encoder and a decoder.

[0008] S2. A data set is selected, and a training set is obtained after processing. The infrared and visible light image fusion model combining the Transformer and the CNN dual encoder in step S1 is trained using the training set.

[0009] S3. The infrared image and the visible light image are input into the infrared and visible light image fusion model combining the Transformer and the CNN dual encoder which has been trained, the infrared features and the visible light features of the infrared image and the visible light image are extracted by the dual encoder, and the features are represented in a top-down manner.

[0010] S4. The features in step S3 are mapped to the original resolution by the decoder, the image features are gradually fused in a bottom-up manner, and a fused image is obtained.

[0011] Further, in step S1, the infrared and visible light image fusion model combining the Transformer and the CNN dual encoder includes the following contents:

[0012] S101. A dual encoder is constructed: the dual encoder includes a local detail feature extraction encoder and a global information extraction encoder.

[0013] The local detail feature extraction encoder includes convolution layers and four feature extraction blocks connected in sequence, the convolution layers are composed of a convolution with a convolution kernel of 4x4, a step of 2 and a padding of 1, and a batch normalization and a LeakyReLU connected in series, the convolution layers are used for downsampling operation and adjusting the channel number and size of the input image.

[0014] The first and second feature extraction blocks are composed of residual modules, the third and fourth feature extraction blocks are composed of residual modules and INN (Invertible Neural networks) connected in series, the channel numbers of the four feature extraction blocks are 16, 32, 64 and 128 respectively, and the output channel numbers are 32, 64, 128 and 256 respectively.

[0015] The invertible neural network includes three invertible residual modules, which include a convolution with a convolution kernel of 1x1, a depthwise separable convolution with a convolution kernel of 3x3 and an activation function (ReLU6).

[0016] The global information extraction encoder includes four Transform Layers connected in series, the input channel numbers of which are 16, 32, 64 and 128 respectively, and the output channel numbers of which are 32, 64, 128 and 256 respectively, the first, second and fourth Transform Layers include two layers of HAT (Hybrid Attention Transformer) structures, wherein the first layer of the first Transform Layer includes a patch embedding operation, a layer normalization operation, a W-MSA module (Windows Multi-Head Self-Attention), a local-non-local attention information enhancement module, a layer normalization operation and a feedforward neural network connected in series, and the second layer includes a layer normalization operation, a SW-MSA module (Shift Windows Multi-Head Self-Attention), a local-non-local attention information enhancement module, a layer normalization operation and a feedforward neural network connected in series.

[0017] The first layer of the second and fourth Transform Layers includes a patch merging operation, a layer normalization operation, a W-MSA module (Windows Multi-Head Self-Attention), a local-non-local attention information enhancement module, a layer normalization operation and a feedforward neural network connected in series, and the second layer includes a layer normalization operation, a SW-MSA module (Shift Windows Multi-Head Self-Attention), a local-non-local attention information enhancement module, a layer normalization operation and a feedforward neural network connected in series.

[0018] The third Transform Layer includes six layers of HAT (Hybrid Attention Transformer) structures, which are composed of three times of stacking of the two layers of HAT (Hybrid Attention Transformer) structures in the second Transform Layer.

[0019] S102, constructing a decoder

[0020] The decoder includes four feature fusion modules and a reconstruction module; each feature fusion module includes an up-sampling operation, a concatenation operation, a deconvolution with a convolution kernel of 4x4, a step of 2 and a padding of 1, a BatchNorm operation and an activation function ReLU connected in sequence.

[0021] The reconstruction module comprises sequentially connected upsampling operations, a convolution with a kernel of 4x4, a convolution with padding of 1, a padding operation, and a hyperbolic tangent function (Tanh).

[0022] S103, the dual encoder extracts features by stepwise downsampling in a top-down manner, and the decoder fuses and reconstructs the features extracted by the dual encoder by stepwise upsampling in a bottom-up manner, to form an infrared and visible light image fusion model combining the dual encoder of the Transformer and the CNN.

[0023] Further, in step S1, the residual module comprises a main path and a residual path, the main path comprising a convolution with a kernel of 1x1, BN, a LeakyReLU activation function, a convolution with a kernel of 3x3, and a convolution with a kernel of 4x4; and the residual path comprising a DSConv and a convolution with a kernel of 4x4.

[0024] Further, in step S1, the local-non-local attention information enhancement module comprises a local information enhancement module and a non-local information enhancement module; wherein the local information enhancement module comprises a concatenation operation, a global average pooling, a depth separable convolution, a Sigmoid function, an element-wise multiplication operation, and an element-wise multiplication operation; and the non-local information enhancement module comprises a LayerNorm operation and a position attention module.

[0025] Further, in step S2, training the infrared and visible light image fusion model combining the dual encoder of the Transformer and the CNN comprises the following sub-steps:

[0026] S201, selecting m pairs of images from a TNO dataset as a dataset, converting the image gray value to [-1, 1], cropping the image using a window of h x w, setting the step size as s, and finally obtaining n pairs of image blocks as a training set, wherein h represents the height of the image, and w represents the width of the image.

[0027] S202, training the infrared and visible light image fusion model combining the dual encoder of the Transformer and the CNN using structural similarity loss and contrastive loss supervision, and obtaining optimal network parameters, the specific formula being:

[0028] L total =λ1L SSIM +λ2L patchNCE

[0029] Wherein, L total represents a loss function, λ1 and λ2 represent hyperparameters, L SSIM represents structural similarity loss, and L patchNCE represents contrastive loss.

[0030] S203, according to the training set and the loss function Ltotal The initial infrared and visible light image fusion model combining the dual encoder of the Transformer and CNN is trained, and the Adam optimizer is used in the training process to update the network model parameters, and the trained infrared and visible light image fusion model combining the dual encoder of the Transformer and CNN is obtained.

[0031] Further, in step S3, the feature extraction using the dual encoder includes the following sub-steps:

[0032] S301, convert the gray values of the infrared image and the visible light image to [-1, 1], and crop the image using a window of h x w, and set the step size to s;

[0033] S302, input the processed infrared image and visible light image in step S301 into the dual encoder, and perform down-sampling operation on the image through the convolution layer to adjust the channel number and size of the image;

[0034] S303, input the image processed in step S302 into four feature extraction blocks, sequentially pass through 1x1 convolution, BN, LeakyReLU activation function, convolution kernel 3x3 convolution, BN and convolution kernel 4x4 convolution operation to obtain the convolution features learned in the main path, and then pass through DSConv and convolution kernel 4x4 convolution operation to obtain the supplementary information features, add the convolution features and the supplementary information features, and then pass through LeakyReLU to obtain the information enhancement features;

[0035] S304, input the information enhancement features obtained in step S303 into the reversible neural network, and divide the features into two parts in the channel dimension: and The feature obtained by the first reversible residual module is added to the feature element by element to obtain the feature The feature passes through the second reversible residual module and is multiplied by the feature element by element to obtain the feature The feature passes through the third reversible residual module to obtain the feature and is added to the feature element by element to obtain the feature The feature and the feature are spliced in the channel dimension to obtain the output feature The specific formula is:

[0036] ​​

[0037]

[0038]

[0039] wherein C represents the complete number of channels, c represents half of the complete number of channels, [1:c] represents the current number of feature channels being 1 to c, [c+1:C] represents the current number of feature channels being c+1 to C, S represents a CNN feature extraction encoder, I represents an arbitrary mapping symbol, k represents the kth reversible network layer number, R n (n = 1, 2, 3) represents a reversible residual module, and exp represents the exponential e.

[0040] The final output infrared local detail feature and the visible light local detail feature.

[0041] The reversible neural network is designed to prevent information loss by reciprocal generation of input and output features, which meets the goal of preserving high-frequency features in the fused image.

[0042] S305, in the first Transformer Layer, the features of the infrared image and the visible light image are respectively subjected to patch embedding operation, then the features are standardized by layer normalization operation, and then the features are input into the W-MSA module, the images are divided into blocks according to the set window size, the self-attention map is obtained, then the local-non-local attention information enhancement module is used to obtain the local-non-local attention information enhancement feature, the enhancement features are added element by element to obtain the attention feature F l , after the layer normalization operation, the feature is obtained through the FFN module, and the feature is added to the attention feature to obtain the output feature F l+1 of the first layer HAT structure; the output feature F l+1 is input into the second layer HAT structure, after patch embedding operation, the feature is standardized by layer normalization operation, the standardized feature is input into the SW-MSA module, the SW-MSA module divides the images into blocks according to the set window size to obtain the self-attention map, and then the local-non-local attention information enhancement module is used to obtain the local-non-local attention information enhancement feature, the local-non-local attention information enhancement features are added element by element to obtain the attention feature F l+1 , the feature obtained by the FFN module after the layer normalization operation of the attention feature is added to the attention feature to obtain the infrared global information feature and the visible light global information feature of the first Transformer Layer, and the specific formula is:

[0043] F l = LG(W-MSA(LN(F l-1 )))+F l-1 ;

[0044] F1 l = FFN(LN(F l ))+ F l ;

[0045] F l+1 = LG(SW-MSA(LN(F1 l )))+ F1 l ;

[0046] F1 l+1 = FFN(LN(F l+1 ))+ F l+1 ;

[0047] wherein, F l-1 represents the features of the infrared image and the visible light image; F l represents the features output by the first layer HAT; F l+1 represents the infrared global information features and the visible light global information features; LG(•) represents a local-non-local information enhancement operation; LN(•) represents a layer normalization operation for normalizing each layer of features; W-MSA(•) represents a window multi-head self-attention module; and SW-MSA(•) represents a shifted window multi-head self-attention module operation, which improves the information interaction ability between windows through two different multi-head self-attention mechanisms, thereby improving the model generalization performance.

[0048] S306, the second, third and fourth Transformer Layers repeat the content of step S305, wherein the third Transformer Layer needs to be repeated three times before being input into the fourth Transformer Layer to output the global information features of the infrared and visible light;

[0049] S307, the double encoder each feature extraction block is added to the infrared and visible light global information features and the infrared and visible light local detail features corresponding to the modal element by element to obtain four groups of pre-fused infrared features and visible light features.

[0050] Further, in step S305, the specific content of the local-non-local attention information enhancement module for extracting features is as follows:

[0051] The input features are respectively subjected to global average pooling, depth separable convolution, sigmoid operation to obtain a local information enhancement map, and then subjected to element-by-element multiplication operation with the original input features to obtain a local information enhancement feature.

[0052] Further, in step S4, obtaining the fusion image by the decoder comprises the following sub-steps:

[0053] S401, after the fourth group of pre-fused infrared features and visible light features are spliced, the third group of pre-fused infrared features and visible light features are spliced in the channel dimension after the up-sampling operation to obtain the third group of reconstruction features, the second group of pre-fused infrared features and visible light features are spliced in the channel dimension after the up-sampling operation of the third group of reconstruction features to obtain the second group of reconstruction features, and the third group of pre-fused infrared features and visible light features are spliced in the channel dimension after the up-sampling operation of the second group of reconstruction features to obtain the reconstruction features.

[0054] S402, the reconstruction features in step S401 are subjected to up-sampling operation, padding operation, 4x4 convolution operation and Tanh operation to finally obtain the reconstructed fusion image.

[0055] The above technical scheme is adopted in the present application, and compared with the prior art, the present application has the following significant technical effects:

[0056] 1. The present application overcomes the limitations of the global information extraction capability of the image fusion method based on the convolutional neural network and the image fusion method based on the generative adversarial network.

[0057] 2. In the U-net framework, the present application embeds the Transformer into the CNN to form a parallel double-encoder structure, which overcomes the shortcomings of the CNN in global modeling, effectively combines local information with global information, improves the quality of the fusion image, and reduces the computational complexity caused by the single Transformer structure.

[0058] 3. The present application introduces a reversible neural network in the local detail feature extraction path, which prevents information loss by mutual generation of input and output features through reversible design, and further improves the high-frequency local detail feature extraction capability.

[0059] 4. The present application develops a local-non-local information enhancement module, which introduces local information induction bias to supplement the learning of non-local information features, improves the anti-interference ability of the model, and enhances the detail information extraction capability.

[0060] 5、The fusion effect of the present application is significantly improved compared with the prior art, the model has good anti-interference ability and generalization ability, can be applied to multi-modal image fusion, and has high application value in the field of image fusion. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 The overall implementation flowchart of the present application.

[0062] Figure 2 The residual module schematic diagram in the local detail feature extraction path.

[0063] Figure 3 The reversible neural network schematic diagram in the local detail feature extraction path.

[0064] Figure 4 The reversible residual module schematic diagram in the reversible neural network.

[0065] Figure 5 The hybrid attention transformer schematic diagram in the global information extraction path.

[0066] Figure 6 The local-non-local attention information enhancement module schematic diagram in the hybrid attention transformer.

[0067] Figure 7 The image reconstruction module schematic diagram in the decoder.

[0068] Figure 8 The infrared image, visible light image and fusion effect comparison chart of each method of the first group of images (sandpath).

[0069] Figure 9 The infrared image, visible light image and fusion effect comparison chart of each method of the second group of images (Kaptein_1123).

[0070] Figure 10 The infrared image, visible light image and fusion effect comparison chart of each method of the third group of images (lake). DETAILED DESCRIPTION

[0071] The technical solutions of the present application will be described clearly and completely in combination with the drawings and specific embodiments in the specification. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. All other embodiments obtained by those skilled in the art based on the embodiments in the present application belong to the scope of protection of the present application.

[0072] The present application proposes an infrared and visible light image fusion method combining a Transformer and a CNN dual encoder, as shown in Figure 1As shown, comprising the following steps:

[0073] S1, based on the U-Net framework, an infrared and visible light image fusion model combining a double encoder of Transformer and CNN is constructed, wherein the fusion network based on the U-Net framework is composed of a double encoder and a decoder, and the specific content is:

[0074] S101, constructing a double encoder: the double encoder includes a local detail feature extraction encoder and a global information extraction encoder.

[0075] Among them, the local detail feature extraction encoder includes a convolution layer and four feature extraction blocks connected in turn, the convolution layer is composed of a convolution with a convolution kernel of 4x4, a step of 2 and a padding of 1, and a batch normalization, LeakyReLU connected in series, the convolution layer is used for downsampling operation, and the channel number and size of the input image are adjusted.

[0076] The second and second feature extraction blocks are composed of a residual module, and the third and fourth feature extraction blocks are composed of a residual module and an INN (Invertible Neural networks, reversible neural network) connected in series, the channel numbers of the four feature extraction blocks are 16, 32, 64 and 128 respectively, and the channel numbers of the outputs are 32, 64, 128 and 256 respectively.

[0077] As shown in Figure 2 , the residual module includes a main path and a residual path, the main path includes a convolution with a convolution kernel of 1x1, a BN, a LeakyReLU activation function, a convolution with a convolution kernel of 3x3, and a convolution with a convolution kernel of 4x4; the residual path includes DSConv and a convolution with a convolution kernel of 4x4.

[0078] As shown in Figure 3 , 4 , the reversible neural network includes three reversible residual modules, which include a convolution with a convolution kernel of 1x1, a depth separable convolution with a convolution kernel of 3x3, and an activation function (ReLU6).

[0079] As shown in Figure 5As shown, the global information extraction encoder includes four TransformerLayers connected in series, the number of input channels of which are 16, 32, 64, and 128, and the number of output channels are 32, 64, 128, and 256, respectively. The first, second, and fourth Transformer Layers include two layers of HAT (Hybrid Attention Transformer) structures, wherein the first layer of the first Transformer Layer includes a patch embedding operation connected in series, a layer normalization operation, a W-MSA module (Windows Multi-Head Self-Attention), a local-non-local attention information enhancement module, a layer normalization operation, and a feedforward neural network, and the second layer includes a layer normalization operation, a SW-MSA module (Shift Windows Multi-Head Self-Attention), a local-non-local attention information enhancement module, a layer normalization operation, and a feedforward neural network connected in series.

[0080] The first layer of the second and fourth Transformer Layers includes a patch merging operation, a layer normalization operation, a W-MSA module (Windows Multi-Head Self-Attention), a local-non-local attention information enhancement module, a layer normalization operation, and a feedforward neural network connected in a serial manner. The second layer includes a layer normalization operation, a SW-MSA module (Shift Windows Multi-Head Self-Attention), a local-non-local attention information enhancement module, a layer normalization operation, and a feedforward neural network connected in a serial manner.

[0081] The third Transformer Layer includes a six-layer hybrid attention transformer structure, which is composed of two layers of HAT (Hybrid Attention Transformer) structures in the second Transformer Layer stacked three times.

[0082] Among them, Figure 6 As shown, the local-non-local attention information enhancement module includes a local information enhancement module and a non-local information enhancement module; the local information enhancement module includes a splicing operation, a global average pooling, a depth-separable convolution, a Sigmoid function, an element-by-element multiplication operation, and an element-by-element multiplication operation; the non-local information enhancement module includes a LayerNorm operation and a position attention module.

[0083] S102, construct a decoder:

[0084] The decoder includes four feature fusion modules and a reconstruction module; each feature fusion module includes an upsampling operation, a splicing operation, a deconvolution with a convolution kernel of 4x4, a step of 2 and padding of 1, a BatchNorm operation and an activation function ReLU connected in sequence.

[0085] As shown in Figure 7 , the reconstruction module includes an upsampling operation, a convolution with a convolution kernel of 4x4, a padding operation and a hyperbolic tangent function (Tanh) connected in sequence.

[0086] S103, the dual encoder extracts features by stepwise downsampling from top to bottom, and the decoder fuses and reconstructs the features extracted by the dual encoder by stepwise upsampling from bottom to top, to form an infrared and visible light image fusion model combining the dual encoder of Transformer and CNN.

[0087] S2, select a data set, process it to obtain a training set, and use the training set to train the infrared and visible light image fusion model combining the dual encoder of Transformer and CNN in step S1, the specific content is:

[0088] S201, select 32 pairs of images from the TNO data set as a data set, convert the image gray value to [-1, 1], use a 128x128 window to crop the image, set the step size to 32, and finally obtain 6184 image blocks as a training set.

[0089] S202, use structural similarity loss and contrast loss to supervise the training of the infrared and visible light image fusion model combining the dual encoder of Transformer and CNN, and obtain the optimal network parameters, the specific formula is:

[0090] L total =λ1L SSIM +λ2L patchNCE

[0091] Wherein, L total represents the loss function, λ1, λ2 represents the hyperparameter, L SSIM represents the structural similarity loss, L patchNCE represents the contrast loss.

[0092] Wherein, the structural similarity loss can evaluate the quality of the fused image in brightness, contrast and structure, and the structural similarity is represented as:

[0093]

[0094] Wherein, μ represents the pixel mean; σxy denotes covariance; σ denotes variance; C1, C2 are set to avoid a small value close to zero, usually 0.01 2 , 0.03 2 .

[0095] The structural similarity loss is represented as:

[0096]

[0097]

[0098]

[0099] where I x , I y and I f represent the visible light image, the infrared image and the fused image respectively; W represents a sliding window with a step of 1; P i represents the value of pixel i; m, n represent the size of the sliding window, which is set to 16x16 in the embodiment.

[0100] The contrast loss is represented as:

[0101]

[0102]

[0103]

[0104] where k represents the encoded feature sample; k + represents a positive sample similar to k; k - represents a negative sample dissimilar to k; τ represents a temperature coefficient, usually τ = 0.07; S represents the number of positions sampled in the image feature layer, where s ∈ {1, 2,..., S}; D s represents any feature sequence in the channel space.

[0105] By calculating the similarity of a certain region in space with positive and negative samples, the most similar parts of the fused image and the source image are effectively retained.

[0106] S203, according to the training set and the loss function L total The initial infrared and visible light image fusion model combining the Transformer and the CNN double encoder is trained. The Adam optimizer is used to update the network model parameters in the training process, the learning rate is set to 0.01, the Epoch is set to 30, and the Batchsize is set to 24. The infrared and visible light image fusion model combining the Transformer and the CNN double encoder is obtained after training.

[0107] S3, input the infrared image and the visible light image into the infrared and visible light image fusion model combined with the trained dual-encoder of the Transformer and the CNN, extract the infrared features and the visible light features of the infrared image and the visible light image pre-fused by the dual-encoder, and perform feature representation in a top-down manner, and the specific content is:

[0108] S301, convert the gray values of the infrared image and the visible light image into [-1, 1], and crop the image by using a window of 128x128, and set the step size to 32.

[0109] S302, input the infrared image and the visible light image processed in step S301 into the dual-encoder, and perform down-sampling operation on the image by using the convolutional layer to adjust the channel number and the size of the image.

[0110] S303, input the image processed in step S302 into four feature extraction blocks, sequentially pass through 1x1 convolution, BN, LeakyReLU activation function, convolution kernel of 3x3 convolution, BN and convolution kernel of 4x4 convolution operation to obtain the convolution features learned in the main path, and then pass through DSConv and convolution kernel of 4x4 convolution operation to obtain the supplementary information features, add the convolution features and the supplementary information features, and then pass through LeakyReLU to obtain the information enhancement features.

[0111] S304, input the information enhancement features obtained in step S303 into the reversible neural network, divide the features into two parts in the channel dimension, and , the feature obtained by the first reversible residual module is added to the feature element by element to obtain the feature , the feature obtained by the second reversible residual module is multiplied with the feature element by element to obtain the feature , and the feature obtained by the third reversible residual module is added to the feature element by element to obtain the feature , and the feature is obtained by the fourth reversible residual module, and is added to the feature element by element to obtain the feature , and the feature is obtained by the fifth reversible residual module, and is added to the feature element by element to obtain the output feature , and the specific formula is:

[0112]

[0113]

[0114]

[0115] wherein C represents the complete number of channels, c represents half of the complete number of channels, [1:c] represents the current number of feature channels is 1 to c, [c+1:C] represents the current number of feature channels is c+1 to C, S represents the CNN feature extraction encoder, I represents an arbitrary mapping symbol, k represents the kth reversible network layer number, R n (n = 1, 2, 3) represents a reversible residual module, and exp represents the exponential e.

[0116] Finally, the infrared local detail feature and the visible light local detail feature are output.

[0117] In the first Transformer Layer, the features of the infrared image and the visible light image are respectively subjected to patch embedding operation, then the features are standardized through layer normalization operation, and then the features are input into the W-MSA module, the images are divided into blocks according to the set window size, the self-attention map is obtained, then the local-non-local attention information enhancement module is passed through to obtain the local-non-local attention information enhancement feature, the enhancement features are added element by element to obtain the attention feature F l , and then the feature is obtained through the layer normalization operation and then the FFN module to obtain the feature, and the feature is added to the attention feature to obtain the output feature F l+1 of the first layer HAT structure; the output feature F l+1 is input into the second layer HAT structure, and after the patch embedding operation, the feature is standardized through the layer normalization operation, the standardized feature is input into the SW-MSA module, the SW-MSA module divides the images into blocks according to the set window size to obtain the self-attention map, and then the local-non-local attention information enhancement module is passed through to obtain the local-non-local attention information enhancement feature, the local-non-local attention information enhancement features are added element by element to obtain the attention feature F l+1 , and then the feature obtained through the layer normalization operation and then the FFN module is added to the attention feature to obtain the infrared global information feature and the visible light global information feature of the first Transformer Layer, and the specific formula is:

[0118] F l = LG(W-MSA(LN(F l-1 )))+F l-1 ;

[0119] F1 l = FFN(LN(F l ))+F l ;

[0120] F l+1= LG(SW-MSA(LN(F1 l )))+F1 l ;

[0121] F1 l+1 = FFN(LN(F l+1 ))+F l+1 ;

[0122] wherein, F l-1 represents the features of the infrared image and the visible light image; F l represents the features output by the first layer HAT; F l+1 represents the infrared global information features and the visible light global information features; LG(·) represents a local-non-local information enhancement operation; LN(·) represents a layer normalization operation for normalizing each layer feature; W-MSA(·) represents a window multi-head self-attention module; and SW-MSA(·) represents a shifted window multi-head self-attention module operation, which improves the information interaction ability between windows through two different multi-head self-attention mechanisms, thereby improving the model generalization performance.

[0123] wherein, the specific content of the local-non-local attention information enhancement module for extracting features is that: the input features respectively pass through global average pooling, depth separable convolution, and Sigmoid operation to obtain a local information enhancement map, and then the local information enhancement map is multiplied by the original input features to obtain a local information enhancement feature; secondly, the input features pass through layer normalization operation and position attention module to obtain a non-local information enhancement feature, and then the local information enhancement feature is added to the non-local information enhancement feature to obtain a local-non-local attention information enhancement feature. This module improves the model anti-interference ability and enhances the detail information extraction ability by introducing a local information induction bias to supplement the learning of non-local information features.

[0124] S306, the second, third and fourth Transformer Layers repeat the content of step S305, wherein the third Transformer Layer needs to be repeated three times before being input into the fourth Transformer Layer to output the global information features of the infrared and visible light.

[0125] S307, the features extracted by each feature extraction block of the dual encoder are added to the infrared and visible light global information features and the infrared and visible light local detail features corresponding to the modal to obtain four groups of pre-fused infrared features and visible light features.

[0126] S4, the features in step S3 are mapped to the original resolution by using the decoder, and the image features are gradually fused in a bottom-up manner to obtain a fused image, and the specific content is:

[0127] S401, as Figure 7As shown, after the fourth group of pre-fused infrared features and visible light features are spliced, the third group of reconstruction features is obtained by splicing the third group of pre-fused infrared features and visible light features in the channel dimension after upsampling operation, the second group of reconstruction features is obtained by splicing the second group of pre-fused infrared features and visible light features in the channel dimension after upsampling operation on the third group of reconstruction features, and the reconstruction features are obtained by splicing the third group of pre-fused infrared features and visible light features in the channel dimension after upsampling operation on the second group of reconstruction features.

[0128] S402, the reconstruction features in step S401 are finally obtained after upsampling operation, padding operation, 4x4 convolution operation, and Tanh operation.

[0129] Further, to verify the image fusion effect of the infrared and visible light image fusion model combined with the dual encoder of the Transformer and the CNN trained by the above method, the trained infrared and visible light image fusion model combined with the dual encoder of the Transformer and the CNN is verified in the embodiment.

[0130] Specifically, in the test phase, 20 groups of images except the training set are selected from the TNO dataset for test verification, 7 typical methods including DenseFuse, U2Fusion, RFN-Nest, SEDRFuse, IFCNN, GANMcC, and CSF are selected for comparison, in addition, 6 indexes including information entropy (EN), average gradient (AG), spatial frequency (SF), mutual information (MI), standard deviation (SD), and visual fidelity (VIF) are used as quantitative evaluation indexes, and the verification result includes qualitative evaluation and quantitative evaluation.

[0131] Qualitative evaluation: Figure 8 、 9 , 10 gives the qualitative comparison result of three groups of typical images (sandpath, Kaptein_1123, and lake). By comparison, it can be found that the fusion method of the application has three advantages. First, for typical infrared targets, such as Figure 8 、 9 in the application, the fusion result has higher luminance thermal radiation information and clearer edge profile than other methods. Second, the fusion result can retain the texture details and background information in the visible light image. For example Figure 8 the brightness of the tree and the sky background, Figure 10The fusion method of the present application can retain clearer detailed information and more obvious background information compared with other methods. Finally, the present application can capture the prominent target of the near-infrared image while retaining the texture details and background features in the visible light image to the maximum extent, and the fused image is clearer and more natural. Compared with the source image and other fusion results, the method of the present application can better retain the prominent target features and rich scene detail information, and the fused image method of the present application can more fully extract fine-grained information, has high image contrast, and is more in line with the human visual effect.

[0132] Quantitative evaluation: Table 1 shows the objective comparison results of 20 groups of images of the TNO data set. The optimal average and the suboptimal average are marked with bold and underlined respectively. It can be seen that the present application achieves the optimal average of EN, MI, SD, VIF and SF, and the AG index is the suboptimal value. The objective experiment shows that the method of the present application has better fusion performance than other methods. The fusion method of the present application achieves the optimal EN and MI, which shows that the fused image of the present application efficiently acquires rich source image information from the source image and effectively reconstructs in the fused image. The maximum SD shows that the fused image of the present application has more dispersed pixels, which shows that the hybrid transformer constructed by the present application can learn image features from local and non-local aspects, extract image detail information, and effectively improve the model anti-interference ability. The maximum VIF shows that the fused image of the present application has better visual effect and more realistic human eye feeling, which is because the fusion method of the present application adopts the Transformer embedded into the CNN to form a double-encoder structure, which can extract global information and local information, and can adaptively fuse different modal features. The SF of the fused image method of the present application is also optimal, which shows that the reversible neural network constructed by the present application can extract high-frequency semantic information from images of different modalities to a certain extent, and prevent information loss caused by mutual generation of input and output features.

[0133] Table 1 Quantitative comparison results of 20 groups of images of the TNO data set

[0134]

[0135]

[0136] The above only describes exemplary embodiments of the present application, and does not limit the patent protection scope of the present application. Any equivalent structure or equivalent flow transformation using the content of the specification and drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An infrared and visible image fusion method combining a dual encoder of a Transformer and a CNN, characterized in that, The application relates to an infrared and visible light image fusion model combining a Transformer and a CNN double encoder based on a U-Net framework. S1. An infrared and visible light image fusion model combining a Transformer and a CNN double encoder is constructed based on a U-Net framework, wherein the model is composed of a double encoder and a decoder; specifically, S101. A double encoder is constructed: the double encoder comprises a local detail feature extraction encoder and a global information extraction encoder; The local detail feature extraction encoder comprises convolution layers and four feature extraction blocks connected in sequence, the convolution layers are composed of convolution with a convolution kernel of 4*4, a step of 2 and padding of 1, and are connected in series with batch normalization and LeakyReLU, the convolution layers are used for downsampling operation and adjusting the channel number and size of an input image; The first and second feature extraction blocks are composed of residual modules, the third and fourth feature extraction blocks are connected in series with residual modules and reversible neural networks, the channel numbers of inputs of the four feature extraction blocks are 16, 32, 64 and 128 respectively, and the channel numbers of outputs are 32, 64, 128 and 256 respectively; The global information extraction encoder comprises four Transformer Layers connected in series, the channel numbers of inputs are 16, 32, 64 and 128 respectively, and the channel numbers of outputs are 32, 64, 128 and 256 respectively; S102. A decoder is constructed: the decoder comprises four feature fusion modules and a reconstruction module; each feature fusion module comprises upsampling operation, splicing operation, convolution with a convolution kernel of 4*4, a step of 2 and padding of 1 connected in sequence, BatchNorm operation and an activation function ReLU; The reconstruction module comprises upsampling operation, convolution with a convolution kernel of 4*4, padding of 1, padding operation and a hyperbolic tangent function connected in sequence; S103. The double encoder extracts features step by step in a top-down manner, the decoder fuses and reconstructs the features extracted by the double encoder step by step in a bottom-up manner to construct the infrared and visible light image fusion model combining the Transformer and the CNN double encoder; S2. A data set is selected, a training set is obtained after processing, and the infrared and visible light image fusion model combining the Transformer and the CNN double encoder in step S1 is trained by using the training set; S3. An infrared image and a visible light image are input into the infrared and visible light image fusion model combining the Transformer and the CNN double encoder which has been trained, infrared features and visible light features of the infrared image and the visible light image pre-fused by the double encoder are extracted, and feature representation is performed in a top-down manner; S4. The feature representation in step S3 is mapped to the original resolution by using the decoder, image features are fused step by step in a bottom-up manner, and a fused image is obtained.

2. The method of claim 1, wherein the method is characterized in that, The residual module comprises a main path and a residual path, the main path comprises convolution with a convolution kernel of 1*1, BN, a LeakyReLU activation function, convolution with a convolution kernel of 3*3 and convolution with a convolution kernel of 4*4; the residual path comprises DSConv and convolution with a convolution kernel of 4*4. The reversible neural network comprises three reversible residual modules, which comprise a convolution with a convolution kernel of 1*1, a depth separable convolution with a convolution kernel of 3*3, and a ReLU6 activation function.

3. The method of claim 1, wherein the method is characterized in that, The first, second and fourth Transformer Layers comprise two layers of mixed attention transformer structures, wherein the first layer of the first Transformer Layer comprises a patch embedding operation, a layer normalization operation, a W-MSA module, a local-non-local attention information enhancement module, a layer normalization operation and a feedforward neural network connected in series, and the second layer comprises a layer normalization operation, a SW-MSA module, a local-non-local attention information enhancement module, a layer normalization operation and a feedforward neural network connected in series; The first layer of the second and fourth Transformer Layers comprises a patch merging operation, a layer normalization operation, a W-MSA module, a local-non-local attention information enhancement module, a layer normalization operation and a feedforward neural network connected in series, and the second layer comprises a layer normalization operation, a SW-MSA module, a local-non-local attention information enhancement module, a layer normalization operation and a feedforward neural network connected in series; The third Transformer Layer comprises six layers of mixed attention transformer structures, which are stacked three times from the two layers of mixed attention transformer structures in the second Transformer Layer.

4. The method of claim 3, wherein the method further comprises: The local-non-local attention information enhancement module comprises a local information enhancement module and a non-local information enhancement module; wherein the local information enhancement module comprises a concatenation operation, a global average pooling, a depth separable convolution, a Sigmoid function, an element-wise multiplication operation and an element-wise multiplication operation; and the non-local information enhancement module comprises a LayerNorm operation and a position attention module.

5. The method of claim 1, wherein the method is characterized in that, In step S2, the infrared and visible image fusion model combining the Transformer and CNN dual encoders comprises the following sub-steps: S201, selecting m pairs of images from the TNO dataset as a dataset, converting the image gray value to [-1, 1], cropping the image using a h*w window, and setting the step size as s, finally obtaining n pairs of image blocks as a training set, wherein h represents the height of the image, and w represents the width of the image; S202, using a structural similarity loss and a contrast loss to supervise the training of the infrared and visible image fusion model combining the Transformer and CNN dual encoders, and obtaining optimal network parameters, and the specific formula is: L total = λ1L SSIM + λ2L patchNCE wherein L total represents a loss function, λ1, λ2 represent hyperparameters, L SSIM represents a structural similarity loss, L patchNCE represents a contrastive loss; S203、According to the training set and the loss function L total The initial infrared and visible light image fusion model combining the Transformer and the CNN double encoder is trained, an Adam optimizer is used in the training process to update the network model parameters, and the trained infrared and visible light image fusion model combining the Transformer and the CNN double encoder is obtained.

6. The method of claim 5, wherein the method further comprises: In step S3, the feature extraction using the dual encoders comprises the following sub-steps: S301, processing the input image according to the operation content of step S201; S302, inputting the processed infrared image and visible light image in step S301 into the dual encoder, and adjusting the channel number and size of the image through the down-sampling operation of the convolution layer; S303, input the image processed in step S302 into four feature extraction blocks, sequentially pass through 1*1 convolution, BN, LeakyReLU activation function, convolution kernel 3*3 convolution, BN and convolution kernel 4*4 convolution operation to obtain the convolution features learned in the main path, and then pass through DSConv and convolution kernel 4*4 convolution operation to obtain the supplementary information features, add the convolution features and the supplementary information features, and then pass through LeakyReLU to obtain the information enhancement features; S304: Input the information enhancement feature obtained in step S303 into the reversible neural network. Divided equally in the channel dimension and Two parts, features Features obtained after the first reversible residual module and features Add element by element to get features feature After the second reversible residual module and the feature Multiply element by element to get features feature After the third reversible residual module, the features are obtained and with the feature Add element by element to get features The features and features The output features are concatenated in the channel dimension The specific formula is: wherein C represents the complete number of channels, c represents half of the complete number of channels, [1:c] represents the current number of feature channels being 1 to c, [c+1:C] represents the current number of feature channels being c+1 to C, S represents a CNN feature extraction encoder, I represents an arbitrary mapping symbol, k represents the kth reversible network layer number, R n (n = 1, 2, 3) represents a reversible residual module, and exp represents the exponential e. Finally output the infrared local detail features and the visible light local detail features; S305、in the first Transform Layer, the features of the infrared image and the visible light image are respectively subjected to patch embedding operation, then the features are standardized through layer normalization operation, and then the features are input into the W-MSA module, the image is blocked according to the set window size, the self-attention map is obtained, then the local-non-local attention information enhancement module is passed through, the local-non-local attention information enhanced feature is obtained, and the enhanced features are added element by element to obtain the attention feature F l , after the layer normalization operation, the feature is obtained through the FFN module, the feature is added to the attention feature to obtain the output feature F l+1 of the first layer HAT structure. The output feature F l+1 The input second layer HAT structure is standardized by a layer normalization operation after a patch embedding operation. The standardized feature is input into an SW-MSA module. The SW-MSA module blocks the image according to a set window size to obtain a self-attention graph. The self-attention graph is input into a local-non-local attention information enhancement module to obtain a local-non-local attention information enhancement feature. The local-non-local attention information enhancement features are added element by element to obtain an attention feature F l+1 After the attention feature is subjected to a layer normalization operation, the feature obtained by the FFN module is added to the attention feature to obtain infrared global information features and visible light global information features of the first Transformer Layer. The specific formula is as follows: F l = LG(W - MSA(LN(F l-1 )) ) + F l-1 ; F1 l = FFN( LN( F l ) ) + F l ; F l+1 = LG(SW-MSA(LN(F1 l )))+F1 l ; F1 l+1 = FFN( LN( F l+1 ))+ F l+1 ; wherein F l-1 denotes the features of the infrared image and the visible light image, F l denotes the features of the first layer HAT output, F l+1 denotes the infrared global information feature and the visible light global information feature, LG(g) denotes a local-non-local information enhancement operation, LN(g) denotes a layer normalization operation, W-MSA(g) denotes a windowed multi-head self-attention module, and SW-MSA(g) denotes a shifted windowed multi-head self-attention module operation; S306, repeat the content of step S305 by the second, third and fourth Transformer Layers, wherein the third Transformer Layer needs to be repeated three times, and then input into the fourth Transformer Layer to output the global information features of infrared and visible light; S307, add each feature extraction block of the dual encoder with the infrared and visible light global information features and the infrared and visible light local detail features corresponding to the modal to obtain four groups of pre-fused infrared features and visible light features.

7. The method of claim 6, wherein the method further comprises: In step S305, the specific content of the local-non-local attention information enhancement module for extracting features is as follows: The input features pass through global average pooling, depth separable convolution and Sigmoid operation to obtain a local information enhancement map, and then pass through element-wise multiplication operation with the original input features to obtain local information enhancement features; The input features pass through layer normalization operation and position attention module to obtain non-local information enhancement features, and then pass through element-wise addition with the local information enhancement features to obtain local-non-local attention information enhancement features.

8. The method of claim 1, wherein the method is characterized in that, In step S4, the fused image obtained by the decoder includes the following sub-steps: S401, after the fourth group of pre-fused infrared features and visible light features are spliced, pass through upsampling operation, and then spliced with the third group of pre-fused infrared features and visible light features in the channel dimension to obtain the third group of reconstruction features, pass the third group of reconstruction features through upsampling operation, and then spliced with the second group of pre-fused infrared features and visible light features in the channel dimension to obtain the second group of reconstruction features, pass the second group of reconstruction features through upsampling operation, and then spliced with the third group of pre-fused infrared features and visible light features in the channel dimension to obtain the reconstruction features; S402, after the reconstruction features in step S401 pass through upsampling operation, padding operation, 4*4 convolution operation and Tanh operation, the final reconstructed fused image is obtained.

Images