An Infrared and Visible Image Fusion Method Based on End-to-End Attention Network

Abstract

The present invention provides an infrared and visible light image fusion method based on an end-to-end attention network, including: preprocessing the infrared and visible light images; constructing an end-to-end attention network; wherein, the attention network includes: an autoencoder network and a channel-spatial dual attention fusion layer, and the autoencoder network includes an encoder-decoder with skip connections; fusing the preprocessed infrared and visible light images based on the attention network. The present invention overcomes the repetitive defect of using the same method to extract features for different source images in traditional fusion methods, and at the same time reduces the limitation of its manually designed fusion strategy, and finally generates a single fusion image containing the feature information of multiple source images. The present invention can be applied to multiple fields such as remote sensing, medical diagnosis, monitoring, and target tracking.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to an infrared and visible light image fusion method based on an end-to-end attention network. Background Art

[0002] Image fusion is an image processing technology for image information enhancement. Due to the theoretical and technical limitations of hardware devices, a single sensor cannot effectively and comprehensively describe scene information under specific shooting settings. For example, visible light images contain more detailed texture information, while infrared images contain more amplitude information. Therefore, image fusion is to combine the complementary information between different source images of the same scene to generate a single image with richer information for application in many fields, such as photographic visualization, target tracking, medical diagnosis, and remote sensing monitoring.

[0003] Generally speaking, image fusion algorithms can be divided into the following two types: traditional methods and deep learning methods. Early image fusion methods all used mathematical transformation methods to generate activity level maps and designed fusion rules in the spatial domain and transform domain. Representative traditional image fusion methods include: multi-scale transform-based methods, sparse representation-based methods, subspace-based methods, saliency-based methods, and total variance-based methods. On the one hand, these traditional fusion algorithms use the same feature extraction and reconstruction methods for different source images without considering the feature differences between different source images, resulting in poor fusion effects. On the other hand, traditional fusion algorithms are designed by humans with fusion criteria, which are too simple, the fusion performance is severely limited, and they can only be applied to specific fusion tasks. In recent years, with the continuous development of deep learning technology, its application in image fusion has become more and more extensive. Currently, deep learning-based fusion algorithms mainly include three categories. One category is the generative adversarial network based on GAN, one category is the network based on the autoencoder (AE), and the last category is based on the conventional convolutional neural network (CNN). First, deep learning-based methods can use different network branches to achieve differential feature extraction, thereby obtaining more targeted features. Second, deep learning-based methods can design their own loss functions and update network parameters through backpropagation of gradients to obtain a more reasonable feature fusion strategy, thereby achieving adaptive feature fusion. Thanks to these advantages, deep learning has promoted great progress in image fusion and obtained performance far exceeding traditional methods.

[0004] Although deep learning has achieved satisfactory results in the field of image fusion, there are still some deficiencies: (1) These deep learning network architectures do not fully consider the intermediate feature layers and only design the loss function starting from the final fused image and the source images. (2) Most fusion algorithms only adopt deep learning models in the feature extraction and feature reconstruction stages, and traditional methods such as adding feature maps, taking the maximum value, and taking the mean are used for feature fusion. (3) Some deep learning models adopt a two-stage training method, which is time-consuming and difficult to train. Summary of the Invention

[0005] To solve the above technical problems, the present invention proposes an infrared and visible light image fusion method based on an end-to-end attention network to improve the image fusion effect.

[0006] To achieve the above object, the present invention provides an infrared and visible light image fusion method based on an end-to-end attention network, including:

[0007] Preprocess the infrared image and the visible light image;

[0008] Construct an end-to-end attention network; wherein, the attention network includes: an autoencoder network and a channel-spatial dual attention fusion layer, and the autoencoder network includes an encoder-decoder with skip connections.

[0009] Fuse the preprocessed infrared image and visible light image based on the attention network.

[0010] Optionally, preprocessing the infrared image and the visible light image includes: converting the infrared image and the visible light image into grayscale images and performing central cropping.

[0011] Optionally, the encoder in the autoencoder network is used to extract multi-scale deep semantic features of the preprocessed image and output an infrared feature map and a visible light feature map; the decoder in the autoencoder network is used to reconstruct the final fused image according to the infrared feature map and the visible light feature map.

[0012] Optionally, the encoder includes a plurality of max-pooling downsampling layers and a plurality of convolutional blocks. The input channel number of the encoder is set based on a first preset number, and the output channel number of the encoder is set based on a second preset number. Each convolutional block of the encoder contains BatchNorm regularization and a RELU activation function;

[0013] The decoder includes a plurality of upsampling layers and a plurality of convolutional blocks. The input channel number of the decoder is set based on a third preset number, and the output channel number of the decoder is set based on a fourth preset number. Each convolutional block of the decoder contains BatchNorm regularization and a RELU activation function.

[0014] Optionally, adding the skip connection to the auto-encoding network includes:

[0015] Connecting the input of each max-pooling layer in the encoder to the output of the up-sampling layer in the decoder, adding a denseblock to the connection path, using different numbers of convolutional blocks to form the denseblock in different connection paths, and setting the output channels of the convolutional blocks based on a fifth preset number.

[0016] Optionally, the channel-spatial dual-attention fusion layer includes: a channel attention module and a spatial attention module;

[0017] In the channel-spatial dual-attention fusion layer, connecting the infrared feature map and the visible light feature map in the channel dimension, respectively inputting the spliced image into the spatial attention module and the channel attention module to obtain a spatial weight map and a channel weight map, multiplying the spatial weight map and the channel weight map by the infrared feature map and the visible light feature map, and then adding the spatial and channel attention fusion features to obtain an intermediate fusion image. On the one hand, an intermediate fusion image is obtained through the intermediate fusion layer, and on the other hand, the intermediate fusion layer can enable the network to focus on the places it needs to focus more. Then, the intermediate fusion image is sent into the decoder of the auto-encoding network to obtain the final fusion image.

[0018] Optionally, constructing the attention network further includes: setting a loss function;

[0019] Setting the loss function includes: adding an SSIM structural similarity metric function, introducing a gradient operator, introducing L2 regularization, designing an objective feature enhancement loss function, and finally performing weighted calculation on each loss.

[0020] Optionally, the SSIM structural similarity metric function includes: a luminance function, a contrast function, and a structure comparison function;

[0021] The luminance function is:

[0022]

[0023] where μ , 2 ,

[0023] , y , , ,

[0024] , i , x , , ,

[0022] ,

[0021] and μ y respectively represent the average luminance of two images, N is the number of pixel points of the picture, X i is the pixel value size, x and y respectively represent two different images, C1 is used to prevent the denominator from being 0, C1 = (k1 * L) 2 , k1 takes 0.01, and L takes 255;

[0024] The contrast function is:

[0025]

[0026] Among them, σ x and σ y respectively represent the standard deviations of two images. C2 is used to prevent the denominator from being 0, and C2 = (k2 * L) 2 , where k2 takes 0.03 and L takes 255;

[0027] The structure comparison function is:

[0028]

[0029] Among them, σ xy represents the covariance of two images. C3 is used to prevent the denominator from being 0, and C3 = C2 / 2;

[0030] The SSIM structural similarity metric function is:

[0031]

[0032] Optionally, the gradient operator is:

[0033] Among them, V is the visible light source image, is the final fused image, is the gradient operator, and || ||1 represents the L1 norm;

[0034] The L2 regularization is: Among them, X is the set unknown, representing the visible light source image or the infrared source image, and || ||2 represents the L2 norm;

[0035] The target feature enhancement loss function is:

[0036]

[0037] Among them, M represents the fusion process at different scales, w e is the weight at different scales, is the fusion result of the feature map at the m-th layer, and are respectively the visible light feature layer and the infrared feature layer at the m-th layer, w vi is the weight of the visible light feature layer, w ir is the weight of the infrared feature layer, and F is the Frobenius norm.

[0038] Optionally, the final loss function is:

[0039]

[0040] Among them, I is the infrared source image, and V is the visible light source image. is the final fused image, and L1 is α1, α2, and α3 are the weights of each loss function respectively.

[0041] Compared with the prior art, the present invention has the following advantages and technical effects:

[0042] 1. The present invention uses an encoder-decoder network structure. In the encoding stage, multi-scale deep features of the input image are fully extracted, and in the decoding stage, the multi-scale deep features are effectively reconstructed. Then, skip connections are further introduced into the autoencoder network to effectively slow down the vanishing gradient and reuse the multi-scale feature layers, which can effectively enhance the network's ability to extract and reconstruct features. Considering that the semantic information contained in features at different scales is different and should not be directly connected, different numbers of convolutional blocks are selected between different connection layers to eliminate and balance this difference.

[0043] 2. The present invention uses a channel-spatial dual attention neural network fusion structure. Different from the previous manually designed fusion strategies, the present invention can effectively preserve the amplitude information of the infrared image and the detailed texture information of the visible light image.

[0044] 3. The present invention designs a new loss function, introduces the SSIM structural similarity function, L2 regularization, gradient operator, and target feature enhancement loss function, which can effectively extract the significant and detailed features of the source image. Adopting an end-to-end network structure and abandoning the manually designed intermediate fusion layer and two-stage training strategy makes the training faster and the fusion result more effective. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The drawings constituting a part of this application are used to provide a further understanding of this application. The schematic embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation to this application. In the drawings:

[0046] Figure 1 is a schematic flow diagram of an infrared and visible light image fusion method based on an end-to-end attention network according to an embodiment of the present invention;

[0047] Figure 2 is a schematic diagram of the attention network structure according to an embodiment of the present invention;

[0048] Figure 3 is a schematic diagram of the spatial-channel dual attention fusion layer structure according to an embodiment of the present invention;

[0049] Figure 4 is an infrared image according to an embodiment of the present invention;

[0050] Figure 5 A visible light image according to an embodiment of the present invention. Detailed implementation manners

[0051] It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments may be combined with each other. The present application will be described in detail below with reference to the drawings and in conjunction with the embodiments.

[0052] It should be noted that the steps shown in the flowchart of the drawings may be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than here.

[0053] Embodiment

[0054] As Figure 1 shown, this embodiment provides an infrared and visible light image fusion method based on an end-to-end attention network, including:

[0055] Preprocess the infrared image and the visible light image;

[0056] Construct an end-to-end attention network; wherein, the attention network includes: an autoencoder network and a channel-spatial dual attention fusion layer, and the autoencoder network includes an encoder-decoder with skip connections;

[0057] Fuse the preprocessed infrared image and visible light image based on the attention network.

[0058] Further, preprocessing the infrared image and the visible light image includes: converting the infrared and visible light images into grayscale images and performing central cropping.

[0059] Further, the autoencoder network includes: an encoder and a decoder;

[0060] The encoder is used to extract multi-scale deep semantic features of the preprocessed image and output an infrared feature map and a visible light feature map; the decoder is used to reconstruct the final fused image according to the infrared feature map and the visible light feature map.

[0061] Further, the encoder is composed of a plurality of max-pooling downsampling layers and a plurality of convolutional blocks, the input channel number of the encoder is set based on a first preset number, the output channel number of the encoder is set based on a second preset number, and each convolutional block of the encoder is followed by BatchNorm regularization and a RELU activation function;

[0062] The decoder consists of a number of upsampling layers and a number of convolutional blocks. The input channel number of the decoder is set based on a third preset number, and the output channel number of the decoder is set based on a fourth preset number. After each convolutional block of the decoder, there are BatchNorm regularization and ReLU activation functions.

[0063] Further, adding the skip connection to the autoencoder network includes:

[0064] Connect the input of each max-pooling layer in the encoder to the output of the upsampling layer in the decoder. Add a denseblock to the connection path. In different connection paths, use different numbers of convolutional blocks to form the denseblock, and set the output channels of the convolutional blocks based on a fifth preset number.

[0065] Further, the channel-spatial dual attention fusion layer includes: a channel attention module and a spatial attention module;

[0066] In the channel-spatial dual attention fusion layer, connect the infrared feature map and the visible light feature map in the channel dimension. The spliced image is respectively input into the spatial attention module and the channel attention module to obtain a spatial weight map and a channel weight map. Multiply the spatial weight map and the channel weight map with the infrared feature map and the visible light feature map, and then add the spatial and channel attention fusion features to obtain an intermediate fusion image. On the one hand, an intermediate fusion image is obtained through the intermediate fusion layer, and on the other hand, the intermediate fusion layer can make the network focus on the places it needs to focus on more. Then send the intermediate fusion image into the decoder of the autoencoder network to obtain the final fusion image.

[0067] Further, constructing the attention network also includes: setting a loss function;

[0068] Setting the loss function includes: adding an SSIM structural similarity metric function, introducing a gradient operator, introducing L2 regularization, designing a target feature enhancement loss function, and finally performing weighted calculation on each loss.

[0069] In this embodiment, an infrared and visible light image fusion method is proposed. The purpose is to fuse complementary and useful information in different modal images to more comprehensively describe the imaging scene. It is divided into the following steps: (1) Construct an autoencoder network with an encoder-decoder to extract the deep semantic information of the input image and reconstruct the fused image; (2) Add skip connections to the autoencoder network, and introduce a denseblock in the skip connections to narrow the difference in the richness of semantic information between the connection layers; (3) Construct a dual-attention fusion layer for channels and space to further retain the texture information of the visible light image and the amplitude information of the infrared image; (4) Design an appropriate loss function, select relevant datasets to train and test the performance of the fusion network. The present invention overcomes the repetitive defect of using the same method to extract features for different source images in traditional fusion methods, and at the same time reduces the limitation of its manually designed fusion strategy. Finally, a single fused image containing the feature information of multiple source images is generated. The invention can be applied to multiple fields such as remote sensing, medical diagnosis, monitoring, and target tracking. The specific implementation steps are as follows:

[0070] Step 1: Construct an autoencoder network with an encoder-decoder. The encoder is used to extract the deep features of the input image, and the decoder network is used to reconstruct the extracted deep features into the final fused image.

[0071] The encoder network consists of 3 max-pooling downsampling layers and 9 ordinary convolutional blocks, and the decoder network consists of 3 upsampling layers and 7 ordinary convolutional blocks. They are connected layer by layer, as Figure 2 shown. In the first convolutional block of the autoencoder network, a 1*1 convolutional kernel is used, and ReflectionPad is used to prevent the appearance of edge artifacts in the fused image. The input channel number is set to 1, and the output channel number is set to 16. The remaining ordinary convolutional modules all use 3*3 convolutional kernels, the stride is set to 1, and 0 is used for padding without changing the image resolution. Among them, the input channel numbers of the encoder are set to 16, 64, 64, 128, 128, 256, 256, 256 respectively, and the output channel numbers are set to 64, 64, 128, 128, 256, 256, 256, 256 respectively. Three max-pooling downsampling layers are set in the encoder stage, and the stride is set to 2. In the decoder stage, bilinear interpolation is used to upsample the size of the feature map by two times. The input channel numbers of the convolutional blocks in the decoding stage are 512, 256, 256, 128, 128, 64 respectively, and the output channel numbers are 256, 128, 128, 64, 64, 64 respectively. After each of the above-mentioned convolutional blocks, there are BatchNorm regularization and RELU activation functions.

[0072] Step 2: Add skip connections to the auto-encoder network to slow down the vanishing gradient and further compensate for the information loss caused by the upsampling and downsampling processes, as well as slow down the vanishing gradient problem of deep neural networks. Considering the semantic information differences between connection layers, they should not be directly connected. Therefore, different convolutional blocks are used on different connection layers to implement skip connections.

[0073] As Figure 2 described, the input of the first maxpooling layer is connected to the output of the third upsampling layer through a skip connection in the channel dimension. Specifically, 4 convolutional blocks are used. The input channels of each convolutional block are set to 64, 64, 128, and 192 respectively, and the output channels are all set to 64. The specific connection method is as follows: The four convolutional blocks are respectively set as A1, A2, A3, and A4. Then the output of A1 is used as the input of A2, the channel connection of A1 and A2 is used as the input of A3, and the channel connection of A1, A2, and A3 is used as the input of A4. Each of the above convolutional blocks is set with a 3*3 convolutional kernel, filled with 0, and the padding is set to 1 without changing the image resolution. Then it is followed by BatchNorm regularization and the RELU activation function. The input of the second max-pooling layer is connected to the output of the second upsampling layer in the same way as above. However, to balance the semantic information differences between the deep and shallow layers, we use three convolutional blocks. The input of the third max-pooling layer is connected to the output of the third upsampling layer in the same way as above, and 2 convolutional blocks are selected.

[0074] Step 3: Construct a channel-spatial dual attention fusion layer (as Figure 3 shown), extract the amplitude and detailed texture information of infrared and visible light images respectively. First, the feature information of the two source images extracted by the encoder is concatenated in the channel dimension, and then sent to the channel and spatial attention layers respectively to obtain the corresponding feature maps, and further added to obtain the intermediate feature layer fusion image.

[0075] Calculate the weights of channel attention: Feed the spliced image (S1) into the Global Average Pooling layer, and then successively pass through two fully connected layers. After the first fully connected layer, the h-swish activation function is used, and the number of output channels is 1 / 4 of the number of channels of S1. After the second convolutional layer, the sigmoid activation function is used to obtain weights in the range of 0-1, and the number of output channels is equal to the number of channels of S1. Finally, multiply the obtained weights by the visible light feature layer to further retain the detailed information of the visible light image. Calculate the weights of spatial attention: Feed the spliced image into the Average Pooling layer and the Max Pooling layer respectively, perform max and average sampling in the channel dimension without changing the image resolution, splice the two output feature layers in the channel dimension, then feed them into a 7*7 convolutional layer with 0 padding, padding set to 3, without changing the image resolution, followed by a sigmoid activation function to obtain a weight distribution map in the range of 0-1, and multiply it by the infrared feature layer to further retain the amplitude information of the infrared source image. Finally, add the feature maps obtained from the two attention structures to get the intermediate fusion image.

[0076] Step 4: Design the loss function: We add the SSIM structural similarity metric function. To further retain the detailed texture information of the visible light, we also introduce the gradient operator, introduce L2 regularization, and finally design the target feature enhancement loss function.

[0077] The above SSIM structural similarity function can better reflect the human visual judgment of the similarity between two images. This similarity function consists of three aspects, namely the brightness, contrast, and structure comparison functions of the images. Among them, the brightness similarity is:

[0078]

[0079] μ x and μ y represent the average brightness of the two images respectively. N is the number of pixel points in the picture, X i is the pixel value size, x and y represent two different images respectively, C1 is used to prevent the denominator from being 0, C1=(k1*L) 2 , k1 takes 0.01, L takes 255; Next is the picture contrast, indicating the severity of the brightness change in the image. The contrast similarity function is set as:

[0080]

[0081] Among them, σ x and σ yRepresents the standard deviation of the two images respectively, C2 is used to prevent the denominator from being 0, C2=(k2*L) 2 ,k2 takes 0.03, L takes 255;

[0082] The structure comparison function is:

[0083]

[0084] The covariance Finally got Among them, C1, C2, and C3 are all to prevent the denominator from being 0. C3 = C2 / 2. The value range of SSIM is -1 to 1. The larger the SSIM value, the higher the similarity of the images. Therefore, the loss function of the final SSIM measurement is L SSIM =1-SSIM.

[0085] The above gradient operator Where V represents the visible light source image, represents the final fused image, is the gradient operator, || ||1 represents the L1 norm; since the visible image has rich texture information, the reconstruction of the visible image is regularized by gradient penalty to ensure texture consistency.

[0086] The above L2 regularization is set to It mainly measures the intensity consistency between the source image and the fused image. X is the set unknown number, representing the visible light grayscale image and the infrared grayscale image, and || ||2 represents the L2 norm.

[0087] The above target feature enhancement loss function is set as

[0088]

[0089] w vi is the weight of the visible light feature layer, w ir is the weight of the infrared feature layer, and F is the Frobenius norm. Because infrared images have more significant target features than visible light images, we design this loss function L2 to constrain the depth features of the fused image, thereby retaining significant features. M is set to 4, representing the fusion process at different scales, w e Represents the weights at different scales. Due to the difference in magnitude at different scales, we will w e Set to [1,10,100,1000] respectively, Represents the fusion result of the m-th layer feature map, and Represent the visible light and infrared feature layers of the mth layer respectively. Since this loss function is mainly used to preserve the significant features of the infrared image, wvi Set smaller than w ir They are 3 and 6 respectively.

[0090] The above calculates the weighted sum of each loss function, where where The final loss function is

[0091] where α1, α2, and α3 are the weight ratios of each loss function respectively, α1, α2, and α3 are set to 2, 2, 10, λ is set to 5, w e is set to [1, 10, 100, 1000], w vi and w ir are set to 3 and 6 respectively.

[0092] Step 5: Select the dataset: The experiment was conducted on three datasets, including TNO, NIR, and FLIR.

[0093] 180 pairs of images were randomly selected from the FLIR dataset as training samples, and one type of infrared and visible light images is shown, for example Figure 4 Figure 5 Before training, all images were converted to grayscale images. At the same time, they were center-cropped to 128×128 pixels. Then, multiple pairs of infrared and visible light images were fed into the above-mentioned network for training, the loss was calculated according to the above-mentioned loss function, and then the network parameters were updated through backpropagation of gradients, where the number of training epochs was set to 120, the Adam optimizer was used, and the learning rate was set to 10 -3 , and the MultiStepLR learning rate adjustment strategy was adopted. Every 40 epochs, the learning rate was multiplied by 0.1. After training, the remaining FLIR data, TNO(40) dataset, and NIR country dataset were used to verify the fusion effect of the model.

[0094] The above is only a preferred specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed in the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An infrared and visible light image fusion method based on an end-to-end attention network, characterized in that Including: Preprocessing the infrared image and the visible light image; Constructing an end-to-end attention network; wherein, the attention network includes: an autoencoder network and a channel-spatial dual attention fusion layer, and the autoencoder network includes an encoder-decoder with skip connections; Fusing the preprocessed infrared image and visible light image based on the attention network; The encoder in the autoencoder network is used to extract multi-scale deep semantic features of the preprocessed image and output an infrared feature map and a visible light feature map; the decoder in the autoencoder network is used to reconstruct the infrared feature map and visible light feature map into a final fused image; The encoder includes a number of max-pooling downsampling layers and a number of convolutional blocks. The input channel number of the encoder is set based on a first preset number, and the output channel number of the encoder is set based on a second preset number. Each convolutional block in the encoder is followed by BatchNorm regularization and a RELU activation function; The decoder includes a number of upsampling layers and a number of convolutional blocks. The input channel number of the decoder is set based on a third preset number, and the output channel number of the decoder is set based on a fourth preset number. Each convolutional block in the decoder is followed by BatchNorm regularization and a RELU activation function; Adding the skip connection to the autoencoder network includes: Connecting the input of each max-pooling downsampling layer in the encoder to the output of the upsampling layer in the decoder, adding a denseblock on the connection path, and using different numbers of convolutional blocks to form the denseblock in different connection paths. The output channel of the convolutional block is set based on a fifth preset number; The channel-spatial dual attention fusion layer includes: a channel attention module and a spatial attention module; In the channel-spatial dual attention fusion layer, the infrared feature map and the visible light feature map are connected in the channel dimension, and the spliced image is respectively input into the spatial attention module and the channel attention module to obtain a spatial weight map and a channel weight map. The spatial weight map and the channel weight map are multiplied by the infrared feature map and the visible light feature map, and then the spatial and channel attention fusion features are added to obtain an intermediate fused image; Constructing the attention network further includes: setting a loss function; Setting the loss function includes: adding an SSIM structural similarity metric function, introducing a gradient operator, introducing L2 regularization, designing a target feature enhancement loss function, and finally performing weighted calculation on each loss.

2. The infrared and visible light image fusion method based on an end-to-end attention network according to claim 1, wherein Preprocessing the infrared image and the visible light image includes: converting the infrared image and the visible light image into grayscale images and performing central cropping.

3. The infrared and visible light image fusion method based on an end-to-end attention network according to claim 1, wherein The SSIM structural similarity metric function includes: a luminance function, a contrast function, and a structure comparison function; The luminance function is: Among them, μ x , μ y respectively represent the average brightness of two images, N is the number of pixel points of the picture, X i is the pixel value size, x and y respectively represent two different images, C1 is used to prevent the denominator from being 0, C1 = (k1 * L) 2 , k1 takes 0.01, and L takes 255; The contrast function is: Among them, σ x and σ y respectively represent the standard deviations of two images. C2 is used to prevent the denominator from being 0, and C2 = (k2 * L) 2 , where k2 is taken as 0.03 and L is taken as 255; The structure comparison function is: Among them, σ xy represents the covariance of two pictures, and C3 is used to prevent the denominator from being 0, where C3 = C2 / 2; The SSIM structural similarity metric function is:

4. The infrared and visible light image fusion method based on an end-to-end attention network according to claim 3, characterized in that The gradient operator is as follows: where V is the visible light source image, is the final fused image, is the gradient operator, and ||||1 represents the L1 norm; The L2 regularization is as follows: where X is the set unknown, representing the visible light source image or the infrared source image, and ||||2 represents the L2 norm; The target feature enhancement loss function is: Among them, M represents the fusion process at different scales, and w e is the weight at different scales, is the fusion result of the feature map at the m-th layer, and are the visible light feature layer and the infrared feature layer at the m-th layer respectively, and w vi is the weight of the visible light feature layer, and w ir is the weight of the infrared feature layer, and F is the Frobenius norm.

5. The infrared and visible light image fusion method based on an end-to-end attention network according to claim 4, wherein The loss function is: Wherein, I is the infrared source image, and V is the visible light source image. is the final fused image, and L1 is α1, α2, and α3 are the weights of each loss function respectively.

Images