An infrared and visible light image fusion algorithm based on an autoencoder

Abstract

The application discloses an infrared and visible light image fusion algorithm based on a self-encoder, and comprises the following steps: S1, an encoder network is constructed by combining a short cross-layer connection with an ECA attention mechanism; S2, registered infrared and visible light images are input into the encoder network, and the encoder and decoder network are trained to obtain an encoder meeting a condition; S3, feature fusion is performed on the input images by using the constructed fusion network, and the fusion network is trained to obtain a fusion network meeting a condition; and S4, the fused images are decoded and reconstructed by using the decoder.The evaluation index of the fused images is significantly improved, the target is clear, the details are prominent, the outline is obvious, and the images meet the human visual perception.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, and particularly relates to an infrared and visible light image fusion algorithm based on an autoencoder. BACKGROUND

[0002] The thermal radiation information of an infrared image is rich, can effectively highlight a significant target, and has strong anti-interference capability, but has the shortcomings of low contrast, poor texture details, and visual blurring. A visible light image contains rich structural information and has good visual effects, but is easily affected by factors such as weather and lighting conditions, and has poor anti-interference capability (Chen Chaoqi, Meng Xiangchao, Shao Feng, Fu Randed. An infrared and visible light image fusion method based on multi-scale low-rank decomposition [J]. Acta Optica Sinica, 2020, 40(11): 72-80). Due to the different characteristics of the two kinds of images, therefore, the infrared and visible light images are fused, and the complementarity of their information is fully utilized, which has wide application value.

[0003] In the past few decades, infrared and visible light image fusion technology has been continuously developed, and these fusion methods can be roughly divided into two categories: traditional methods and deep learning-based methods. Typical representatives of traditional methods include multi-scale transform-based methods (Xia Jingming, Chen Yiming, Chen Yicai, et al. Infrared and visible light image fusion based on sparse representation and NSCT-PCNN [J]. Optoelectronics and Control, 2018, 25(06): 1-6) and sparse representation-based methods (Li S T, Yang B, Hu J W. Performance comparison of different multi-resolution transforms for image fusion [J]. Information Fusion, 2010, 12(2): 74-84); the multi-scale transform-based method is to extract multi-scale features from the source image, then use a fusion strategy to fuse the features, and finally reconstruct the fusion image through the decoder. The sparse representation-based method is to first construct a dictionary, then obtain the representation coefficients through the dictionary, then obtain the fusion coefficients through a fusion strategy, and finally reconstruct the fusion image through the two kinds of coefficients. The traditional method is relatively simple, and the information loss of the fusion image is more serious compared with the source image.

[0004] With the maturity of deep learning technology, it has been introduced into the field of infrared and visible light image fusion, significantly improving the quality of fused images. Deep learning-based methods can be mainly divided into generative adversarial networks (GANs), autoencoders, and convolutional neural networks (CNNs). Recently, L. Tang (Tang LF, Yuan JT, Ma JY. Image fusion in the loop of high-level vision tasks: A semantic-aware real-time infrared and visible image fusion network[J]. Information Fusion, 2022, 82.28-42) et al. proposed the first image fusion framework combining high-level vision tasks from a new perspective, based on deep learning. This method was first proposed by Prabhakar et al. in 2017. Although the proposed fusion framework (DeepFuse) improved fusion performance, it was difficult to extract salient features and suffered from loss of detail information. Subsequently, some scholars incorporated dense blocks into DeepFuse, proposing a new fusion framework, DenseFuse. DenseFuse can extract salient features better, and the loss of detail information is greatly reduced. Since then, autoencoder-based methods have gradually matured. As the method continues to improve, its effectiveness gradually increases, but two drawbacks still exist:

[0005] 1) The target in the fused image is not prominent, details are lost, and the visual experience is poor;

[0006] 2) The fused image cannot balance infrared and visible light information well, and the image becomes gray and dark. Summary of the Invention

[0007] Purpose of the invention: The purpose of this invention is to provide an autoencoder-based infrared and visible light image fusion algorithm that provides good visual effects, clear targets, prominent details, and distinct outlines.

[0008] Technical solution: The infrared and visible light image fusion algorithm of the present invention includes the following steps:

[0009] S1 constructs an encoder network by combining short cross-layer connections with the ECA attention mechanism;

[0010] S2, input the registered infrared and visible light images into the encoder network, and train the encoder and decoder networks to obtain an encoder that meets the conditions;

[0011] S3, for the input image, feature fusion is performed through the constructed fusion network, and the fusion network is trained to obtain a fusion network that meets the conditions;

[0012] S4, the fused image is decoded by a decoder to reconstruct a picture.

[0013] Further, in step S1, the encoder is composed of a convolutional layer and an ECA attention mechanism, and the input image is feature extracted on four scales; the convolutional layer is connected with the ECA attention mechanism through a short cross-layer.

[0014] Further, in step S2, an auto-encoder loss function L auto is used to train the auto-encoder network, L auto is defined as follows,

[0015] L auto = L pixel + λL ssim

[0016] Wherein, L pixel represents the pixel loss between the input image and the output image, L ssim represents the structural similarity loss between the input image and the output image; λ is the weighting parameter between L pixel and L ssim .

[0017] L pixel loss calculation expression is as follows:

[0018]

[0019] Wherein, is the Frobenius norm;

[0020] L ssim loss calculation expression is as follows:

[0021] L ssim = 1-SSIM(Output,Input)

[0022] Wherein, SSIM(.) is the structural similarity measure.

[0023] Further, in step S3, the fusion network is composed of LReLU activation function and 3x3 convolutional layer, Tanh activation function and 1x1 convolutional layer, attention mechanism CBAM and 3x3 convolutional layer; the basic block composed of three 3x3 convolution and LReLU activation function is used to extract the features of the source image; the 5x5 convolutional layer and the CBAM attention mechanism are used to deeply extract the detail information of the source image, and the extracted detail feature information and the feature information extracted by the three basic blocks are spliced with each other; finally, the basic block composed of Tanh activation function and 1x1 convolutional layer is used for convolution and output.

[0024] Further, in step S3, the fusion strategy loss function L is used to train the fusion network CRN CRN defined as follows:

[0025] L CRN = alpha * L detail + L feature

[0026] Wherein, L detail and L feature represent the background detail preservation loss function and the target feature enhancement loss function respectively, alpha is a trade-off parameter; L detail The calculation expression is as follows:

[0027] L detail = 1-SSIM(O, I vi )

[0028] Wherein, SSIM(.) is the structural similarity measure, I vi represents the visible light image;

[0029] L feature The calculation expression is as follows:

[0030]

[0031] In the formula, m is the number of multi-scale deep features; w1 is a trade-off parameter vector for balancing the loss amplitude; is the fusion feature, w vi is the relative influence coefficient of the visible light feature , and w ir is the relative influence coefficient of the infrared feature .

[0032] Compared with the prior art, the present application has the following remarkable effects:

[0033] 1、The improved ECA attention mechanism is introduced into the encoder, the fusion strategy CRN is designed, and the CBAM attention mechanism is introduced, so that the problems of detail information loss and infrared and visible light information preservation balance are solved to a certain extent;

[0034] 2、The fusion strategy designed by the present application can better preserve the thermal radiation information in the infrared image and the texture details in the visible light image; compared with the existing five fusion algorithms on the public data set, the present application not only has good visual effect, but also the experimental results show that the image evaluation index after fusion is improved significantly, the target is clear, the details are prominent, the outline is obvious, and it conforms to human visual perception. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 ​A schematic diagram of the network overall structure of the present application;

[0036] Figure 2 A schematic diagram of the encoder structure of the present application;

[0037] Figure 3 A schematic diagram of the ECA attention mechanism of the present application;

[0038] Figure 4 A schematic diagram of the decoder structure of the present application;

[0039] Figure 5 A schematic diagram of the fusion network of the present application;

[0040] Figure 6 A schematic diagram of the CBAM attention mechanism of the present application;

[0041] Figure 7 A comparison diagram of the hedge fusion results under different algorithms;

[0042] Figure 8 A comparison diagram of the helicopter fusion results under different algorithms;

[0043] Figure 9 A comparison diagram of the vehicle and person fusion results under different algorithms;

[0044] Figure 10 A comparison diagram of the person fusion results under different algorithms;

[0045] Figure 11 A comparison diagram of the tent and person fusion results under different algorithms. DETAILED DESCRIPTION

[0046] The present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0047] The application proposes an infrared and visible light image fusion algorithm based on an autoencoder, introduces an improved efficient channel attention mechanism (ECA) in the encoder part, and introduces a hybrid attention mechanism module (Convolutional Block Attention Module, CBAM) in the fusion strategy part. In order to fully extract infrared and visible light features, a CRN fusion block is designed, which is composed of an LReLU activation function and a 3x3 convolution layer, a Tanh activation function and a 1x1 convolution layer, an attention mechanism CBAM and a 3x3 convolution layer. Finally, the decoder completes the decoding of the fused feature image, and finally reconstructs the fused image. Among them, the CRN network adopts a basic block composed of three 3x3 convolutions and LReLU activation functions to extract features from the source image, and experiments show that the three times of extraction effect and the training time cost are optimal, and the model activated by the LReLU activation function can better mine related features and fit feature information. In view of the problem that the detail information is not fully retained and the extraction speed is slow, the method of the application designs a 5x5 convolution layer and a CBAM attention mechanism to deeply extract the detail information of the source image, and the response speed is faster. At the same time, the extracted feature information and the feature information extracted by the three basic blocks are spliced with each other, so that the details can be retained while the fusion speed is improved. Finally, the basic block composed of a Tanh activation function and a 1x1 convolution layer is convolved and output.

[0048] The overall structure of the network of the application is shown in Figure 1 First, the registered infrared and visible light images are input into the encoder, and then the feature fusion is performed through the convolution residual network CRN, and after the fusion, the image is reconstructed through the decoder. In Figure 1 , I ir and I vis are input infrared and visible light images, and O f represents the output fused image.

[0049] The application adopts a two-stage training method, the first stage trains the encoding and decoding network, and the second stage trains the fusion network. In the training of the encoder and the decoder, the infrared image and the visible light image are registered and sent into the encoding and decoding network. The input image is first encoded and the feature is extracted by the encoder, and finally the image is decoded and reconstructed by the decoder. In the second stage, the parameter weight of the first stage encoding and decoding is directly used to train the fusion network. In order to improve the quality of the fused image and retain the detail features and thermal radiation information of the source image, the application designs four CRN fusion blocks to fuse the multi-scale deep features extracted by the encoder. CRN is simple, efficient and learnable. It can be directly fused into the autoencoder structure, reducing the trouble of manual design.

[0050] (I) Coding and decoding structure

[0051] The coding and decoding network mainly consists of an encoder and a decoder.

[0052] The structure of the encoder is shown in Figure 2 . The encoder consists of a convolutional layer and an ECA attention mechanism. The encoder extracts features of the input image at four scales to better obtain shallow features and deep features. Through the combination of short cross-layer connections and the ECA attention mechanism, the details of the source image can be fully read. In Figure 2 , for the convolutional layer, "k x k, (in, out)" means that the kernel size is k x k, the input channel is in, and the output channel is out. The ECA attention mechanism is introduced in the encoder, and the ECA attention mechanism is shown in Figure 3 . ECA is a parameter involved in a significant performance improvement. Through the ablation experiment, this module is important for improving the quality of the fused image.

[0053] The detailed structure of the decoder is shown in Figure 4 . Decoder Block (DB) represents a decoder convolutional block, which has two convolutional layers. In each row, these blocks are connected through short connections similar to the dense block architecture. In addition, the decoder structure also adopts cross-layer connections to retain more multi-scale deep features and details in the source image. The output of the network is the fused image after multi-scale feature reconstruction.

[0054] (II) Fusion network

[0055] The structure of the fusion network CRN is shown in Figure 5 . In the figure, I ir and I vis represent the input infrared image and the visible light image respectively, and O f represents the output image. The fusion network mainly consists of 3 x 3 convolutional layers and LReLU activation functions. The model after LReLU activation can better mine relevant features and fit feature information. After passing through the convolutional layer and the LReLU activation function, the attention mechanism CBAM is introduced. This attention mechanism can act on the channel and spatial dimensions at the same time, so the present application seamlessly integrates it into the fusion network and trains it together. Through the ablation experiment, this attention mechanism has a certain effect on improving the quality of the fused image. The detailed structure diagram of CBAM is shown in Figure 6 .

[0056] In order to preserve more details of the fused image, a 5x5 convolutional layer is used in the fusion strategy part to directly read the infrared and visible light image source information, which is spliced with the activation function and the features read by the convolutional layer. This operation improves the fusion speed and to some extent reduces the information loss problem.

[0057] (Three) loss function

[0058] The loss function of the application is divided into an autoencoder loss function and a fusion strategy loss function.

[0059] (31) Autoencoder loss function

[0060] The application uses an autoencoder loss function to train the autoencoder network, L auto is defined as follows,

[0061] L auto = L pixel + λL ssim (1)

[0062] wherein, L pixel and L ssim represent the pixel loss and structural similarity loss between the input image and the output image. λ is the weighting parameter between L pixel and L ssim L pixel loss is calculated by equation (2),

[0063]

[0064] wherein, is the Frobenius norm, and output and input represent the output and input images respectively. The similarity between the reconstructed image and the input image at the pixel level is constrained by L pixel loss. L ssim loss is calculated by equation (3),

[0065] L ssim = 1-SSIM(Output,Input) (3)

[0066] wherein, SSIM(.) is the structural similarity measure, which quantifies the structural similarity of two images.

[0067] (32) Fusion strategy loss function

[0068] The training of the CRN aims to implement a completely learnable fusion strategy. In the second stage, the encoder and decoder are fixed, and the CRN is trained with a suitable loss function. In order to better train the CRN, the application uses a fusion strategy loss function L CRN , which is defined as follows:

[0069] L CRN = aL detail + L feature (4)

[0070] where L detail and L feature represent the background detail preservation loss function and the target feature enhancement loss function, respectively. a is a trade-off parameter. Since most of the background detail information in the fused image comes from the visible light image, the purpose of L detail is to preserve the detail information and structural features in the visible light image, which is defined as:

[0071] L detail = 1-SSIM(O, I vi ) (5)

[0072] where I vi denotes the visible light image.

[0073] Since the infrared image contains more significant target features than the visible light image, the loss function L feature is designed to constrain the fused deep features to preserve the significant structures. This L feature is defined as:

[0074]

[0075] In equation (6), m is the number of multi-scale deep features, which is set to 4. w1 is a trade-off parameter vector for balancing the loss magnitude due to the size difference between the scales. For the fused features, w vi is the relative influence coefficient of the visible light features , and w ir is the relative influence coefficient of the infrared features . Since the visible light information is constrained by L detail , and the purpose of L feature is to preserve the significant features from the infrared image, w ir is usually larger than w vi .

[0076] (IV) Experimental results and analysis

[0077] In the implementation, the attention mechanism and the fusion strategy CRN are subjected to an ablation experiment, which fully demonstrates the effectiveness of the algorithm proposed in the present application. Finally, the algorithm of the present application is compared with other existing five algorithms, including:

[0078] Existing algorithm 1: Gradient transfer and total variation minimization (GTF) (Ma J Y, Chen C, Li C, Huang J. Infrared and visible image fusion via gradient transfer and total variation minimization[J]. Information Fusion, 2016, 31. 100-109.);

[0079] Existing algorithm 2, Fusion generative adversarial network (FusionGAN) (Ma J Y, Yu W, Liang P W, Li C, Jiang J J. FusionGAN: A generative adversarial network for infrared and visible image fusion[J]. Information Fusion, 2018, 48. 11-26);

[0080] Existing algorithm 3, End-to-end residual fusion network for infrared and visible images (RFN-Nest) (Li H, Wu X J, Kittler J. RFN-Nest: An end-to-end residual fusion network for infrared and visible images[J]. Information Fusion, 2021, 73. 72-86);

[0081] Existing algorithm 4, Deep image decomposition for infrared and visible image fusion (DIDFuse) (Zhao Z, Xu S, Zhang C, et al. DIDFuse: Deep Image Decomposition for Infrared and Visible Image Fusion[J]. 2020. DOI: 10.24963 / ijcai.2020 / 135);

[0082] The existing algorithm 5 is unsupervised misaligned infrared and visible image fusion based on cross-modality image generation and registration (UMF-CMGR) (Wang D, Liu J Y, et al. Unsupervised Misaligned Infrared and Visible Image Fusion via Cross-Modality Image Generation and Registration [EB / OL] 2022-05-25, https: / / arXiv.org / abs / 2205.11876);

[0083] Seven indicators are used to objectively evaluate the quality of the fused pictures, respectively: spatial frequency (SF), average gradient (AG), standard deviation (SD), correlation coefficient (CC), difference correlation and (SCD), visual fidelity (VIF), and peak signal-to-noise ratio (PSNR).

[0084] (41) Experimental setup

[0085] Since the TNO dataset contains rich military scenes, such as helicopters, houses, tanks, people, forests, vehicles, etc., the TNO dataset is very suitable for the data set requirements in the field of infrared and visible image fusion, and is the most authoritative data set in this research field. Therefore, the training set and the test set of the present application are derived from the TNO dataset. First, 43 pairs of infrared and visible images of different scenes are selected as the basic training set, and the step size is set to 12 and the sub-image size is set to 256x256 by expanding the data set. Finally, 24024 pairs of infrared and visible images are obtained as the training set. In order to make the test results more reasonable and fair, 42 pairs of images are selected as the test set, and the average value of the obtained results is taken. In this embodiment, Pytorch is used to build the experimental environment, and the training is carried out on an NVIDIA RTX3080 GPU, with an initial learning rate of 1e-4, a batch_size of 4, and an epoch of 4.

[0086] (42) Comparative experiment

[0087] In order to illustrate the effectiveness of the method of the present application, the algorithm of the present application is compared with five published algorithms in the subjective aspect. Among the five selected comparison algorithms, in addition to the GTF being a traditional method, FusionGAN, DIDFuse, UMF-CMGR, and RFN-Nest are all deep learning methods, and they are all methods published in recent years.

[0088] (421) Subjective evaluation

[0089] Figure 7A fence image from the TNO dataset is selected to show the fusion results of different algorithms. From the Figure 7 fusion images of GTF and FusionGAN, it can be seen that the fence information in the source image is severely lost. The GTF algorithm also does not retain the vegetation information in the visible light image. The DIDFuse algorithm fused image is relatively blurred, and the infrared information of the fence and vegetation is partially missing, making the texture boundary unclear. The UMF-CMGR, RFN-Nest and the algorithm of the present application fused image quality is relatively good. But intuitively, it can be observed that the image fused by the RFN-Nest algorithm is dark due to the loss of infrared information. The UMF-CMGR algorithm loses visible light information, so the overall image is bright, and the present application method balances the retention of infrared and visible light information, so the brightness level is between the two algorithms.

[0090] In order to better observe the detail information, Figure 8 The helicopter fusion results under different algorithms are selected. From the intuitive point of view, only the algorithm of the present application can better retain the background information. Although DIDFuse can also retain the background information, only the background information in the lower left corner is well retained, and the background information in other parts is still lost. In addition, in the black box, the helicopter landing gear under the UMF-CMGR algorithm, the infrared and visible light information of the landing gear is severely lost, and intuitively almost no landing gear can be observed. Under the GTF and DIDFuse algorithms, the infrared information of the landing gear is lost, so the landing gear details are blurred and the texture is not clear. The background of the fusion image under the RFN-Nest algorithm is overall dark, and the background information is not clear enough, so the algorithm of the present application is better in terms of visual effect.

[0091] In order to illustrate the universality of the algorithm of the present application, three typical images from the TNO dataset are selected to be fused under different algorithms, Figure 9 , Figure 10 , Figure 11 fusion results. By comparing the results of other algorithms, from the subjective aspect, the image fused by the algorithm of the present application is clear, the details are prominent, and the outline is obvious, which is more in line with human visual perception.

Claims

1. An autoencoder-based infrared and visible image fusion algorithm, characterized in that, The method comprises the following steps: S1, an encoder network is constructed by combining a short cross-layer connection with an ECA attention mechanism; S2, the registered infrared and visible light images are input into the encoder network, and the encoder and decoder network are trained to obtain an encoder meeting a condition; wherein the autoencoder loss function is used to train the autoencoder network, is defined as follows, wherein, represents a pixel loss between the input image and the output image, represents a structural similarity loss between the input image and the output image; is and a trade-off parameter between the two. The loss calculation expression is as follows: wherein is the Frobenius norm; The loss calculation expression is as follows: wherein is a structural similarity metric; S3, feature fusion is performed on the input images through the constructed fusion network, and the fusion network is trained to obtain a fusion network meeting a condition; The fusion network is composed of LReLU activation function and convolutional layer, Tanh activation function and convolutional layer, attention mechanism CBAM, convolutional layer; adopt three times convolution and LReLU activation function composed of basic blocks to extract features of the source image; through convolutional layer and CBAM attention mechanism to deeply extract the detail information of the source image, and splice the extracted detail feature information and the feature information extracted by the three basic blocks; finally, the basic blocks composed of Tanh activation function and convolutional layer are convolved and output. Wherein, the fusion strategy loss function is used to train the fusion network, and the fusion strategy loss function is defined as follows: wherein, and respectively represent the background detail preservation loss function and the target feature enhancement loss function, is a trade-off parameter; The calculation expression is as follows: wherein I vi represents a visible light image; The calculation expression is as follows: In the formula, is the number of multi-scale depth features; is a weighting parameter vector for balancing loss magnitudes; is a fused feature, is a visible light feature a relative influence coefficient of, is an infrared feature a relative influence coefficient of; S4, the fused images are decoded and reconstructed by the decoder.

2. The algorithm for infrared and visible image fusion based on auto-encoder according to claim 1, characterized in that, In step S1, the encoder is composed of a convolution layer and an ECA attention mechanism, and feature extraction is performed on the input images at four scales; the convolution layer is connected with the ECA attention mechanism through a short cross-layer.

Images