End-to-end infrared and visible light image fusion method based on auto-encoder

Abstract

The invention provides an end-to-end infrared and visible light image fusion method based on an auto-encoder, aiming at the defect that a fusion strategy is manually designed in an infrared and visible light image fusion algorithm based on the auto-encoder. The method mainly comprises the following four steps: firstly, acquiring infrared and visible light images, and making a training set; then, an infrared and visible light image fusion model framework based on an auto-encoder is established, an encoder is used for extracting shallow layer features, a residual fusion module is used for extracting and fusing multi-scale depth features, and a decoder is used for reconstructing a fused image; thirdly, training the frame according to a training set and a loss function to obtain an infrared and visible light image fusion model; and finally, sending a to-be-fused source image into the trained model to obtain a fused image. Experimental results show that compared with other seven methods, the method disclosed by the invention improves fusion quality while avoiding manual design.

Description

Technical Field

[0001] The present invention relates to image fusion technology in the field of digital image processing, and in particular to an infrared and visible light image fusion method based on an autoencoder. Background Art

[0002] With the advancement of science and technology, human beings' demand for more comprehensive image information is also increasing. Sensors of different types or under different optical settings cannot obtain comprehensive information of the scene. Each sensor has its own advantages and limitations. For example, infrared sensors use the thermal radiation of objects to form images, which are less affected by factors such as light, dust and fog, but the generated image quality is low. The images obtained by visible light sensors contain rich detailed textures, but are easily affected by the environment. Therefore, the use of multiple sensors can collect information of the same scene from different angles, so as to obtain more comprehensive and accurate information. Image fusion technology can integrate complementary information into a single image. This technology has important application value in military, medical, geological and other fields.

[0003] Existing image fusion algorithms can be divided into two categories: algorithms based on traditional image processing and algorithms based on deep learning. According to the applied network framework, infrared and visible light image fusion algorithms based on deep learning can be divided into three subcategories, namely, fusion methods using convolutional neural networks (CNNs), fusion methods based on generative adversarial networks (GANs), and fusion methods based on autoencoders (AEs). Existing AE-based fusion algorithms have achieved good fusion effects, but there are still some limitations. Such methods mostly use manually designed fusion strategies to achieve feature fusion, but they are difficult to apply to deep features, which limits the further development of such methods. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide an end-to-end infrared and visible light image fusion method based on autoencoder, which further improves the image fusion effect while avoiding manual design of fusion strategy.

[0005] The present invention specifically adopts the following technical solutions:

[0006] An end-to-end autoencoder-based infrared and visible light image fusion method, characterized by:

[0007] a. Designing an end-to-end fusion network based on the autoencoder can realize the fusion of infrared and visible light images and generate natural and information-rich fused images.

[0008] b. The fusion module of the architecture is learnable, which can avoid manually designed fusion strategies.

[0009] c. The residual fusion module is designed based on the spatial pyramid of dilated convolution, which can fully extract and fuse the multi-scale features of the source image.

[0010] The method mainly includes the following steps:

[0011] (1) Obtain infrared and visible light source images from a set data set, and the acquired infrared source images and visible light source images correspond to each other. The source images are organized into a training set.

[0012] (2) Establish an infrared and visible light image fusion model framework based on an autoencoder. The infrared and visible light image fusion model framework includes an encoder, a residual fusion module, and a decoder. The encoder is used to extract shallow features, the residual fusion module is used to extract multi-scale deep features and fuse them, and the decoder is used to reconstruct the fused image.

[0013] (3) The infrared and visible light image fusion model framework is trained according to the training set and the loss function to obtain the infrared and visible light fusion model.

[0014] (4) The infrared and visible light images to be fused are sent to the infrared and visible light image fusion model trained in step (3), and two encoders are used to extract shallow features of the two source images respectively. Then, the residual fusion module extracts multi-scale features and fully fuses them. Finally, the decoder outputs the fused image.

[0015] The beneficial effects of the present invention are:

[0016] (1) The learnable residual fusion module can avoid the manual design of fusion strategies.

[0017] (2) The atrous convolution is introduced into the fusion network, which can more fully extract and fuse the multi-scale features in the source image.

[0018] (3) The fusion results obtained by this method are more natural and retain richer information. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic diagram of the fusion network framework of the present invention.

[0020] Figure 2 This is a schematic diagram of the encoder framework.

[0021] Figure 3 Schematic diagram of the multi-level fusion block framework.

[0022] Figure 4 Schematic diagram of the decoder framework.

[0023] Figure 5 Schematic diagram of subjective comparison experimental results between the method of the present invention and seven other methods. DETAILED DESCRIPTION

[0024] The present invention is further described below in conjunction with the accompanying drawings. It is necessary to point out here that the following embodiments are only used to further illustrate the present invention and cannot be understood as limiting the scope of protection of the present invention. Persons skilled in the art in this field may make some non-essential improvements and adjustments to the present invention based on the above content of the present invention, which still fall within the scope of protection of the present invention.

[0025] Figure 1 Specifically, an end-to-end autoencoder-based infrared and visible light image fusion method is involved, which can be divided into the following steps:

[0026] (1) Obtain infrared and visible light source images from a set data set, and the acquired infrared source images and visible light source images correspond to each other. The source images are organized into a training set.

[0027] (2) Establish an infrared and visible light image fusion model framework based on an autoencoder. The infrared and visible light image fusion model framework includes an encoder, a residual fusion module, and a decoder. The encoder is used to extract shallow features, the residual fusion module is used to extract multi-scale deep features and fuse them, and the decoder is used to reconstruct the fused image.

[0028] (3) The infrared and visible light image fusion model framework is trained according to the training set and the loss function to obtain the infrared and visible light fusion model.

[0029] (4) The infrared and visible light images to be fused are sent to the infrared and visible light image fusion model trained in step (3), and two encoders are used to extract shallow features of the two source images respectively. Then, the residual fusion module extracts multi-scale features and fully fuses them. Finally, the decoder outputs the fused image.

[0030] Specifically, the infrared and visible light image fusion framework in step (2) includes three parts: an encoder, a residual fusion module and a decoder.

[0031] The encoder consists of four convolutional layers containing 3×3 filters, such as Figure 2 As shown in Figure 1, it is used to extract shallow features in the source image. In order to improve the vanishing gradient and reuse the features extracted from the previous layers, the DenseBlock proposed in Densefuse is applied, and a direct connection is established between each layer and each previous layer. The number of channels of the feature map obtained in each layer is 16, and the stride of all convolutional layers is set to 1.

[0032] The main purpose of the residual fusion module is to extract features containing information of different scales and fully fuse them. The features of the visible light image and infrared image extracted by the previous two encoders are represented as en_vi and en_ir respectively. Then en_vi and en_ir are concatenated, and the concatenated features are used as the input of the residual fusion module. The residual fusion module contains three stacked multi-level fusion blocks (HFB). The structure of HFB is as follows: Figure 3 As shown in Figure 2. In this block, a spatial pyramid dilated convolution is introduced to efficiently fuse multi-scale features.

[0033] In each HFB module, the first step is to send the input features to the traditional 3×3 convolution layer to compress the number of channels of the input feature map. The second step is to convolve the feature map input in the previous step using K convolution kernels of size 3×3 with different expansion rates, with expansion rates of 2 k-1 , k = {1,2,…,K}. The dilated convolution can increase the receptive field. Therefore, without introducing additional parameters, the effective receptive field is expanded from 3×3 to 3+2 (2 k-1 -1). In the network structure of the present invention, K is set to 4. Since different dilated convolutions can obtain different receptive fields, multi-scale depth feature maps of the source image can be extracted, and information loss caused by downsampling can be avoided. Figure 3 The strategy shown in the figure starts with the feature map output by the smallest hole convolution kernel, and feature maps of different scales are stacked step by step. This operation can effectively improve the grid effect without introducing new parameters and increasing the computational burden too much. Then the summed K features are merged, and the number of channels of the concatenated feature map is reduced to 64 by using a 1×1 standard convolution. At the same time, in order to make full use of shallow feature information and ensure the gradient transmission of the network, the fusion module adds a direct connection from input to output. Finally, the output of the fusion module is a feature map containing multi-scale features.

[0034] The training strategy in step (3) shown in the figure includes two stages, namely, the training of the autoencoder network and the training of the residual fusion module network. The first step is to train the autoencoder, the purpose of which is to extract shallow features and reconstruct the input image. After fixing the parameters of the autoencoder, the second stage is to train the residual fusion module, the purpose of which is to extract multi-scale features and fully fuse them.

[0035] The loss function in step (3) shown includes the autoencoder loss function and the residual fusion module loss function.

[0036] The loss function of the autoencoder consists of two parts, namely the pixel loss function and the structural similarity loss function, and the formula is defined as follows:

[0037] L(I,O)=L pix(I,O)+λL ssim (I,O) (1)

[0038] Among them, I represents the input image, O represents the output reconstructed image, and λ is a hyperparameter used to balance the weights of the two parts of the loss function.

[0039] Loss function L pix It is used to measure the pixel error between the reconstructed image and the input image. The loss function is calculated by the mean square error function, and the formula is as follows:

[0040] L pix =MSE(O,I) (2)

[0041] Where I represents the input image and O represents the output reconstructed image. MSE represents the mean square error function, which is defined as follows:

[0042]

[0043] L ssim The loss function is set based on the structural similarity index metric (SSIM). This loss function can constrain the reconstructed image to be similar to the source image in terms of brightness, structure, and contrast. The formula is defined as follows:

[0044] L ssim =1-SSIM(I,O) (4)

[0045] Where I and O represent the input image and output image respectively.

[0046] The loss function L of the residual fusion module f , which is defined as follows:

[0047] L f =L feature +L ssim (5)

[0048] The definition of the structural similarity loss function is expressed as:

[0049] L ssim =1-SSIM(I f ,I vi ) (6)

[0050] Among them, I f represents the fused image, I vi Represents a visible light image.

[0051] In order to fully integrate the multi-scale features extracted in the previous stage, the feature loss function L feature The formula is defined as follows:

[0052]

[0053] Where en_vi and en_ir are the features obtained by inputting the infrared image and the visible light image into the dual-branch encoder respectively. is the output of the residual fusion module. ir and w vi It is a parameter that controls the relative influence of infrared and visible light features in the fused feature map.

[0054] To better illustrate the effectiveness of the present invention, the present invention is used to realize infrared and visible light image fusion on a public data set, and the present invention is compared with some classic and cutting-edge fusion methods in subjective and objective experiments. The comparison methods include CBF, GTF, CNN, FusionGAN, DDcGAN, NestFuse, and RFN-Nest. In the objective comparison experiment, a total of eight indicators are selected, namely, correlation coefficient (CC), mean square error (MSE), peak signal-to-noise ratio (PSNR), gradient-based fusion performance (Q AB / F ), Sum of Correlation Coefficients of Difference (SCD), Feature Mutual Information (FMI), Multi-Scale Structural Similarity Index Measure (MS-SSIM), and Structural Similarity Index Measure (SSIM).

[0055] Figure 5 A schematic diagram of the subjective experimental comparison results of the present invention and other seven methods is shown. Four scenes are selected here, namely "person", "ship", "jeep" and "bunker" from top to bottom. Visually, these methods can fuse the thermal radiation information of infrared images and the detail information in visible light images to a certain extent. However, compared with other methods, the present invention has obvious advantages and is more visually consistent with the source image because the present invention can effectively extract and fuse multi-scale features. First, the fusion result of the present invention can maintain the main thermal radiation information of the infrared image, which is crucial for subsequent image processing. Second, the fusion result of the present invention contains rich details of the infrared and visible light images. Third, the target in the fused image of the present invention is clear. In addition, the result of the present invention is free of distortion and artifacts.

[0056] Table 1 is a schematic diagram of the objective comparison experimental results of the present invention and other methods. Among them, the best value of the objective image fusion index is indicated in bold. The second best value is indicated in italics. From the statistical values, it can be seen that the present invention obtains the best values ​​in MSE, PSNR and SSIM indicators, and ranks second in CC, SCD and MS-SSIM indicators. AB / Fand FMI, the present invention ranks third. The best value of MSE shows that the result of the present invention can achieve a greater similarity with the source image. The best value of SSIM shows that the present invention maintains a high structural similarity. The good performance on PSNR means that the fusion result obtained by the present invention is more natural and contains less noise. The good result on FMI means that the method proposed by the present invention can retain more feature information in the fused image.

[0057] In summary, both the objective evaluation results and the subjective evaluation results show that the present invention has more obvious advantages than other methods.

[0058] Table 1 Objective comparison results of other seven methods and the present invention

[0059] Evaluation indicators CBF GTF CNN FusionGAN NestFuse DDcGAN RFN The present invention CC 0.3895 0.3512 0.4622 0.4555 0.4933 0.4926 0.5337 0.5329 MSE 0.0407 0.0528 0.0466 0.0514 0.0425 0.0495 0.0413 0.0365 PSNR 62.9124 61.7470 62.3089 61.6192 62.5252 61.4534 62.7127 63.2323 Qabf 0.3948 0.3562 0.4890 0.2226 0.5181 0.3441 0.3215 0.4322 SCD 1.2554 0.9138 1.6688 1.3335 1.6687 1.5095 1.7303 1.7231 FMI_PIX 0.8685 0.8987 0.9133 0.8887 0.9086 0.8773 0.9024 0.9034 FMI_DCT 0.1942 0.2360 0.2480 0.3655 0.3529 0.3826 0.2758 0.3739 MS-SSIM 0.7025 0.8247 0.9428 0.7537 0.8930 0.7705 0.9111 0.9346 SSIM 0.7370 0.8538 0.9274 0.8166 0.9159 0.8086 0.8752 0.9461

Claims

1. An end-to-end autoencoder-based infrared and visible light image fusion method, characterized in that The following steps are involved: (1) Obtain infrared and visible light source images from a set data set, the obtained infrared source images and visible light source images correspond to each other, and then organize the source images into a training set; (2) Establishing an infrared and visible light image fusion model framework based on an autoencoder, wherein the infrared and visible light image fusion model framework includes an encoder, a residual fusion module, and a decoder, wherein the encoder is used to extract shallow features, the residual fusion module is used to extract multi-scale deep features and fuse them, and the decoder is used to reconstruct a fused image; (3) Train the infrared and visible light image fusion model framework according to the training set and the loss function to obtain the infrared and visible light fusion model; (4) The infrared and visible light images to be fused are sent to the infrared and visible light image fusion model trained in step (3), and two encoders are used to extract shallow features of the two source images respectively. Then, the residual fusion module extracts multi-scale features and fully fuses them. Finally, the decoder outputs the fused image.

2. According to claim 1, an end-to-end autoencoder-based infrared and visible light image fusion method is characterized in that The infrared and visible light image fusion framework described in step (2) includes an encoder, a residual fusion module, and a decoder, all of which are composed of neural networks, avoiding manually designed fusion strategies. The end-to-end network framework can realize the fusion of infrared and visible light images and generate more natural and information-rich fused images.

3. The end-to-end infrared and visible light image fusion method based on autoencoder according to claim 1 is characterized in that The residual fusion module described in step (2) concatenates the features en_vi and en_ir of the visible light image and infrared image extracted by the previous two encoders, and the concatenated features are used as the input of the residual fusion module. The residual fusion module includes three stacked multi-level fusion blocks (HFBs). The structure of the HFB is shown in FIG3 , which introduces the dilated convolution of the spatial pyramid, which can efficiently extract and fuse multi-scale features.

4. The end-to-end autoencoder-based infrared and visible light image fusion method according to claim 1 is characterized in that The model training phase described in step (3) is divided into two steps. First, an autoencoder is trained to extract shallow features of the source image and image reconstruction. After fixing the autoencoder parameters, the residual fusion module is trained in the middle. The purpose of this part is to extract multi-scale features and fully integrate them.

Images