End-to-end visible light image fusion network in infrared and night scenes

Abstract

The invention provides a visible light image fusion network in an end-to-end infrared and night scene, and relates to the technical field of image fusion, and the method comprises the following steps: 1, constructing the visible light image fusion network in the end-to-end infrared and night scene; step 2, acquiring a training set of an infrared and visible light image fusion network; step 3, using the training set to train a visible light image fusion network in an end-to-end infrared and night scene; 4, acquiring a test set of the infrared and visible light image fusion network; and 5, testing the test set by using the trained end-to-end visible light image fusion network in the infrared and night scenes to obtain a fusion image with high visual quality.

Description

Technical Field

[0001] The present invention relates to the field of image fusion technology, and in particular to an end-to-end infrared and visible light image fusion network in nighttime scenes. Background Art

[0002] Infrared and visible light image fusion is an enhancement technique that integrates key information from infrared and visible light images to produce a fused image. Due to the strong complementarity between the source image information, infrared and visible light image fusion has been widely used in practical scenarios such as autonomous driving and advanced vision tasks such as semantic segmentation and object detection. Therefore, it is of great significance to develop a method for fusion of infrared and visible light images in night scenes to achieve image enhancement and thereby improve the performance of tasks such as video surveillance and autonomous driving. Traditional infrared and visible light image fusion methods perform well in certain scenarios, but they generally rely on hand-crafted features and rules and have limited generalization capabilities. In recent years, the introduction of deep learning to the field of infrared and visible light fusion has brought new breakthroughs and achieved good fusion results. However, several challenges remain to be addressed.

[0003] First, infrared and visible light image fusion in nighttime scenes often uses a two-stage training framework: in the first stage, a network is trained to enhance the visible light image, and in the second stage, a network is trained to fuse the enhanced visible light image with the infrared image. Training the model in stages can achieve local optimality but not global optimality. Second, the fusion task fails to account for the uneven brightness of different regions of visible light. The method does not adaptively adjust the brightness distribution of the visible light image, resulting in overexposure or underexposure of certain areas. Overexposed areas dominate the fusion process, resulting in the neglect of significant information in the infrared image. Underexposure leads to a loss of detail in the fused image, impacting subsequent tasks. Summary of the Invention

[0004] In order to meet the above challenges, the present invention provides an end-to-end infrared and visible light image fusion network for night scenes. Specifically, we propose an exposure correction module for visible light images in combination with the logarithmic enhancement method. First, the visible light image is decomposed into three channels of R, G, and B. The three channels are divided by region and the brightness distribution of each region is estimated. Then, according to the brightness distribution, the low-exposure area is enhanced and the over-exposure area is compressed so that the visible light image gradually approaches normal exposure. Secondly, the features of the visible light image and the infrared image after exposure correction are extracted through an encoder. In order to fully integrate the key information in the infrared features and the visible light features after exposure correction, we use information modulation to assign weights and supplement features to the two features.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] The present invention provides an end-to-end infrared and visible light image fusion network for night scenes, comprising the following steps:

[0007] Step 1: Build an end-to-end infrared and visible light image fusion network for night scenes;

[0008] Step 2: Obtain the training set of the infrared and visible light image fusion network;

[0009] Step 3: Use the training set to train an end-to-end infrared and visible light image fusion network for night scenes;

[0010] Step 4: Obtain the test set of the infrared and visible light image fusion network;

[0011] Step 5: Use the trained end-to-end infrared and visible light image fusion network under night scenes to test the test set to obtain a fused image with high visual quality.

[0012] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0013] 1. The end-to-end infrared and visible light image fusion network for night scenes of the present invention achieves global optimization of image enhancement and fusion tasks by training the network with dual constraints of the loss function on the exposure-corrected visible light image and the fused image.

[0014] 2. The exposure correction module of the present invention can compress the over-exposure area and enhance the under-exposure area so that the visible light image gradually approaches normal exposure.

[0015] 3. The feature fusion module of the present invention can effectively fuse the key information on infrared and visible light images. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 Flowchart of the present invention

[0017] Figure 2 This is the framework structure diagram of the present invention

[0018] Figure 3 This is the structural diagram of the exposure correction module

[0019] Figure 4 This is the structural diagram of the feature fusion module DETAILED DESCRIPTION

[0020] The embodiments of the present invention are described in detail below with reference to the accompanying drawings and examples.

[0021] The present invention provides an end-to-end infrared and nighttime visible light image fusion network, which relates to the field of image fusion technology. The present invention includes the following steps:

[0022] Step 1: Build an end-to-end infrared and visible light image fusion network for night scenes; the network model mainly consists of four parts: exposure correction module, encoder, feature fusion module and decoder. The exposure correction module consists of three branches. Each branch consists of two serial components and a logarithmic operation. Each component includes a 3×3 convolution, a 5×5 convolution and a ReLU activation function in sequence. The encoder includes three serial 3×3 convolutions and LRelu activation functions in sequence. The specific structure of the feature fusion module is as follows: Figure 4 As shown in the figure, each feature passes through a separate branch, which consists of a 3×3 convolution, a sigmoid activation function, a multiplication operation, a 3×3 convolution, a sigmoid activation function, and an addition operation. Finally, the two updated features are concatenated through a channel concatenation operation. The decoder consists of two serial 3×3 convolutions and LReLU activation functions, and a set of 3×3 convolutions and tanh activation functions.

[0023] Step 2: Obtain the training set of the infrared and visible light image fusion network;

[0024] 1083 pairs of infrared and visible light images were selected from the public infrared and visible light image fusion dataset MSRS training set, and the obtained image pairs were used to construct the training set of the infrared and visible light image fusion network.

[0025] Step 3: Use the training set to train an end-to-end infrared and visible light image fusion network for night scenes;

[0026] In this application, the training process of the end-to-end infrared and visible light image fusion network in night scenes includes the following steps:

[0027] Step 3.1: Read infrared image and visible light image I vis as input.

[0028] Read the image pairs in the end-to-end infrared and visible light image fusion network training set (I ir ,I vis ), normalize the pixel values ​​of the image to [0,1].

[0029] Step 3.2: Send the read visible light image to the exposure correction module to obtain the visible light image after exposure correction Then the infrared image I ir Convert to grayscale image Convert the exposure-corrected visible light image to the YCbCr color space and extract the Y channel image of the exposure-corrected visible light image Get the image pair of infrared grayscale image and visible light Y channel image after exposure correction

[0030] Step 3.3: Send the infrared grayscale image and the exposure-corrected visible light Y channel image to the encoder to extract their corresponding features. ir and

[0031] Step 3.4: Extract the features ir and Send it to the feature fusion module to get the fused feature feature fusion .

[0032] Step 3.5: Fusion feature fusion The Y channel map of the fused image is reconstructed into the decoder, and then the Y channel map is combined with the exposure-corrected visible light images Cb and Cr to reconstruct the YCbCr color channel map of the fused image, which is then converted into an RGB image to obtain the final fusion result.

[0033] Step 3.6: According to the loss function Calculate the loss value of the end-to-end infrared and visible light image fusion network under night scenes, use the Adam optimizer to optimize the loss gradient and backpropagate, and update the model parameters of the end-to-end infrared and visible light image fusion network under night scenes; the loss function in this application The calculation formula is shown in formula (1):

[0034]

[0035] In formula (1), and The loss used to constrain the exposure-corrected visible light image is spatial consistency loss, exposure control loss, color constancy loss, and smoothness loss. and The constraints used to fusion images are: texture loss, intensity loss and color loss.

[0036] In formula (1), the spatial consistency loss The calculation formula is shown in formula (2):

[0037]

[0038] In Equation (2), N represents the number of pixels in the region, and v(i) represents the four adjacent regions (upper, lower, left, and right) centered on region i. We denote the average intensity values ​​of the adjacent regions of the output and input images as Y and I, respectively.

[0039] In formula (1), exposure control loss The calculation formula is shown in formula (3):

[0040]

[0041] In formula (3), M represents the number of non-overlapping local regions of size 16×16, Y represents the average intensity value of the local region in the output image, and E represents the good exposure level.

[0042] In formula (1), the color constancy loss The calculation formula is shown in formula (4):

[0043]

[0044] Among them J p represents the average intensity of the p channel.

[0045] In formula (1), the lighting smoothness loss The calculation formula is shown in formula (5):

[0046]

[0047] In formula (5), Z is the number of iterations, and Represents horizontal and vertical gradient operations respectively.

[0048] In formula (1), the edge loss The calculation formula is shown in formula (6):

[0049]

[0050] In formula (6), W and H represent the width and height of the image respectively. represents the Sobel gradient operator, max() represents element-wise maximum selection, and ||·||1 represents the L1 norm.

[0051] In formula (1), the strength loss The calculation formula is shown in formula (7):

[0052]

[0053] Step 3.7: Repeat steps 3.1 to 3.6 until the number of iterations epoch on the entire training set reaches the set threshold, and a trained end-to-end infrared and visible light image fusion network for night scenes is obtained. In this embodiment, the iteration threshold during the training of the end-to-end infrared and visible light image fusion network for night scenes is 10. In this experimental example, the training of the end-to-end infrared and visible light image fusion network for night scenes is based on the NVIDIA A4000 GPU chip and uses the Adam optimizer.

[0054] Step 4: Obtain the test set of the infrared and visible light image fusion network;

[0055] 361 visible light images were selected from the MSRS test set, and the obtained visible light images were used to construct a test set for the brightness enhancement network.

[0056] Step 5: Use the trained end-to-end infrared and visible light image fusion network in night scenes to test the test set to obtain a fused image with high visual quality.

[0057] The image pairs in the test set of the infrared and visible light image fusion network are sent to the trained end-to-end infrared and visible light image fusion network under night scenes to obtain the fused image.

Claims

1. Build an end-to-end infrared and visible light image fusion network for night scenes; The network model consists of four main components: an exposure correction module, an encoder, a feature fusion module, and a decoder. The exposure correction module consists of three branches, each consisting of two serial components and a logarithmic operation. Each component sequentially includes a 3×3 convolution, a 5×5 convolution, and a Reluctant Luminance (ReLU) activation function. The encoder sequentially includes three serial 3×3 convolutions and an LReluctant Luminance (LReLU) activation function. The detailed structure of the feature fusion module is shown in Figure 4. Each feature passes through a separate branch, which sequentially includes a 3×3 convolution, a sigmoid activation function, a multiplication operation, a 3×3 convolution, a sigmoid activation function, and an addition operation. Finally, the two updated features are concatenated through a channel-wise concatenation operation. The decoder sequentially includes two serial 3×3 convolutions and an LReluctant Luminance (LReLU) activation function, and a set of 3×3 convolutions and a tanh activation function.

2. Obtain a training set for the infrared and visible light image fusion network; 1083 pairs of infrared and visible light images were selected from the public infrared and visible light image fusion dataset MSRS training set, and the obtained image pairs were used to construct the training set of the infrared and visible light image fusion network.

3. Use the training set to train an end-to-end infrared and visible light image fusion network for night scenes; In this application, the training process of the end-to-end infrared and visible light image fusion network in night scenes includes the following steps: Step S1: Read infrared image and visible light image I vis as input. Read the image pairs in the end-to-end infrared and visible light image fusion network training set (I ir ,I vis ), normalize the pixel values ​​of the image to [0,1]. Step S2: Send the read visible light image to the exposure correction module to obtain the visible light image after exposure correction Then the infrared image I ir Convert to grayscale image Convert the exposure-corrected visible light image to the YCbCr color space and extract the Y channel image of the exposure-corrected visible light image Get the image pair of infrared grayscale image and visible light Y channel image after exposure correction Step S3: Send the infrared grayscale image and the exposure-corrected visible light Y channel image to the encoder to extract their corresponding features. ir and Step S4: Extract the feature ir and Send it to the feature fusion module to get the fused feature feature fusion . Step S5: Fusion feature fusion The Y channel map of the fused image is reconstructed into the decoder, and then the Y channel map is combined with the exposure-corrected visible light images Cb and Cr to reconstruct the YCbCr color channel map of the fused image, which is then converted into an RGB image to obtain the final fusion result. Step S6: According to the loss function Calculate the loss value of the end-to-end infrared and visible light image fusion network for night scenes, use the Adam optimizer to optimize the loss gradient and backpropagate, and update the model parameters of the end-to-end infrared and visible light image fusion network for night scenes; Step S7: Repeat steps 3.1 to 3.6 until the number of iterations epoch on the entire training set reaches the set threshold, and a trained end-to-end infrared and visible light image fusion network for night scenes is obtained. In this embodiment, the threshold number of iterations during the training of the end-to-end infrared and visible light image fusion network for night scenes is 10. In this experimental example, the training of the end-to-end infrared and visible light image fusion network for night scenes is based on the NVIDIA A4000 GPU chip and uses the Adam optimizer.

4. Obtain a test set for the infrared and visible light image fusion network; 361 visible light images were selected from the MSRS test set, and the obtained visible light images were used to construct a test set for the brightness enhancement network.

5. Use the trained end-to-end infrared and visible light image fusion network in night scenes to test the test set and obtain a fused image with high visual quality. The image pairs in the test set of the infrared and visible light image fusion network are sent to the trained end-to-end infrared and visible light image fusion network under night scenes to obtain the fused image.

Images