Fusion method of infrared image and visible light image

Abstract

The invention provides a fusion method for an infrared image and a visible light image, and belongs to the technical field of computer vision and artificial intelligence, and the method comprises the steps: S1, constructing an image fusion network model for generating a fusion image; s2, acquiring the infrared image and the corresponding visible light image to form a training data set, and training the fusion network model by using a loss function by using the training data set, the loss function is related to the structural similarity loss of the visible light image and the fusion image, the structural similarity loss of a semantic thermodynamic diagram mask area based on the infrared image and the corresponding visible light image, the image intensity loss and the image gradient loss; and S3, inputting an infrared image to be fused and a corresponding visible light image into the fusion network model trained in the step S2 to obtain a final fusion image. According to the method, manual operation is not needed, the computing power requirement is lowered, real-time image processing can be achieved, information is not prone to being lost, and efficiency is higher.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and artificial intelligence, and specifically relates to a method for fusing an infrared image with a visible light image. Background Art

[0002] In recent years, the fusion technology of infrared and visible light images has rapidly emerged in the field of computer vision and has become the focus of researchers. The core of this technology is to effectively integrate the information of multi-source images and give full play to the respective advantages of infrared and visible light images to generate images with richer information and higher definition, thereby more accurately reflecting the scene content. This technology not only improves the quality of the image, but also provides more comprehensive information support for scene recognition and target detection. It is widely used in industrial monitoring, security and remote perception.

[0003] In the civilian field, an important application of infrared and visible light image fusion technology is the automotive night vision system, which can greatly improve driving safety, especially in severe weather such as fog and rainstorm. Infrared images are good at capturing the thermal radiation of objects and can work stably under the condition of little influence of lighting changes and obstructions, which is suitable for night or low visibility environments. However, since infrared images lack the details and texture information of objects, their effect when used alone is limited. In contrast, visible light images can provide delicate textures and rich details, but they have difficulty in identifying targets in low light environments or complex backgrounds. After fusing these two images, the advantages of both can be combined to generate images containing rich details and stable thermal radiation information, thereby significantly improving target detection and environmental perception capabilities. In recent years, the rise of deep learning technology has brought new methods for the fusion of infrared and visible light images. Compared with traditional manual feature extraction, the image fusion method based on deep learning can automatically extract multi-level features in images, greatly simplifying the traditional process and improving the fusion effect. Through advanced technologies such as convolutional neural networks (CNN), generative adversarial networks (GAN), and Transformer, the fused image can not only retain the key information of multi-source images, but also enhance the overall performance of the image, significantly improving the performance of computer vision tasks. Overall, the outstanding performance of infrared and visible light image fusion technology in complex scenes makes it a key technology for improving target recognition, image enhancement, and environmental perception, and it is expected to play an important role in a wider range of applications in the future. However, the existing deep learning-based fusion technology still has defects: the salient targets are based on manual calibration, which is time-consuming and labor-intensive in actual use; the model has a large number of parameters and cannot cope with real-time image processing, requires high computing power, and is prone to losing infrared information of small targets at a distance. Summary of the invention

[0004] The purpose of the present invention is to propose a method for fusing infrared images and visible light images, which does not require manual operation, reduces computing power requirements, can realize real-time processing of images, is not prone to information loss, and is more efficient.

[0005] The present invention is achieved through the following technical solutions:

[0006] A method for fusing an infrared image and a visible light image comprises the following steps:

[0007] Step S1, constructing an image fusion network model, the image fusion network model comprising two deep convolutional networks, a cross convolutional network respectively connected to the middle layers of the two deep convolutional networks, a fusion module, and an image reconstruction network, the two deep convolutional networks are respectively used to extract semantic information of infrared images and visible light images to generate infrared image features and visible light image features, the cross convolutional network is used to generate complementary features of the infrared image and the visible light image, the fusion module is used to fuse the infrared image features, the visible light image features and the complementary features to generate fused features, and the image reconstruction network is used to reconstruct the fused features to generate a fused image;

[0008] Step S2, obtaining an infrared image and a corresponding visible light image to form a training data set, and using the training data set to train the fusion network model with a loss function, where the loss function is related to a structural similarity loss between the visible light image and the fused image, a structural similarity loss based on a semantic heat map mask area of ​​the infrared image and the corresponding visible light image, an image intensity loss, and an image gradient loss;

[0009] Step S3: input the infrared image to be fused and the corresponding visible light image into the fusion network model trained in step S2 to obtain the final fused image.

[0010] Furthermore, in step S1, the deep convolutional network includes an initialized convolutional module and a star module connected in sequence, the star module includes a first deep convolutional layer, an intermediate layer and a jump connection layer connected in sequence, the intermediate layer is composed of two intermediate convolutions connected in parallel, the jump connection layer is composed of a Sobel operator connected head to tail and the first convolutional layer, and the cross convolutional network includes a second convolutional layer and a second deep convolutional layer connected in sequence, and the second convolutional layer is respectively connected to the intermediate layers of the two deep convolutional networks.

[0011] Furthermore, in step S1, the image reconstruction network includes two first convolution modules and a second convolution module connected in sequence, the first convolution module includes a third convolution layer and a LeakyRelu activation function, and the second convolution module includes a fourth convolution layer and a Tanh activation function.

[0012] Furthermore, in the step S1, the fusion module multiplies the infrared image feature and the visible light image feature in the channel dimension, and then adds the complementary feature to generate the fused feature.

[0013] Furthermore, in step S2, the infrared image and the corresponding visible light image are input into a multimodal semantic segmentation neural network, and a semantic heat map mask is generated through a reverse gradient propagation mechanism to form a semantic heat map mask area of ​​the infrared image and the corresponding visible light image.

[0014] Furthermore, in step S2, the loss function is expressed as ,in, represents the structural similarity loss between the visible light image and the fused image, represents the structural similarity loss of the semantic heat map mask region based on the infrared image and the corresponding visible light image, represents the image intensity loss, represents the image gradient loss, and Indicates the weights calculated based on the structural similarity after the infrared image and the corresponding visible light image are input into the multimodal semantic segmentation neural network and the convolution feature map is visualized through forward propagation.

[0015] Furthermore, the structural similarity loss between the visible light image and the fused image is expressed as ,in, represents a visible light image, represents the fused image, , represents the mean of the sliding window set in the visible light image, represents the mean of the sliding window set in the fused image, for and The covariance between and is a constant, for The corresponding variance, for The corresponding variance, The value range is [-1,1].

[0016] Furthermore, the structural similarity loss of the semantic heat map mask area based on the infrared image and the corresponding visible light image is expressed as ,in, represents an infrared image, represents the fused image, , H is the height of the fused image, W is the width of the fused image, For SSIM( ) is the value of the structural similarity of the pixel coordinates (i, j) calculated, The value of the mask for pixel coordinate (i, j).

[0017] Furthermore, the image intensity loss is expressed as ,in, represents the pixel coordinates (i, j) of the infrared image, represents the pixel coordinates (i, j) of the visible light image, represents the pixel coordinates (i, j) of the fused image, H and W represent the height and width of the infrared image, visible light image, and fused image, respectively. Indicates finding the absolute value.

[0018] Furthermore, the image gradient loss is expressed as ,in, represents the gradient of the image, represents the pixel coordinates (i, j) of the infrared image, represents the pixel coordinates (i, j) of the visible light image, represents the pixel coordinates (i, j) of the fused image, H and W represent the height and width of the infrared image, visible light image, and fused image, respectively. Indicates finding the absolute value.

[0019] The present invention has the following beneficial effects:

[0020] 1. The image fusion network model constructed by the present invention includes two deep convolutional networks for extracting semantic information to generate infrared image features and visible light image features, a cross convolutional network connected to the middle layer of the two deep convolutional networks to generate complementary features of the infrared image and the visible light image, a fusion module for fusing the infrared image features, the visible light image features and the complementary features to generate fusion features, and an image reconstruction network for reconstructing the fusion features to generate a fusion image. In the training of the image fusion network model, a visible light image and a corresponding infrared image are obtained to form an image pair, and the image pair forms a training data set. The loss used in the training is The function is related to the structural similarity loss between the visible light image and the fused image, the structural similarity loss based on the semantic heat map mask area of ​​the infrared image and the corresponding visible light image, the image intensity loss and the image gradient loss, so that the image fusion network model can automatically balance the clarity and texture of the infrared target and the structural information and texture of the visible light image, thereby generating a high-quality fused image with clear infrared targets and complete information, realizing real-time processing of images, not prone to information loss, and no manual operation is required in the process, avoiding the problems of relying on standard reference images and requiring subsequent optimization processing in the prior art, and solving the problems of complex algorithms and low efficiency in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The present invention is further described in detail below with reference to the accompanying drawings.

[0022] Figure 1 It is a flow chart of the present invention.

[0023] Figure 2 It is a fusion schematic diagram of the present invention.

[0024] Figure 3 Schematic diagram of the network structure of the image fusion network model of the present invention.

[0025] Figure 4 This is a diagram of the network structure of the multimodal semantic segmentation neural network of the present invention.

[0026] Figure 5 This is a diagram of the training parameter table of the image fusion network model of the present invention.

[0027] Figure 6 Schematic diagram of three groups of image pairs according to the present invention.

[0028] Figure 7 The present invention corresponds to Figure 6 Schematic diagram of the fused image of the three image pairs in . DETAILED DESCRIPTION

[0029] like Figure 1 and Figure 2 As shown, the method for fusing infrared images with visible light images is characterized by comprising the following steps:

[0030] Step S1, constructing an image fusion network model, the image fusion network model comprising two deep convolutional networks, a cross convolutional network respectively connected to the middle layers of the two deep convolutional networks, a fusion module, and an image reconstruction network, the two deep convolutional networks are respectively used to extract semantic information of infrared images and visible light images to generate infrared image features and visible light image features, the cross convolutional network is used to generate complementary features of the infrared image and the visible light image, the fusion module is used to fuse the infrared image features, the visible light image features and the complementary features to generate fused features, and the image reconstruction network is used to reconstruct the fused features to generate a fused image;

[0031] More specifically, if Figure 3 As shown in the figure, the deep convolution network includes an initialization convolution module (IC) and a star module (Star) connected in sequence. The initialization convolution module includes a convolution layer and a deep learning layer connected in sequence, which is used to perform streamlined image processing on the original image. The star module can project low-dimensional semantic information to a higher dimension to extract more advanced semantic information and richer texture details.

[0032] The star-shaped module includes a first deep convolutional layer, an intermediate layer and a jump connection layer connected in sequence. The intermediate layer is composed of two parallel intermediate convolutions. The jump connection layer is composed of a Sobel operator connected to the head and the tail and the first convolutional layer. The cross convolutional network includes a second convolutional layer and a second deep convolutional layer connected in sequence. The second convolutional layer is connected to the intermediate layers of the two deep convolutional networks respectively.

[0033] The convolution kernel of the initialization convolution module is 3*3 convolution, the number of convolution kernels is 32, the step size is 1, and the padding size is 1; the convolution kernel of the first depth convolution layer is 7*7, the number of convolution kernels is 64, the step size is 1, and the padding size is 3; the convolution kernel of the intermediate convolution is 3*3, the number of convolutions is 160, the step size is 1, and there is no padding.

[0034] The fusion module multiplies the infrared image features and the visible light image features bit by bit in the channel dimension, and then adds the multiplication result and the complementary features bit by bit to generate a fusion feature. More specifically, the function implemented by the fusion module is implemented by the image fusion network model, that is, when the infrared image features, visible light image features and complementary features are obtained, the image fusion network model performs a fusion operation to generate a fusion feature.

[0035] The image reconstruction network includes two first convolution modules (C1 layer and C2 layer) and a second convolution module (C3 layer) connected in sequence. The first convolution module includes the third convolution layer and the LeakyRelu activation function, and the second convolution module includes the fourth convolution layer and the Tanh activation function. The convolution kernel of the third convolution layer is 3*3, the number of convolutions is 8, the step size is 1, and the padding size is 1. The convolution kernel of the fourth convolution layer is 3*3, the number of convolutions is 1, the step size is 1, and the padding size is 1. Tanh and LeakyReLU are used as the activation functions of the network to speed up the convergence of the network model and generate a direct mapping from the source image to the final fusion image, so as to obtain a high-quality fusion image and avoid manual operation.

[0036] Step S2, obtaining an infrared image and a corresponding visible light image to form a training data set, and using the training data set to train the fusion network model with a loss function, where the loss function is related to a structural similarity loss between the visible light image and the fused image, a structural similarity loss based on a semantic heat map mask area of ​​the infrared image and the corresponding visible light image, an image intensity loss, and an image gradient loss;

[0037] The infrared image and the corresponding visible light image are input into the multimodal semantic segmentation neural network, and a semantic heat map mask is generated through the reverse gradient propagation mechanism to form a semantic heat map mask area of ​​the infrared image and the corresponding visible light image. The process of obtaining the mask area is a prior art. Figure 4As shown in the figure, the multimodal semantic segmentation neural network is a neural network composed of an initialized convolution operation module and a densely connected convolution module and a deconvolution group. The convolution kernel of the initial convolution operation module is a 7*7 convolution, the number of convolution kernels is 64, the step size is 2, the padding is 3, and the initial convolution operation module is sequentially connected to the batch norm function, the ReLU function and the MaxPool layer. The convolution module is taken from the four densely connected convolution modules of Denesenet121 and three downsampling modules consisting of 1*1 convolution and 2*2 average pooling. The deconvolution group contains 5 layers of deconvolution. The convolution kernel of the first layer of deconvolution is 2*2 deconvolution, the number of convolution kernels is 512, the step size is 2, and the padding is 0. The convolution kernel of the second layer of deconvolution is 2*2 deconvolution, the number of convolution kernels is 256, the step size is 2, and the padding is 0. The convolution kernel of the third layer of deconvolution is 2*2 deconvolution, the number of convolution kernels is 128, the step size is 2, and the padding is 0. The convolution kernel of the fourth layer of deconvolution is 2*2 deconvolution, the number of convolution kernels is 64, the step size is 2, and the padding is 0. The convolution kernel of the fifth layer of deconvolution is 2*2 deconvolution, the number of convolution kernels is 9, the step size is 2, and the padding is 0.

[0038] The loss function is specifically expressed as ,in, represents the structural similarity loss between the visible light image and the fused image, represents the structural similarity loss of the semantic heat map mask region based on the infrared image and the corresponding visible light image, represents the image intensity loss, represents the image gradient loss, and Respectively represent the infrared image and the corresponding visible light image input multimodal semantic segmentation neural network, and after forward propagation to visualize the convolution feature map, the weights calculated according to the structural similarity are: , , where A is the infrared image feature map, C is the visible light image feature map, and B is the infrared visible light image feature map that combines the infrared image features and the visible light image features. The specific calculation of SSIM (A, B) is the same as the following same.

[0039] The structural similarity loss between the visible light image and the fused image is expressed as ,in, represents a visible light image, represents the fused image, Represents the input visible light image The fused image with the output The structural similarity of different sliding windows at corresponding positions, represents the mean of the sliding window set in the visible light image, represents the mean of the sliding window set in the fused image, for and The covariance between and is a constant, for The corresponding variance, for The corresponding variance, The value range is [-1,1].

[0040] The structural similarity loss of the semantic heat map mask area based on the infrared image and the corresponding visible light image is expressed as ,in, represents an infrared image, represents the fused image, , H is the height of the fused image, W is the width of the fused image, For SSIM( ) is the value of the structural similarity of the pixel coordinates (i, j) calculated, is the value of the mask at pixel coordinate (i, j). The semantic heatmap mask region calculated in the multimodal semantic segmentation neural network is represented as ,in , Represents the feature map of the convolutional layer in the multimodal semantic segmentation neural network, k is the channel index, N is the feature map space size, It is a feature map where is the value of (i,j), is the importance weight of category c to the kth feature map channel, obtained by calculating the gradient of the channel and performing global average pooling. Generally speaking, in image segmentation neural networks, logits are generated for each pixel. , is the sum of the class pixels in the network's area of ​​interest.

[0041] The image intensity loss is expressed as ,in, represents the pixel coordinates (i, j) of the infrared image, represents the pixel coordinates (i, j) of the visible light image, represents the pixel coordinates (i, j) of the fused image, H and W represent the height and width of the infrared image, visible light image, and fused image, respectively. Indicates finding the absolute value.

[0042] The image gradient loss is expressed as ,in, represents the gradient of the image, represents the pixel coordinates (i, j) of the infrared image, represents the pixel coordinates (i, j) of the visible light image, represents the pixel coordinate (i, j) of the fused image, H and W represent the height and width of the infrared image, visible light image, and fused image, respectively.

[0043] like Figure 5 is a training parameter table diagram of this embodiment, Indicates finding the absolute value.

[0044] Step S3, inputting the infrared image to be fused and the corresponding visible light image into the fusion network model trained in step S2 to obtain a final fused image;

[0045] like Figure 6 The figure shows a schematic diagram of three sets of image pairs (visible light images and their corresponding infrared images). Figure 7 To correspond to Figure 6 Schematic diagram of fused images of three groups of image pairs. It can be seen that the fusion effect of the present invention is good and no information is lost.

[0046] The above description is only a preferred embodiment of the present invention, and therefore cannot be used to limit the scope of implementation of the present invention. That is, equivalent changes and modifications made according to the scope of the patent application of the present invention and the contents of the specification should still fall within the scope covered by the patent of the present invention.

Claims

1. A method for fusing an infrared image and a visible light image, characterized in that: The steps include: Step S1, constructing an image fusion network model, the image fusion network model comprising two deep convolutional networks, a cross convolutional network respectively connected to the middle layers of the two deep convolutional networks, a fusion module, and an image reconstruction network, the two deep convolutional networks are respectively used to extract semantic information of infrared images and visible light images to generate infrared image features and visible light image features, the cross convolutional network is used to generate complementary features of the infrared image and the visible light image, the fusion module is used to fuse the infrared image features, the visible light image features and the complementary features to generate fused features, and the image reconstruction network is used to reconstruct the fused features to generate a fused image; Step S2, obtaining an infrared image and a corresponding visible light image to form a training data set, and using the training data set to train the fusion network model with a loss function, where the loss function is related to a structural similarity loss between the visible light image and the fused image, a structural similarity loss based on a semantic heat map mask area of ​​the infrared image and the corresponding visible light image, an image intensity loss, and an image gradient loss; Step S3: input the infrared image to be fused and the corresponding visible light image into the fusion network model trained in step S2 to obtain the final fused image.

2. The method for fusing an infrared image and a visible light image according to claim 1, characterized in that: In step S1, the deep convolutional network includes an initialized convolutional module and a star module connected in sequence, the star module includes a first deep convolutional layer, an intermediate layer and a jump connection layer connected in sequence, the intermediate layer is composed of two parallel intermediate convolutions, the jump connection layer is composed of a Sobel operator connected head to tail and the first convolutional layer, the cross convolutional network includes a second convolutional layer and a second deep convolutional layer connected in sequence, and the second convolutional layer is respectively connected to the intermediate layers of the two deep convolutional networks.

3. The method for fusing infrared images and visible light images according to claim 2, characterized in that: In step S1, the image reconstruction network includes two first convolution modules and a second convolution module connected in sequence, the first convolution module includes a third convolution layer and a LeakyRelu activation function, and the second convolution module includes a fourth convolution layer and a Tanh activation function.

4. The method for fusing an infrared image and a visible light image according to claim 1, 2 or 3, characterized in that: In the step S1, the fusion module multiplies the infrared image feature and the visible light image feature in the channel dimension, and then adds the complementary feature to generate the fusion feature.

5. The method for fusing an infrared image and a visible light image according to claim 1, 2 or 3, characterized in that: In step S2, the infrared image and the corresponding visible light image are input into a multimodal semantic segmentation neural network, and a semantic heat map mask is generated through a reverse gradient propagation mechanism to form a semantic heat map mask area of ​​the infrared image and the corresponding visible light image.

6. The method for fusing infrared images and visible light images according to claim 5, characterized in that: In step S2, the loss function is expressed as ,in, represents the structural similarity loss between the visible light image and the fused image, represents the structural similarity loss of the semantic heat map mask region based on the infrared image and the corresponding visible light image, represents the image intensity loss, represents the image gradient loss, and Indicates the weights calculated based on the structural similarity after the infrared image and the corresponding visible light image are input into the multimodal semantic segmentation neural network and the convolution feature map is visualized through forward propagation.

7. The method for fusing infrared images and visible light images according to claim 6, characterized in that: The structural similarity loss between the visible light image and the fused image is expressed as ,in, represents a visible light image, represents the fused image, , represents the mean of the sliding window set in the visible light image, represents the mean of the sliding window set in the fused image, for and The covariance between and is a constant, for The corresponding variance, for The corresponding variance, The value range is [-1,1].

8. The method for fusing infrared images and visible light images according to claim 7, characterized in that: The structural similarity loss of the semantic heat map mask area based on the infrared image and the corresponding visible light image is expressed as ,in, represents an infrared image, represents the fused image, , H is the height of the fused image, W is the width of the fused image, For SSIM( ) is the numerical value of the structural similarity of the pixel coordinates (i, j) after calculation, The value of the mask for pixel coordinate (i, j).

9. The method for fusing infrared images and visible light images according to claim 6, characterized in that: The image intensity loss is expressed as ,in, represents the pixel coordinates (i, j) of the infrared image, represents the pixel coordinates (i, j) of the visible light image, represents the pixel coordinates (i, j) of the fused image, H and W represent the height and width of the infrared image, visible light image, and fused image, respectively. Indicates absolute value.

10. The method for fusing infrared images and visible light images according to claim 6, characterized in that: The image gradient loss is expressed as ,in, represents the gradient of the image, represents the pixel coordinates (i, j) of the infrared image, represents the pixel coordinates (i, j) of the visible light image, represents the pixel coordinates (i, j) of the fused image, H and W represent the height and width of the infrared image, visible light image, and fused image, respectively. Indicates absolute value.

Images