An infrared and visible light image fusion method based on a diffusion model

Abstract

The application discloses an infrared light and visible light image fusion method based on a diffusion model, and relates to the technical field of image fusion.The application is designed based on a denoising diffusion probability model (DDPM), in order to accurately predict noise and efficiently train the model, a light-weight U-Net architecture is designed as a conditional noise predictor, starting from a pure noise image, taking infrared light and visible light images as conditions, using DDPM to perform multiple denoising operations through iteration, and restoring the distribution of the fusion image, so that the purpose of fusing the infrared light and the visible light images is achieved.This method does not depend on various specific fusion rules and complex feature extraction networks, and overcomes many algorithm design and training difficulties in the existing fusion methods.The experimental results show that the fusion method proposed in the application can generate high-quality infrared light and visible light fusion images.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a method for fusing infrared light images and visible light images. BACKGROUND

[0002] Due to the hardware limitations of imaging devices, a sensor under a single type or a single setting cannot fully represent the imaging scene. The image captured by a visible light camera conforms to human visual perception and has rich texture details and spatial resolution, but in poor environmental conditions such as insufficient light, rain and fog, the imaging quality will decrease significantly, making it difficult to identify target information. Thermal infrared camera imaging relies on the thermal radiation information of the target and has better smoke penetration ability and is not affected by light conditions, and is suitable for all-weather conditions, such as performing night mountain or fire scene rescue tasks, when the thermal infrared camera has great advantages, but the image itself lacks details and cannot reflect scene information. Therefore, there are great limitations in relying on single-class images to perform visual tasks. How to use image fusion technology to organically integrate the image information collected under different sensors to generate images with stronger robustness, more abundant information and better human eye perception has become a hot spot in the field of image processing.

[0003] In order to solve the challenges in image fusion and avoid manual and tedious fusion scheme design. Among them, the method based on generative adversarial network (GAN) stands out as an end-to-end method with excellent performance. The workflow of the GAN-based fusion method mainly involves a generator that generates a fusion image by integrating information from different domains, and a discriminator that is responsible for evaluating the similarity of the probability distribution between the generated image and the source image. Although the GAN-based image fusion model has produced satisfactory fusion images, they are plagued by several problems, including unstable training process and mode collapse. These challenges result in unreasonable distribution and low quality of the generated fusion images, which seriously affects the practicality of the GAN-based method. In recent years, the denoising diffusion probability model (DDPM) has attracted much attention. It has good mathematical interpretability and high-quality generation results, and has achieved great success in image generation. However, in multi-modal image fusion, the diffusion model cannot achieve direct image fusion due to the lack of true values. This is a great challenge for the current application of diffusion models in multi-modal image fusion. SUMMARY

[0004] The purpose of the present application is to provide an image fusion method based on a diffusion model for infrared light images and visible light images.

[0005] The technical scheme adopted by the present application is as follows:

[0006] A method for fusing infrared light and visible light images, specifically:

[0007] Constructing a dataset, including registered infrared image I ir , visible light image I vi and fusion prior image F0, wherein the fusion prior image represents the general result of image fusion, obtained by the target search function f ts (F j ) for the training process of the model.

[0008] Forward noise adding, reverse denoising diffusion process is performed on the fusion prior image F0:

[0009] In the forward process, the fusion prior image F0 is repeatedly added with Gaussian noise through the variance table [β1, β2, β3,..., β T ] determined by the variance value range, and F0 will gradually degenerate into a Gaussian noise image F T within T time steps, where T is defined as the maximum time step, t ∈ [1, T], T = 2000; this forward diffusion process q is formulated as:

[0010]

[0011] Wherein α t = 1-β t , N represents the Gaussian normal distribution; through the repeated superposition step of the Gaussian noise α t of small variance, F0 gradually becomes a Gaussian image F t similar to the standard Gaussian form;

[0012] In the reverse diffusion process, the infrared image I ir and the visible light image I vi are introduced as input conditions of the diffusion model to reconstruct the fusion prior image F0 from the Gaussian image F t , and a neural network p θ with mean μ θ is used for each reverse diffusion step and is written as:

[0013]

[0014] Wherein the mean μ θ is formulated as:

[0015]

[0016] Wherein ε θ is the noise of the estimated output of the neural network p θ ; the role of the neural network p θ is to learn the modeling of the Gaussian normal distribution N at each time step; at the same time, the neural network takes the infrared image Iir and visible light image I vi As a condition, to reconstruct the fusion prior image F0, the learning goal is established as follows:

[0017] L t =||ε t -ε θ (F t ,I ir ,I vi ,t)|| 2

[0018] Through continuous training, the value of the learning goal L t is reduced.

[0019] The trained diffusion model is inputted with an infrared visible light image pair of any resolution, and the diffusion model automatically outputs a fusion image that retains the rich texture and color information of the visible light image and the prominent target information of the infrared image through continuous iteration.

[0020] The size of the input picture sample is (H, W, C), where H represents the height of the picture, W represents the width of the picture, and C represents the number of channels of the picture. The infrared image I ir , the visible light image I vi and the Gaussian image F t are spliced together in the channel dimension as the input of the neural network noise predictor.

[0021] The noise predictor epsilon θ is a U-net type neural network, which converts the number of feature map channels to C through a convolution module. epsilon θ includes a down sampling (Down Sample) module, a mid step (Mid Step) module and an up sampling (Up Sample) module. The down sampling and up sampling modules contain four steps, which are composed of several residual blocks, two down sampling layers and two up sampling layers. In this paper, two-dimensional convolution with a step of 2 is used to realize down sampling, and two-dimensional transpose convolution is used to realize up sampling. In addition, the output of each down sampling step is added to the input of the corresponding up sampling step to fully utilize the image features. For the time step t, the position embedding module in the Transformer is used to convert the time step t into a time embedding t e , and t e is added to each residual block. It is worth noting that compared with the noise predictor in the original paper of the diffusion model, the noise predictor in this paper is lightweight, which can significantly reduce the training cost.

[0022] Compared with the prior art, the present application has the following advantages:

[0023] 1. The application proposes an infrared and visible image fusion method based on diffusion model. The method creatively introduces a fusion prior image to guide the training in view of the lack of true value in image fusion tasks. The fusion knowledge prior is a general distribution of infrared and visible image fusion results, which contains fusion results from various strategies, and the fusion prior image is sampled from the distribution.

[0024] 2. The diffusion model network used in the application can fuse infrared and visible images to obtain a fusion image combining the information of the two modalities, which can help improve the performance of advanced vision tasks such as target tracking, target detection and semantic segmentation.

[0025] 3. The training framework proposed by the application can be extended to other related image fusion fields to solve similar problems and has universality. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 is the data set of the method.

[0027] Figure 2 is the forward noise adding process of the method.

[0028] Figure 3 is the architecture of the conditional noise predictor of the method.

[0029] Figure 4 is the image fusion schematic process of the method.

[0030] Figure 5 is the training process schematic diagram of the method.

[0031] Figure 6 is the inference process schematic diagram of the method.

[0032] Figure 7 is the schematic diagram of the fusion image generated by the method. DETAILED DESCRIPTION

[0033] The technical solutions of the application will be further described below in combination with the drawings. The meanings of the parameters are shown in Table 1.

[0034] Table 1

[0035]

[0036]

[0037] This embodiment takes infrared, visible and fusion prior images in the data set as examples to illustrate the image fusion method based on diffusion model.

[0038] As Figure 1As shown, the dataset used in this invention consists of registered infrared images, visible light images, and fused prior images. The infrared and visible light images are used as input to the noise predictor, and the fused prior images are used for the training process of the noise predictor.

[0039] like Figure 2 As shown, noise is continuously added to the fused prior image F0, using the formula... Obtain the noise image F at time t t As can be seen, with the continuous increase of time steps, the original fused prior image F0 is continuously "destroyed" and eventually becomes a Gaussian noise image F. t Our ultimate goal is to predict the noise added to the fused image at time t during the inference phase, using infrared and visible light images as conditions, thereby achieving the reconstruction of the fused image. For the infrared and visible light image pairs used for inference, the reconstruction process is the same as the image fusion process.

[0040] like Figure 3 As shown, infrared, visible light, and Gaussian noise images are stitched together and used as input to the noise predictor. The U-net-type network output predicts the noise ε added at time t. θ (F t ,I ir ,I vi ,t), and the noise ε added at time t. t For comparison, the noise predictor is trained by minimizing the following loss function:

[0041] L t =||ε t -ε θ (F t ,I ir ,I vi ,t)|| 2 .

[0042] like Figure 4 As shown, during the inference phase, infrared and visible light images, along with a noisy image F of the same size as them, are used. t-1 These are concatenated together and used as input to the trained noise predictor, which will output the predicted noise ε at time t. pred Through formula Obtain the Gaussian noise image F from the previous time step. t-1 The process continues iteratively until the fused image F0 is obtained.

[0043] The training process of this invention is as follows: Figure 5 As shown, the input is: the fused prior image F0 processed through the forward process of the diffusion model. The obtained noise image F t Infrared image Iir visible light image I vi The three pictures are spliced on the channel level. The denoising module (DM) is a conditional noise predictor ε θ (F t , I ir , I vi , t) is used to predict the noise ε added in the forward process t The learning target L is established t =||ε t -ε θ (F t , I ir , I vi , t)| 2 .

[0044] The inference stage of the present application is shown in Figure 6 , the input is: infrared image I ir , visible light image I vi and noise image F T , F T ~N(0, I). The trained model outputs the predicted noise ε at time T pred , and the denoised image F T-1 is obtained through formula . The model is iterated through formula , and finally outputs the fused image , thereby realizing the reconstruction of the fused prior image F0.

[0045] As shown in Figure 7 , under poor lighting conditions, the visible light image is degraded and it is difficult to capture the pedestrian target, while the infrared image can reflect the thermal radiation information and clearly show the position information of the pedestrian, but the texture information of the background is incomplete. The fusion method adopted by the present application can better combine the information of the two modal images, generate an image with stronger robustness, more abundant information and helpful to human eye perception, and improve the performance of other advanced vision tasks.

[0046] It is obvious to those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. The scope of the present application is defined by the appended claims rather than the above description, and therefore all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present application.

Claims

1. An infrared and visible light fusion method based on a diffusion model, characterized in that: Step one, constructing dataset, including a pair of registered infrared image I ir and visible light image I vi and their corresponding fusion prior image F0, wherein the fusion prior image represents the general result of image fusion, used in the training process of the model; the fusion prior image F0 is a pre-generated fusion image, as the true value of model training, so that the image fusion task changes from an unsupervised task to a supervised task; the distribution of the fusion image F0 is obtained through the target search function , wherein ω n is the weight of different indicators, T n represents the evaluation function, T n (F j ) represents the evaluation score of the given jth sample, F j is the sample image generated by multiple methods; Step two, forward noise adding and reverse noise removing diffusion processing are performed on the fusion prior image F0: In the forward process, the variance table [β1, β2,..., β t ,...,β T ] controls the amount of noise added at each time step, β t Gradually increasing with time step t, it means adding more noise at each time step; repeat adding Gaussian noise to the fused prior image F0, F0 will gradually degenerate into a Gaussian noise image F t , where T is defined as the maximum time step length, t ∈ [1, T], T = 1000; this forward diffusion process q is formulated as: where α t = 1 - β t , N denotes a Gaussian normal distribution, I is an identity matrix, is a coefficient in the diffusion process, which controls the noise intensity at each time step, denotes the noise variance in the forward diffusion process, which is calculated from α t . In the backward diffusion process, a fused prior image F0is reconstructed from the Gaussian image F by gradually denoising t ir vi and the visible light image Ias input conditions for the diffusion model, each backward diffusion step uses a neural network p θ θ written as:​​​ wherein Mean μ θ is formulated as: where ε θ is a neural network p θ estimating the noise of the output; the neural network p θ learns a modeling of a Gaussian normal distribution N at each time step; simultaneously this neural network is used to reconstruct a fused prior image F0 with the infrared image I ir and the visible light image I vi as conditions, establishing a learning goal as follows: L t = E(||ε t - ε θ (F t , I ir , I vi , t)| 2 ) By constant training, reduce the value of learning goal L t . Step three, output of the diffusion model network: An infrared and visible light image pair of any resolution is input into the trained diffusion model, and the diffusion model automatically outputs a fusion image which retains the rich texture and color information of the visible light image and the prominent target information of the infrared image through continuous iteration.

2. The method of claim 1, wherein, The size of the input picture sample is (H, W, C), wherein H represents the height of the picture, W represents the width of the picture, and C represents the number of channels of the picture; the input picture is spliced together in the channel dimension as the input of the neural network noise predictor.

3. The method of claim 2, wherein, Noise predictor ε θ is a U-net type neural network, which converts the channel number of the feature map to C through the convolution module; ε θ includes a down-sampling module, an intermediate module and an up-sampling module; the down-sampling and up-sampling modules contain four steps, which are composed of several residual blocks, two down-sampling layers and two up-sampling layers; a two-dimensional convolution with a step of 2 is used to realize down-sampling, and a two-dimensional transposed convolution is used to realize up-sampling; in addition, the output of each down-sampling step is added to the input of the corresponding up-sampling step to make full use of image features; for the time step t, the time step t is converted into a time embedding t e by using the position embedding module in the Transformer, and t e is added to each residual block.

Images