Infrared and visible light image fusion target detection method based on diffusion generation driving

Abstract

The invention belongs to the technical field of target detection of computer vision, and discloses an infrared and visible light image fusion target detection method based on diffusion generation drive, which comprises the following steps of: constructing a data set for conditional diffusion model training based on a disclosed infrared and visible light image data set M3FD and a low light and normal light image data set LOL; pre-training the conditional diffusion model for the low-quality image data to obtain collaborative diffusion generation priori of the low-quality image; a prior fusion image is generated based on collaborative diffusion, and fusion features are obtained; and performing target detection based on the generated drive fusion image to generate a final target detection result. According to the infrared and visible light image fusion target detection method based on diffusion generation driving, the noise influence caused by low-quality images is weakened, the low-quality infrared and visible light images can be effectively processed, and accurate target detection is carried out.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection in computer vision, and in particular to a target detection method based on infrared and visible light image fusion driven by diffusion generation. Background Art

[0002] As an important research direction in the field of computer vision, target detection has a wide range of applications in many fields of real life. However, in many practical scenarios such as security monitoring and intelligent driving, it is often necessary to maintain efficient target detection capabilities under different weather conditions during the day and night. At the same time, the quality of the collected images may be low due to limitations such as the environment and equipment. Faced with such challenges, target detection methods based on high-quality images collected by a single sensor can no longer meet the needs. Therefore, a large number of target detection methods based on multi-sensor data have become a new research hotspot, which to a certain extent make up for the shortcomings of limited information collected by a single sensor and improve the accuracy of target detection.

[0003] Currently, the most typical and common multi-sensor fusion source is the combination of infrared and visible light images. By combining the heat source target information perceived by the infrared sensor with the object reflected light imaging captured by the visible light sensor, the two sensors can complement each other's advantages, effectively integrate, synthesize and explore the feature information of the two modal images, highlight the target while enhancing the understanding of the scene, and realizing target detection in all weather and all scenarios.

[0004] Although existing infrared and visible light image fusion target detection methods have achieved some success in the fusion of multi-sensor image data, these methods mainly rely on data from ideal scenes and acquisition conditions for training. Although some methods focus on image fusion target detection under low-light conditions, they still face challenges in detection technology under the combined influence of multiple factors. When dealing with challenging application scenarios, such as complex environments caused by multiple factors such as noise, low contrast, low resolution, and low light, they often perform poorly. Summary of the invention

[0005] The purpose of the present invention is to provide a target detection method for infrared and visible light image fusion based on diffusion generation drive, which can effectively process low-quality infrared and visible light images and perform accurate target detection.

[0006] To achieve the above object, the present invention provides a method for detecting infrared and visible light image fusion targets based on diffusion generation drive, comprising the following steps: Step S1, based on the public infrared and visible light image dataset M3FD and the low-light and normal light image dataset LOL, construct a low-quality infrared image dataset, a high-quality infrared image dataset, a low-quality visible light image dataset, and a high-quality visible light image dataset to obtain a dataset for conditional diffusion model training; Step S2, pre-training the conditional diffusion model for low-quality image data to obtain a collaborative diffusion generation prior for low-quality images; Step S3, generating a priori fusion image based on cooperative diffusion to obtain fusion features; Step S4: Based on the generated driving fusion image, target detection is performed to generate a final target detection result.

[0007] Preferably, in step S1, a low-quality infrared and visible light image dataset for training is constructed using a public infrared and visible light image dataset M3FD and a low-light and normal light image dataset LOL; Among them, all infrared images of the M3FD dataset are subjected to simulated degradation processing to construct a low-quality infrared image dataset with noise and low resolution characteristics. The specific process is as follows: By using a downsampling factor of 0.25, the image resolution is reduced to a quarter of the original image, and stripe noise is added to the downsampled image. The noise intensity is set to ,in It is a measure of noise intensity; The original infrared images of the M3FD dataset correspond to the high-quality infrared image dataset; The low-light images of the LOL dataset are directly used as a low-quality visible light image dataset with low resolution and low-light characteristics; The normal light images of the LOL dataset correspond to the high-quality infrared image dataset.

[0008] Preferably, in step S2, a conditional diffusion model for low-quality image data is pre-trained to obtain a collaborative diffusion generation prior for low-quality images. The specific process is as follows: Step S21: First, define a low-quality image as follows: ; in, Represents low-quality image samples; Including infrared images and visible light images ; Second, define a high-quality image as follows: ; in, Represents a high-quality image sample; Step S22: In the forward diffusion stage, define the distribution of data as follows: ; in, Follow the distribution ; Represents a raw data; Represents the original data The probability distribution of Step S23: gradually inject Gaussian noise in the forward diffusion stage to achieve diffusion time steps gradually destroy the original data ; is the total number of steps in the entire diffusion process, which is set to 1000; Step S24: predefine the Markov chain to satisfy the following distribution: ; in, is the state of the sample at time step t; represents normal distribution; represents a probability distribution; and is the noise adjustment parameter in the diffusion process, which is controlled in the The intensity of adding noise in each time step is related to ; represents the identity matrix; At the same time, the marginal distribution is derived as follows: ; in, represents the original data sample; , ;when Trending time step hour, tends to 0, Normal distribution , and the forward process ends at the same time; Step S25, reverse process from Gaussian noise The probability distribution of the generated image is shown below after Markov chain denoising step by step: ; in, Indicates that the condition and Next, time step A random variable The probability distribution of are the parameters of the model; variance is a time-dependent constant; is at the time step The predicted mean value at ; and The calculation formula is as follows: ; ; Among them, the state of the sample at time step t With low quality images After splicing, the input parameters are Noise estimation network In , the degraded image will provide semantic and structural information for sample generation; Step S26, optimizing the target, as shown below: ; in, express expectations; Indicates that at time step the added noise vector; express The square of the norm; Step S27. Finally, in the inference stage of low-quality image restoration, the low-quality image will be used as a diffusion condition to guide the model to iterate step by step starting from standard Gaussian noise to generate a high-quality source image.

[0009] Preferably, in step S3, a priori fusion images are generated based on cooperative diffusion to obtain fusion features, and the specific process is as follows: Step S31: First, initialize the fused image sample from Gaussian noise ,definition , represents the fused image; the mean value during sampling Compute the update as follows: ; in, Representation includes low-quality images of infrared and visible light modalities; Represents the state of the fused image at time step t; Representatives and The mode-dependent High-quality samples; is the combined weight in the reverse process and satisfies ; By taking the current fusion sample , high quality samples , and the weight of the previous step Combined and input into the weight prediction network It is calculated as follows: ; in, For the The mode at the time step The unnormalized weights of Step S32: The mode at the time step The unnormalized weights of Standardize and get the final combined weight , as shown below: ; in, Indicates The mode at the time step The unnormalized weights e exponential form; at the same time, in this image task, since there are only two modalities, the weight of the corresponding other modality is directly defined as ; Step S33: Weight prediction network The loss function , as shown below: ; in, Represents the time step Predicted fused image; represents the maximum aggregation strategy; represents the gradient operator; express norm; It is a high-quality source image restored based on the diffusion model for low-quality images obtained through pre-training in the previous stage.

[0010] Preferably, in step S4, target detection is performed based on the generated drive fusion image to generate a final target detection result. The specific process is as follows: After the image fusion step based on the combined prior until the network converges, a clear fusion feature is finally obtained. , the fusion features Incoming task header To extract task-specific information, the task head uses the YOLOv5 architecture; Finally, after post-processing, the final prediction result is obtained. , as shown below: ; in, Represents the task head The post-processing process includes the category label, corresponding bounding box coordinates and confidence score of each detected object.

[0011] Therefore, the present invention adopts the above-mentioned infrared and visible light image fusion target detection method based on diffusion generation drive, which reduces the noise impact brought by low-quality images, can effectively process low-quality infrared and visible light images, and perform accurate target detection.

[0012] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a flow chart of the infrared and visible light image fusion target detection method based on diffusion generation drive of the present invention; Figure 2 It is a flow chart of pre-training a conditional diffusion model for low-quality image data according to the present invention; Figure 3 It is a flow chart of image fusion and target detection based on collaborative diffusion to generate priors in the present invention. DETAILED DESCRIPTION

[0014] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0015] like Figure 1 As shown, the infrared and visible light image fusion target detection method based on diffusion generation drive includes the following steps: Step S1, based on the public infrared and visible light image dataset M3FD and the low-light and normal light image dataset LOL, construct a low-quality infrared image dataset, a high-quality infrared image dataset, a low-quality visible light image dataset, and a high-quality visible light image dataset to obtain a dataset for conditional diffusion model training; Step S2, pre-training the conditional diffusion model for low-quality image data to obtain a collaborative diffusion generation prior for low-quality images; Step S3, generating a priori fusion image based on cooperative diffusion to obtain fusion features; Step S4: Based on the generated driving fusion image, target detection is performed to generate a final target detection result.

[0016] Example Step S1: Based on the public infrared and visible light image dataset M3FD and the low-light and normal light image dataset LOL, construct a low-quality infrared image dataset, a high-quality infrared image dataset, a low-quality visible light image dataset, and a high-quality visible light image dataset to obtain a dataset for conditional diffusion model training.

[0017] The public infrared and visible light image dataset M3FD and low-light and normal light image dataset LOL are used. Both datasets include various scenes in a variety of real environments. In order to make the method proposed in this method more effective in reducing the noise impact brought by low-quality images, in the image preprocessing process, low-quality infrared and visible light image datasets for training will be constructed first.

[0018] Among them, all infrared images of the M3FD dataset are subjected to simulated degradation processing to construct a low-quality infrared image dataset with mainly noise and low resolution characteristics; the original infrared images correspond to the high-quality infrared image dataset. Specifically, by using a downsampling factor of 0.25, the image resolution is reduced to a quarter of the original image, and stripe noise is added to the downsampled image, and the noise intensity is set to ,in It is an indicator to measure the noise intensity. The larger the value, the stronger the noise.

[0019] The low-light images of the LOL dataset have the characteristics of low contrast and low illumination, and can be directly used as a low-quality visible light image dataset with mainly low resolution and low illumination characteristics; the normal light images of the LOL dataset correspond to a high-quality visible light image dataset.

[0020] Step S2: pre-training the conditional diffusion model for low-quality image data to obtain the collaborative diffusion generation prior of the low-quality image.

[0021] In the training phase of low-quality image restoration, the two infrared and visible light image datasets constructed in the previous phase are used to train a denoising diffusion probability model with image restoration capabilities, such as Figure 2 shown.

[0022] Step S21: First, define a low-quality image as follows: ; in, Represents low-quality image samples; Including infrared images and visible light images .

[0023] Second, define a high-quality image as follows: ; in, Represents a high-quality image sample.

[0024] Step S22: In the forward diffusion stage, define the distribution of data as follows: ; in, Follow the distribution ; Represents a raw data; Represents the original data The probability distribution of .

[0025] Step S23: gradually inject Gaussian noise in the forward diffusion stage to achieve diffusion time steps gradually destroy the original data ; is the total number of steps in the entire diffusion process, which is set to 1000.

[0026] Step S24: predefine the Markov chain to satisfy the following distribution: ; in, is the state of the sample at time step t; represents normal distribution; represents a probability distribution; and is the noise adjustment parameter in the diffusion process, which is controlled in the The intensity of adding noise in each time step is related to ; Represents the identity matrix.

[0027] At the same time, the marginal distribution is derived as follows: ; in, represents the original data sample; , ;when Trending time step hour, tends to 0, Normal distribution , and the forward process ends.

[0028] Step S25, reverse process from Gaussian noise The probability distribution of the generated image is shown below after Markov chain denoising step by step: ; in, Indicates that the condition and Next, time step A random variable The probability distribution of are the parameters of the model; variance is a time-dependent constant; is at the time step The predicted mean value at .

[0029] and The calculation formula is as follows: ; ; Among them, the state of the sample at time step t With low quality images After splicing, the input parameters are Noise estimation network In this paper, the degraded image will provide semantic and structural information for sample generation.

[0030] Step S26, optimizing the target, as shown below: ; in, express expectations; Indicates that at time step the added noise vector; express The square of the norm.

[0031] Step S27. Finally, in the inference stage of low-quality image restoration, the low-quality image will be used as a diffusion condition to guide the model to iterate step by step starting from standard Gaussian noise and finally generate a high-quality source image.

[0032] Step S3: Generate a priori fusion image based on cooperative diffusion to obtain fusion features.

[0033] In the image fusion generation stage, after pre-training the diffusion model for low-quality images, the prior distribution of unimodal high-quality images is obtained. The prior is determined by the noise predicted at each sampling step, and combining the noise from multiple diffusion models can generate a result that contains their respective prior distributions.

[0034] Therefore, by flexibly combining the noise generated by the pre-trained diffusion model, image fusion based on the generative model is achieved, such as Figure 3 shown.

[0035] Step S31: First, initialize the fused image sample from Gaussian noise ,definition , represents the fused image. The mean value during sampling Compute the update as follows: ; in, Representation includes low-quality images of infrared and visible light modalities; Represents the state of the fused image at time step t; Representatives and The mode-dependent High-quality samples; is the combined weight in the reverse process and satisfies .

[0036] By taking the current fusion sample , high quality samples , and the weight of the previous step Combined and input into the weight prediction network It is calculated as follows: ; in, For the The mode at the time step The unnormalized weights of .

[0037] Step S32: Then, The mode at the time step The unnormalized weights Standardize and get the final combined weight , as shown below: ; in, Indicates The mode at the time step The unnormalized weights e exponential form; at the same time, in this image task, since there are only two modalities, the weight of the corresponding other modality can be directly defined as .

[0038] Step S33: Weight prediction network The loss function , as shown below: ; in, Represents the time step Predicted fused image; represents the maximum aggregation strategy; represents the gradient operator; express norm; It is a high-quality source image restored based on the diffusion model for low-quality images obtained through pre-training in the previous stage.

[0039] Step S4: Based on the generated driving fusion image, target detection is performed to generate a final target detection result.

[0040] After the image fusion step based on the combined prior until the network converges, a clear fusion feature can be finally obtained. , the fusion features Incoming task header To extract task-specific information, the task head uses the YOLOv5 architecture.

[0041] Finally, after post-processing, the final prediction result is obtained. , as shown below: ; in, Represents the task head The post-processing process includes the category label, corresponding bounding box coordinates and confidence score of each detected object.

[0042] Therefore, the present invention adopts the above-mentioned infrared and visible light image fusion target detection method based on diffusion generation drive, which reduces the noise impact brought by low-quality images, can effectively process low-quality infrared and visible light images, and perform accurate target detection.

[0043] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A target detection method based on infrared and visible light image fusion driven by diffusion generation, characterized in that: The following steps are involved: Step S1, based on the public infrared and visible light image dataset M3FD and the low-light and normal light image dataset LOL, construct a low-quality infrared image dataset, a high-quality infrared image dataset, a low-quality visible light image dataset, and a high-quality visible light image dataset to obtain a dataset for conditional diffusion model training; Among them, all infrared images of the M3FD dataset are subjected to simulated degradation processing to construct a low-quality infrared image dataset with noise and low resolution characteristics. The specific process is as follows: By using a downsampling factor of 0.25, the image resolution is reduced to a quarter of the original image, and stripe noise is added to the downsampled image. The noise intensity is set to ,in It is a measure of noise intensity; The original infrared images of the M3FD dataset correspond to the high-quality infrared image dataset; The low-light images of the LOL dataset are directly used as a low-quality visible light image dataset with low resolution and low-light characteristics; The normal light images of the LOL dataset correspond to the high-quality infrared image dataset; Step S2, pre-training the conditional diffusion model for low-quality image data to obtain a collaborative diffusion generation prior for low-quality images; Step S3, generating a priori fusion image based on cooperative diffusion to obtain fusion features; Step S4: Based on the generated driving fusion image, target detection is performed to generate a final target detection result.

2. The infrared and visible light image fusion target detection method based on diffusion generation drive according to claim 1 is characterized in that: In step S2, the conditional diffusion model for low-quality image data is pre-trained to obtain the co-diffusion generation prior of low-quality images. The specific process is as follows: Step S21: First, define a low-quality image as follows: ； in, Represents low-quality image samples; Including infrared images and visible light images ; Second, define a high-quality image as follows: ； in, Represents a high-quality image sample; Step S22: In the forward diffusion stage, define the distribution of data as follows: ； in, Follow the distribution ; Represents a raw data; Represents the original data The probability distribution of Step S23: gradually inject Gaussian noise in the forward diffusion stage to achieve diffusion time steps gradually destroy the original data ; is the total number of steps in the entire diffusion process, which is set to 1000; Step S24: predefine the Markov chain to satisfy the following distribution: ； in, is the state of the sample at time step t; represents normal distribution; represents a probability distribution; and is the noise adjustment parameter in the diffusion process, which is controlled in the The intensity of adding noise in each time step is related to ; represents the identity matrix; At the same time, the marginal distribution is derived as follows: ； in, represents the original data sample; , ;when Trending time step hour, tends to 0, Normal distribution , and the forward process ends at the same time; Step S25, reverse process from Gaussian noise The probability distribution of the generated image is shown below after Markov chain denoising step by step: ； in, Indicates that the condition and Next, time step A random variable The probability distribution of are the parameters of the model; variance is a time-dependent constant; is at the time step The predicted mean value at ; and The calculation formula is as follows: ； ； Among them, the state of the sample at time step t With low quality images After splicing, the input parameters are Noise estimation network In , the degraded image will provide semantic and structural information for sample generation; Step S26, optimizing the target, as shown below: ； in, express expectations; Indicates that at time step the added noise vector; express The square of the norm; Step S27. Finally, in the inference stage of low-quality image restoration, the low-quality image will be used as a diffusion condition to guide the model to iterate step by step starting from standard Gaussian noise to generate a high-quality source image.

3. The infrared and visible light image fusion target detection method based on diffusion generation drive according to claim 1 is characterized in that: In step S3, a priori fusion image is generated based on cooperative diffusion to obtain fusion features. The specific process is as follows: Step S31: First, initialize the fused image sample from Gaussian noise ,definition , represents the fused image; the mean value during sampling Compute the update as follows: ； in, Representation includes low-quality images of infrared and visible light modalities; Represents the state of the fused image at time step t; Representatives and The mode-dependent High-quality samples; is the combined weight in the reverse process and satisfies ; By taking the current fusion sample , high quality samples , and the weight of the previous step Combined and input into the weight prediction network It is calculated as follows: ； in, For the The mode at the time step The unnormalized weights of Step S32, then, The mode at the time step The unnormalized weights Standardize and get the final combined weight , as shown below: ； in, Indicates The mode at the time step The unnormalized weights e exponential form; at the same time, in this image task, since there are only two modalities, the weight of the corresponding other modality is directly defined as ; Step S33: Weight prediction network The loss function , as shown below: ； in, Represents the time step Predicted fused image; represents the maximum aggregation strategy; represents the gradient operator; express norm; It is a high-quality source image restored based on the diffusion model for low-quality images obtained through pre-training in the previous stage.

4. The infrared and visible light image fusion target detection method based on diffusion generation drive according to claim 1 is characterized in that: In step S4, target detection is performed based on the generated drive fusion image to generate the final target detection result. The specific process is as follows: After the image fusion step based on the combined prior until the network converges, the fusion feature is finally obtained. , the fusion features Incoming task header To extract task-specific information, the task head uses the YOLOv5 architecture; Finally, after post-processing, the final prediction result is obtained. , as shown below: ； in, Represents the task head The post-processing process includes the category label, corresponding bounding box coordinates and confidence score of each detected object.

Images