Image defogging method based on physical prior
By combining the Unet model with physical priors and a transmittance map-refined diffusion model, the efficiency and generalization problems of existing image dehazing methods under complex haze conditions are solved, achieving efficient and realistic image dehazing results.
Patent Information
- Application Number
- CN202411076740.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-08-07
AI Technical Summary
Existing image dehazing methods are ineffective in handling complex haze conditions, especially those based on physical models which are computationally complex and time-consuming. Deep learning methods require a large amount of labeled data and computational resources and have insufficient generalization ability, while prior-based methods lack versatility.
By employing the Unet model combined with physical priors, synthetic and real training sets are constructed, and a pre-trained physical model is used to generate preliminary dehazed images, transmittance images, and atmospheric light images of synthetic foggy images. Combined with transmittance map refinement and diffusion models, multiple rounds of denoising training are conducted, and a joint loss function is introduced to improve the model's generalization ability.
It effectively removes haze, restores clear images, improves dehazing efficiency, preserves the original image effect in hazy areas, generates visually more realistic images, and demonstrates good generalization performance on different datasets.
Smart Images

Figure CN119006339A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to image processing technology, and in particular to an image defogging method based on physical priori. Background Art
[0002] Due to the presence of haze, outdoor images often have low contrast and limited visibility, which is not conducive to the progress of subsequent computer vision tasks. Therefore, image dehazing technology has attracted more and more attention, and its purpose is to restore clear haze-free images from blurred hazy images.
[0003] In recent years, the image dehazing methods in traditional technologies are mainly divided into the following three types:
[0004] 1. Image dehazing based on physical model:
[0005] Image dehazing methods based on physical models mainly remove haze from images by simulating the scattering and absorption of light in the atmosphere. These methods provide a theoretically reliable framework that can generate results that are highly interpretable and easy to analyze.
[0006] However, physical models require accurate estimation of atmospheric light and transmittance, which may be difficult to implement in practice, especially when haze conditions are complex or highly variable. In addition, the calculation process of physical models is usually complex and may take a long time, which poses a challenge to real-time applications.
[0007] 2. Image dehazing based on priors:
[0008] Prior-based image dehazing methods use the statistical properties of clear images to estimate the transmission map, and then use the scattering model to remove haze. Such methods include dark channel prior, color attenuation prior, and color line prior, which infer the distribution of haze through specific image characteristics. These methods are relatively simple, easy to implement, and perform well in specific scenarios.
[0009] However, the hand-crafted prior assumptions are not always applicable to all haze images, which may lead to limitations in the dehazing effect. For images that do not meet the prior assumptions, these methods may perform poorly, and need to adjust the priors according to different types of images, which is not very universal.
[0010] 3. Image dehazing based on deep learning:
[0011] Image dehazing methods based on deep learning have become increasingly popular in recent years. These methods use a large amount of annotated data and powerful convolutional neural networks (CNNs) to automatically learn features for image dehazing. Deep learning methods can generate clear images directly from input hazy images without additional parameter estimation, and have excellent performance and strong versatility. They can handle complex haze conditions and a variety of image scenes and have strong robustness.
[0012] However, these methods require a large amount of labeled data for training, and the cost of data acquisition and labeling is high. In addition, the training time of deep learning models is long and requires high computing resources. Under certain extreme conditions, these methods may produce artifacts or distortions, affecting the overall quality of the dehazing effect.
[0013] In addition, when the above-mentioned image dehazing methods in traditional techniques are generalized to the dehazing task of real foggy images, the distribution gap between synthetic data and real data may cause a significant performance degradation of the model, that is, the model lacks generalization ability. Summary of the invention
[0014] The technical problem to be solved by the present invention is to propose an image defogging method based on physical priors to improve the effect of image defogging.
[0015] The technical solution adopted by the present invention to solve the above technical problems is:
[0016] A physical prior-based image defogging method comprises: inputting a photographed foggy image to be processed into a trained denoising model to obtain a denoised fog-free image; the denoising model adopts a Unet model, which is characterized in that:
[0017] The training of the denoising model includes the following steps:
[0018] A1. Constructing a synthetic training set:
[0019] Extracting a real fog-free image; obtaining a synthetic foggy image by using a synthesis method based on the real fog-free image; generating a preliminary defogging picture, a transmittance picture and an atmospheric light picture of the synthetic foggy image by using a pre-trained physical model based on the synthetic foggy image; forming a synthetic sample from the corresponding real fog-free image, the synthetic foggy image and the preliminary defogging picture, the transmittance picture and the atmospheric light picture of the synthetic foggy image, and forming a synthetic training set from the synthetic samples;
[0020] A2, inputting a synthetic sample, and randomly selecting a round less than or equal to K as a calculation round, where K is the number of denoising iterations of a preset denoising model; generating random noise, and using the denoising model to denoise the preliminary defogging image of the input synthetic sample using the random noise to obtain a noisy image of the calculation round;
[0021] A3, inputting the noisy image obtained in step A2 and the preliminary defogging image and transmittance image of the corresponding synthetic sample into the denoising model to obtain the calculation noise and calculation transmittance image of the calculation round; the calculation transmittance image is obtained by the output of the upsampling part through the fully connected layer, and the calculation noise is obtained by the denoising model input and output calculation;
[0022] A4, using the calculated transmittance image obtained in step A3 and the preliminary defogging image and atmospheric light image of the corresponding synthetic sample, using the pre-trained physical model used to construct the training set, to obtain a reconstructed image of the calculation round;
[0023] A5. Calculate the reconstruction loss using the reconstructed image obtained in step A4 and the synthetic foggy image of the corresponding synthetic sample; calculate the noise prediction loss using the calculated noise obtained in step A3 and the random noise generated in step A2; and update the denoising model using the reconstruction loss and the noise prediction loss;
[0024] A6. Generate a random noise image D 0 , and use the denoising model updated in step A5 as the initial model to derive the noise map D 0 Perform K rounds of denoising to obtain the denoising result D K ; Each round of denoising includes:
[0025] A61, input the denoising result D obtained in the k-1th round k , input it and the preliminary defogging image and transmittance image of the corresponding synthetic sample into the denoising model to obtain the calculated noise and calculated transmittance image of the kth round;
[0026] A62, using the noise model, based on the denoising result D input in step A61 k and the k-th round of calculation noise obtained in step A61 to generate the k-th round of denoising result D k+1 ;
[0027] A7. Denoising result D obtained using step A6 K And the corresponding real fog-free image, calculate the denoising loss, and update the denoising model;
[0028] A8. Determine whether the preset conditions are met. If so, obtain a trained denoising model; otherwise, return to step A2.
[0029] Furthermore, step A1 also includes constructing a real training set, including:
[0030] Extract the real foggy image corresponding to the real fog-free image, and generate the preliminary defogging picture, transmittance picture and atmospheric light picture of the real foggy image based on the real foggy image by using the pre-trained physical model; the corresponding real foggy image, the real foggy image and the preliminary defogging picture, transmittance picture and atmospheric light picture of the real foggy image constitute the real sample, and the real sample constitutes the real training set;
[0031] The training of the denoising model also includes:
[0032] A9. Use the trained denoising model obtained in step A8 as the first model; copy the first model and fine-tune its parameters to use it as the second model;
[0033] A10, input the real sample, and randomly select a round less than or equal to K as the calculation round; generate random noise, and use the random noise to add noise to the preliminary defogging image of the input real sample through the noise adding model to obtain the noisy image of the calculation round;
[0034] A11, input the noisy image obtained in step A10 and the preliminary defogging image and transmittance image of the corresponding real sample into the first model to obtain the first feature map of the calculation round; input into the second model to obtain the calculated noise, calculated transmittance image and second feature map of the calculation round; the first feature map and the second feature map are the outputs of the upsampling parts of the first model and the second model respectively;
[0035] A12, using the calculated transmittance image obtained in step A11 and the preliminary defogging image and atmospheric light image of the corresponding real sample, using the pre-trained physical model used to construct the training set, to obtain a reconstructed image of the calculation round;
[0036] A13. Calculate the joint loss and update the second model; the joint loss includes reconstruction loss, noise prediction loss and memory loss; the reconstruction loss is calculated using the reconstructed image obtained in step A12 and the real foggy image of the corresponding real sample; the noise prediction loss is calculated using the calculated noise obtained in step A11 and the random noise generated in step A10; the memory loss is calculated using the first feature map and the second feature map obtained in step A11;
[0037] A14. Generate a random noise image D 0 , and use the second model updated in step A13 as the initial model to analyze the noise map D 0 Perform K rounds of denoising to obtain the denoising result D K ; Each round of denoising includes:
[0038] A141, input the denoising result D of the k-1th round k, input it and the preliminary dehazed image and transmittance image of the corresponding real sample into the second model to obtain the calculated noise and calculated transmittance image of the kth round;
[0039] A142, using the noise model, based on the denoising result D input in step A141 k and the calculated noise of the kth round obtained in step A141 to generate the denoising result D of the kth round k+1 ;
[0040] A15. Denoising result D obtained using step A14 K and the corresponding real haze-free image, calculate the denoising loss, and update the second model;
[0041] A16. Determine whether the preset condition is met. If so, use the second model that has completed training as the final denoising model; otherwise, return to step A10.
[0042] Furthermore, the pre-trained physical model used to construct the training set is the SLP model.
[0043] Furthermore, random noise is generated, and the preliminary defogging image is denoised using the random noise through a denoising model to obtain a noisy image of calculation round k, including:
[0044] First, generate a random noise that follows a standard normal distribution; then, add noise according to the following formula:
[0045]
[0046] Among them, α k is a constant hyperparameter of the kth round, which gradually increases with the increase of round k within the set value range; J represents the corresponding preliminary dehazed image, and ε represents random noise.
[0047] Furthermore, according to the following formula, using the noise model, based on the denoising result D obtained in the k-1th round k and the calculated noise of the kth round, generating the initial denoising result D of the kth round k+1 :
[0048]
[0049] Among them, α k is the constant hyperparameter of the kth round, which gradually increases with the increase of round k within the set value range; e k represents the computational noise of the kth round.
[0050] Furthermore, in step A62, if the current round is the first round, the calculated transmittance image obtained in step A61 is used as the weight, and based on the generated denoising result Dk+1 And the initial defogging picture of the corresponding sample, the generated denoising result D k+1 Update and obtain the updated denoising result D k+1 ;
[0051] In step A142, if the current round is the first round, the calculated transmittance image obtained in step A141 is used as the weight, and the generated denoising result D k+1 And the initial defogging picture of the corresponding sample, the generated denoising result D k+1 Update and obtain the updated denoising result D k+1 .
[0052] Furthermore, the calculated transmittance image is used as the weight, based on the generated denoising result D k+1 And the initial defogging image of the corresponding sample, the generated denoising result D is calculated according to the following formula k+1 Update and obtain the updated denoising result D k+1 :
[0053] D k+1 =D k+1 *(1-T k )+J*T k
[0054] Among them, D on the left side of the equal sign k+1 is the denoising result of the kth round after the update, and D on the right side of the equal sign k+1 is the denoising result of the kth round generated by the denoising model, T k represents the calculated transmittance image of the kth round, and J represents the corresponding preliminary dehazed image.
[0055] Furthermore, the reconstruction loss L rec for:
[0056] L rec =||I p ,I|| 2
[0057] Among them, ||·|| 2 represents the L2 loss function; I p represents the reconstructed image, and I represents the foggy image of the corresponding sample;
[0058] The noise prediction loss L dif for:
[0059] L dif =||e k ,ε|| 2
[0060] Among them, ||·|| 2represents the L2 loss function; e k represents computational noise, and ε represents random noise;
[0061] The denoising loss L de for:
[0062] L de =||D K , J 0 || 2
[0063] Among them, ||·|| 2 represents the L2 loss function; D K represents the denoising result obtained by the denoising model after K rounds of denoising, J 0 Represents a real haze-free image.
[0064] Furthermore, the memory loss L mem for:
[0065] L mem =||f t1 , f t2 || 2
[0066] Among them, ||·|| 2 represents the L2 loss function; f t1 represents the first feature map, f t2 Represents the first feature map.
[0067] Furthermore, in step A13, the joint loss also includes a dark channel prior loss L dcp and the bright channel prior loss L bcp ;
[0068] The dark channel prior loss L dcp for:
[0069]
[0070] The bright channel prior loss L bcp for:
[0071] L bcp =||T b , T t2 || 2
[0072] Among them, T d represents the transmission map of the real image obtained based on the dark channel prior derivation, Represents T d The Laplace matrix, T brepresents the transmission map of the real image obtained based on the bright channel prior derivation, γ is a hyperparameter, and the superscript T represents the matrix transpose; T t2 It represents the calculated transmittance image obtained by step A11; ||·|| 2 Represents the L2 loss function;
[0073] The T d The derivation of T includes: first, calculating the minimum value of the RGB components of each pixel in the image; then, using the corrosion operation to solve the minimum value of the preset window centered on each pixel, so as to obtain the dark channel image; the T b The derivation includes: first, calculating the maximum value of the RGB components of each pixel in the image; then, using the expansion operation to solve the maximum value of a preset window centered on each pixel, thereby obtaining a bright channel image.
[0074] The beneficial effects of the present invention are:
[0075] The present invention organically combines the image defogging task with the transmittance map, refinement perception and diffusion model.
[0076] First of all, the transmittance map is crucial for defogging, indicating the exact location of the haze area. The present invention models defogging as a joint task to pursue shadow-free images and refinement of the transmittance map, wherein the refinement of the transmittance map is used as an auxiliary task of the defogging task and is divided into two stages: in the first stage, the transmittance picture of the foggy image is estimated using a pre-trained physical model; in the second stage, the diffusion model is used to supplement the information. Therefore, the transmittance map can be fully utilized to reflect the fog concentration information of different areas in the image, and the areas with dense fog are processed intensively to restore the details lost in the areas with dense fog, which can not only improve the efficiency of defogging, but also retain the original image effect of the areas with light fog, avoiding information loss and affecting the image quality after defogging.
[0077] Secondly, through the excellent ability of the diffusion model in image generation, it is possible to generate details lost in the foggy areas of the image, making the dehazed image more perceptually realistic and achieving satisfactory visual results.
[0078] In addition, considering that the model obtained by training on a single dataset has limitations in generalization ability, the present invention proposes the idea of refined perception, first training on a synthetic dataset and then fine-tuning on a real dataset. By adding a joint loss function, the obtained model can achieve good generalization on different datasets and has very good processing effects on real images with different distributions. BRIEF DESCRIPTION OF THE DRAWINGS
[0079] Figure 1 is a training flow chart of a denoising model in an embodiment of the present invention;
[0080] Figure 2 This is the training flow chart of the denoising model on the synthetic training set;
[0081] Figure 3 This is the training flow chart of the denoising model on the real training set. DETAILED DESCRIPTION
[0082] The present invention aims to provide an image defogging method based on physical priors, which organically combines the image defogging task with the transmittance map, refinement perception and diffusion model to improve the image defogging effect.
[0083] In the scheme of the present invention, firstly, a synthetic foggy image is obtained by a synthesis method based on a real fog-free image, and a pre-trained physical model is used to generate a preliminary defogging picture, a transmittance picture and an atmospheric light picture, which together with the real fog-free image constitute a synthetic training sample; then, in the process of training the denoising model using the synthetic training set, the preliminary defogging picture in the synthetic training sample is first denoised, and the noisy picture is denoised using the denoising model to predict the calculated noise and the calculated transmittance picture, and then the pre-trained physical model is used to reconstruct the preliminary defogging picture according to the calculated transmittance picture, and then the reconstruction loss and the noise prediction loss are calculated, and the denoising model is updated, and then the updated denoising model is used to denoise the random noise picture, and the denoising loss is calculated according to the denoising result and the corresponding real fog-free image, and the denoising model is updated again according to the denoising loss.
[0084] By using the refinement of the transmittance map as an auxiliary task for the defogging task, the transmittance map can be fully utilized to reflect the fog concentration information of different areas in the image, and the areas with dense fog are processed intensively to restore the lost details in the dense fog areas. This can not only improve the efficiency of defogging, but also preserve the original image effect in the areas with light fog, avoiding information loss and affecting the quality of the defogged image. At the same time, the excellent ability of the diffusion model in image generation is cleverly used to generate the lost details in the dense fog areas of the image, making the defogged image more realistic in perception.
[0085] In addition, considering the generalization performance, after training on the synthetic dataset, the real dataset is also introduced to retrain the initially trained denoising model for fine-tuning. Specifically, the denoising model trained on the synthetic training set is used as the first model, and the model after fine-tuning the parameters of the first model is used as the second model. The initial defogging image in the real training sample is first denoised, and the noisy image, the initial defogging image and the transmittance image are used as input. The first model is used to obtain the first feature map, and the second model is used to generate the second feature map and predict the calculated noise and calculated transmittance images; then the pre-trained physical model is used to reconstruct the initial defogging image according to the calculated transmittance image, and then the reconstruction loss, noise prediction loss, and memory loss are calculated, and the second model is updated; then, the random noise image is denoised using the updated second model, and the denoising loss is calculated based on the denoising result and the corresponding real fog-free image, and the second model is updated again based on the denoising loss.
[0086] Example
[0087] This embodiment provides an image defogging method based on physical priors, including: inputting a foggy image to be processed obtained by shooting into a trained denoising model to obtain a denoised fog-free image; the denoising model adopts a Unet model.
[0088] The training process of the denoising model can be found in Figure 1 , including the following steps:
[0089] S1, construct synthetic training sets and real training sets;
[0090] In this step, the data source of the synthetic training set is real fog-free images, that is, images of real scenes in real life taken by a camera without fog. The data source of the real training set is real fog-free images and real foggy images, that is, images of real scenes in real life taken by a camera without fog and with fog at different times.
[0091] For the construction of the synthetic training set, first, based on the real fog-free image J0, a synthetic foggy image I is obtained by using a synthesis method; based on the synthetic foggy image I, a preliminary defogging picture J, a transmittance picture T and an atmospheric light picture A of the synthetic foggy image are generated by using a pre-trained physical model; the corresponding real fog-free image J0, the synthetic foggy image, and the preliminary defogging picture J, the transmittance picture T and the atmospheric light picture A of the synthetic foggy image constitute synthetic samples, and the synthetic samples constitute a synthetic training set.
[0092] For the construction of the real training set, first, extract the real foggy image corresponding to the real fog-free image J0 Using pre-trained physical models, based on real foggy images Generate realistic foggy images The initial defogging image J, transmittance image T and atmospheric light image A; the corresponding real fog-free image J0, real foggy image And real foggy images The initial dehazed image J, transmittance image T and atmospheric light image A constitute real samples, and the real samples constitute the real training set.
[0093] In an exemplary solution, the above physical model adopts the existing SLP model, which is derived from "Single Image Dehazing Using Saturation Line Prior". It takes a synthetic foggy image or a real foggy image as input, and obtains the preliminary dehazed image J, transmittance image T and atmospheric light image A of the corresponding foggy image through the three steps of saturation line construction-transmittance refinement-scene restoration. Among them, the atmospheric light image A reflects the brightness of the background light of the image, and the transmittance image T reflects the concentration of fog in different areas of the image. The transmittance is low in places with thick fog, the pixel value is small, and the darker the image area is, the transmittance is high in places with light fog, the pixel value is large, and the brighter the image area is.
[0094] The reason why this embodiment uses a pre-trained physical model to process the foggy image is to obtain the background light brightness information in the foggy image and the fog concentration information in different areas of the image, so as to help accurately obtain the noise, that is, the location of the fog, and improve the defogging effect.
[0095] S2, training the denoising model on the synthetic training set to obtain a preliminarily trained denoising model;
[0096] In this step, the training process on the synthetic training set is as follows: Figure 2 As shown, it includes the following specific sub-steps:
[0097] S21, random noise addition to the initial dehazed image;
[0098] In this sub-step, a synthetic sample is input, and a round k less than or equal to K is randomly selected as the calculation round, where K is the number of denoising iterations of the preset denoising model, and in this embodiment, K=1000. Then, random noise is generated, and the preliminary defogging image of the input synthetic sample is denoised by the random noise through the denoising model to obtain the denoised image of the calculation round.
[0099] The noise intensity of the added noise gradually increases with the increase of the number of randomly obtained calculation rounds. In an exemplary solution, for the calculation round k, the specific process of adding noise is:
[0100] First, generate a random noise that follows a standard normal distribution; then, add noise according to the following formula:
[0101]
[0102] Among them, α k is a constant hyperparameter of the kth round, which gradually increases with the increase of round k within the set value range; J represents the corresponding preliminary dehazed image, and ε represents random noise.
[0103] S22, denoising output prediction;
[0104] In this sub-step, the noise-added image J obtained in step S21 is k And the preliminary defogging image J and transmittance image T of the corresponding synthetic sample are input into the denoising model to obtain the calculation noise e of the calculation round k And calculate the transmittance image T k The calculated transmittance image is obtained by the output of the upsampling part of the Unet model through the fully connected layer, and the calculated noise is obtained by calculating the input and output of the denoising model. The formula is expressed as:
[0105] e k , T k =∈ θ (J k , J, T)
[0106] Among them, ∈θ represents the denoising model.
[0107] S23, reconstructing the preliminary defogging image;
[0108] In this step, the calculated transmittance image T obtained in step S22 is k The preliminary dehazed image J and the atmospheric light image A of the corresponding synthetic samples are used to obtain the reconstructed image Ip of the calculation round by using the pre-trained physical model used to construct the training set.
[0109] S24, calculating reconstruction loss and noise prediction loss, and updating the denoising model;
[0110] In this sub-step, first, the reconstructed image I obtained in step S23 is used p And the synthetic foggy image I of the corresponding synthetic sample, calculate the reconstruction loss L rec :
[0111] L rec =||I p ,I|| 2
[0112] Among them, ||·|| 2 represents the L2 loss function; I p represents the reconstructed image, and I represents the foggy image of the corresponding sample.
[0113] Then, the calculated noise e obtained in step S22 is used k and the random noise ε generated in step S21, calculate the noise prediction loss L dif :
[0114] L dif =||e k ,ε|| 2
[0115] Among them, ||·|| 2 represents the L2 loss function; e k represents computational noise, and ε represents random noise.
[0116] Finally, using the reconstruction loss L rec and noise prediction loss L dif , and perform reverse update on the parameters of the denoising model.
[0117] S25, performing K rounds of denoising on the random noise image to obtain a denoising result;
[0118] In this substep, a random noise image D is generated. 0 , and use the denoising model updated in step S24 as the initial model to denoise the noise map D 0 Perform K rounds of denoising to obtain the denoising result D x .
[0119] Among them, each round of denoising includes:
[0120] S251, input the denoising result D obtained in the k-1th round k , input it and the preliminary defogging image and transmittance image of the corresponding synthetic sample into the denoising model to obtain the calculated noise and calculated transmittance image of the kth round;
[0121] S252, using the noise model, based on the denoising result D input in step S251 k and the calculated noise of the kth round obtained in step S251 to generate the denoising result D of the kth round k+1 :
[0122]
[0123] Among them, α k is the constant hyperparameter of the kth round, which gradually increases with the increase of round k within the set value range; e k represents the computational noise of the kth round.
[0124] It should be noted that since the transmittance image reflects the density of fog in the foggy image, we hope that the model can focus on processing the dense fog area; on the contrary, we do not want the model to process this part too much in the light fog area, otherwise it will lead to the loss of fog-free information of the original image. Therefore, we can further use the transmittance map as a weight to achieve the use of the model to generate dense fog areas and retain the effect of the original image in light fog areas without losing the information of the original image.
[0125] That is, the calculated transmittance image is used as the weight, based on the generated denoising result D k+1 And the initial defogging image of the corresponding sample, the generated denoising result D is calculated according to the following formula k+1 Update and obtain the updated denoising result D k+1 :
[0126] D k+1 =D k+1 *(1-T k )+J*T k
[0127] Among them, D on the left side of the equal sign k+1 is the denoising result of the kth round after the update, and D on the right side of the equal sign k+1 is the denoising result of the kth round generated by the denoising model, T k represents the calculated transmittance image of the kth round, and J represents the corresponding preliminary defogging image.
[0128] In order to accelerate the convergence of the model and avoid the denoising result D in each round k+1 The artifacts are introduced by updating. In this embodiment, only when the current round is the first round, the denoising result D is updated according to the above formula. k+1 Update, and use the updated denoising results and the preliminary defogging pictures and transmittance pictures of the corresponding synthetic samples as the input of the next round of denoising model;
[0129] For situations other than the first round, the denoising results of the next round are not updated after being generated. The denoising results of the next round and the preliminary defogging images and transmittance images of the corresponding synthetic samples are directly used as the input of the denoising model of the next round.
[0130] S26, calculating denoising loss and updating denoising model;
[0131] In this sub-step, the denoising result D obtained in step S25 is used. K and the corresponding real haze-free image J 0 , calculate the denoising loss L de , the formula is as follows:
[0132] L de =||D K , J0 || 2
[0133] Among them, ||·|| 2 represents the L2 loss function; D K represents the denoising result obtained by the denoising model after K rounds of denoising, J 0 Represents a real haze-free image.
[0134] Then, according to the denoising loss L de Update the denoising model in reverse order.
[0135] S27, determining whether a preset condition is met;
[0136] In this sub-step, it is determined whether the preset conditions are met, such as the preset number of iterations on the synthetic training set or the denoising loss L. de If the value fluctuates within a certain range, the training on the synthetic training set is completed to obtain a preliminary trained denoising model; otherwise, return to step S21 and continue iterating.
[0137] S3, retraining the initially trained denoising model using the real training set;
[0138] In this step, in order to make the denoising model have stronger generalization performance in real scenes, after obtaining the initially trained denoising model in step S2, it is necessary to retrain the initially trained denoising model using the real training set.
[0139] Before retraining, the denoising model trained above is used as the first model, denoted by ∈ θ1 ; Copy the first model and fine-tune its parameters as the second model, denoted by ∈ θ2 ; Change the two outputs of the denoising model to three outputs, namely the noise output, the transmittance image and the feature map image; During the training process, the first model ∈ θ1 The parameters of are frozen and will not be updated, while the second model ∈ θ2 The parameters will be updated.
[0140] The process of retraining the initially trained denoising model using the real training set can be found in Figure 3 , which includes the following sub-steps:
[0141] S31, random noise addition to the preliminary dehazed image;
[0142] This sub-step is similar to sub-step S21, except that step S21 inputs synthetic training samples, while this sub-step inputs real training samples.
[0143] Specifically, a real training sample is input, and a round less than or equal to K is randomly selected as the calculation round, where K is the number of denoising iterations of the preset denoising model, which is 1000 in this embodiment. Then, random noise is generated, and the random noise is used to add noise to the preliminary defogging image of the input synthetic sample through the denoising model to obtain the noisy image of the calculation round.
[0144] The noise intensity of the added noise gradually increases with the increase of the calculation round. In an exemplary solution, for the calculation round k, the specific process of adding noise is:
[0145] First, generate a random noise that follows a standard normal distribution; then, add noise according to the following formula:
[0146]
[0147] Among them, α k is a constant hyperparameter of the kth round, which gradually increases with the increase of round k within the set value range; J represents the corresponding preliminary dehazed image in the real training sample, and ε represents random noise.
[0148] S32, denoising output prediction;
[0149] In this sub-step, the noise-added image J obtained in step S31 is k And the initial dehazed image J and transmittance image T of the corresponding real sample, input the first model ∈ θ1 , get the first feature map f of the calculation round k1 ; Input the second model ∈ θ2 , get the computational noise e of the computational round k , calculate the transmittance image T k and the second feature map f k2 ; The first feature map and the second feature map are respectively the first model ∈ θ1 and the second model ∈ θ2 The direct output of the upsampling part is expressed as:
[0150] f k1 =∈ θ1 (J k , J, T)
[0151] e k , T k , f k2 =∈ θ2 (J k , J, T)
[0152] Among them, f k1 is the first feature map of round k, f k2is the second feature map of round k, J is the preliminary dehazed image of the real sample, T is the transmittance image of the real sample, and J k To calculate the noise image of round k, T k is the calculated transmittance image of round k, e k is the computational noise of round k.
[0153] S33, reconstructing the preliminary defogging image;
[0154] In this step, the calculated transmittance image T obtained in step S32 is k And the preliminary dehazed image J and the atmospheric light image A of the corresponding real samples, using the pre-trained physical model used to construct the training set, to obtain the reconstructed image of the calculation round.
[0155] S34, calculating the joint loss and updating the second model;
[0156] In this substep, the joint loss is calculated as follows:
[0157] The reconstruction loss L is calculated using the reconstructed image obtained in step S33 and the real foggy image of the corresponding real sample. rec :
[0158]
[0159] Among them, ||·|| 2 represents the L2 loss function; I p Represents the reconstructed image, represents the real foggy image corresponding to the real sample.
[0160] The noise prediction loss L is calculated by using the calculation noise obtained in step S32 and the random noise generated in step S31 dif :
[0161] Ldi f =||e k ,ε|| 2
[0162] Among them, ||·|| 2 represents the L2 loss function; e k represents computational noise, and ε represents random noise.
[0163] The memory loss L is calculated using the first characteristic map and the second characteristic map obtained in step S32. mem :
[0164] L mem =||f t1 , f t2 || 2
[0165] Among them, ||·|| 2 represents the L2 loss function; f t1 represents the first feature map, f t2 Represents the first feature map.
[0166] In addition, it is generally believed that a real fog-free image conforms to certain statistical prior principles, such as dark channel prior (DCP) and bright channel prior (BCP). Therefore, this embodiment adopts a prior-based method to derive the transmission map of the real image, and uses the transmission map of the real image to supervise the transmittance map, thereby improving the accuracy of the transmittance map in expressing the regional fog concentration information, thereby improving the defogging effect.
[0167] Therefore, the joint loss in this embodiment also considers two losses, namely, the dark channel prior loss L dcp and the bright channel prior loss L bcp ;
[0168] The dark channel prior loss L dcp for:
[0169]
[0170] The bright channel prior loss L bcp for:
[0171] L bcp =||T b , T t2 || 2
[0172] In, T d represents the transmission map of the real image obtained based on the dark channel prior derivation, Represents T d The Laplace matrix, T b represents the transmission map of the real image obtained based on the bright channel prior derivation, γ is a hyperparameter, and the superscript T represents the matrix transpose; T t2 It represents the calculated transmittance image obtained by step A11; ||·|| 2 Represents the L2 loss function;
[0173] The T d The derivation of T includes: first, calculating the minimum value of the RGB components of each pixel in the image; then, using the corrosion operation to solve the minimum value of the preset window centered on each pixel, so as to obtain the dark channel image; the T b The derivation includes: first, calculating the maximum value of the RGB components of each pixel in the image; then, using the expansion operation to solve the maximum value of a preset window centered on each pixel, thereby obtaining a bright channel image.
[0174] Finally, the above five losses are calculated, namely L rec , L dif , L mem , L dcp , L bcp The second model is updated backward based on the joint loss.
[0175] S35, performing K rounds of denoising on the random noise image to obtain a denoising result;
[0176] In this substep, a random noise image D is generated. 0 , and use the second model updated in step S34 as the initial model to analyze the noise map D 0 Perform K rounds of denoising to obtain the denoising result D K ;
[0177] Among them, each round of denoising includes:
[0178] S351, input the denoising result D of the k-1th round k , input it and the preliminary dehazed image and transmittance image of the corresponding real sample into the second model to obtain the calculated noise and calculated transmittance image of the kth round;
[0179] S352, using the noise model, based on the denoising result D input in step S351 k The k-th round of calculation noise obtained in step S351 is combined to generate the k-th round of denoising result D k+1 .
[0180] Similarly, similar to step S25, in order to allow the model to focus on the foggy areas in the image and retain the original information of the light foggy areas, and to accelerate the convergence of the model and avoid introducing artifacts, in the first round, the calculated transmittance image is used as the weight, and the generated denoising result D is used as the weight. k+1 And the initial defogging picture corresponding to the real sample, the generated denoising result D k+1 Update and obtain the updated denoising result D k+1 .
[0181] In this sub-step, the calculation formula is the same as step S25.
[0182] S36, calculating denoising loss and updating the second model;
[0183] In this sub-step, the denoising result D obtained in step S35 is used. K and the corresponding real haze-free image J 0 , calculate the denoising loss L de :
[0184] L de=||D K , J 0 || 2
[0185] Among them, ||·|| 2 represents the L2 loss function; D K represents the denoising result obtained by the denoising model after K rounds of denoising, J 0 Represents a real haze-free image.
[0186] Finally, using the denoising loss L de Perform the reverse update on the second model.
[0187] S37, judging whether the preset conditions are met;
[0188] In this sub-step, it is determined whether the preset conditions are met, such as the preset number of iterations on the real training set or the denoising loss L. de If it fluctuates within a certain range, then the training on the real training set is completed to obtain the final denoising model; otherwise, return to step S31 and continue iterating.
[0189] Experimental verification:
[0190] In order to compare the effects of the scheme provided in this embodiment with other schemes in the prior art on image dehazing, different schemes are used for testing on different test sets. Among them, the schemes in the prior art use ten schemes, namely SLP, DCP, D4, SID, Dehamer, Deformer, FFA, SCANet, FC, and FSDGN, and compare them with the scheme of this embodiment (abbreviated as Ours).
[0191] The test sets include SOTS, DENSE-HAZE, NH-HAZE, and NH-HAZE2. The SOTS test set consists of 500 indoor images and 500 outdoor images; the DenseHaze test set contains five real outdoor fog images; the NH-HAZE test set contains five real outdoor fog images; and the NH-HAZE2 test set contains 25 real outdoor fog images.
[0192] The evaluation index is measured by three indicators: PSNR (peak signal-to-noise ratio), SSIM (structural similarity index), and LPIPS (perceptual similarity loss). PSNR: Based on error statistics, it quantifies the difference between the original image and the reconstructed image to evaluate the reconstruction quality or compression quality of the image. The larger the value, the better; SSIM: It evaluates the image quality by measuring the similarity of the two images in terms of brightness, contrast, and structure. The larger the value, the better; LPIPS: A perceptual similarity metric based on a deep learning model, used to evaluate the perceptual similarity difference of images. The smaller the value, the better.
[0193] The test results are shown in Table 1 below.
[0194] Table 1: Comparison of the effects of different solutions on different test sets
[0195]
[0196] It can be seen from the above table that the solution of this embodiment is not optimal except for a few indicators on individual test sets. For example, on the NH-Haze test set, for the PSNR indicator, the test result value of this embodiment is 20.37, slightly lower than 20.66 of the Dehamer solution, ranking second. On the NH-Haze2 test set, for the LPIPS indicator, the test result value of this embodiment is 0.1578, which is not as good as 0.1266 of the SCANet algorithm on this indicator of the test set and 0.1363 of the FC algorithm on this indicator of the test set, ranking third. In addition, the test results of the solution of this embodiment are all optimal. It can be seen that the solution of this embodiment has good performance in defogging effect.
[0197] Finally, it should be noted that the above embodiments are only preferred implementations and are not intended to limit the present invention. It should be pointed out that for those skilled in the art, several modifications, equivalent replacements, improvements, etc. can be made without departing from the scope of the present invention and the scope of protection of the claims, and all of these should be included in the protection scope of the present invention.
Claims
1. A physical prior-based image dehazing method, include: The foggy image to be processed is input into the trained denoising model to obtain a denoised fog-free image; The denoising model adopts the Unet model, which is characterized by: The training of the denoising model includes the following steps: A1. Constructing a synthetic training set: Extracting a real fog-free image; obtaining a synthetic foggy image by using a synthesis method based on the real fog-free image; generating a preliminary defogging picture, a transmittance picture and an atmospheric light picture of the synthetic foggy image by using a pre-trained physical model based on the synthetic foggy image; forming a synthetic sample from the corresponding real fog-free image, the synthetic foggy image and the preliminary defogging picture, the transmittance picture and the atmospheric light picture of the synthetic foggy image, and forming a synthetic training set from the synthetic samples; A2, inputting a synthetic sample, and randomly selecting a round less than or equal to K as a calculation round, where K is the number of denoising iterations of a preset denoising model; generating random noise, and using the denoising model to denoise the preliminary defogging image of the input synthetic sample using the random noise to obtain a noisy image of the calculation round; A3, inputting the noisy image obtained in step A2 and the preliminary defogging image and transmittance image of the corresponding synthetic sample into the denoising model to obtain the calculation noise and calculation transmittance image of the calculation round; the calculation transmittance image is obtained by the output of the upsampling part through the fully connected layer, and the calculation noise is obtained by the denoising model input and output calculation; A4, using the calculated transmittance image obtained in step A3 and the preliminary defogging image and atmospheric light image of the corresponding synthetic sample, using the pre-trained physical model used to construct the training set, to obtain a reconstructed image of the calculation round; A5. Calculate the reconstruction loss using the reconstructed image obtained in step A4 and the synthetic foggy image of the corresponding synthetic sample; calculate the noise prediction loss using the calculated noise obtained in step A3 and the random noise generated in step A2; and update the denoising model using the reconstruction loss and the noise prediction loss; A6. Generate a random noise image D 0 , and use the denoising model updated in step A5 as the initial model to derive the noise map D 0 Perform K rounds of denoising to obtain the denoising result D K ; Each round of denoising includes: A61, input the denoising result D obtained in the k-1th round k , input it and the preliminary defogging image and transmittance image of the corresponding synthetic sample into the denoising model to obtain the calculated noise and calculated transmittance image of the kth round; A62, using the noise model, based on the denoising result D input in step A61 k and the k-th round of calculation noise obtained in step A61 to generate the k-th round of denoising result D k+1 ; A7. Denoising result D obtained using step A6 K And the corresponding real fog-free image, calculate the denoising loss, and update the denoising model; A8. Determine whether the preset conditions are met. If so, obtain a trained denoising model; otherwise, return to step A2.
2. The image defogging method based on physical priors as claimed in claim 1, It is characterized in that Step A1 also includes constructing a real training set, including: Extract the real foggy image corresponding to the real fog-free image, and generate the preliminary defogging picture, transmittance picture and atmospheric light picture of the real foggy image based on the real foggy image by using the pre-trained physical model; the corresponding real foggy image, the real foggy image and the preliminary defogging picture, transmittance picture and atmospheric light picture of the real foggy image constitute the real sample, and the real sample constitutes the real training set; The training of the denoising model also includes: A9. Use the trained denoising model obtained in step A8 as the first model; copy the first model and fine-tune its parameters to use it as the second model; A10, input the real sample, and randomly select a round less than or equal to K as the calculation round; generate random noise, and use the random noise to add noise to the preliminary defogging image of the input real sample through the noise adding model to obtain the noisy image of the calculation round; A11, input the noisy image obtained in step A10 and the preliminary defogging image and transmittance image of the corresponding real sample into the first model to obtain the first feature map of the calculation round; input into the second model to obtain the calculated noise, calculated transmittance image and second feature map of the calculation round; the first feature map and the second feature map are the outputs of the upsampling parts of the first model and the second model respectively; A12, using the calculated transmittance image obtained in step A11 and the preliminary defogging image and atmospheric light image of the corresponding real sample, using the pre-trained physical model used to construct the training set, to obtain a reconstructed image of the calculation round; A13. Calculate the joint loss and update the second model; the joint loss includes reconstruction loss, noise prediction loss and memory loss; the reconstruction loss is calculated using the reconstructed image obtained in step A12 and the real foggy image of the corresponding real sample; the noise prediction loss is calculated using the calculated noise obtained in step A11 and the random noise generated in step A10; the memory loss is calculated using the first feature map and the second feature map obtained in step A11; A14. Generate a random noise image D 0 , and use the second model updated in step A13 as the initial model to analyze the noise map D 0 Perform K rounds of denoising to obtain the denoising result D K ; Each round of denoising includes: A141, input the denoising result D of the k-1th round k , input it and the preliminary dehazed image and transmittance image of the corresponding real sample into the second model to obtain the calculated noise and calculated transmittance image of the kth round; A142, using the noise model, based on the denoising result D input in step A141 k and the calculated noise of the kth round obtained in step A141 to generate the denoising result D of the kth round k+1 ; A15. Denoising result D obtained using step A14 K and the corresponding real haze-free image, calculate the denoising loss, and update the second model; A16. Determine whether the preset condition is met. If so, use the second model that has completed training as the final denoising model; otherwise, return to step A10.
3. The image defogging method based on physical priors as claimed in claim 1 or 2, It is characterized in that The pre-trained physical model used to construct the training set is the SLP model.
4. The image defogging method based on physical priors as claimed in claim 1 or 2, It is characterized in that Generate random noise, and use the random noise to add noise to the preliminary defogging image through the noise model to obtain the noisy image of calculation round k, including: First, generate a random noise that follows a standard normal distribution; then, add noise according to the following formula: Among them, α k is a constant hyperparameter of the kth round, which gradually increases with the increase of round k within the set value range; J represents the corresponding preliminary dehazed image, and ε represents random noise.
5. The image defogging method based on physical priors as claimed in claim 1 or 2, It is characterized in that According to the following formula, using the noise model, based on the denoising result D obtained in the k-1th round k and the calculated noise of the kth round, generating the initial denoising result D of the kth round k+1 : Among them, α k is the constant hyperparameter of the kth round, which gradually increases with the increase of round k within the set value range; e k represents the computational noise of the kth round.
6. The image defogging method based on physical priors as claimed in claim 1 or 2, It is characterized in that In step A62, if the current round is the first round, the calculated transmittance image obtained in step A61 is used as the weight, and the generated denoising result D k+1 And the initial defogging picture of the corresponding sample, the generated denoising result D k+1 Update and obtain the updated denoising result D k+1 ; In step A142, if the current round is the first round, the calculated transmittance image obtained in step A141 is used as the weight, and the generated denoising result D k+1 And the initial defogging picture of the corresponding sample, the generated denoising result D k+1 Update and obtain the updated denoising result D k+1 .
7. The image defogging method based on physical priors as claimed in claim 6, It is characterized in that The calculated transmittance image is used as the weight, based on the generated denoising result D k+1 And the initial defogging image of the corresponding sample, the generated denoising result D is calculated according to the following formula k+1 Update and obtain the updated denoising result D k+1 : D k+1 =D k+1 *(1-T k )+J*T k Among them, D on the left side of the equal sign k+1 is the denoising result of the kth round after the update, and D on the right side of the equal sign k+1 is the denoising result of the kth round generated by the denoising model, T k represents the calculated transmittance image of the kth round, and J represents the corresponding preliminary dehazed image.
8. The image defogging method based on physical priors as claimed in claim 1 or 2, It is characterized in that The reconstruction loss L rec for: L rec =||I p ,I|| 2 Among them, ||·|| 2 represents the L2 loss function; I p represents the reconstructed image, and I represents the foggy image of the corresponding sample; The noise prediction loss L dif for: L dif =||e k ,e|| 2 Among them, ||·|| 2 represents the L2 loss function; e k represents computational noise, and ε represents random noise; The denoising loss L de for: L de =||D K ,J 0 || 2 Among them, ||·|| 2 represents the L2 loss function; D K represents the denoising result obtained by the denoising model after K rounds of denoising, J 0 Represents a real haze-free image.
9. The image defogging method based on physical priors as claimed in claim 2, It is characterized in that The memory loss L mem for: L mem =||f t1 ,f t2 || 2 Among them, ||·|| 2 represents the L2 loss function; f t1 represents the first feature map, f t2 Represents the first feature map.
10. The image defogging method based on physical priors as claimed in claim 2, It is characterized in that In step A13, the joint loss also includes a dark channel prior loss L dcp and the bright channel prior loss L bcp ; The dark channel prior loss L dcp for: The bright channel prior loss L bcp for: L bcp =||T b ,T t2 || 2 Among them, T d represents the transmission map of the real image obtained based on the dark channel prior derivation, Represents T d The Laplace matrix, T b represents the transmission map of the real image obtained based on the bright channel prior derivation, γ is a hyperparameter, and the superscript T represents the matrix transpose; T t2 It represents the calculated transmittance image obtained by step A11; ||·|| 2 Represents the L2 loss function; The T d The derivation of T includes: first, calculating the minimum value of the RGB components of each pixel in the image; then, using the corrosion operation to solve the minimum value of the preset window centered on each pixel, so as to obtain the dark channel image; the T b The derivation includes: first, calculating the maximum value of the RGB components of each pixel in the image; then, using the expansion operation to solve the maximum value of a preset window centered on each pixel, thereby obtaining a bright channel image.
Citation Information
Patent Citations
Image defogging method and system, electronic equipment and storage medium
CN110197469A
Single image defogging method based on unsupervised learning
CN111861939A
Image defogging processing method and device based on deep learning
CN116452469A
Image defogging method and device, equipment and storage medium
CN116977215A
Ship defogging method and system based on atmospheric scattering prior diffusion model
CN117789041A
Cited By
Model training method and device, image reconstruction method and device, electronic equipment and medium
CN120411686A
Model training method, image reconstruction method, device, electronic device and medium
CN120411686B
Image restoration method and device based on residual denoising diffusion model
CN121563847A
Image Restoration Method and Apparatus Based on Residual Denoising Diffusion Model
CN121563847B
