Image defogging method based on physical prior

Abstract

The present application relates to image processing technology, it discloses a kind of based on physical priori image defogging method, first, based on real fog-free image, construct including preliminary defogging picture, transmissivity picture and atmospheric light picture synthetic training sample;Then, first, add noise and denoising are carried out, calculate noise and calculate transmissivity picture are predicted, then based on the prediction result is reconstructed, after that, calculate reconstruction loss, noise prediction loss, update to denoising model;Then, using the denoising model obtained by updating to random noise graph is denoised, according to denoising result and real fog-free image calculate loss, according to loss update to denoising model.It will be refined to transmissivity graph as the auxiliary task of defogging task, not only can improve the efficiency of defogging, also can retain the original map effect of fog light area;Meanwhile, cleverly through diffusion model, generate the details lost in image fog dense area, so that the image after defogging is more real in perception.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to an image defogging method based on physical priori. BACKGROUND

[0002] Due to the existence of haze, outdoor images often have low contrast and limited visibility, which is not conducive to the progress of subsequent computer vision tasks. Therefore, image defogging technology has attracted more and more attention, and the purpose is to restore a clear haze-free image from a blurred foggy image.

[0003] In recent years, the image defogging methods in traditional technology mainly include the following three kinds:

[0004] 1. Image defogging based on physical model:

[0005] The image defogging method based on physical model mainly removes the haze in the image by simulating the scattering and absorption process of light in the atmosphere. These methods provide a theoretically reliable framework, which can generate results with strong interpretability and easy analysis.

[0006] However, the physical model needs to accurately estimate the atmospheric light and the transmittance, which may encounter difficulties in practical applications, especially in complex or highly variable haze conditions. In addition, the calculation process of the physical model is usually complex, which may lead to a long time-consuming, which poses a challenge to real-time applications.

[0007] 2. Image defogging based on priori:

[0008] The image defogging method based on priori uses the statistical properties of clear images to estimate the transmission map, and then uses the scattering model to remove the haze. This kind of method includes dark channel priori, color attenuation priori and color line priori, etc., which infers the distribution of haze through specific image characteristics. These methods are relatively simple to implement, and perform well in specific scenarios.

[0009] However, the manually designed priori assumptions are not always suitable for all foggy images, which may lead to limitations in defogging effect. For images that do not meet the priori assumptions, the performance of these methods may be poor, and the priori needs to be adjusted according to different types of images, which is not strong in universality.

[0010] 3. Image defogging based on deep learning:

[0011] Image dehazing methods based on deep learning have become increasingly popular in recent years. These methods utilize large amounts of labeled data and powerful convolutional neural networks (CNNs) to automatically learn features for image dehazing. Deep learning methods can directly generate clear images from input hazy images without additional parameter estimation, showing excellent performance and strong versatility. They can handle complex fog conditions and various image scenes with strong robustness.

[0012] However, these methods require a large amount of labeled data for training, and the cost of data acquisition and labeling is high. In addition, the training time of deep learning models is long, and the requirement for computing resources is high. In some extreme conditions, these methods may produce artifacts or distortions, affecting the overall quality of the dehazing effect.

[0013] In addition, when the above-mentioned image dehazing methods in the conventional technology are extended to the dehazing task of real foggy images, the distribution gap between synthetic data and real data may cause a significant performance decline of the model, i.e., the model lacks generalization ability. SUMMARY

[0014] The technical problem to be solved by the present application is to provide an image dehazing method based on physical priors to improve the effect of image dehazing.

[0015] The technical solution adopted by the present application to solve the above technical problem is:

[0016] An image dehazing method based on physical priors, comprising: inputting a to-be-processed foggy image obtained by shooting into a trained de-noising model to obtain a de-noised fog-free image; the de-noising model adopts a Unet model, characterized in that,

[0017] The training of the de-noising model comprises the following steps:

[0018] A1, constructing a synthetic training set:

[0019] extracting a real fog-free image; based on the real fog-free image, using a synthetic method to obtain a synthetic foggy image; using a pre-trained physical model to generate a preliminary dehazing picture, a transmittance picture and an atmospheric light picture of the synthetic foggy image based on the synthetic foggy image; the synthetic sample is composed of the corresponding real fog-free image, the synthetic foggy image and the preliminary dehazing picture, the transmittance picture and the atmospheric light picture of the synthetic foggy image, and the synthetic training set is composed of the synthetic sample;

[0020] A2, inputting the synthetic sample and randomly taking a round less than or equal to K as a calculation round, wherein K is a preset de-noising iteration number of the de-noising model; generating random noise, and using the random noise to add noise to the preliminary dehazing picture of the input synthetic sample through a noise adding model to obtain a noise-added picture of the calculation round;

[0021] A3, input the noisy picture obtained in step A2, the preliminary defogging picture and the transmittance picture of the corresponding synthetic sample of the noisy picture into the denoising model to obtain a calculation round of calculation noise and calculation transmittance picture; the calculation transmittance picture is obtained from the output of the up-sampling part through a full connection layer, and the calculation noise is obtained from the input and output calculation of the denoising model;

[0022] A4, use the pre-trained physical model used to build the training set to obtain a reconstruction picture of a calculation round from the calculation transmittance picture obtained in step A3 and the preliminary defogging picture and the atmospheric light picture of the corresponding synthetic sample of the noisy picture;

[0023] A5, calculate a reconstruction loss from the reconstruction picture obtained in step A4 and the synthetic foggy image of the corresponding synthetic sample; calculate a noise prediction loss from the calculation noise obtained in step A3 and the random noise generated in step A2; and update the denoising model according to the reconstruction loss and the noise prediction loss;

[0024] A6, generate a random noise picture D0, and use the denoising model obtained by updating in step A5 as an initial model to perform K rounds of denoising on the noise picture D0 to obtain a denoising result D K ; wherein the denoising of each round comprises:

[0025] A61, input the denoising result D k obtained in the (k-1)th round, and input the preliminary defogging picture and the transmittance picture of the corresponding synthetic sample into the denoising model to obtain a calculation round of calculation noise and calculation transmittance picture;

[0026] A62, use the noise adding model to generate the denoising result D k of the kth round based on the denoising result D k+1 input in step A61 and the calculation noise of the kth round obtained in step A61;

[0027] A7, calculate a denoising loss from the denoising result D K obtained in step A6 and the corresponding real fog-free image, and update the denoising model;

[0028] A8, determine whether a preset condition is reached, if yes, obtain the denoising model trained, and if not, return to step A2.

[0029] Further, step A1 further comprises building a real training set, comprising:

[0030] extracting a real hazy image corresponding to a real haze-free image, using a pre-trained physical model, based on the real hazy image, generating a preliminary dehazing picture, a transmittance picture and an atmospheric light picture of the real hazy image; the real sample is composed of the corresponding real haze-free image, the real hazy image and the preliminary dehazing picture, the transmittance picture and the atmospheric light picture of the real hazy image, and the real training set is composed of the real sample;

[0031] The training of the denoising model further comprises:

[0032] A9, the trained denoising model obtained in step A8 is taken as a first model; the first model is copied and its parameters are fine-tuned to obtain a second model;

[0033] A10, input the real sample, and randomly take a round less than or equal to K as a calculation round; generate random noise, and use the random noise to add noise to the preliminary dehazing picture of the input real sample through the noise adding model to obtain a noise-added picture of the calculation round;

[0034] A11, input the noise-added picture obtained in step A10, the preliminary dehazing picture and the transmittance picture of the corresponding real sample into the first model to obtain a first feature map of the calculation round; input the second model to obtain a calculation noise, a calculation transmittance picture and a second feature map of the calculation round; the first feature map and the second feature map are respectively the outputs of the up-sampling parts of the first model and the second model;

[0035] A12, use the pre-trained physical model used to build the training set to obtain a reconstructed picture of the calculation round by inputting the calculation transmittance picture obtained in step A11 and the preliminary dehazing picture and the atmospheric light picture of the corresponding real sample;

[0036] A13, calculate a joint loss to update the second model; the joint loss comprises a reconstruction loss, a noise prediction loss and a memory loss; the reconstruction loss is calculated by using the reconstructed picture obtained in step A12 and the real hazy image of the corresponding real sample; the noise prediction loss is calculated by using the calculation noise obtained in step A11 and the random noise generated in step A10; the memory loss is calculated by using the first feature map and the second feature map obtained in step A11;

[0037] A14, generate a random noise picture D0, and use the second model updated in step A13 as an initial model to denoise the noise picture D0 for K rounds to obtain a denoising result D K ; wherein, the denoising of each round comprises:

[0038] A141, input the denoising result D kinputting the preliminary defogging picture and the transmittance picture of the corresponding real sample into the second model to obtain the calculated noise of the kth round and the calculated transmittance picture of the kth round;

[0039] A142, using the noise adding model, generating the denoising result D of the kth round based on the denoising result D input in step A141 and the calculated noise of the kth round obtained in step A141 k k+1 ;

[0040] A15, using the denoising result D obtained in step A14 K and the corresponding real haze-free image, calculating a denoising loss, and updating the second model;

[0041] A16, determining whether a preset condition is reached, if yes, taking the trained second model as the finally obtained denoising model; otherwise, returning to step A10.

[0042] Further, the pre-trained physical model used for constructing the training set is an SLP model.

[0043] Further, a random noise is generated, and the random noise is used to add noise to the preliminary defogging picture by the noise adding model to obtain the noise-added picture of the kth round of calculation, including:

[0044] First, a random noise subject to a standard normal distribution is generated; then, noise is added according to the following formula:

[0045]

[0046] wherein, α k is a constant hyperparameter of the kth round, which gradually increases with the increase of the round k in a set value range; J represents the corresponding preliminary defogging picture, and ε represents the random noise.

[0047] Further, the denoising result D k of the kth round is generated based on the denoising result D k+1 of the k-1th round and the calculated noise of the kth round by the noise adding model according to the following formula:

[0048]

[0049] wherein, α k is a constant hyperparameter of the kth round, which gradually increases with the increase of the round k in a set value range; e k represents the calculated noise of the kth round.

[0050] Further, in the step A62, if the current round is the first round, the calculated transmittance picture obtained in the step A61 is taken as the weight, and the generated denoising result D​k+1 and the preliminary dehaze picture of the corresponding sample, the generated de-noising result D k+1 is updated to obtain an updated de-noising result D k+1 .

[0051] In the step A142, if the current round is the first round, the calculated transmittance picture obtained in the step A141 is used as the weight, and the generated de-noising result D k+1 is updated based on the generated de-noising result D k+1 and the preliminary dehaze picture of the corresponding sample to obtain an updated de-noising result D k+1 .

[0052] Further, the calculated transmittance picture is used as the weight, and the generated de-noising result D k+1 is updated based on the generated de-noising result D k+1 and the preliminary dehaze picture of the corresponding sample according to the following formula to obtain an updated de-noising result D k+1 :

[0053] D k+1 = D k+1 *(1-T k )+J*T k

[0054] wherein D k+1 on the left side of the equal sign is the updated de-noising result of the kth round, D k+1 on the right side of the equal sign is the de-noising result of the kth round generated by using the noise adding model, T k represents the calculated transmittance picture of the kth round, and J represents the corresponding preliminary dehaze picture.

[0055] Further, the reconstruction loss L rec is:

[0056] L rec =||I p , I|| 2

[0057] wherein ||·|| 2 represents an L2 loss function; I p represents a reconstructed picture, and I represents a foggy image of the corresponding sample;

[0058] the noise prediction loss L dif is:

[0059] L dif =||e k , ε|| 2

[0060] wherein ||·|| 2represents L2 loss function; e k represents calculating noise, and ε represents random noise;

[0061] The denoising loss L de is:

[0062] L de =||D K , J0 2

[0063] wherein, ||·|| 2 represents L2 loss function; D K represents a denoising result obtained after K rounds of denoising of the denoising model, and J0represents a real fog-free image.

[0064] Further, the memory loss L mem is:

[0065] L mem =||f t1 , f t2 || 2

[0066] wherein, ||·|| 2 represents L2 loss function; f t1 represents a first feature map, and f t2 represents the first feature map.

[0067] Further, in step A13, the joint loss further comprises an dark channel prior loss L dcp and a light channel prior loss L bcp .

[0068] The dark channel prior loss L dcp is:

[0069]

[0070] The light channel prior loss L bcp is:

[0071] L bcp =||T b , T t2 || 2

[0072] wherein, T d represents a transmission map of a real image obtained based on a dark channel prior derivation, represents a Laplace matrix of T d , T b represents a transmission map of a real image obtained based on a light channel prior derivation, and γ is a hyperparameter; the superscript T represents matrix transposition; Tt2 denotes a calculated transmittance picture obtained by step A11; 2 denotes L2 loss function;

[0073] The T d derivation, comprising: first, calculating the minimum value in the RGB component of each pixel in the image; then, solving the minimum value of the preset window centered on each pixel by using the erosion operation, thereby obtaining the dark channel image; the T b derivation, comprising: first, calculating the maximum value in the RGB component of each pixel in the image; then, solving the maximum value of the preset window centered on each pixel by using the inflation operation, thereby obtaining the bright channel image.

[0074] The beneficial effects of the present application are:

[0075] The present application combines the image defogging task with the transmittance map, refined perception and diffusion model.

[0076] Firstly, the transmittance map is crucial for defogging, indicating the exact position of the haze area. The present application models the defogging as a joint task to pursue the refinement of the shadow-free image and the transmittance map, wherein the refinement of the transmittance map is taken as an auxiliary task of the defogging task and is divided into two stages: in the first stage, the transmittance picture of the foggy image is estimated by using the pre-trained physical model; in the second stage, the diffusion model is used for information supplement. Therefore, the transmittance map can be fully utilized to reflect the fog concentration information of different areas in the image, the areas with heavy fog are processed emphatically, and the details lost in the heavy fog area are restored, which not only improves the defogging efficiency, but also preserves the original image effect of the area with light fog, avoiding information loss and affecting the image quality after defogging.

[0077] Secondly, through the excellent ability of the diffusion model in image generation, the details lost in the image foggy area can be generated, so that the defogged image is more realistic in perception and satisfactory visual results can be obtained.

[0078] In addition, considering that the model trained on a single data set has limitations in generalization ability, the present application proposes the idea of refined perception, which first trains on a synthetic data set and then fine-tunes on a real data set, and through the increased joint loss function, the obtained model can achieve good generalization on different data sets and has very good processing effect on real images with different distributions. BRIEF DESCRIPTION OF DRAWINGS

[0079] Figure 1 It is the training flowchart of the denoising model in the embodiment of the present application;

[0080] Figure 2A training flowchart of the denoising model on the synthetic training set;

[0081] Figure 3 A training flowchart of the denoising model on the real training set. DETAILED DESCRIPTION

[0082] The present application aims to provide a physical prior-based image defogging method, which organically combines the image defogging task with the transmittance map, refined perception and diffusion model, and improves the image defogging effect.

[0083] In the present application, first, based on the real haze-free image, a synthetic haze image is obtained by a synthesis method, a pre-trained physical model is used to generate a preliminary defogging picture, a transmittance picture and an atmospheric light picture, which together with the real haze-free image constitute a synthetic training sample; then, in the process of training the denoising model using the synthetic training set, the preliminary defogging picture in the synthetic training sample is first denoised, the denoising model is used to denoise the denoised picture, the calculated noise and the calculated transmittance picture are predicted, then the pre-trained physical model is used to reconstruct the preliminary defogging picture according to the calculated transmittance picture, then the reconstruction loss and the noise prediction loss are calculated, and the denoising model is updated, then the denoising model is used to denoise the random noise picture, the denoising loss is calculated according to the denoising result and the corresponding real haze-free image, and the denoising model is updated again according to the denoising loss.

[0084] The refinement of the transmittance map is used as an auxiliary task of the defogging task, which can fully utilize the transmittance map to reflect the fog concentration information of different regions in the image, and can focus on processing the regions with high fog concentration to restore the lost details in the high fog concentration regions, which not only improves the defogging efficiency, but also preserves the original image effect of the low fog concentration regions, avoids information loss and affects the image quality after defogging. At the same time, the diffusion model is used to generate the lost details in the image fog concentration region, which makes the defogged image more realistic in perception.

[0085] In addition, considering the generalization performance, after training on the synthetic data set, a real data set is also introduced to retrain the preliminarily trained denoising model to realize fine-tuning. Specifically, the denoising model trained on the synthetic training set is taken as a first model, a model fine-tuned on the parameters of the first model is taken as a second model, the preliminarily defogged picture in the real training sample is first added with noise, the image added with noise, the preliminarily defogged picture and the transmittance picture are taken as inputs, the first feature map is obtained by using the first model, the second feature map is generated by using the second model and the calculated noise and the calculated transmittance picture are predicted; then, the pre-trained physical model is adopted to reconstruct the preliminarily defogged picture according to the calculated transmittance picture, then the reconstruction loss, the noise prediction loss and the memory loss are calculated, and the second model is updated; then, the second model after the update is used to denoise the random noise picture, the denoising loss is calculated according to the denoising result and the corresponding real haze-free image, and the second model is updated again according to the denoising loss.

[0086] Embodiment

[0087] The embodiment provides a physical prior-based image defogging method, comprising: inputting a to-be-processed haze image obtained by shooting into a denoising model trained to obtain a haze-free image after denoising; and the denoising model adopts an Unet model.

[0088] The training process of the denoising model is shown in Figure 1 , and comprises the following steps.

[0089] S1, constructing a synthetic training set and a real training set;

[0090] In this step, the data source of the synthetic training set is a real haze-free image, that is, an image without haze in a real scene photographed by a camera. The data source of the real training set is a real haze-free image and a real haze image, that is, images without haze and with haze at different times in a real scene photographed by a camera.

[0091] For the construction of the synthetic training set, first, based on the real haze-free image J0, a synthetic haze image I is obtained by using a synthesis method; the pre-trained physical model is used to generate the preliminarily defogged picture J, the transmittance picture T and the atmospheric light picture A of the synthetic haze image I based on the synthetic haze image I; the corresponding real haze-free image J0, the synthetic haze image, and the preliminarily defogged picture J, the transmittance picture T and the atmospheric light picture A of the synthetic haze image are used to form a synthetic sample, and the synthetic training set is formed by the synthetic sample.

[0092] For the construction of the real training set, first, the real haze image corresponding to the real haze-free image J0 is extracted The pre-trained physical model is used to generate the preliminarily defogged picture J, the transmittance picture T and the atmospheric light picture A of the real haze image based on the real haze image Generating a real hazy image a preliminary dehazing picture J, a transmittance picture T and an atmospheric light picture A of the real hazy image A real sample is constituted by the preliminary dehazing picture J, the transmittance picture T and the atmospheric light picture A of the real hazy image and a real training set is constituted by the real sample.

[0093] In an exemplary scheme, the above-mentioned physical model adopts an existing SLP model, which is derived from Single Image Dehazing Using Saturation Line Prior, to obtain the preliminary dehazing picture J, the transmittance picture T and the atmospheric light picture A of the corresponding hazy picture through the three steps of saturation line construction-transmittance refinement-scene restoration, taking a synthetic hazy image or a real hazy image as input. The atmospheric light picture A reflects the brightness of the background light of the picture, and the transmittance picture T reflects the concentration of the fog in different regions of the picture. The place with high fog concentration has low transmittance and large pixel value, and the image region is dark. The place with low fog concentration has high transmittance and small pixel value, and the image region is bright.

[0094] The reason why the pre-trained physical model is used to process the hazy image in this embodiment is to obtain the brightness information of the background light in the hazy image and the fog concentration information in different regions of the picture, so as to help accurately obtain the noise, i.e., the location of the fog, and improve the dehazing effect.

[0095] S2, training the denoising model on the synthetic training set to obtain a preliminary trained denoising model;

[0096] In this step, the training process on the synthetic training set is as shown in Figure 2 and includes the following specific sub-steps:

[0097] S21, randomly adding noise to the preliminary dehazing picture;

[0098] In this sub-step, the synthetic sample is input, and a round k less than or equal to K is randomly taken as the calculation round, wherein K is the preset denoising iteration number of the denoising model, and K=1000 in this embodiment. Then, a random noise is generated, and the random noise is used to add noise to the preliminary dehazing picture of the input synthetic sample through the noise adding model to obtain the noise-added picture of the calculation round.

[0099] The noise intensity of the noise addition gradually increases with the increase of the randomly taken calculation round value. In an exemplary scheme, for the calculation round k, the specific process of noise addition is as follows:

[0100] First, a random noise obeying the standard normal distribution is generated; then, noise addition is performed according to the following formula:

[0101]

[0102] wherein, α k is a constant hyper-parameter of the kth round, which is gradually increased with the increase of the round k in a set value range; J represents the corresponding preliminary defogging picture, and ε represents random noise.

[0103] S22, denoising output prediction;

[0104] In this sub-step, the noisy picture J k obtained in step S21 and the preliminary defogging picture J and the transmittance picture T corresponding to the synthetic sample are input into the denoising model to obtain the calculation noise e k and the calculation transmittance picture T k of the calculation round; the calculation transmittance picture is obtained by full connection layer from the output of the up-sampling part of the Unet model, and the calculation noise is obtained by input and output calculation of the denoising model, and the formula is represented as:

[0105] e k , T k =∈ θ (J k , J, T)

[0106] wherein, ∈θ represents the denoising model.

[0107] S23, reconstructing the preliminary defogging picture;

[0108] In this sub-step, the calculation transmittance picture T k obtained in step S22 and the preliminary defogging picture J and the atmospheric light picture A corresponding to the synthetic sample are used to obtain the reconstructed picture Ip of the calculation round by using the pre-trained physical model used to build the training set.

[0109] S24, calculating the reconstruction loss, the noise prediction loss, and updating the denoising model;

[0110] In this sub-step, first, the reconstructed picture I p obtained in step S23 and the synthetic fog image I corresponding to the synthetic sample are used to calculate the reconstruction loss L rec :

[0111] L rec =||I p , I|| 2

[0112] wherein, ||·|| 2 represents the L2 loss loss function; I p represents the reconstructed picture, and I represents the fog image corresponding to the sample.

[0113] Then, the calculation noise e k and the random noise ε generated in step S21, the noise prediction loss L dif :

[0114] L dif =||e k ,ε|| 2

[0115] wherein, ||·|| 2 represents the L2 loss function; e k represents the calculation noise, and ε represents the random noise.

[0116] Finally, the denoising model is updated in parameters by using the reconstruction loss L rec and the noise prediction loss L dif .

[0117] S25, K rounds of denoising are performed on the random noise map to obtain a denoised result;

[0118] In this sub-step, a random noise map D0 is generated, and the denoising model obtained by updating in step S24 is used as an initial model to perform K rounds of denoising on the noise map D0 to obtain a denoised result D x .

[0119] Wherein, each round of denoising includes:

[0120] S251, input the denoised result D k obtained in the k-1th round, and input the preliminary defogging picture and the transmittance picture corresponding to the synthesized sample into the denoising model to obtain the calculation noise and the calculation transmittance picture in the kth round;

[0121] S252, use the noise adding model to generate the denoised result D k in the kth round based on the denoised result D k+1 input in step S251 and the calculation noise in the kth round obtained in step S251:

[0122]

[0123] wherein, α k is a constant hyperparameter in the kth round, which gradually increases with the increase of the round k in a set value range; e k represents the calculation noise in the kth round.

[0124] It should be noted that since the transmittance picture reflects the density of fog in the foggy image, for the area with thick fog, the model is expected to focus on processing; on the contrary, for the area with light fog, the model is not expected to process too much, otherwise the original image information will be lost. Therefore, further, the transmittance picture can be used as a weight to generate the model for the area with thick fog, and the original image effect is retained for the area with light fog, without losing the original image information.

[0125] That is, taking the calculated transmittance picture as a weight, the generated denoising result D k+1 and the preliminary defogging picture of the corresponding sample are updated according to the following formula to obtain the updated denoising result D k+1 . k+1 :

[0126] D k+1 =D k+1 *(1-T k )+J*T k

[0127] Wherein, D k+1 on the left side of the equal sign is the updated denoising result of the kth round, D k+1 on the right side of the equal sign is the denoising result of the kth round generated by the noise adding model, T k represents the kth round of calculated transmittance picture, and J represents the corresponding preliminary defogging picture.

[0128] In order to accelerate the convergence of the model, while avoiding the introduction of artifacts by updating the denoising result D k+1 in each round, in the embodiment, the denoising result D k+1 is updated according to the above formula only when the current round is the first round, and the updated denoising result and the preliminary defogging picture and the transmittance picture of the corresponding synthetic sample are used as the input of the denoising model of the next round.

[0129] For non-first rounds, after generating the denoising result of the next round, the denoising result is not updated, and the denoising result of the next round and the preliminary defogging picture and the transmittance picture of the corresponding synthetic sample are directly used as the input of the denoising model of the next round.

[0130] S26, calculate the denoising loss and update the denoising model;

[0131] In this sub-step, the denoising result D K obtained in step S25 and the corresponding real fog-free image J0 are used to calculate the denoising loss L de , and the formula is as follows:

[0132] L de =||D K , J0|| 2

[0133] Among them, ||·|| 2 D represents the L2 loss function; K J0 represents the denoising result obtained after K rounds of denoising by the denoising model, and J0 represents the real fog-free image.

[0134] Then, based on the denoising loss L de The denoising model is updated in reverse.

[0135] S27. Determine whether the preset conditions have been met;

[0136] In this sub-step, it is determined whether preset conditions have been met, such as reaching a preset number of iterations on the synthetic training set, or the denoising loss L. de If the value fluctuates within a certain range, then the training on the synthetic training set is completed, and a preliminary denoising model is obtained; otherwise, return to step S21 and continue iterating.

[0137] S3. Retrain the initially trained denoising model using the real training set;

[0138] In this step, in order to make the denoising model have strong generalization performance in real scenes, after obtaining the initially trained denoising model in step S2, it is necessary to retrain the initially trained denoising model using the real training set.

[0139] Before retraining, the denoising model trained above is used as the first model, denoted as ∈ θ1 Copy the first model and fine-tune its parameters to create a second model, denoted as ∈ θ2 The two outputs of the denoising model are changed to three outputs: noise output, transmittance image, and feature map image. During training, the first model ∈ θ1 The parameters of the first model are frozen and will not be updated, while the second model ∈ θ2 The parameters will be updated.

[0140] The process of retraining the initially trained denoising model using a real training set is described in [link to documentation]. Figure 3 It includes the following sub-steps:

[0141] S31. Randomly add noise to the initial dehazed image;

[0142] This sub-step is similar to sub-step S21, except that the input to step S21 is a synthetic training sample, while the input to this sub-step is a real training sample.

[0143] Specifically, input a real training sample, and randomly take a round less than or equal to K as a calculation round, where K is a preset denoising iteration number of a denoising model, and in this embodiment, K is 1000. Then, generate random noise, add noise to the preliminary dehazing picture of the input synthetic sample by using a noise adding model to obtain a noisy picture of the calculation round.

[0144] The noise intensity of the noise adding gradually increases with the increase of the calculation round. In an exemplary scheme, for the calculation round k, the specific process of noise adding is as follows:

[0145] First, generate a random noise subject to a standard normal distribution; then, add noise according to the following formula:

[0146]

[0147] Wherein, α k is a constant hyperparameter of the kth round, which gradually increases with the increase of the round k in a set value range; J represents the corresponding preliminary dehazing picture in the real training sample, and ε represents the random noise.

[0148] S32, denoising output prediction;

[0149] In this sub-step, input the noisy picture J k obtained in step S31, the preliminary dehazing picture J and the transmittance picture T of the corresponding real sample into the first model ∈ θ1 to obtain the first feature map f k1 of the calculation round; input the second model ∈ θ2 to obtain the calculation noise e k , the calculation transmittance picture T k and the second feature map f k2 of the calculation round; the first feature map and the second feature map are respectively the direct outputs of the up-sampling parts of the first model ∈ θ1 and the second model ∈ θ2 , and the formula is represented as:

[0150] f k1 =∈ θ1 (J k , J, T)

[0151] e k , T k , f k2 =∈ θ2 (J k , J, T)

[0152] Wherein, f k1 is the first feature map of the calculation round k, and f k2To calculate the second feature map of the round k, J is the preliminary defogging picture of the real sample, T is the transmittance picture of the real sample, J k To calculate the noisy picture of the round k, T k To calculate the calculated transmittance picture of the round k, e k To calculate the calculated noise of the round k.

[0153] S33, reconstructing the preliminary defogging picture;

[0154] In this sub-step, the calculated transmittance picture T k obtained in step S32, and the preliminary defogging picture J and the atmospheric light picture A of the corresponding real sample, use the pre-trained physical model used to build the training set to obtain the reconstructed picture of the calculation round.

[0155] S34, calculating the joint loss and updating the second model;

[0156] In this sub-step, the joint loss is calculated as follows:

[0157] The reconstructed picture obtained in step S33 and the real foggy image of the corresponding real sample are used to calculate the reconstruction loss L rec :

[0158]

[0159] Where, ||·|| 2 represents the L2 loss loss function; I p represents the reconstructed picture, represents the real foggy image of the corresponding real sample.

[0160] The calculated noise obtained in step S32 and the random noise generated in step S31 are used to calculate the noise prediction loss L dif :

[0161] Ldi f = ||e k , ε 2

[0162] Where, ||·|| 2 represents the L2 loss loss function; e k represents the calculated noise, and ε represents the random noise.

[0163] The first feature map and the second feature map obtained in step S32 are used to calculate the memory loss L mem :

[0164] L mem = ||f t1 , f t2 || 2

[0165] wherein ||·|| denotes L2 loss function; f 2 denotes the first feature map, f t1 denotes the first feature map. t2 denotes the first feature map.

[0166] In addition, it is generally believed that the real haze-free image conforms to certain statistical prior principles, such as dark channel prior (DCP) and bright channel prior (BCP), therefore, the embodiment adopts a prior-based method to derive the transmission map of the real image, and uses the transmission map of the real image to supervise the transmittance map, so as to improve the expression accuracy of the transmittance map on the regional haze concentration information, thereby improving the haze removal effect.

[0167] Therefore, the joint loss in the embodiment also considers two kinds of losses, namely, dark channel prior loss L dcp and bright channel prior loss L bcp .

[0168] The dark channel prior loss L dcp is:

[0169]

[0170] The bright channel prior loss L bcp is:

[0171] L bcp =||T b , T t2 || 2

[0172] wherein T d represents the transmission map of the real image obtained based on the dark channel prior, represents the Laplacian matrix of T d , T b represents the transmission map of the real image obtained based on the bright channel prior, and γ is a hyperparameter; the superscript T represents matrix transposition; T t2 denotes the calculated transmittance picture obtained in step A11; ||·|| denotes L2 loss function; 2 denotes L2 loss function;

[0173] The derivation of T d includes: first, calculating the minimum value of each pixel RGB component in the image; then, solving the minimum value of a preset window centered at each pixel by using an erosion operation, thereby obtaining a dark channel image; the derivation of T b includes: first, calculating the maximum value of each pixel RGB component in the image; then, solving the maximum value of a preset window centered at each pixel by using an inflation operation, thereby obtaining a bright channel image.

[0174] Finally, the above five losses, i.e., L rec , L dif , L mem , L dcp , and L bcp , are calculated, and the second model is updated in reverse based on the joint loss.

[0175] S35, K rounds of denoising are performed on the random noise map to obtain a denoised result;

[0176] In this sub-step, a random noise map D0 is generated, and the second model obtained by updating in step S34 is used as an initial model to perform K rounds of denoising on the noise map D0 to obtain a denoised result D K .

[0177] Each round of denoising includes:

[0178] S351, the denoised result D k of the k-1th round is input, and the preliminary defogging picture and the transmittance picture corresponding to the real sample are input into the second model to obtain the calculated noise and the calculated transmittance picture of the kth round;

[0179] S352, a noise adding model is used to generate the denoised result D k of the kth round based on the denoised result D k+1 input in step S351 and the calculated noise of the kth round obtained in step S351.

[0180] Similarly, similar to step S25, in order to enable the model to focus on the area with heavy fog in the picture for intensive processing while preserving the original information of the area with light fog, and in consideration of accelerating the convergence of the model and avoiding the introduction of artifacts, in the first round, the generated denoised result D k+1 is updated based on the generated denoised result D k+1 and the preliminary defogging picture corresponding to the real sample, using the calculated transmittance picture as a weight, to obtain an updated denoised result D k+1 .

[0181] In this sub-step, the calculation formula is the same as that in step S25.

[0182] S36, a denoising loss is calculated, and the second model is updated;

[0183] In this sub-step, the denoised result D K obtained in step S35 and the corresponding real fog-free image J0 are used to calculate a denoising loss L de :

[0184] L de =||D K , J0|| 2

[0185] wherein ||·|| denotes L2 loss function; D 2 denotes L2 loss function; D K denotes the denoising result obtained after K rounds of denoising of the denoising model, and J0 denotes the real haze-free image.

[0186] Finally, the denoising loss L de The second model is updated in reverse.

[0187] S37, determine whether a preset condition is reached;

[0188] In this sub-step, it is determined whether a preset condition is reached, such as reaching a preset number of iteration rounds on the real training set, or the denoising loss L de fluctuates within a certain range, if yes, the training on the real training set is completed, and a final denoising model is obtained; otherwise, return to step S31 to continue iteration.

[0189] Experimental verification:

[0190] In order to compare the effects of the scheme provided in the embodiment with other schemes in the prior art on image defogging, different schemes are tested on different test sets, wherein the schemes in the prior art adopt SLP, DCP, D4, SID, Dehamer, Deformer, FFA, SCANet, FC, and FSDGN, which are compared with the scheme (referred to as Ours) of the embodiment.

[0191] The test set adopts SOTS, DENSE-HAZE, NH-HAZE, and NH-HAZE2, the SOTS test set is composed of 500 indoor images and 500 outdoor images; the DenseHaze test set contains five outdoor real fog images; the NH-HAZE test set contains five outdoor real fog images; and the NH-HAZE2 test set contains 25 outdoor real fog images.

[0192] The evaluation indexes adopt three indexes of PSNR (peak signal-to-noise ratio), SSIM (structural similarity index), and LPIPS (perceptual similarity loss) to measure. PSNR: based on error statistics, the difference between the original image and the reconstructed image is quantified to evaluate the reconstruction quality or compression quality of the image, the larger the value is, the better; SSIM: by measuring the similarity of two images in brightness, contrast, and structure, the image quality is evaluated, the larger the value is, the better; LPIPS: based on the perceptual similarity measurement of a deep learning model, used to evaluate the perceptual similarity difference of images, the smaller the value is, the better.

[0193] The test results are shown in Table 1 below.

[0194] Table 1: Comparison of effects of different schemes tested on different test sets

[0195]

[0196] As can be seen from the above table, in addition to a few indicators not being optimal on individual test sets, such as on the NH-Haze test set, for the PSNR indicator, the test result value of the scheme of the present embodiment is 20.37, slightly lower than 20.66 of the Dehamer scheme, ranking second, on the NH-Haze2 test set, for the LPIPS indicator, the test result value of the present embodiment is 0.1578, not as good as 0.1266 of the SCANet algorithm on this test set for this indicator and 0.1363 of the FC algorithm on this test set for this indicator, ranking third, in addition, the test results of the scheme of the present embodiment are all optimal. It can be seen that the scheme of the present embodiment has good performance in the effect of dehazing.

[0197] Finally, it should be noted that the above embodiments are only preferred embodiments and are not intended to limit the present application. It should be noted that for those skilled in the art, without departing from the scope of the present application and the scope of protection of the claims, a number of modifications, equivalent replacements, improvements, etc. can be made, which should be included in the scope of protection of the present application.

Claims

1. A physical prior based image dehazing method, comprising: The obtained foggy image is input into the trained denoising model to obtain a de-noised haze-free image. The denoising model adopts an Unet model, and the Unet model is characterized in that, The training of the denoising model comprises the following steps: A1, constructing a synthetic training set: extracting a real haze-free image; based on the real haze-free image, a synthetic foggy image is obtained by using a synthesis method; a preliminary de-hazing picture, a transmittance picture and an atmospheric light picture of the synthetic foggy image are generated based on the synthetic foggy image by using a pre-trained physical model; a synthetic sample is formed by the corresponding real haze-free image, the synthetic foggy image and the preliminary de-hazing picture, the transmittance picture and the atmospheric light picture of the synthetic foggy image, and the synthetic training set is formed by the synthetic sample; A2, inputting the synthetic sample and randomly taking a round less than or equal to K as a calculation round, wherein K is a preset denoising iteration number of the denoising model; generating random noise, and adding noise to the preliminary de-hazing picture of the input synthetic sample by using the random noise through a noise adding model to obtain a noise-added picture of the calculation round; A3, inputting the noise-added picture obtained in step A2, the preliminary de-hazing picture and the transmittance picture of the corresponding synthetic sample into the denoising model to obtain a calculation noise and a calculation transmittance picture of the calculation round; the calculation transmittance picture is obtained by the output of the up-sampling part through a full connection layer, and the calculation noise is obtained by the input and output calculation of the denoising model; A4, using the pre-trained physical model used for constructing the training set, the calculation transmittance picture obtained in step A3 and the preliminary de-hazing picture and the atmospheric light picture of the corresponding synthetic sample are used to obtain a reconstructed picture of the calculation round; A5, using the reconstructed picture obtained in step A4 and the synthetic foggy image of the corresponding synthetic sample, a reconstruction loss is calculated; using the calculation noise obtained in step A3 and the random noise generated in step A2, a noise prediction loss is calculated; the denoising model is updated by using the reconstruction loss and the noise prediction loss; A6, generating a random noise map D0, and using the denoising model updated by step A5 as the initial model, performing K rounds of denoising on the noise map D0 to obtain a denoising result D K ; wherein the denoising of each round comprises: A61, input the de-noising result D obtained in the k-1th round k The preliminary defogging picture and the transmittance picture corresponding to the synthetic sample are input into the de-noising model to obtain the calculated noise picture and the calculated transmittance picture in the kth round; A62, using the noise adding model, based on the denoising result D input in step A61 k and the calculated noise of the kth round obtained in step A61, to generate the denoising result D of the kth round k+1 ; A7、using the denoising result D obtained in step A6 K and the corresponding real non-fog image, calculate the denoising loss, update the denoising model; A8, determining whether a preset condition is reached, if yes, a denoising model completed training is obtained; otherwise, returning to step A2.

2. The image de-hazing method based on a physical prior according to claim 1, characterized in that, In step A1, a real training set is further constructed, comprising: extracting a real haze-free image; based on the real haze-free image, a synthetic foggy image is obtained by using a synthesis method; a preliminary de-hazing picture, a transmittance picture and an atmospheric light picture of the synthetic foggy image are generated based on the synthetic foggy image by using a pre-trained physical model; a synthetic sample is formed by the corresponding real haze-free image, the synthetic foggy image and the preliminary de-hazing picture, the transmittance picture and the atmospheric light picture of the synthetic foggy image, and the synthetic training set is formed by the synthetic sample; The training of the denoising model further comprises: A9, taking the denoising model completed training obtained in step A8 as a first model; copying the first model and fine-tuning the parameters of the first model to obtain a second model; A10, inputting the real sample and randomly taking a round less than or equal to K as a calculation round; generating random noise, and adding noise to the preliminary de-hazing picture of the input real sample by using the random noise through a noise adding model to obtain a noise-added picture of the calculation round; A11, input the noisy picture obtained in step A10 and the preliminary defogging picture and the transmittance picture of the corresponding real sample into the first model to obtain the first feature map of the calculation round; input into the second model to obtain the calculation noise, the calculation transmittance picture and the second feature map of the calculation round; the first feature map and the second feature map are respectively the outputs of the up-sampling part of the first model and the second model; A12, using the pre-training physical model used to build the training set, input the calculation transmittance picture obtained in step A11 and the preliminary defogging picture and the atmospheric light picture of the corresponding real sample to obtain the reconstructed picture of the calculation round; A13, calculate the joint loss and update the second model; the joint loss includes the reconstruction loss, the noise prediction loss and the memory loss; the reconstruction loss is calculated using the reconstructed picture obtained in step A12 and the real hazy image of the corresponding real sample; the noise prediction loss is calculated using the calculation noise obtained in step A11 and the random noise generated in step A10; the memory loss is calculated using the first feature map and the second feature map obtained in step A11; A14. generating a random noise map D0, and performing K rounds of denoising on the noise map D0using the second model updated in step A13 as the initial model to obtain a denoising result D K ; wherein each round of denoising comprises: A141、input the de-noising result D of the k-1th round k The preliminary defogging picture and the transmittance picture corresponding to the real sample are input into the second model to obtain the calculated noise and the calculated transmittance picture of the kth round; A142、using the noise adding model, generating the kth-round denoising result D based on the denoising result D input in step A141 k and the kth-round calculated noise obtained in step A141, to generate the kth-round denoising result D k+1 ; A15、using the de-noising result D obtained in step A14 K and the corresponding real non-fog image, calculate the de-noising loss, update the second model; A16, determine whether the preset condition is reached, if yes, the second model completed training is used as the final obtained denoising model; otherwise, return to step A10.

3. The physical prior based image dehazing method of claim 1 or 2, wherein, The pre-training physical model used to build the training set is an SLP model.

4. The physical prior-based image defogging method according to claim 1 or 2, characterized in that, generating random noise, using random noise to add noise to the preliminary defogging picture through the noise adding model to obtain the noisy picture of the calculation round k, comprising: first, generating a random noise obeying the standard normal distribution; then, adding noise according to the following formula: wherein, α k is a constant hyper-parameter of the kth round, which gradually increases with the increase of the round k in a set value range; J represents the corresponding preliminary defogging picture, and ε represents random noise.

5. The physical prior-based image defogging method according to claim 1 or 2, characterized in that, Using the noise adding model, based on the de-noising result D obtained in the k-1th round, the initial de-noising result D of the kth round is generated according to the following formula k and the calculated noise of the kth round k+1 : wherein α k is a constant hyper-parameter of the kth round, which gradually increases with the increase of the round k in a set value range; e k represents the calculation noise of the kth round.

6. The physical prior-based image defogging method according to claim 1 or 2, characterized in that, In the step A62, if the current round is the first round, the calculated transmittance picture obtained in the step A61 is taken as the weight, and the generated denoising result D k+1 is updated based on the generated preliminary defogging picture of the corresponding sample and the denoising result D k+1 , to obtain an updated denoising result D k+1 . In the step A142, if the current round is the first round, the calculated transmittance picture obtained in the step A141 is taken as the weight, and the generated denoising result D k+1 is updated based on the generated preliminary defogging picture of the corresponding sample and the denoising result D k+1 , to obtain an updated denoising result D k+1 .

7. The physical prior-based image defogging method according to claim 6, characterized in that, Based on the generated denoised result D k+1 and the preliminary defogging picture of the corresponding sample, the generated denoised result D k+1 is updated according to the following formula to obtain the updated denoised result D k+1 : D k+1 = D k+1 *(1-T k )+J*T k where D k+1 is the updated kth round of de-noised results, D k+1 is the kth round of de-noised results generated using the noise adding model, T k represents the kth round of calculated transmission rate pictures, and J represents the corresponding preliminary defogging pictures.

8. The physical prior-based image defogging method according to claim 1 or 2, characterized in that, The reconstruction loss L rec is: L rec =||I p ,I|| 2 Among them, ||·|| 2 I represents the L2 loss function; p This represents the reconstructed image, where I represents the foggy image of the corresponding sample; The noise prediction loss L dif is: L dif =||e k ,ε|| 2 where ||·|| denotes the L2 loss function; e 2 denotes the L2 loss function; e k denotes the L2 loss function; e The denoising loss L de is: L de =||D K , J0| 2 where ||·|| denotes the L2 loss function; D 2 denotes the L2 loss function; D K denotes the denoising result obtained after K rounds of denoising of the denoising model, and J0denotes the real haze-free image.

9. The physical prior-based image defogging method according to claim 2, characterized in that, The memory loss L mem is: L mem =||f t1 , f t2 || 2 Among them, ||·|| 2 f represents the L2 loss function; t1 Let f represent the first feature map. t2 This represents the first feature map.

10. The physical prior-based image defogging method according to claim 2, characterized in that, In step A13, the joint loss further includes a dark channel prior loss L dcp and a light channel prior loss L bcp ; The dark channel prior loss L dcp is: The explicit channel prior loss L bcp is: L bcp =||T b ,T t2 || 2 Among them, T d The transmission graph represents the real image obtained based on the prior derivation of the dark channel. Represents T d The Laplace matrix, T b This represents the transmission graph of the real image obtained based on the open-channel prior derivation, where γ is a hyperparameter and the superscript T denotes matrix transpose; T t2 This represents the calculated transmittance image obtained in step A11; ||·|| 2 This represents the L2 loss function; The T d derivation, comprising: first, calculating the minimum value in the RGB component of each pixel in the image; then, solving the minimum value of a preset window centered on each pixel by using the erosion operation, thereby obtaining the dark channel image; the T b derivation, comprising: first, calculating the maximum value in the RGB component of each pixel in the image; then, solving the maximum value of a preset window centered on each pixel by using the inflation operation, thereby obtaining the bright channel image.

Images