A unified image defogging and denoising method based on unsupervised learning

Abstract

This invention discloses a unified image dehazing and denoising method based on unsupervised learning, relating to the field of digital image processing technology, and includes the following steps: S1, processing the original haze image I... haze The input is fed into a dehazing module trained using the dark channel prior DCP method and generative adversarial training to obtain a dehazed, noisy image J. dehaze S2, take the original smog image I haze The input is fed into a downsampling-based denoising module to obtain a denoised hazy image J. noise S3, the images J obtained from S1 and S2 dehaze and image J noise The input is fed into the image fusion module, which fuses the dehazed and denoised images using region similarity as the objective function, and outputs a fused image J that balances dehazing and denoising. fused The method of this invention mainly consists of an unsupervised dehazing module, an unsupervised denoising module, and a region similarity fusion strategy to restore visibility and suppress depth-dependent noise propagation in the dehazing module. The method of this invention outperforms other state-of-the-art dehazing and denoising methods in terms of PSNR, SSIM, and subjective visual effects.

Description

Technical Field

[0001] This invention relates to the field of digital image processing technology, specifically to a unified image dehazing and denoising method based on unsupervised learning. Background Technology

[0002] Under severe smog conditions, atmospheric turbidity (such as particles and water droplets) greatly absorbs and scatters light, resulting in poor visibility and color artifacts in captured images. Furthermore, during image acquisition, limitations of electronic equipment and the external environment inevitably introduce irregularly distributed and amplitude-dependent random noise into blurred images. These image degradations make it difficult to perform advanced computer vision tasks such as object detection and image understanding.

[0003] Defogging methods can be broadly classified into two categories: methods based on manual priors and learning-based methods.

[0004] (1) Methods based on manual priors are generally based on atmospheric scattering models. He et al. proposed the dark channel prior (DCP) method to estimate the transmittance and atmospheric light values ​​of haze images. The DCP method is mainly expressed by the following formula:

[0005] I(x)=J(x)T(x)+A(x)(1-T(x));

[0006] Where J(x) is the clean image, T(x) is the transmission function, and A(x) is the ambient light.

[0007] Zhu et al. obtained the scene depth of hazy images based on color attenuation priors and used the estimated parameters to remove haze from the images. Qin et al. used feature extraction methods to remove haze. Salazar-Colores et al. integrated DCP and mathematical morphology operations to effectively evaluate the transmission graph. Liu et al. proposed nonlocal total variational regularization (NLTV) to refine the transmission graph.

[0008] (2) Learning-based dehazing methods have developed with the popularization of CNNs. Zhang et al. proposed the DCPDN dehazing method, which does not rely on physical scattering models. Cai et al. applied Dehazenet for end-to-end image dehazing. Li et al. used the AOD-Net dehazing network. Engin et al. used Cycle-Gan to estimate sharp images from blurry images. Zhang et al. proposed a fast and accurate multi-scale dehazing network, FAMED-Net, to estimate sharp images. Want et al. proposed AIPNet, which uses multi-scale CNNs to restore blurry images. Liu et al. iteratively solved the dehazing problem and used CNNs to generate outputs as inputs for the next iteration. Although these methods are effective in restoring image visibility, they are sensitive to noise, and since noise is unavoidable in natural images, the above methods may not achieve satisfactory visual results.

[0009] Noise reduction methods are mainly divided into two types: supervised and unsupervised.

[0010] (1) Supervised image denoising: Supervised methods learn a denoiser on a set of sharp / noisy image pairs. Zhang et al. proposed DnCNN, which combines convolutional neural networks with residual learning, for blind denoising. Gu et al. proposed a self-guided architecture top-down SGN network to handle image denoising tasks. Guo et al. proposed CBDNet, which contains two subnetworks for noise estimation and non-blind denoising in blind image denoising, respectively. Jia et al. proposed the FOCNet image denoising method by partially solving the optimal control problem. Yang et al. combined a set of simple basis denoisers to form a more efficient image denoiser. These deep denoisers require a large number of aligned sharp / noisy image pairs as training data, which is difficult to obtain. However, our method focuses on training with unpaired data.

[0011] (2) Unsupervised Image Denoising: Lehtinen et al. proposed an unsupervised learning-based image denoising method, Noise2Noise, instead of training with clear / noisy image pairs. Huang et al. proposed the Neighbor2Neighbor image denoising method. Quan et al. introduced an image denoising method that utilizes Bernoulli sampling to expand the training data. Dmitry proposed an image denoising method based on depth image prior (DIP); Kostadin et al. proposed a BM3D image denoising strategy based on sparse representation. Noise2Void (N2V) and Noise2Self (N2S) provide a denoising method based on self-predictive loss learning. While these denoising models can reduce noise in blurred images, they may cause blurring of some useful details.

[0012] There are two main types of merging defogging and noise reduction operations:

[0013] (1) Denoising before dehazing: For example, Schechner et al. added a local penalty term to the transmittance proportionally in Equation 1 to suppress noise during the dehazing process. Kaftory et al. proposed a distance-dependent regularization operator to remove noise before image dehazing. Matlin proposed a method to remove haze and noise from a single image using the BM3D denoising algorithm.

[0014] (2) Dehazing before denoising: For example, Jidesh et al. used the dark channel prior DCP method to estimate the medium transmittance and atmospheric light of hazy images, and then performed dehazing and denoising methods under Gamma distribution noise settings and linear blur artifacts. Lan et al. used DCP to remove haze and estimated blur / noise interference to recover the underlying image. Shen et al. proposed an iterative method to suppress noise and improve dehazing performance. However, cascaded processing rarely utilizes the correlation between the two operations, and the estimation error of the first-level model will be amplified in the second-level model.

[0015] Therefore, the present invention aims to provide a unified image dehazing and denoising method based on unsupervised learning to solve the above problems. Summary of the Invention

[0016] The purpose of this invention is to solve the above-mentioned problems and provide a unified image dehazing and denoising method based on unsupervised learning, which can efficiently remove haze and noise from hazy images.

[0017] To achieve the above objectives, the technical solution of the present invention is as follows:

[0018] A unified image dehazing and denoising method based on unsupervised learning includes the following steps:

[0019] S1, take the original smog image I haze The input is fed into a dehazing module trained using the dark channel prior DCP method and generative adversarial training to obtain a dehazed, noisy image J. dehaze ;

[0020] S2, Transfer the original smog image I haze The image J is input to a downsampling-based denoising module for denoising, thus obtaining the denoised hazy image. noise ;

[0021] S3, the images J obtained from S1 and S2 dehaze and image J noise The input is fed into the image fusion module, which fuses the dehazed and denoised images using region similarity as the objective function, and outputs a fused image J that balances dehazing and denoising. fused .

[0022] Furthermore, in step S1, the dehazed noisy image J dehaze The specific method for generating it is as follows:

[0023] S1-1. Generate ambient light A using the dark channel prior DCP method; initially reconstruct the image J. DCP and transport mapping T DCP ;

[0024] S1-2, Transfer mapping T DCP High-precision transmission images T are generated through a transmission sensing network. En (x), and using T En (x), atmospheric ambient light A(x), and the original haze image I haze Obtain a dehazed and noisy image.

[0025] S1-3, Use discriminator D to process the dehazed and noisy image J dehaze The evaluation is performed, a loss function is constructed, and gradients are backpropagated for training.

[0026] Furthermore, the downsampling in step S2 includes maximum downsampling and average downsampling. The downsampling operation divides the noisy image y into [W / k]×[H / k], and adjacent elements are randomly selected in a k×k pixel unit. Then, the maximum or average operation is selected for the pixels in each unit as the new image pixel.

[0027] Further, the denoising method in step S2 is as follows: by comparing and reconstructing the images before and after denoising and before and after sampling, a loss function is constructed for network training; the constructed loss function includes: the average downsampled image is denoised and reconstructed into the maximum downsampled image, the maximum downsampled image is denoised and reconstructed into the average downsampled image, and the consistency between the original image after denoising and the image after maximum / average downsampling and the image after direct maximum / average downsampling without denoising.

[0028] Furthermore, the specific steps of the noise reduction module in step S2 are as follows:

[0029] S2-1. Use the average downsampling g and the maximum downsampling g′ to generate noise sub-maps g(y) and g′(y) respectively;

[0030] S2-2, Denoising network f using shared weights θ (·) Denoise the original noisy image y, the average downsampled noise sub-image g(y), and the maximum downsampled noise sub-image g′(y) respectively, and obtain the result f. θ (y), f θ (g(y)) and f θ (g′(y));

[0031] S2-3, regarding the f obtained in step S2-2 θ (y) Downsampling is performed using the average downsampler g and the maximum downsampler g′ from step S2-1 to obtain g(f) θ (y)) and g′(f θ (y));

[0032] S2-4. Calculate the reconstruction loss L rec1 L rec2 And consistent loss L con1 L con2 The gradient is calculated based on the total loss function L and the network weights are changed by backpropagation.

[0033] S2-5. Repeat steps S2-1 to S2-4.

[0034] Furthermore, in step S3, guided by the objective function, a Bregman-based optimization method is used to obtain the pre-fused image. The specific steps of the fusion module are as follows:

[0035] S3-1, By minimizing the objective function:

[0036]

[0037] To generate and fuse images J pre-fuse Where f represents the prefused image, ▽ represents the gradient calculation, and λ is the hyperparameter of the gradient feature constraint.

[0038] S3-2, Calculate the image pairs (J) respectively. denoise J pre-fuse ) and (J dehaze J pre-fuse The similarity between regions is used to obtain the region similarity map S. denoise-p With S dehaze-p ;

[0039] S3-3, The similarity map S of the regions denoise-p With S dehaze-p The fusion weights W are obtained by using the normalized exponential function softmax(·). denoise With W dehaze To make it satisfy W denoise +W dehaze =1;

[0040] S3-4. Calculate the final fused image J according to the following formula. fused :

[0041] J fused =J denoise lW denoise +J dehaze lW dehaze ,

[0042] Here, l represents element-wise multiplication.

[0043] Furthermore, the formula for calculating the region similarity in the fusion module described in step S3 is as follows:

[0044] S denoise-p =g(J denoise J pre-fuse (,N,T),

[0045] S dehaze-p =g(J dehaze J pre-fuse (,N,T),

[0046]

[0047] Where g(·) is the Local Structural Similarity Index (SSIM) operator between the corresponding image patches of each image pair; (i,j) represents the image pair (J denoise J pre-fuse ) and (J dehaze J pre-fuse ), μ i σ represents the average pixel value of image i. i σ represents the standard deviation of image i. ij Let c1 represent the covariance of the image pair (i,j), with c1 value being 0.02 and c2 value being 0.03.

[0048] In summary, the solution of this invention adds image fusion and other techniques to the dark channel prior dehazing algorithm and unsupervised denoising algorithm, thus completing the task of simultaneously dehazing and denoising images.

[0049] The problem solved and the significance of the present invention are as follows:

[0050] Because hazy images captured in scenes with scattering media (i.e., fog, mist, or smoke) suffer from reduced visibility, most existing haze image enhancement methods perform dehazing and denoising in stages. A major drawback is that estimation errors from the previous stage must propagate and amplify in subsequent stages, such as noise amplification after dehazing. To address this inconsistent degradation problem, this invention proposes a unified image dehazing and denoising algorithm based on unsupervised learning to simultaneously remove haze and suppress noise from a single haze image.

[0051] The method of this invention mainly consists of an unsupervised dehazing module, an unsupervised denoising module, and a region similarity fusion strategy. The present invention proposes an unsupervised transport-aware dehazing module to restore visibility and suppress depth-dependent noise propagation in the dehazing module. Furthermore, the method of this invention designs an unsupervised network of mean / maximum subsamplers in the denoising module, and proposes a region similarity fusion strategy to calculate the final result in order to fully utilize the correlation and complementarity between previous outputs. Moreover, the method of this invention outperforms other state-of-the-art dehazing and denoising methods in terms of PSNR, SSIM, and subjective visual effect.

[0052] Compared with existing technologies, the beneficial effects of this solution are:

[0053] (1) The method of the present invention is far superior to most state-of-the-art dehazing and denoising methods in terms of quantitative indicators Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM).

[0054] (2) The method of the present invention fully considers the correlation and complementarity between image dehazing and denoising, and unifies the two tasks of image dehazing and denoising while ensuring the effectiveness of the method.

[0055] (3) The method of the present invention has obvious advantages and robustness in image enhancement of blurred haze images.

[0056] (4) The method of the present invention effectively improves the performance of fuzzy image detection and recognition tasks. Attached Figure Description

[0057] Figure 1 This is a diagram illustrating the overall framework of a unified image dehazing and denoising algorithm based on unsupervised learning in an embodiment of the present invention.

[0058] Figure 2 This is a framework diagram of the image dehazing sub-algorithm module in an embodiment of the present invention;

[0059] Figure 3 This is a framework diagram of the image denoising sub-algorithm module in an embodiment of the present invention;

[0060] Figure 4 This is a schematic diagram of the average / maximum subsampler in an embodiment of the present invention;

[0061] Figure 5 This is a framework diagram of the image fusion sub-algorithm module in an embodiment of the present invention;

[0062] Figure 6 This is a practical example of the effect of a unified image dehazing and denoising algorithm based on unsupervised learning in an embodiment of the present invention. Detailed Implementation

[0063] To enable those skilled in the art to better understand the present invention, the technical solution of the present invention will be described in further detail below with reference to the embodiments and accompanying drawings. Obviously, the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort should fall within the scope of protection of the present invention.

[0064] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the embodiments.

[0065] Example:

[0066] This invention provides a unified image dehazing and denoising algorithm based on unsupervised learning (framework as follows). Figure 1 As shown), it includes the following steps:

[0067] Step 1: Extract the original smog image I haze Input defogging module (framework as follows) Figure 2 As shown), the ambient light A and the preliminary reconstructed image J are generated using the dark channel prior DCP method. DCP and transport mapping T DCP The specific generation method is determined by the following formula:

[0068]

[0069]

[0070]

[0071] Step 2: Then put T DCP High-precision transmission images T are generated through a transmission sensing network. En (x), and using T En (x), atmospheric ambient light A(x), and the original haze image I haze The dehazed image J is obtained from the following formula. dehaze :

[0072]

[0073] The transmission-aware network uses standard coding.

[0074]

[0075] The decoding structure and specific network parameters are shown in the table below:

[0076] Step 3: To make the recovered dehazed image closer to the clean image, the unpaired dehazed image J... dehaze and clean image J clear Simultaneously input to discriminator D, to distinguish J dehaze The defogging effect was evaluated. The table below shows the detailed parameters of the discriminator network D:

[0077]

[0078] Construct the loss function L total Training is performed using backpropagation of gradients. The loss function is as follows:

[0079]

[0080]

[0081] L idt =||I haze -I rec ||1,

[0082] Step 4: Extract the original smog image I haze Input noise reduction module (framework as follows) Figure 3 As shown in the image, the instantiated haze image is represented by y. Noise submaps g(y) and g′(y) are generated using the average downsampling method g and the maximum downsampling method g′, respectively.

[0083] The principle of average / maximum downsampling is as follows: Figure 4 As shown, with k=2, the noisy image y is divided into [W / k]×[H / k], and adjacent elements are randomly selected into a 2×2 pixel unit, as shown in yellow and green. In the average subsampler g, the average of two adjacent elements is calculated, which is the pixel value of the average subimage g(y). In the maximum subsampler g′(·), the maximum value of two adjacent elements is selected, which is the pixel value of the maximum subimage g′(y).

[0084] Step 5: Denoising network f using shared weights θ (·) Denoise the original noisy image y, the average downsampled noise sub-image g(y), and the maximum downsampled noise sub-image g′(y) respectively, and obtain the result f. θ (y), f θ (g(y)) and f θ (g′(y)).

[0085] Step 6: Apply the f obtained in Step 5 θ (y) Use the average downsampler g and the maximum downsampler g′ from step 4 to perform downsampling, and construct g(f) θ (y)) and g′(f θ (y)).

[0086] Step 7: The loss function needs to include four parts: f θ The reconstruction loss L for reconstructing (g(y)) into g′(y) is... rec1 f θ The reconstruction loss L of (g′(y)) to g(y) is... rec2 , measuring g(f θ The consistency loss L between (y) and g(y) con1 and measure g′(f θ The consistency loss L between (y) and g′(y) con2 Calculate the reconstruction loss L rec1 L rec2 And consistent loss L con1 L con2 The gradient is calculated based on the total loss function L and backpropagation is used for training. Finally, the trained network parameters are used to infer and generate a denoised image J. denoise Specifically, the total loss function is as follows:

[0087]

[0088] Among them, f θ (·) is the denoising network. σ and ξ are hyperparameters used to control the proportion of consistency loss. It is worth noting that we did not increase g(f) θ (y)) and g′(f θ The gradient of (y)). The adjustment of σ and ξ is gradual, by gradually increasing them to the set values ​​to make the training process more stable.

[0089] Step 8: Dehaze the image J obtained in Step 3. dehaze The denoised image J obtained in step 7 denoise Input image fusion module (framework such as) Figure 5 As shown, image fusion is achieved by minimizing the objective function between the images to be fused. Specifically, grayscale information and gradient features of the input images are extracted, and a Bregman-based optimization method is used to obtain the pre-fused image J. pre-fuse The objective function L that needs to be minimized is... f As shown below:

[0090]

[0091] Where f represents the pre-fused image, ▽ represents calculating its gradient; λ is the hyperparameter of the gradient feature constraint, set to 10. 2 .

[0092] Step 9: Calculate the image pairs (J) separately. denoise J pre-fuse) and (J dehaze J pre-fuse The similarity between regions is used to obtain the region similarity map S. denoise-p With S dehaze-p The higher the score in the region similarity map, the closer the image is to that region. pre-f The higher the similarity between regions, the better. Region similarity is calculated as follows:

[0093] S denoise-p =g(J denoise J pre-fuse (,N,T)

[0094] S dehaze-p =g(J dehaze J pre-fuse (,N,T)

[0095]

[0096] Where g(·) is the Local Structural Similarity Index (SSIM) operator between the corresponding image patches of each image pair; (i,j) represents the image pair (J denoise J pre-fuse ) and (J dehaze J pre-fuse ), μ i σ represents the average pixel value of image i. i σ represents the standard deviation of image i. ij Let c1 represent the covariance of the image pair (i,j), with c1 value being 0.02 and c2 value being 0.03.

[0097] Step 10: Transfer the region similarity map S denoise-p With S dehaze-p The fusion weights W are obtained by using the normalized exponential function softmax(·). denoise With W dehaze To make it satisfy W denoise +W dehaze =1. That is, we have the following expression:

[0098]

[0099] Step 11: Use the weight parameters W obtained in Step 10 denoise With W dehaze The final fused image J is calculated according to the following formula. fused :

[0100] J fused =J denoise lW denoise +J dehaze lW dehaze ,

[0101] Here, l represents element-wise multiplication.

[0102] The final fusion comparison image is as follows Figure 6 As shown.

[0103] Through the above embodiments of the present invention, the method of the present invention mainly consists of an unsupervised dehazing module, an unsupervised denoising module, and a region similarity fusion strategy. The present invention proposes an unsupervised transport-aware dehazing module to restore visibility and suppress depth-dependent noise propagation in the dehazing module. Furthermore, the method of the present invention designs an unsupervised network of mean / maximum subsamplers in the denoising module, and proposes a region similarity fusion strategy to calculate the final result in order to fully utilize the correlation and complementarity between previous outputs. Moreover, the method of the present invention outperforms other state-of-the-art dehazing and denoising methods in terms of PSNR, SSIM, and subjective visual effects.

[0104] The above specific embodiments are merely explanations of the present invention and are not intended to limit the present invention. After reading this specification, those skilled in the art can make modifications to these embodiments without contributing any inventive step, but as long as they are within the scope of the claims of the present invention, they are protected by patent law.

Claims

1. A unified image defogging and denoising method based on unsupervised learning, characterized in that: The method comprises the following steps: S1, the original haze picture input to the dark channel prior (DCP) based and the generative adversarial trained dehazing module, obtaining the dehazed noisy image ; S2, the original haze picture is input to a denoising module based on down-sampling for denoising, i.e. obtaining a denoised haze image ; S3, the image obtained in S1 and S2 are fused and image The dehazed image and the de-noised image are fused by taking the regional similarity as a target function in the image fusion module, and a fused image considering dehazing and de-noising is output , comprising: In step S3, a pre-fusion image is obtained by using a Bregman-based optimization method guided by the target function, and the specific method steps of the fusion module are as follows: S3-1, minimizing the target function: to generate a pre-fusion image ; wherein denotes the pre-fusion image, represents a gradient; is a hyperparameter of the gradient feature constraint; S3-2, respectively calculate the region similarity between the image pair With the region similarity, and obtain a region similarity map With ; S3-3, region similarity map with by a normalized exponential function to obtain fusion weights with such that ; S3-4. Calculate the final fused image according to the following formula : , wherein denotes an element-wise multiplication.

2. The unified image defogging and denoising method based on unsupervised learning according to claim 1, characterized in that: The dehazed noisy image in step S1 The specific method for generating is: S1-1, generating atmospheric ambient light using a dark channel prior (DCP) method , preliminary restoration of the image and transmission mapping ; S1-2, mapping the transmission Generating high-precision transmission images through transmission-aware networks , and using , atmospheric ambient light , and original hazy pictures , to obtain a dehazed noisy image ; S1-3, using discriminator to defog a noisy image evaluation, construct loss function, backpropagation gradient training.

3. The unified image defogging and denoising method based on unsupervised learning according to claim 1, characterized in that: The downsampling in step S2 includes maximum downsampling and average downsampling, and the downsampling operation divides the noise image into , adjacent elements are randomly selected in a pixel unit, and the maximum or average operation is selected for each pixel in the unit as a new image pixel point.

4. The unified image defogging and denoising method based on unsupervised learning of claim 1, characterized in that: The denoising method in step S2 is as follows: by comparing and reconstructing the images before and after denoising and before and after sampling, a loss function is constructed for network training. The constructed loss function comprises: consistency between the average down-sampling image denoising reconstruction into the maximum down-sampling image, the maximum down-sampling image denoising reconstruction into the average down-sampling image, and the original image after denoising and the image directly subjected to maximum / average down-sampling without denoising.

5. The unified image dehazing and de-noising method based on unsupervised learning of claim 1, wherein: The specific method steps of the denoising module in step S2 are as follows: S2-1, using an average down-sampler and a maximum down-sampler respectively generate noise sub-graphs and ; S2-2, denoising network using shared weight Denoise the original noise image , the average down-sampling noise sub-image , and the maximum down-sampling noise sub-image , respectively, to obtain the results , , and ; S2-3, downsampling the result obtained in step S2-2 using the average down-sampler in step S2-1 and the maximum down-sampler to obtain and ; S2-4, compute reconstruction loss and consistency loss , compute gradient and backpropagate changes to network weights according to total loss function ​ S2-5, repeating steps S2-1 to S2-4.

6. The unified image dehazing and de-noising method based on unsupervised learning of claim 1, wherein: The regional similarity calculation formula in the fusion module in step S3 is as follows: ， ， ； wherein, is a local structural similarity index (SSIM) operator between respective image patches of each image pair; represents an image pair and , denotes the pixel mean of the image , denotes the standard deviation of the image , denotes the covariance of the image pair , has a value of 0.02, has a value of 0.03.

Images