A real scene visual enhancement method for simultaneous dehazing and resolution improvement

Abstract

The visual enhancement method of a real scene that simultaneously removes haze and improves resolution described in the present invention first uses a rank-one prior to obtain an initial defogging picture and a scattered light map, and then combines the rank-one prior defogging and a super-resolution deep neural network prior for processing various image degradation problems under the framework of a variational model, and uses a semi-quadratic splitting (HQS) optimization algorithm to alternately iteratively estimate the scattered light map for removing haze and seek a high-quality visual enhancement image through a deep super-resolution network. The method described in the present invention can also handle other complex scenes and improve the clarity and recognizability of the acquired image.

Description

Technical Field

[0001] The invention relates to the technical field of image processing, and in particular to a real scene visual enhancement method for simultaneously removing fog and improving resolution. Background Art

[0002] When capturing images in bad weather (sandstorms or haze) and different imaging conditions (underwater or low illumination), due to the large amount of particles suspended in the environmental medium (such as haze, sand and dust) or due to the absorption, attenuation, discoloration and interference of water impurities when the light penetrates the water, the observed images will have low visibility, quality degradation and color distortion. These undesirable degradations will affect the visibility of the image, and then affect subsequent image processing tasks such as object recognition, road monitoring systems, remote sensing systems, etc. Therefore, under complex imaging conditions such as haze, sandstorms, underwater or low illumination, visibility enhancement of outdoor images has always been an important task in computer vision.

[0003] At present, based on traditional models or neural network training, domestic and foreign scholars in the field of computer vision have proposed many algorithms related to dehazing and image super-resolution. First of all, in the field of image dehazing, the most famous image dehazing method is the image dehazing method based on dark channel prior (DCP) proposed by Kaiming He. The dark channel prior is based on the observation that in most non-sky patches, the intensity of some pixels in at least one color channel is very low, close to zero in fog-free outdoor images. In addition, in recent years, many deep learning network methods for dehazing have also been widely proposed, among which Cai et al. proposed a DehazeNet for estimating medium transmission maps, Li et al. proposed an end-to-end AOD-Net that can directly output fog-free images, and some use adversarial generative networks (HardGAN) to solve the dehazing problem.

[0004] Although the above algorithms have achieved good defogging effects, many restored haze-free images are still damaged by compression artifacts and block artifacts; in addition, the noise that originally existed in the original image will be amplified during the defogging process, which will reduce the clarity and visualization of the final restored image. Considering that the image super-resolution algorithm can use higher-dimensional image information to achieve better image restoration, the defogging image can be regarded as a low-resolution image, and the image super-resolution method can be used to improve the image resolution to obtain more details and reduce artifacts and noise.

[0005] Single image super-resolution (SISR) can generally be divided into the following types: interpolation-based methods, prior-based optimization methods, and neural network-based methods. Among the interpolation-based methods, commonly used methods are bilinear and bicubic interpolators, Lanczos and nearest neighbor, etc. These methods are convenient, but have limited performance in blurred situations. Among the prior-based methods, Yang proposed an efficient sparse representation prior for image super-resolution, etc. Many neural network methods are also used to process low-resolution images, such as IRCN N, SRM D, SRCN N, VDSR, etc.

[0006] Although the above work can achieve good results in image dehazing or image super-resolution tasks, the restored images either cannot achieve the dehazing effect well or are damaged by artifacts and noise, making it difficult to obtain clear non-degraded images under harsh environmental conditions. Therefore, it is a technical problem that needs to be solved to reduce artifacts and noise in dehazed images, improve image resolution, and obtain clearer visual effect images while removing haze. Summary of the invention

[0007] To solve the above technical problems, the present invention is based on a rank-one prior and a super-resolution deep neural network for processing various image degradation problems as a prior, and proposes a visual enhancement method for real scenes that simultaneously removes haze and improves image resolution under the framework of a variational model. While obtaining a dehazed image, the resolution of the image is improved to obtain a clearer visual effect.

[0008] The method for simultaneously removing fog and improving resolution of a real scene visual enhancement according to the present invention comprises the following steps:

[0009] S1. Input haze image I(x) and obtain the initial scattered light map based on rank-one prior and the initial dehazed image J(x);

[0010] S2. Based on the variational model framework, an optimization problem is proposed by combining rank-one prior and super-resolution deep neural network prior.

[0011] S3, by introducing auxiliary variables u and v, adding penalty terms, and based on the semi-quadratic splitting method, a new unconstrained optimization problem is obtained;

[0012] S4. Using the alternating minimization algorithm, the new unconstrained optimization problem is solved to obtain the final visually enhanced image.

[0013] Furthermore, S1 is specifically:

[0014] S1-1. Calculate the unified radiance S of each R, G, and B channel based on the input haze image I(x) u, which reflects uniform ambient light:

[0015]

[0016] Ω=m×n, c={R, G, B},

[0017] Where x is the pixel position in the image, c refers to any one of the three channels R, G, and B, and I c (x) is the color channel map of the haze image, m and n are the corresponding length and width of the haze image, and Ω is the area of ​​the haze image;

[0018] S1-2, Regularized unified radiance S u , and obtain a unified spectrum S describing the spectral characteristics of the ambient light source nu :

[0019]

[0020] S1-3. Obtain the initial scattered light map based on the rank-one prior:

[0021]

[0022] Pick The indices of the pixels in the highest 0.1% norm in , and the pixel mean values ​​in I(x) corresponding to these pixel indices are taken as the global atmospheric light A, and the initial dehazed image J(x) is obtained as:

[0023]

[0024] where x is the pixel position in the image, ω∈(0, 1] is a residual parameter, and t0 is a small positive number used to avoid division by zero in the denominator.

[0025] Furthermore, in S2, based on the variational model framework, combined with a rank-one prior and a super-resolution deep neural network prior, the optimization problem proposed is:

[0026]

[0027] in, represents the convolution operation between the assumed blur kernel k and the high-resolution image J to be restored, ↓ s is a subsequent downsampling operation with a scaling factor of s; the first term in the formula Is a measure of the scattered light map that needs to be updated For the initial scattered light map The data fidelity term, the second term is the data fidelity term obtained according to the physical model of dehazing, Φ(J) is the super-resolution network trained by SRMD (super-resolution network for multiple degradations), and is used as the model prior in a plug-and-play manner; α, β are the parameters controlling the fidelity term, μ is the parameter controlling the variational term, and λ is the parameter controlling the super-resolution network prior term; Here, Approximating two discrete differential operators using backward differences with periodic boundary conditions and

[0028]

[0029] in, express The pixel value of the i-th row and j-th column in .

[0030] Furthermore, in S3, by introducing auxiliary variables u and v and adding penalty terms, based on the semi-quadratic splitting (HQS) method, a new unconstrained optimization problem is obtained:

[0031]

[0032] Among them, γ and σ are penalty parameters. When γ and σ approach infinity, the result of S3 tends to the result of S2.

[0033] Furthermore, in S4, the alternating minimization algorithm is used to obtain the following sub-problems, and then the scattered light map is iteratively solved respectively: , auxiliary variables u and v, low-resolution image in, is the updated scattered light map at the k+1th iteration, u k +1 、v k+1 is the auxiliary variable updated at the k+1th iteration, is the updated low-resolution image at the k+1th iteration:

[0034]

[0035] S4-1, using the split-Bregman method to solve, we get The results and the displayed solution at the k+1th iteration based on the x-direction and the y-direction

[0036]

[0037] in, and are Fourier transform and inverse Fourier transform, respectively. Yes The complex conjugate operator of ;

[0038]

[0039]

[0040] here and They are gradient operators based on the x-direction and y-direction respectively. and are auxiliary variables updated at the kth iteration based on the x-direction and the y-direction, and are the parameters updated in the kth iteration based on the x-direction and the y-direction respectively;

[0041]

[0042] in,

[0043] S4-2, directly obtain the closed-form solution,

[0044]

[0045] In order to obtain a more natural dehazed image, the parameter ω∈(0, 1] is introduced, and the above formula can be further written as:

[0046]

[0047] S4-3. Sub-questions It can be further written as:

[0048]

[0049] Here, the final visually enhanced image Expressed as the low-resolution image v, blur kernel k, noise level Function of the scaling factor s and parameter λ:

[0050]

[0051] Where Θ represents the parameters of MAP inference, and the mapping function Learned by the CNN-based super-resolution network SRMD.

[0052] The beneficial effects described in the present invention are as follows: the present invention first uses a rank-one prior to obtain an initial dehazed image and a scattered light map, and then, under the framework of a variational model, combines the rank-one prior dehazing and a super-resolution deep neural network prior for processing a variety of image degradation problems, and uses a half-quadratic splitting (HQS) optimization algorithm to alternately iteratively estimate the scattered light map for removing haze and seek high-quality visual enhancement images through a deep super-resolution network; not only can the adverse effects of haze be greatly removed, but also artifacts can be reduced and the appearance of noise can be suppressed, and finally a high-quality image with clearer details and higher visualization can be obtained. In addition, the proposed method can also handle other complex scenes, that is, image restoration under different weather and imaging conditions, such as sandstorm weather, underwater imaging, and low-light imaging, with high robustness.

[0053] The present invention can be further applied as image processing software to other image processing tasks, such as road monitoring systems, underwater detection, monitoring systems under low illumination conditions at night, and target recognition, to improve the clarity and recognizability of acquired images in complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 Flowchart for obtaining the initial scattered light map and dehazed image using rank-one prior;

[0055] Figure 2 A flowchart of an image visual enhancement algorithm of the present invention that combines a rank-one prior and a super-resolution deep neural network prior for processing various image degradation problems under a variational model;

[0056] Figure 3 The experimental results on foggy days, (a) is a foggy image, (b) is an image restored using the dark channel method, (c) is an image restored by directly using the rank-one prior, and (d) is a visually enhanced image obtained by the method adopted by the present invention;

[0057] Figure 4 The experimental results of sandstorm weather, (a) is a sandstorm weather image, (b) is an image restored by using the famous dark channel method, (c) is an image restored by directly using the rank-one prior, and (d) is a visually enhanced image obtained by the method adopted by the present invention;

[0058] Figure 5 The image restoration experiment results under different weather and imaging conditions, where (a) is foggy, (b) is dusty, (c) is underwater, and (d) is low-light. DETAILED DESCRIPTION

[0059] In order to make the contents of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments in conjunction with the accompanying drawings.

[0060] The method for simultaneously removing fog and improving resolution of a real scene visual enhancement according to the present invention comprises the following steps:

[0061] S1. Input haze image I(x) and obtain the initial scattered light map based on rank-one prior and the initial dehazed image J(x);

[0062] The mathematical model describing the degradation caused by the absorption and scattering of light by particles in the atmosphere is usually:

[0063]

[0064] Where x is the pixel position in the image, represents the dot product, I(x) and J(x) are the image to be dehazed and the haze-free image to be restored, A is the global atmospheric light, and t(x) is the air refractive index.

[0065] The present invention does not consider solving the air refractive index t(x), but instead considers solving the scattered light map describing the part of the ambient scattered light that affects the imaging. Furthermore, we can Rewrite the model to solve the haze-free image J(x) to be restored:

[0066]

[0067] Among them, ω∈(0,1] is a residual parameter, which in practice keeps a small amount of haze for distant objects, t0 is a small positive number to avoid division by zero, t0=0.001; Based on the above model, in order to obtain the restored dehazed image, it is necessary to solve the scattered light map and global atmospheric light A.

[0068] like Figure 1 , the specific steps of S1 are:

[0069] S1-1. Calculate the unified radiance S of each R, G, and B channel based on the input haze image I(x) u , which reflects uniform ambient light:

[0070]

[0071] Ω=m×n, c={R, G, B},

[0072] Among them, m and n are the sizes of the image;

[0073] S1-2, Regularized unified radiance Su , and obtain a unified spectrum S describing the spectral characteristics of the ambient light source nu :

[0074]

[0075] S1-3. Obtain the initial scattered light map based on the rank-one prior:

[0076]

[0077] Pick The indices of the pixels in the highest 0.1% norm in , and the pixel mean values ​​in I(x) corresponding to these pixel indices are taken as the global atmospheric light A, and the initial dehazed image J(x) is obtained as:

[0078]

[0079] where x is the pixel position in the image, ω∈(0, 1] is a residual parameter, and t0 is a small positive number used to avoid division by zero in the denominator.

[0080] S2. Based on the variational model framework, an optimization problem is proposed by combining a rank-one prior and a super-resolution deep neural network prior.

[0081] After obtaining the initial scattered light map After obtaining the initial dehazed image J(x), the present invention combines the rank-one prior dehazing and a super-resolution deep neural network SRMD prior for processing various image degradation problems through a variational model framework, and proposes the following optimization problem:

[0082]

[0083] in, represents the convolution operation between the assumed blur kernel k and the high-resolution image J to be restored, ↓ s is a subsequent downsampling operation with a scaling factor of s; the first term in the formula is a measure of for The first term is the data fidelity term, the second term is the data fidelity term obtained according to the physical model of dehazing, Φ(J) is a super-resolution network trained by SRMD (super-resolution network for multiple degradations), and is used as a model prior in a Plug-and-Play manner; α and β are parameters that control the fidelity term, μ is a parameter that controls the variational term, and λ is a parameter that controls the super-resolution network prior.

[0084] here, Approximating two discrete differential operators using backward differences with periodic boundary conditions and

[0085]

[0086] in, express The pixel value of the i-th row and j-th column in .

[0087] S3, by introducing auxiliary variables u and v, adding penalty terms, and based on the semi-quadratic splitting (HQS) method, a new unconstrained optimization problem is obtained;

[0088]

[0089] Among them, γ and σ are penalty parameters. When γ and σ approach infinity, the result of formula (4) approaches formula (3).

[0090] S4. Solve the new unconstrained optimization problem using the alternating minimization algorithm to obtain the final visually enhanced image; specifically:

[0091] Using the alternating minimization algorithm, we can get the following sub-problems, and then iterate to solve the scattered light map Auxiliary variables u and v, low-resolution image

[0092]

[0093] Solution (5): We can use the split-Bregman method to solve:

[0094]

[0095] Alternating Solution and(u x ,u y ):

[0096]

[0097] Since the right side of equation (9) is differentiable, using fast Fourier transform (FFT), we can get The results:

[0098]

[0099] in,

[0100]

[0101] Using the generalized contraction formula, solving formula (10), we obtain The display solution of :

[0102]

[0103] in,

[0104] Solve (6): We can directly obtain the closed-form solution,

[0105]

[0106] In order to obtain a more natural dehazed image, similar to model (2), the parameter ω∈(0, 1] is introduced, and formula (13) can be further written as:

[0107]

[0108] Solution (7): Formula (7) can first be further written as

[0109]

[0110] Here, the final visually enhanced image Expressed as the low-resolution image v, blur kernel k, noise level Function of the scaling factor s and parameter λ:

[0111]

[0112] Where Θ represents the parameters of MAP inference, and the mapping function Learned by the cNN-based super-resolution network SRMD.

[0113] At this point, according to the above iterative optimization solution process, the optimized scattered light map can be obtained And the final dehazed and improved resolution visually enhanced image

[0114] like Figure 3 As shown, in haze weather, compared with the well-known dark channel prior method, the method described in the present invention is as follows Figure 3 As shown in (d), no block artifacts and fog will appear when processing the sky and non-sky areas, and the restored image details are clearer. Figure 3 (b) The dark channel prior produces undesirable block artifacts and noise during processing, and the overall image effect is blurred. In addition, compared with the image directly restored using only the rank-one prior, the present invention greatly removes haze while reducing artifacts and suppressing the appearance of noise.

[0115] Figure 4 It shows that in dusty weather, dark channel prior methods such as Figure 4As shown in (b), the dust cannot be removed well. The image restored directly using the rank-one prior is as follows: Figure 4 As shown in (c), although the dust can be removed, the noise existing in the original image is also amplified. The method described in the present invention is as follows Figure 4 As shown in (d), the dust is removed while the noise is suppressed, and the restored image is clearer.

[0116] Figure 5 The robustness of the present invention is demonstrated. When processing degraded images in different weather conditions and conditions, the method of the present invention can still achieve good results and restore more details.

[0117] The variational model described in the present invention not only adds the total variation term, but also adds the rank-one prior and the super-resolution deep neural network prior into the same optimization problem, wherein the total variation term can effectively remove the undesirable noise in the image, rely on gradient descent to smooth the image, effectively smooth the image inside the image, and obtain as sharp an edge as possible at the edge of the image; the rank-one prior can effectively remove the adverse effects of haze and dust in the image; the super-resolution deep neural network can effectively remove the adverse effects of compression artifacts, block artifacts and noise, and enhance the visibility of the image. Based on this, the visual enhancement method for real scenes that simultaneously removes haze and improves image resolution proposed by the present invention under the framework of the variational model can improve the resolution of the image while obtaining the dehazed image, thereby obtaining a clearer visual effect.

[0118] The above description is only a preferred embodiment of the present invention and is not intended to be a further limitation of the present invention. All equivalent changes made using the contents of the present specification and drawings are within the protection scope of the present invention.

Claims

1. A method for simultaneously removing fog and improving resolution of real scene visual enhancement, characterized in that: The method steps are: S1. Input haze image I(x) and obtain the initial scattered light map based on rank-one prior and the initial dehazed image J(x); S2. Based on the variational model framework, combined with the rank-one prior and the super-resolution deep neural network prior, an optimization problem is proposed, specifically: in, represents the convolution operation between the assumed blur kernel k and the high-resolution image J to be restored, ↓ s is a subsequent downsampling operation with a scaling factor of s; the first term in the formula Is a measure of the scattered light map that needs to be updated For the initial scattered light map The data fidelity term, the second term is the data fidelity term obtained according to the physical model of dehazing, Φ(J) is the super-resolution network trained by SRMD, and the Plug-and-Play method is used as the model prior; α, β are the parameters controlling the fidelity term, μ is the parameter controlling the variational term, and λ is the parameter controlling the super-resolution network prior term; Here, Approximating two discrete differential operators using backward differences with periodic boundary conditions and in, express The pixel value of the i-th row and j-th column in; S3. By introducing auxiliary variables u and v and adding penalty terms, a new unconstrained optimization problem is obtained based on the semi-quadratic splitting method, which is: Among them, γ and σ are penalty parameters. When γ and σ approach infinity, the result of S3 tends to the result of S2. S4. Using the alternating minimization algorithm, the new unconstrained optimization problem is solved to obtain the final visually enhanced image.

2. A method for simultaneously removing fog and improving resolution of real scene visual enhancement according to claim 1, characterized in that: S1 is specifically: S1-1. Calculate the unified radiance s of each R, G, and B channel based on the input haze image I(x) u : Ω=m×n,c={R,G,B} Where x is the pixel position in the image, c refers to any one of the three channels R, G, and B, and I c (x) is the color channel map of the haze image, m and n are the length and width of the haze image, and Ω is the area of ​​the haze image; S1-2, Regularized unified radiance s u , and obtain a unified spectrum S describing the spectral characteristics of the ambient light source nu : S1-3. Obtain the initial scattered light map based on the rank-one prior: Pick The indices of the pixels in the highest 0.1% norm in , and the pixel mean values ​​in I(x) corresponding to these pixel indices are taken as the global atmospheric light A, and the initial dehazed image J(x) is obtained as: where x is the pixel position in the image, ω∈(0, 1] is a residual parameter, and t0 is a small positive number used to avoid division by zero in the denominator.

3. The method for simultaneously removing fog and improving resolution of a real scene visual enhancement according to claim 1, characterized in that: In S4, the alternating minimization algorithm is used to obtain the following sub-problems, and then the scattered light map is iteratively solved Auxiliary variables u and v, low-resolution image in, is the updated scattered light map at the k+1th iteration, u k+1 、v k+1 is the auxiliary variable updated at the k+1th iteration, is the updated low-resolution image at the k+1th iteration: S4-1, using the split-Bregman method to solve, we get The results and the displayed solution at the k+1th iteration based on the x-direction and the y-direction in, and are Fourier transform and inverse Fourier transform, respectively. Yes The complex conjugate operator of here and They are gradient operators based on the x-direction and y-direction respectively. and are auxiliary variables updated at the kth iteration based on the x-direction and the y-direction, and are the parameters updated in the kth iteration based on the x-direction and the y-direction respectively; in, S4-2, directly obtain the closed-form solution, In order to obtain a more natural dehazed image, the parameter ω∈(0, 1] is introduced, and the above formula is further written as: S4-3. Sub-questions Further written as: Here, the final visually enhanced image Expressed as the low-resolution image v, blur kernel k, noise level Function of the scaling factor s and parameter λ: Where Θ represents the parameters of MAP inference, and the mapping function Learned by the CNN-based super-resolution network SRMD.

Images