Deep Network Image De-raining Method Driven by Gradient Prior

Abstract

The present invention relates to the technical field of image restoration, and discloses a method for removing rain from images based on a gradient prior-driven deep network, which includes: Step 1: Obtain a damaged image to be processed taken on a rainy day; Step 2: Integrate the model-driven method and the deep neural network method. When constructing the deep network framework model, introduce gradient domain features to construct an error control term for the model-driven deep network, and construct a gradient prior-driven deep network rain removal model; Step 3: Solve the gradient prior-driven deep network rain removal model, and output a rain streak image and a restored clear image. Compared with the prior art, on the basis of combining the model-driven method and deep learning compatibility, when constructing the deep network framework model, the present invention introduces gradient domain features to construct an error control term for the model-driven deep network, improving the image rain removal ability and the information restoration quality.

Description

Technical Field

[0001] The present invention relates to the technical field of image restoration, and particularly to a method for removing rain from images based on a gradient prior-driven deep network. Background Art

[0002] Optoelectronic imaging systems for outdoor operations are widely used in fields such as transportation, detection, target tracking, and recognition. However, outdoor operations of imaging systems are inevitably affected by weather factors. For example, the traces of raindrops falling will affect the quality of the final image acquisition, thus affecting the subsequent analysis and use of image information. To remove the influence of rain streaks and restore image target information as much as possible, researchers have proposed numerous methods, including model-driven methods and deep neural network-based methods. Model-driven methods require artificial setting of corresponding image prior knowledge to represent and protect image target information, and their optimization process requires a large amount of time and the search for appropriate model parameters. These factors restrict the improvement of image processing quality. Deep neural network-based methods have problems such as the need for a large number of training samples and the inability to explain how the network structure represents and protects images. When the processing object deviates greatly from the training samples, the processing effect of the neural network method is greatly reduced.

[0003] Although existing methods have combined model methods and deep learning methods, such as a single-image rain removal method based on deep learning and model driving, CN 111462014 A, which constructs a new network structure, this method uses information in the image spatial domain when constructing the model. According to existing research: a method for removing rain from traffic surveillance images based on anisotropic sparse gradients, CN201910783723.6, it is found that there are obvious gradient features between the image and the rain. Currently, no method considers using information in other domains of the image to further improve the rain removal and information restoration capabilities of the model-driven network. Summary of the Invention

[0004] Object of the Invention: Aiming at the problems existing in the prior art, the present invention provides a method for removing rain from images based on a gradient prior-driven deep network. On the basis of combining model-driven methods and deep learning compatibility, when constructing the deep network framework model, gradient domain features are introduced to construct an error control term for the model-driven deep network, improving the rain removal ability and information restoration quality of the image.

[0005] Technical Solution: The present invention provides a method for removing rain from images based on a gradient prior-driven deep network, including the following steps:

[0006] Step 1: Obtain the damaged image to be processed taken on a rainy day;

[0007] Step 2: Integrate the model-driven method and the deep neural network method. When constructing the deep network framework model, introduce gradient domain features to construct the error control term of the model-driven deep network, and construct the following gradient prior-driven deep network de-raining model:

[0008]

[0009] where \(o\), \(r\), and \(u\) respectively represent the damaged image taken on a rainy day, the rain streak image, and the restored clear image. and respectively represent the overall gradient operator, vertical direction gradient operator, and horizontal direction gradient operator of the image. \(\Psi(u)\) and \(\Phi(r)\) respectively represent the regularization term of the background image and the regularization term of the rain streak image; \(\alpha_1\), \(\alpha_2\), \(\lambda_1\), and \(\lambda_2\) respectively represent non-negative regularization coefficients. represents the L2 norm; is the error control term;

[0010] Step 3: Input the damaged image to be processed into the gradient prior-driven deep network de-raining model, and solve the gradient prior-driven deep network de-raining model to output the rain streak image \(r\) k+1 and the restored clear image \(u\) k+1 .

[0011] Furthermore, when solving in Step 3, first convert the bivariate solution problem in the model into two univariate solution problems:

[0012]

[0013]

[0014] Furthermore, the solution of the rain streak image \(r\) k+1 is as follows:

[0015]

[0016] where is the proximal gradient operator, which is used to control the regularization prior \(\Phi(r)\) of the image. The regularization prior function \(\Phi(r)\) is obtained through deep learning; the proximal gradient operator is composed of a 4-layer residual convolutional neural network.

[0017] Furthermore, the solution of the restored clear image \(u\) k+1 is as follows:

[0018]

[0019] where \(\eta_2\) is the iteration step size of \(u\) k+1 ,

[0020]

[0021] is the proximal gradient operator, which is used to control the regularization prior Φ(r) of the image. The regularization prior function Φ(r) is obtained through deep learning; the proximal gradient operator is composed of a 4-layer residual convolutional neural network.

[0022] Furthermore, the proximal gradient operator is composed of a 4-layer residual convolutional neural network. Each layer of the residual network structure is composed of two convolutional layers and a rectified linear unit (Relu) layer. Each convolutional layer performs convolution operations with 64 convolutional kernels of size 3*3. After the first convolutional layer operation, it passes through the rectified linear unit (Relu) layer and then performs the second convolutional layer operation.

[0023] Beneficial effects:

[0024] The present invention combines the advantages of model-driven methods and deep neural network methods, and introduces gradient domain information to enhance the error control ability of the model-driven network. This network not only makes the network structure interpretable, but also does not require prior knowledge design and parameter tuning, and uses gradient information to strengthen the information representation ability of the model-driven deep network, effectively improving the image de-raining ability and information restoration quality. Brief description of the drawings

[0025] Figure 1 is the residual network structure diagram of the present invention;

[0026] Figure 2 is the specific flowchart of the residual module of the present invention;

[0027] Figure 3 is the comparison chart of de-raining and information restoration performance between the present invention and other methods; among them, (a) clear reference chart; (b) rain-degraded chart; (c) sparse representation restoration chart; (d) deep learning restoration chart; (e) directional gradient restoration chart; (f) restoration chart of the present invention. Specific implementation manners

[0028] The present invention will be further described below with reference to the drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and cannot be used to limit the protection scope of the present invention.

[0029] The traditional mathematical model of obtaining an image in the spatial domain on a rainy day is:

[0030] o = r + u (1)

[0031] In the formula, o, r, and u respectively represent the damaged image taken on a rainy day, the rain streak image, and the clear image to be restored.

[0032] If the image has gradient domain information, then in the gradient domain space, the mathematical model of its rainy-day image gradient domain can be expressed as:

[0033]

[0034] In the formula, the gradient operator and respectively represent the vertical direction gradient operator and the horizontal direction gradient operator.

[0035] The existing publicly available model-driven depth rain removal models (CN 111462014 A) are:

[0036]

[0037] In the formula, M is the sparse coefficient, C is the convolutional dictionary, Φ(M) and Ψ(u) are the regularization terms of rain and clear images, is the model error control term of the model-driven deep network, which is used to control the rain information image obtained after training the model-driven deep network and the background image u to be close to the true solution. Since the regularization terms Φ(M) and Ψ(u) in formula (3) adaptively obtain the rain layer and background information through deep learning, the error control term has an important impact on the processing result of formula (3).

[0038] To further enhance the image rain removal ability and background information restoration, the present invention discloses a gradient prior-driven deep network image rain removal method, which combines the model-driven method and the deep neural network method. When constructing the deep network framework model, gradient domain features are introduced to construct the error control term of the model-driven deep network, and a gradient prior-driven deep network rain removal model is proposed. The specific rain removal method includes the following steps:

[0039] Step 1: Obtain the damaged image to be processed taken on a rainy day.

[0040] Step 2: Combine the model-driven method and the deep neural network method. When constructing the deep network framework model, introduce gradient domain features to construct the error control term of the model-driven deep network, and construct the following gradient prior-driven deep network rain removal model:

[0041]

[0042] Among them, o, r, and u respectively represent the damaged image taken on a rainy day, the rain streak image, and the clear image to be restored, and respectively represent the overall gradient operator, vertical direction gradient operator, and horizontal direction gradient operator of the image, Ψ(u) and Φ(r) respectively represent the regularization term of the background image and the regularization term of the rain streak image; α1, α2, λ1, and λ2 respectively represent non - negative regularization coefficients, represents the L2 norm; is the error control term.

[0043] Step 3: Input the damaged image to be processed into the gradient - prior - driven deep network de - raining model, and solve the gradient - prior - driven deep network de - raining model to output the rain streak image r k+1 and the restored clear image u k+1 .

[0044] The bivariate solution problem in Equation (4) can be converted into the following two univariate solution problems:

[0045]

[0046]

[0047] Equation (5) is equivalent to the quadratic approximation formula, that is:

[0048]

[0049] where,

[0050]

[0051] η1 is the step size, <·, ·> is the inner - product operator, represents the first - order derivative of g(r k ) with respect to r k .

[0052] The inner - product expansion of Equation (7) is equivalent to:

[0053]

[0054] The solution of Equation (8) is:

[0055]

[0056] where, is the proximal gradient operator, which is used to control the regularization prior Φ(r) of the image. Since the regularization prior function Φ(r) here has no specific mathematical form and is obtained through deep learning. The proximal gradient operator is composed of a 4 - layer residual convolutional neural network. See Appendix Figure 1 .

[0057] Equation (6) uses the same method as Equation (5) to obtain the result about uk+1 Solution:

[0058]

[0059] where η2 is u k+1 Iteration step size

[0060]

[0061] Residual network, for its structure diagram, see Appendix Figure 1 , which is composed of 4-layer residual convolutional neural network. Each layer of the residual network structure is composed of two convolutional layers and a rectified linear unit (Relu) layer. Each convolutional layer performs convolution operation with 64 convolutional kernels of size 3*3. After the first convolutional layer operation, it passes through the rectified linear unit (Relu) layer and then performs the second convolutional layer operation.

[0062] According to the above solutions for r k+1 and u k+1 , the algorithm flow of the depth network image de-raining method driven by gradient prior is as follows:

[0063]

[0064]

[0065] In the above formula, η = 0.9 is the learning rate. Other parameters in the above de-raining are automatically obtained by the designed model through training on the training samples during the experiment, without the need for parameter tuning or manual setting. Using the above method for de-raining, the de-raining and information restoration performances of the present invention and other methods are as Figure 3 , and it can be found from the figure that the de-raining effect of the method of the present invention is better and the information restoration quality is higher.

[0066] To better describe the performance of the de-raining method of the present invention, see Table 1. The de-raining performance is compared using the evaluation values PSNR and SSIM. It can be clearly seen from the table that the evaluation values PSNR and SSIM of the method of the present invention are higher, reflecting better image de-raining and information restoration quality.

[0067] Table 1 Comparison of de-raining performance evaluation values of the method of the present invention and other methods for simulated images

[0068] Method Sparse Representation Directional Gradient Deep Learning Patent Method PSNR 20.28dB 28.73dB 28.94dB 29.16dB SSIM 0.7511 0.8934 0.8969 0.8992 .

[0069] The above embodiments are only for illustrating the technical concept and characteristics of the present invention, and their purpose is to enable those who are familiar with this technology to understand the content of the present invention and implement it accordingly, and cannot be used to limit the protection scope of the present invention. Any equivalent transformation or modification made according to the spirit and essence of the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for removing rain from images in a deep network driven by gradient prior, characterized in that It includes the following steps: Step 1: Obtain the damaged image to be processed taken on a rainy day; Step 2: Integrate the model-driven method and the deep neural network method. When constructing the deep network framework model, introduce gradient domain features to construct the error control term of the model-driven deep network, and construct the following rain removal model driven by gradient prior: Among them, o, r, and u respectively represent the damaged image taken on a rainy day, the rain streak image, and the clear image to be restored. and respectively represent the overall gradient operator, the vertical gradient operator, and the horizontal gradient operator of the image. Ψ(u) and Φ(r) respectively represent the regularization term of the background image and the regularization term of the rain streak image; α1, α2, λ1, and λ2 respectively represent non-negative regularization coefficients. represents the L2 norm; is the error control term; Step 3: Input the damaged image to be processed into the gradient prior-driven deep network de-raining model, solve the gradient prior-driven deep network de-raining model, and output the rain streak image r k+1 and the restored clear image u k+1 .

2. The method for removing rain from a deep network image based on gradient prior driving according to claim 1, wherein When solving the said Step 3, first convert the bivariate solution problem in the model into two univariate solution problems:

3. The method for removing rain from a deep network image based on gradient prior driving according to claim 2, wherein The rain mark image r k+1 has the following solution: Among them, is the proximal gradient operator, which is used to control the regularization prior Φ(r) of the image. The regularization prior function Φ(r) is obtained through deep learning; the proximal gradient operator is composed of a 4-layer residual convolutional neural network.

4. The method for removing rain from images of a deep network driven by gradient prior according to claim 2, wherein The restored clear image u k+1 has the following solution: where η2 is u k+1 the iteration step size is the proximal gradient operator, which is used to control the regularization prior Φ(r) of the image. The regularization prior function Φ(r) is obtained through deep learning; the proximal gradient operator is composed of a 4-layer residual convolutional neural network.

5. The method for removing rain from a deep network image driven by gradient prior according to claim 3 or 4, characterized in that, The proximal gradient operator is composed of a 4-layer residual convolutional neural network. Each layer of the residual network structure consists of two convolutional layers and a rectified linear unit (Relu) layer. Each convolutional layer performs a convolution operation with 64 convolutional kernels of size 3*3. After the first convolutional layer operation, it passes through the rectified linear unit (Relu) layer and then undergoes the second convolutional layer operation.

Images