An image deraining method based on physical modeling and generative adversarial learning

Abstract

The present invention belongs to the field of image processing and computer vision, and proposes an image deraining method based on physical modeling generative adversarial learning, and realizes generative adversarial learning through a double-layer paradigm model. The generative adversarial network is mainly composed of two parts: a generator and a discriminator. The generator generates the target result by using the input random noise, and the discriminator discriminates the result generated by the generator until the discriminator believes that the result of the generator meets the target requirement. The traditional generative adversarial network is completely data-driven, and the functions of the generator and the discriminator are realized through a convolutional neural network. However, due to the particularity of the deraining task, a large number of data pairs are difficult to obtain. Therefore, we realize generative adversarial learning from the perspective of model optimization. Based on the means of the double-layer paradigm, the generator and the discriminator are modeled respectively through the upper and lower layers of the model, and the optimal deraining result is obtained through the upper and lower layers of adversarial learning.

Description

Technical Field

[0001] The invention belongs to the field of image processing and computer vision, and relates to an image deraining method based on physical modeling and generative adversarial learning. Background Art

[0002] With the development of computer hardware equipment, the processing speed of image and video data on PC and mobile terminals has been greatly improved, and the application scope of image and video data in people's daily life has also been increased, which has brought many conveniences to human production and life, such as: facial recognition technology when clocking in at work, reversing images required when driving, traffic safety monitoring systems, etc. Along with this, many technologies related to image equipment, video processing, etc. are also constantly developing and progressing, which in turn promotes the application scope of the entire image field to become wider and wider. However, one limitation in the field of vision that cannot be ignored is the impact of bad weather. For example, in rainy conditions, not only will people's naked eye recognition of objects be affected, but the clarity and visibility of data acquisition equipment such as cameras and cameras will also be greatly reduced. Therefore, algorithms based on image data, especially image processing algorithms collected in outdoor environments, will be affected. In general applications, we rarely consider the impact of outdoor weather when operating on actually captured images and videos. This will affect the effectiveness and stability of the computer vision system. The image finally obtained by the system becomes blurred and unclear, with reduced contrast, reduced or lost details, resulting in reduced image quality, increased image noise, and occlusion and loss of important background information.

[0003] Therefore, rain removal has important practical significance for image processing and has great development prospects. It can not only be used as a separate task, such as restoring license plates taken on rainy days in video surveillance systems, but can also be used in the preprocessing steps of many computer vision system tasks to prevent inaccurate information caused by loss of details, and ensure the accuracy and effectiveness of the computer vision system. However, because rain is randomly distributed in space and has no regularity, and rain is falling at high speed, it is easy to be confused with moving objects. Therefore, rain removal, that is, the detection and removal of raindrops, is very difficult.

[0004] At present, many excellent algorithms have been proposed for the image deraining problem, which can be essentially divided into two categories: deraining methods based on traditional models and deraining methods based on deep learning.

[0005] The image deraining method based on traditional models mainly uses model priors to model rain lines and natural backgrounds, and uses the model to characterize the characteristics of rain lines and separate signals. Yu Li et al. published "Rain Streak Removal using Layer Priors" in the 2016 "IEEE Conference on Computer Vision and Pattern Recognition" to model the prior information of background images and rain lines, and process rain lines of different scales. However, since this method only fits the distribution of rain lines through multiple Gaussian components, its ability is limited and the reliability of this method is low. CN108765327A uses discriminative sparse coding to characterize rain lines, but this will also cause a certain degree of color distortion in the output image. The development of deep learning has brought new solutions to image deraining from a data-driven perspective. CN110838095A uses a cyclic dense network to remove rain streaks and extracts sparse rain lines through iterations of a multi-layer network. CN110503609A introduces a non-local information perception network, which can automatically learn global information through training to help the repair network restore a clearer image; and introduces a local information perception network to effectively avoid the over-enhancement phenomenon in rainless areas, so that the repair network only focuses on the raindrop occlusion area. CN110378849A proposes an image defogging and deraining method based on a deep residual network. The method uses a nonlinear conversion module to extract high-level image features; and then decodes the feature map through a decoding module to obtain the residual between the rainless image and the rainy image. CN111815526A provides a rain streak removal method for rainy images based on image filtering and CNN, which solves the problems that the existing deraining methods are difficult to accurately characterize the rain model, the rain streaks cannot be removed cleanly, and the details are easily lost. Although the algorithm based on deep learning solves the deraining problem to a certain extent, it requires a large amount of rain-no rain data pairs for training to ensure the robustness of the algorithm. However, it is very difficult to collect rainy and non-rainy data pairs at the same time under natural conditions, and the types of existing data are relatively single. Therefore, the effectiveness of current deep learning-based image deraining methods in practical applications is very limited. It is only effective for scenes with similar characteristics to the training data, and cannot solve the real scenes with large differences from the training data characteristics well. Summary of the invention

[0006] The present invention establishes an image deraining method based on physical modeling generative adversarial learning, and realizes generative adversarial learning through a two-layer paradigm model. Specifically, the generative adversarial network is mainly composed of two parts: a generator and a discriminator. The generator generates the target result by using the input random noise, and the discriminator discriminates the result generated by the generator until the discriminator believes that the result of the generator meets the target requirement. The traditional generative adversarial network is completely data-driven, and the functions of the generator and the discriminator are realized through convolutional neural networks. However, due to the particularity of the deraining task, a large number of data pairs are difficult to obtain. Therefore, we realize generative adversarial learning from the perspective of model optimization. Based on the means of the two-layer paradigm, the generator and the discriminator are modeled respectively through the upper and lower layers of models, and the adversarial learning of the upper and lower layers is used to obtain the optimal deraining result.

[0007] The specific plan includes the following steps:

[0008] A method for image deraining based on physical model generation adversarial learning, the steps are as follows:

[0009] The first step is to model the task based on the two-level paradigm optimization;

[0010] In the second step, the lower-layer generator module decomposes the problem that is difficult to solve directly into multiple sub-problems that are easy to solve through semi-quadratic optimization. In addition, an implicit prior is introduced to characterize the rain line features, and a clear background is obtained after removing the visible rain lines.

[0011] In the third step, the upper generator module judges the result obtained in the second step until the generator considers that the result is a clear rain-free background.

[0012] The beneficial effects of the present invention are:

[0013] 1) The present invention experiments with generative adversarial learning from the perspective of double-layer optimization to solve the image deraining problem. The functions of the generator and the discriminator are realized respectively through the upper and lower sub-problems, and the adversarial learning of generation and discrimination is realized through the game of the upper and lower sub-problems, so as to finally obtain a clear and silent image.

[0014] 2) The generator of the present invention is designed based on the traditional model, and the discriminator is based on the data-driven network model, which realizes the combination of the traditional model and deep learning, makes up for the limited ability of the traditional model to depict rain lines, and solves the disadvantage of insufficient training data for deep learning; the two complement each other to achieve effective image deraining. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is the overall flow chart of the program.

[0016] Figure 2Schematic diagram showing the results of the present invention on real data, (a) represents the input of a real rain map, and (b) represents the experimental results of the present invention.

[0017] Figure 3 Schematic diagram showing the results of the present invention on synthetic data, (a) represents the input synthetic rain map, and (b) represents the experimental results of the present invention. DETAILED DESCRIPTION

[0018] The present invention is based on a two-layer paradigm optimization, takes a single rainy image as input, and uses the domain knowledge model and implicit prior in the lower layer problem to realize the characterization of the rain line in the image, and realizes the deraining to obtain preliminary results. Then the discriminator of the upper layer problem uses the network model obtained by data drive to discriminate the result obtained by the lower layer, judge whether it is a clear rain-free image and return a 0-1 numerical result. If it does not meet the requirements, the output result of the discriminator is returned in the form of a gradient, and a variable update is performed to assist the lower layer problem to continue deraining. If the result is clear and rain-free, the result of this iteration is output as the final result obtained by the algorithm. The specific implementation plan is as follows:

[0019] The specific process of the solution network is as follows Figure 1 , the specific operations are as follows:

[0020] The first step is to generate adversarial modeling for the image deraining task based on a two-layer paradigm optimization.

[0021] y represents the observed rainy image, and x represents the clear image without rain to be restored. The two-layer paradigm optimization consists of two layers of models, the lower layer as a generator, as shown in formula (2), which uses domain knowledge to model the image deraining task by means of maximum a posteriori probability estimation; the upper layer as a generator, as shown in formula (1), judges the deraining result generated by the lower layer to determine whether it is a clear background without rain line interference, which is specifically expressed as:

[0022] max x φ x (x) (1)

[0023] stx∈argmin x F(x,y)+G x (x) (2)

[0024] In the second step, the lower-layer generator uses implicit priors to perform rain line characterization and rain removal;

[0025] The lower-layer generator module, that is, formula (2), F(x, y) is a data item used to ensure the consistency of the output result and the input scene; G x(x) represents the prior constraints of the rain removal task. In order to solve the underlying problem, the present invention decomposes the problem that is difficult to solve directly into multiple sub-problems that are easy to solve by means of semi-quadratic splitting optimization. Specifically, it is expressed as:

[0026]

[0027] s represents the auxiliary variable introduced by semi-quadratic optimization. By separating the variables in formula (3), we can obtain the following two sub-problems about x and s, which are specifically expressed as follows:

[0028]

[0029]

[0030] G x (s) is a model prior specific to the rain removal task. In order to make up for the lack of the model's ability to depict rain lines, the invention introduces a data-driven implicit prior to depict the rain line features. By fitting the complex and irregular rain line distribution through a large number of parameters in the convolutional neural network, a clear background with visible rain lines removed is directly obtained, which is specifically expressed as:

[0031] s=N(x;w) (6)

[0032] Where N represents a convolutional neural network, and w represents the parameters in the network. After obtaining s, the generated rain-free image x can be obtained through formula (4).

[0033] In the third step, the upper generator module judges the results obtained in the second step;

[0034] The upper discriminator module, i.e., formula (1), uses a pre-trained discriminator to judge the result generated by the generator. The discriminator takes the generated result as input and outputs a value of 0-1, where 0 means that the generated result is a clear and clean background; 1 means that the result still has visible rain lines. The gradient of the output result is used as the step information to update the upper layer result, and the deraining result obtained by the upper layer in this iteration can be obtained. Specifically expressed as:

[0035]

[0036] in represents the gradient operation, φ x represents the pre-trained discriminant prior, and x obtained by formula (7) represents the deraining result of this iteration obtained by the upper layer problem.

[0037] After obtaining the output results of the upper and lower layers respectively, the final deraining result of the algorithm is obtained by taking into account the results of the upper and lower layers through an adjustable parameter, which is specifically expressed as:

[0038] x=αx down +(1-α)x up (8)

[0039] x down ,x up They represent the deraining results of the lower layer and the upper layer obtained by formula (4) and formula (7) respectively. So far, we have obtained the deraining result of this iteration, and the final result output by the invention is the deraining image when the upper layer discriminator in a certain iteration judges it as a real clear rain-free image. This result realizes the removal of rain lines from rainy images through model optimization and discrimination. The experimental results show that the method of the invention is more effective than the existing deraining algorithms and has a more obvious deraining effect.

Claims

1. An image deraining method based on physical modeling and generative adversarial learning, characterized in that: Here are the steps: The first step is to model the task based on the two-level paradigm optimization; y represents the observed rainy image, and x represents the rain-free background to be restored. The two-layer paradigm optimization consists of two layers, the lower layer as a generator, which models the image deraining task by means of maximum a posteriori probability estimation. The upper layer acts as a generator to judge the deraining result generated by the lower layer to determine whether it is a clear background without rain line interference. The specific expression is: max x φ x (x) (1) s.t.x∈argmin x F(x,y)+G x (x) (2) In the second step, the lower-layer generator uses implicit priors to perform rain line characterization and rain removal; The lower generator module, i.e., formula (2), decomposes the problem that is difficult to solve directly into multiple sub-problems that are easy to solve through semi-quadratic optimization; specifically, it can be expressed as: And the implicit prior is introduced to characterize the rain line features, and a clear background with visible rain lines removed is obtained, which is specifically expressed as: s=N(x;w) (6) Among them, the s sub-problem is directly solved by the neural network N with the help of implicit priors, and w represents the parameters in the network; In the third step, the upper generator module judges the results obtained in the second step; The upper discriminator module, i.e., formula (1), uses a pre-trained discriminator to judge the result generated by the generator. The discriminator takes the generated result as input and outputs a value of 0-1, where 0 indicates that the generated result is a clear and clean background; 1 indicates that the result still has visible rain lines. The gradient of the output result is used as the step information to update the upper result, and the deraining result obtained by the upper layer can be obtained.

2. The image deraining method based on physical modeling generative adversarial learning as claimed in claim 1, characterized in that: The third step is specifically expressed as: in represents the gradient, φ x represents the discriminant prior; After obtaining the output results of the upper and lower layers respectively, the final deraining result of the algorithm is obtained by taking into account the results of the upper and lower layers, which is specifically expressed as: x=αx down +(1-a)x up x down ,x up They represent the intermediate results obtained from the lower layer and the upper layer respectively; at this point, the final deraining result is obtained, which realizes the removal of rain lines from rainy images through model optimization and discrimination.

3. The image deraining method based on physical modeling generative adversarial learning as claimed in claim 1 or 2, characterized in that: The second step is to decompose and directly solve the underlying problem model by means of semi-quadratic optimization and the introduction of implicit priors:

4. The image deraining method based on physical modeling generative adversarial learning as claimed in claim 1 or 2, characterized in that: In the third step, the data-driven discriminant network is used to discriminate the generated deraining results to determine whether they meet the clear and rain-free conditions, and return a 0-1 result to indicate its clarity.

5. The image deraining method based on physical modeling generative adversarial learning as claimed in claim 3, characterized in that: In the third step, the data-driven discriminant network is used to discriminate the generated deraining results to determine whether they meet the clear and rain-free conditions, and return a 0-1 result to indicate its clarity.

Images