Image rain removal method based on generative adversarial network and convolutional neural network

Abstract

The application discloses a kind of based on image rain-removing method of generative adversarial network and convolutional neural network, comprising: according to rain-free image, utilize random noise to synthesize rainy image;The pre-processing of synthesized rainy image, and the multiple rainy images after pre-processing constitute data set;The generator and discriminator in the generative adversarial network GAN are trained using the data set, and the rain-removed image generated using the generative adversarial network model is used to train convolutional neural network CNN;Generative adversarial network GAN and convolutional neural network CNN are alternately trained until reaching preset condition, and trained generative adversarial network GAN and trained convolutional neural network CNN are obtained;The image to be removed is input into the generator in trained generative adversarial network and is removed, to obtain the image after rain-removing;The image after rain-removing is input into trained convolutional neural network, to obtain the rain-removed image after final optimization.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and relates to an image rain removal method based on a generative adversarial network (GAN) and a convolutional neural network (CNN). BACKGROUND

[0002] Since raindrops and rain marks often block or blur the key information of outdoor shooting images, the rain removal task of images or videos is useful and necessary and has good value in the field of computer vision, such as target detection and identification, and an effective rain removal technology can usually help images or videos to better deliver more accurate information detection or identification results.

[0003] Many times, rain marks and water particles in the background form a thin veil due to rain accumulation, which significantly reduces the contrast and visibility of the scene. The artifacts introduced by rain streaks have an adverse effect on the performance of computer vision algorithms, such as object detection and identification, due to image degradation caused by rain streaks and rain accumulation. Therefore, we need to perform post-processing on images, and the research on image noise removal is of great practical significance. At present, video and single image rain removal methods are roughly divided into model-based methods and data-driven methods.

[0004] Video-based rain removal methods can find the differences between adjacent frames, and often use the pixel information blocked by rain marks that can be obtained in the front and rear frames, so that rain marks can be relatively easily removed from videos, but this requires the camera to be in a relatively static state during video shooting. Single image rain removal methods mostly use cost functions to represent rain and background layers through the development of various priors. Since 2017, single image rain removal has begun to be studied based on deep learning, such as convolutional neural networks, recurrent neural networks, and generative adversarial networks, which have shown good performance in rain removal. Video rain removal can utilize the existing redundant temporal information images in adjacent frames, which helps to identify and remove rain marks. For the video rain removal task, there are still certain problems, and most model-driven methods cannot be directly applied to real-time streaming video data. At the same time, deep learning methods require a large amount of supervised video, show high computational complexity in the training stage, and they cannot guarantee good rain removal performance, especially in complex scenes.

[0005] However, due to the nature of the ill-posed and less availability, removing rain streaks from a single image faces difficult challenges. SUMMARY

[0006] Purpose: In order to overcome the deficiencies in the prior art, the application provides an image rain removal method based on a generative adversarial network and a convolutional neural network.

[0007] Technical solution: To solve the above technical problems, the technical solution adopted by the present application is:

[0008] In the first aspect, the present application provides an image rain removal method based on a generative adversarial network and a convolutional neural network, comprising:

[0009] Step S1: According to the rain-free image, use random noise to synthesize the rainy image;

[0010] Step S2: Preprocess the synthesized rainy image, and the preprocessed multiple rainy images constitute a data set;

[0011] Step S3: Train the generator and discriminator in the generative adversarial network GAN using the data set, and train the convolutional neural network CNN using the rain-removed image generated by the generative adversarial network model; alternately train the generative adversarial network GAN and the convolutional neural network CNN until the preset condition is reached, and obtain the trained generative adversarial network GAN and the trained convolutional neural network CNN;

[0012] Step S4: input the image to be removed into the generator of the trained generative adversarial network for rain removal, and obtain the rain-removed image; input the rain-removed image into the trained convolutional neural network to obtain the final optimized rain-removed image.

[0013] In some embodiments, step S1: according to the rain-free image, use random noise to synthesize the rainy image, comprising:

[0014] S11: Generate random noise of different densities to simulate different sizes of rainfall;

[0015] S12: Use uniform random numbers and threshold values to control the level of noise, simulate rain of different sizes, densities and directions by elongating the noise, rotating the direction of the noise, etc., and obtain the generated raindrop noise;

[0016] S13: Superimpose the generated raindrop noise and the rain-free image, and use the pixels of each channel to operate with the generated raindrop noise to obtain a simulated rainy scene and synthesize an ideal rainy image.

[0017] Further, in some embodiments, in S13, the operation of using the pixels of each channel with the generated raindrop noise comprises: keeping the original image at the position of black noise, modifying the position of the raindrop noise to the texture of the weighted raindrop noise, and assigning the noise value.

[0018] In some embodiments, in step S2, the synthesized rainy image is preprocessed, comprising: adjusting the size of the synthesized rainy image to 512x512.

[0019] In some embodiments, in step S3, the data set is used to train a generator and a discriminator in a generative adversarial network, including:

[0020] In step S31, the rainy images in the data set are input into the generator for training, to obtain a generated rain-free image I clear ;

[0021] In step S32, the rain-free image I clear generated by the generator is input into the discriminator for discriminative training,

[0022] The steps S31 to S32 are repeated until the generator and the discriminator meet the preset requirements.

[0023] Further, step S31 includes:

[0024] A rainy image I is randomly sampled from the data set as a model input, and a guided filter is used to obtain a filtered background image I base and a rain detail image I detail =I-I base ;

[0025] The rain detail image I detail and the background image I base are input into the generator for training, to obtain a clean rain-free image I clear generated by the generator.

[0026] Further, in the training process, the generative adversarial network adopts a perceptual loss function:

[0027] perceptual loss = content_loss + λadverserial_loss

[0028] Wherein, λ is a hyperparameter to control the balance of the two losses;

[0029] The content loss content_loss is:

[0030] content_loss = ||VGG(I truth )-VGG(I clear )||2

[0031] VGG is a convolutional neural network CNN, VGG(*) is a feature map of the VGG network, and ||*||2 is an L-2 norm matrix;

[0032] The adversarial loss adverserial_loss is:

[0033] adverserial-loss = ∑[log(D(I truth))+log(1-G(Ic lear ))

[0034] wherein D is a discriminator, G is a generator, I truth is a label of the rain-free image I clear generated by the generator.

[0035] In some embodiments, training a convolutional neural network CNN with the rain-removed images generated by the generative adversarial network model comprises:

[0036] During the training process, the convolutional neural network adopts a mean square error loss function MSE:

[0037] MSE=||I truth -I clear ||2

[0038] wherein I truth is a label of the rain-free image I clear generated by the generator, and ||*||2 is an L-2 norm matrix.

[0039] Alternately training the GAN and CNN networks, combining the perceptual function and the MSE loss function to solve the entire optimization problem can avoid the problems of excessively smooth images and artifacts.

[0040] In a second aspect, the present application provides an image rain-removal device based on a generative adversarial network and a convolutional neural network, comprising a processor and a storage medium.

[0041] The storage medium is configured to store instructions.

[0042] The processor is configured to operate according to the instructions to perform the steps of the method according to the first aspect.

[0043] In a third aspect, the present application provides a storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the method according to the first aspect.

[0044] In a fourth aspect, the present application provides an apparatus comprising,

[0045] one or more processors, one or more memories, and one or more programs, wherein the one or more programs are stored in the one or more memories and configured to be executed by the one or more processors, and the one or more programs include instructions for performing any of the methods according to the first aspect.

[0046] Beneficial effects: the image rain removal method based on the generative adversarial network and the convolutional neural network provided by the application has the following advantages: the unsupervised real rainy day image is added to the network training, the supervised synthetic data and the unsupervised real data are fed into the network at the same time, for the supervised part, the random noise is used to synthesize the rain image: the random noise is used to superimpose the simulated raindrop motion track of the image, the operation of lengthening and rotating direction is performed on the noise, the rainwater of different sizes and directions is simulated, the size, length, inclination degree and quantity of the raindrop can be changed by changing the corresponding value, finally the generated raindrop noise and the original image are superimposed to obtain the simulated rainy scene, for the synthesized rain image, the network combining CNN and GAN is used for image rain removal, wherein the GAN is used to eliminate the structural rain streaks, the CNN is used to restore the details in the original image, the CNN and the generator enjoy the same weight, the synthesized rainy image is filtered out by the guided filter base and I detail GAN is composed of two CNNs: generator and discriminator, the generated false image after rain removal is generated by the generator, and then the discriminator discriminates the false image and the real image, and then promotes the generator to generate a false image closer to the real image, so that the finally generated rain removal image achieves the best rain removal effect. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 is a flowchart of the image rain removal method based on the generative adversarial network and the convolutional neural network according to an embodiment of the application;

[0048] Figure 2 is a comparison chart of SSIM and PSNR corresponding to different training times of the image after image rain removal, the original image for synthesizing the rain image and the rainy image synthesized by using random noise according to an embodiment of the application. DETAILED DESCRIPTION

[0049] The application will be further described below in combination with the drawings and embodiments. The following embodiments are only used to more clearly illustrate the technical solutions of the application, and cannot be used to limit the protection scope of the application.

[0050] In the description of the application, if several meanings are one or more, the meaning of multiple is two or more, greater than, less than, more than, etc. are understood as not including the number, above, below, within, etc. are understood as including the number. If it is described as first, second, it is only used to distinguish the technical features for the purpose, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of indicated technical features or implicitly indicating the sequence of indicated technical features.

[0051] In the description of the present application, the description of the terms "one embodiment", "some embodiments", "illustrative embodiment", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0052] Single image rain removal is actually a classification process that regards the image as two layers: a rain-free layer and a rain layer, and then separates the rain layer from the original image to leave a rain-free image. Model-based methods are divided into sparse coding and Gaussian mixture model (GMM) methods, and data-driven methods are divided into deep convolutional neural network methods, generative adversarial networks and semi / supervised learning methods. Deep learning-based methods are becoming more and more popular, and future research will focus on establishing semi / supervised methods, conducting domain synthesis, creating reasonable evaluation quality indicators for rain removal, and exploring the potential of rain removal in practical applications.

[0053] The present application tests synthetic images and real images, and uses a network combining CNN and GAN to remove rain from images. GAN is used to eliminate structural rain streaks, and CNN is used to restore details in the original image. CNN and the generator enjoy the same weight. The synthetic rainy image is filtered through a guided filter. GAN is composed of two CNNs: a generator and a discriminator. The generator generates a false image after rain removal, and the discriminator discriminates between the false image and the real image. The input of the discrimination network is the real sample or the output of the generation network, and the purpose is to distinguish the output of the generation network from the real sample as much as possible. The generation network tries to deceive the discrimination network as much as possible. The two networks are in mutual opposition, and the parameters are constantly adjusted through the loss to optimize the generation network and the discrimination network, so that the discrimination network cannot judge whether the output result of the generation network is real. Further, the generator generates a false image that is closer to the real image, and the final generated rain-removed image achieves the best rain-removal effect. The method is compared with other methods, and PSNR and SSIM are used to evaluate the quality of the rain-removed image.

[0054] Embodiment 1

[0055] An image rain removal method based on a generative adversarial network and a convolutional neural network, comprising:

[0056] Step S1: synthesizing a rainy image from a rain-free image according to random noise;

[0057] Step S2: preprocessing the synthesized rainy image, and the preprocessed multiple rainy images constitute a data set;

[0058] Step S3: training a generator and a discriminator in a generative adversarial network (GAN) using the dataset, training a convolutional neural network (CNN) using a rain-removed image generated by the generative adversarial network model, and alternately training the generative adversarial network (GAN) and the convolutional neural network (CNN) until a preset condition is reached to obtain a trained generative adversarial network (GAN) and a trained convolutional neural network (CNN);

[0059] Step S4: inputting an image to be rain-removed into the generator of the trained generative adversarial network for rain removal to obtain a rain-removed image, and inputting the rain-removed image into the trained convolutional neural network to obtain a finally optimized rain-removed image.

[0060] In some embodiments, the model uses Adam to optimize its weights, the weight β1 of the generative adversarial network is 0.9, the weight β2 of the convolutional neural network is 0.99, the learning rate of the GAN network is 0.001, and the learning rate of the CNN network is 0.0001.

[0061] The GAN network realizes the elimination of structural rain streaks, and in order to better restore the details in the image, the CNN network is used to retain the texture and color in the image, and the details in the image are restored by using a smaller learning rate, more comprehensive training data and a more fine-grained loss function, and the GAN and CNN networks are alternately trained.

[0062] In some embodiments, step S1: synthesizing a rainy image from a rain-free image using random noise, comprises:

[0063] S11: generating random noise of different densities to simulate different sizes of rainfall;

[0064] S12: using uniform random numbers and a threshold to control the level of noise, and simulating rain of different sizes, densities and directions by elongating the noise, rotating the direction of the noise, etc., to obtain generated raindrop noise;

[0065] S13: superimposing the generated raindrop noise and the rain-free image, and using the pixels of each channel to operate with the generated raindrop noise to obtain a simulated rainy scene and synthesize an ideal rainy image.

[0066] Further, in some embodiments, in S13, the operation of using the pixels of each channel to operate with the generated raindrop noise comprises: keeping the original image at positions where the noise is black, modifying the positions of the raindrop noise to the texture of the weighted raindrop noise, and assigning the noise value.

[0067] In some embodiments, in step S2, the synthesized rainy image is preprocessed, comprising: adjusting the size of the synthesized rainy image to 512x512.

[0068] In some embodiments, step S3, training the generator and discriminator in the generative adversarial network using the dataset, includes:

[0069] Step S31: Input the rainy images from the dataset into the generator for training to obtain the generated rainless image I. clear ;

[0070] Step S32: Generate the rainless map I generated by the generator. clear Input the discriminator for discrimination training.

[0071] Repeat steps S31 to S32 until the generator and discriminator meet the preset requirements.

[0072] Further, step S31 includes:

[0073] Rainy images I are randomly sampled from the dataset and used as model input. A guided filter is then used to obtain the filtered background image I. base And rain detail image I detail =II base ;

[0074] Rain detail image I detail and background image I base The generator is trained using the input, resulting in a clean, rain-free image I generated by the generator. clear Using rain detail image I base Instead of using the original rain image, the image construction process can be simplified by generating a clean, rain-free image. clear No rain map I clear It is necessary to mimic the real samples in the training set as closely as possible.

[0075] The loss function of the GAN network combines content loss and adversarial loss. The content loss is calculated by optimizing the distance between the real image and the rain image in the VGG feature space. Intuitively, since the content loss and adversarial loss are designed for classification, they are more suitable for removing structural rain stripes.

[0076] Furthermore, during training, the generative adversarial network employs a perceptual loss function:

[0077] perceptual loss=content_loss+λadverserial_loss

[0078] Here, λ is a hyperparameter that controls the balance between the two losses;

[0079] The content loss is:

[0080] content_loss = ||VGG(I truth ) - VGG(I clear )||2

[0081] VGG is a convolutional neural network CNN, VGG(*) is the feature map of VGG network, ||*||2 is L-2 norm matrix;

[0082] The adverserial_loss is:

[0083] adverserial_loss = ∑[log(D(I truth ))+log(1-G(I clear ))]

[0084] Wherein, D is the discriminator, G is the generator, I truth is the label of the rain-free image I clear generated by the generator.

[0085] The CNN network is used to reconstruct the image from the GAN network to retain the color and texture and remove the defects caused by the perceptual loss. Since the mean square error is more sensitive to pixel-level information, the CNN network adopts the mean square error as its loss function.

[0086] In some embodiments, in step S3, the rain-removed image generated by the generative adversarial network model is used to train the convolutional neural network CNN, including:

[0087] The convolutional neural network adopts the mean square minimum error loss function MSE:

[0088] MSE = ||I truth -I clear ||2

[0089] Wherein, I truth is the label of the rain-free image I clear generated by the generator, and ||*||2 is an L-2 norm matrix.

[0090] The GAN and CNN networks are alternately trained, the perceptual function and the MSE loss function are combined to solve the whole optimization problem, and the problems of too smooth image and artifacts can be avoided.

[0091] In some embodiments, two index structures, structural similarity index (SSIM) and peak signal-to-noise ratio (PSNR), are used to assist in measuring the effect of image rain removal.

[0092] Given two images x and y, the luminance, contrast and structure between the two are respectively as follows:

[0093]

[0094]

[0095]

[0096] SSIM(x, y) = [l(x, y) α ·c(x, y) β ·s(x, y) γ ]

[0097] Given a gray image I of size m x n and a noise map K, the mean square error (MSE) formula is as follows:

[0098]

[0099] Definition:

[0100]

[0101] As shown in Figure 2 , it is the SSIM and PSNR comparison chart corresponding to the different training times of the image after image rain removal, the original image for synthesizing the rain image, and the rain image synthesized by random noise.

[0102] Embodiment 2

[0103] In a second aspect, the embodiment provides an image rain removal device based on a generative adversarial network and a convolutional neural network, comprising a processor and a storage medium.

[0104] The storage medium is used to store instructions.

[0105] The processor is used to operate according to the instructions to perform the steps of the method according to embodiment 1.

[0106] Embodiment 3

[0107] In a third aspect, the embodiment provides a storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the method of embodiment 1.

[0108] Embodiment 4

[0109] In a fourth aspect, the present application provides a device comprising,

[0110] One or more processors, one or more memories, and one or more programs, wherein the one or more programs are stored in the one or more memories and configured to be executed by the one or more processors, the one or more programs including instructions for performing any of the methods described in embodiment 1.

[0111] As will be appreciated by one skilled in the art, embodiments of the present application can be provided as methods, systems, or computer program products. Accordingly, embodiments of the present application can be embodied in the form of entire hardware embodiments, entire software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present application can be in the form of computer program products that are implemented on one or more computer-usable storage media (including, but not limited to, magnetic disk storage, CD-ROM, optical storage, and the like) that contain computer usable program code.

[0112] The present application is described in reference to the flowchart illustrations and / or block diagrams according to the methods, apparatus (systems), and computer program products of embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks.

[0113] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks.

[0114] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks.

[0115] The above merely describes the preferred embodiments of the present application, and it should be pointed out that those skilled in the art can make several improvements and refinements without departing from the principles of the present application, and these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. An image rain removal method based on a generative adversarial network and a convolutional neural network, characterized in that, The method comprises the following steps: Step S1: synthesizing a rainy image from a non-rainy image by using random noise; Step S2: preprocessing the synthesized rainy image, and the preprocessed multiple rainy images forming a data set; Step S3: training a generator and a discriminator in a generative adversarial network by using the data set, and training a convolutional neural network by using a de-raining image generated by the generative adversarial network model; alternately training the generative adversarial network and the convolutional neural network until a preset condition is reached, to obtain a trained generative adversarial network and a trained convolutional neural network; Step S4: inputting an image to be de-rained into the generator in the trained generative adversarial network for de-raining, to obtain a de-rained image; and inputting the de-rained image into the trained convolutional neural network, to obtain a final optimized de-rained image; In step S3, the training of the generator and the discriminator in the generative adversarial network by using the data set comprises: Step S31, input the rainy images in the dataset into the generator for training to obtain the generated rain-free image I clear , specifically including: randomly sampling rainy images I from the dataset as model input, using a guided filter to obtain a filtered background image I base and a rain detail image I detail = I-I base ; input the rain detail image I detail and the background image I base into the generator for training to obtain a clean rain-free image I clear generated by the generator; Step S32, the rain-free map I generated by the generator clear The discriminator is input for discriminative training, cyclic steps S31 to S32 until the generator and the discriminator meet a preset requirement; In the training process, the generative adversarial network adopts a perceptual loss function perceptualloss: perceptualloss = content_loss + λadverserial_loss; wherein λ is a hyperparameter for controlling the balance between the two losses; The content loss content_loss is: content_loss = ||VGG(I truth ) - VGG(I clear )||2; VGG is a convolutional neural network CNN, VGG(*) is a feature map of the VGG network, and ||*||2 is an L-2 norm matrix; The adverserial loss adverserial_loss is: adverserial_loss =∑ [log(D(I truth ))+log(1-G(I clear ))] ; where D is the discriminator, G is the generator, I truth is the label of the rain-free image I clear generated by the generator. 2.The image rain removal method based on a generative adversarial network and a convolutional neural network according to claim 1, wherein, Step S1: synthesizing a rainy image from a non-rainy image by using random noise, comprising: S11: generating random noise of different densities to simulate different amounts of rainfall; S12: using uniform random numbers and a threshold to control the level of noise, and simulating rain of different sizes, densities and directions by elongating the noise, rotating the direction of the noise, etc., to obtain generated raindrop noise; S13: superimposing the generated raindrop noise and the non-rainy image, and performing operations on the generated raindrop noise by using the pixels of each channel, to obtain a simulated rainy scene and synthesize an ideal rainy image. 3.The image rain removal method based on a generative adversarial network and a convolutional neural network according to claim 2, characterized in that, In S13, the operation on the generated raindrop noise by using the pixels of each channel comprises: keeping the original image at positions where the noise is black, modifying the positions of the raindrop noise to the texture of the weighted raindrop noise, and assigning a noise value. 4.The image rain removal method based on generative adversarial network and convolutional neural network according to claim 1, wherein, In step S2, the preprocessing of the synthesized rainy image comprises: adjusting the size of the synthesized rainy image to 512x512. 5.The image rain removal method based on generative adversarial network and convolutional neural network according to claim 1, wherein, The training of the convolutional neural network by using the de-raining image generated by the generative adversarial network model comprises: In the training process, the convolutional neural network adopts a mean square error loss function MSE: MSE = ||I truth - I clear ||2; where I truth is the label of the rain-free image I clear generated by the generator, and ||*||2is the L-2 norm matrix.

6. An image rain removal device based on a generative adversarial network and a convolutional neural network, characterized in that, The computer program is executed by the processor to implement the steps of the method according to any one of claims 1 to 5. The computer program is executed by the processor to implement the steps of the method according to any one of claims 1 to 5. ​ 7. A storage medium having stored thereon a computer program, characterized in that ​

Images