Image raindrop removing method based on Attentive GAN
Through the image raindrop removal method based on Attentive GAN, the raindrop image is processed using adversarial network and smooth mask matrix, the problem of image blur in rainy and snowy weather is solved, high-resolution raindrop image recovery is achieved, and the autonomous driving perception is improved. performance.
Patent Information
- Application Number
- CN202411193381.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2024-11-26
AI Technical Summary
In rainy and snowy weather, raindrops or sewage on car cameras will cause blurred images, affecting the road conditions detection and target detection ahead in autonomous driving perception. It is difficult for the existing technology to effectively remove raindrops and restore high-resolution clear images.
Using the Attentive GAN-based image raindrop removal method, the adversarial network model is constructed by obtaining raindrops and clear graph data sets for preprocessing, including generating networks and discriminative networks, training generators and discriminators, and using smooth mask matrix and attention The mechanism generates a clear image of the removal of raindrops.
It realizes the removal of image raindrops in autonomous driving perception that relies on pure vision, and obtains high-resolution clear images without raindrops, which significantly improves the effects of road conditions detection and target detection ahead, and improves the performance of autonomous driving under rainy and snowy conditions.
Smart Images

Figure CN119027334A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to an image raindrop removal method based on Attentive GAN. Background Art
[0002] The car's camera is used to input the road conditions and images ahead in vision-based autonomous driving perception, which can realize functions such as pedestrian collision warning, lane keeping, forward collision warning, and even replace the exterior mirrors with cameras. However, when the vehicle is traveling in rainy and snowy weather or wading through muddy ditches, raindrops or sewage will adhere to the camera, and some areas of the image will become blurred and distorted, which will significantly reduce the visibility of the scene and affect the detection of road conditions ahead and target detection. In addition, LiDAR is more susceptible to rainy and snowy weather than cameras, and it cannot make up for the shortcomings of cameras at this time. Attentive GAN is currently an innovative deep learning model that improves the performance of GAN in image processing tasks by introducing an attention mechanism. It performs well in raindrop removal, image restoration, and other aspects, and has broad application prospects.
[0003] Therefore, it is necessary to propose an image deraining method based on Attentive GAN to remove image raindrops and obtain high-resolution raindrop-free clear images in autonomous driving perception that relies on pure vision. Summary of the invention
[0004] The purpose of the present invention is to provide an image raindrop removal method based on Attentive GAN, which can remove image raindrops and obtain a high-resolution raindrop-free clear image in the autonomous driving perception relying on pure vision, which is beneficial to the vehicle's front road condition detection and target detection.
[0005] To achieve the above object, the present invention adopts an image raindrop removal method based on Attentive GAN, which includes the following steps:
[0006] Get the Attentive GAN dataset containing raindrop images and clear images;
[0007] Preprocess the data in the data set and obtain a smooth mask matrix;
[0008] Build a raindrop removal model based on an adversarial network; the adversarial network consists of a generation network and a discriminant network;
[0009] Get the loss functions of the generator and discriminator;
[0010] Train the raindrop removal network model to obtain the final model parameters;
[0011] The image with raindrops is input into the trained raindrop removal network model to obtain a clear image without raindrops.
[0012] Among them, in the step of obtaining the dataset containing raindrop images and clear images in Attentive GAN:
[0013] Get the raindrop dataset in Attentive GAN, which contains a training set of 861 pairs of raindrop images and clear images, a test set test_A of 58 pairs of images, and a test set test_B of 249 pairs of images; the test_A test set is a subset of the test_B test set.
[0014] Among them, in the step of preprocessing the data in the data set and obtaining the smoothed mask matrix:
[0015] Zoom the image in the dataset, calculate the smooth mask matrix M based on the raindrop image and the corresponding clear image in the dataset, set the matrix value of the position with raindrops in the smooth mask matrix M to 1, and gradually transition from 1 to 0 near the raindrops; the calculation method is:
[0016] When the sum of the differences between the RGB three-channel pixel values at the pixel x position of the image with raindrops and the image without raindrops is greater than 30, set M(x) = 1;
[0017] When sums<15, let M(x)=0; when 15≤sums≤30, let M(x)=sums / 30.
[0018] Among them, in the step of building a raindrop removal model based on an adversarial network, wherein the adversarial network is composed of a generation network and a discrimination network:
[0019] The generating network is used to generate an image without raindrops from an input image with raindrops, and the discriminating network is used to discriminate whether the image without raindrops is real.
[0020] Among them, in the step of obtaining the loss function of the generator and the discriminator:
[0021] The loss function of the generator is expressed as:
[0022]
[0023] Where G is the generating network, D is the discriminating network, I is the input image with raindrops, and R is the clear true value image;
[0024] The loss function of the discriminator is expressed as:
[0025] L D (O,R,A N )=-log(D(R))-log(1-D(O))+γLmap (O,R,A N );
[0026] Where O is the final output image of the generative network, R is the sample extracted from the true value image, and A N is the attention map generated by the Nth module.
[0027] Among them, in the step of training the raindrop removal network model and obtaining the final model parameters:
[0028] The raindrop network model is trained, and the loss function is minimized using the gradient optimization method to obtain the final model parameters.
[0029] The present invention discloses an image raindrop removal method based on Attentive GAN, which comprises the following steps: obtaining a data set including a raindrop image and a clear image in Attentive GAN; preprocessing the data in the data set and obtaining a smooth mask matrix; constructing a raindrop removal model based on an adversarial network; wherein the adversarial network is composed of a generator network and a discriminator network; obtaining the loss functions of the generator and the discriminator; training the raindrop removal network model and obtaining the final model parameters; inputting the image with raindrops into the trained raindrop removal network model to obtain a clear image with raindrops removed; and in the above-mentioned manner, in the automatic driving perception relying on pure vision, raindrops on the image can be removed and a high-resolution clear image without raindrops can be obtained, which is beneficial to the detection of the road condition ahead and the target detection of the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0031] Figure 1 It is a flowchart of the steps of the image raindrop removal method based on Attentive GAN of the present invention.
[0032] Figure 2 It is a schematic diagram of the calculation result of the smoothing mask matrix M of the present invention.
[0033] Figure 3 It is a structural diagram of the Resnet residual module of the present invention.
[0034] Figure 4 It is a schematic diagram of obtaining a clear image with raindrops removed according to the present invention. DETAILED DESCRIPTION
[0035] See also Figures 1 to 4 The present invention provides an image raindrop removal method based on Attentive GAN, comprising the following steps:
[0036] S100: Get the dataset containing raindrop images and clear images in Attentive GAN.
[0037] In this embodiment, a raindrop dataset in Attentive GAN is obtained, which includes a training set of 861 pairs of raindrop images and clear images, a test_A test set of 58 pairs of images, and a test_B test set of 249 pairs of images; wherein the test_A test set is a subset of the test_B test set.
[0038] S200: preprocessing the data in the data set and obtaining a smoothing mask matrix.
[0039] In this embodiment, the images in the data set are scaled, and the 720×480 images in the original data set are scaled into 480×360 images. Based on the raindrop image and the corresponding clear image in the data set, a smooth mask matrix M is calculated to guide the generation of the attention map. The matrix value at the position with raindrops in the smooth mask matrix M is set to 1, and the value at the position near the raindrops gradually transitions from 1 to 0, such as Figure 2 As shown, from Figure 2 (a) Figure 2 (b) Figure 2 (c) Figure 2 (d) Thus, the details at the edge of the original raindrop position in the final image are more natural and realistic. The obtained smooth mask matrix is as follows: Figure 2 (d) is shown; the calculation method is:
[0040] When the sum of the differences between the RGB three-channel pixel values at the pixel x position of the image with raindrops and the image without raindrops is greater than 30, set M(x) = 1;
[0041] When sums<15, let M(x)=0; when 15≤sums≤30, let M(x)=sums / 30.
[0042] S300: constructing a raindrop removal model based on an adversarial network; wherein the adversarial network is composed of a generation network and a discrimination network.
[0043] In this embodiment, the adversarial network is composed of a generation network and a discrimination network. The generation network is used to generate an image without raindrops from an input image with raindrops, and the discrimination network is used to determine whether the image without raindrops is real.
[0044] Given an input image with raindrops, the generative network attempts to generate an image without raindrops that is as real, natural, and clear as possible; while the discriminative network verifies whether the image generated by the generative network looks real and natural.
[0045] The generative network consists of a generative-recurrent network and a contextual autoencoder. The generative-recurrent network is used to locate the areas in the input image that need attention, that is, to obtain the attention map. These areas are mainly raindrops and their edge areas, which help the autoencoder obtain the texture information of the areas blocked by raindrops, so as to better restore the local details of the reconstructed image.
[0046] The generative-recurrent network consists of four neural network modules with the same structure from left to right, which are considered as four stages. Each stage consists of five Resnet residual modules, one LSTM module, a 3×3 standard convolutional layer and a ReLU activation function from top to bottom. The structure of the Resnet residual module is as follows: Figure 3 As shown in the figure, it contains two sets of 3×3 convolution and ReLU activation function. Each layer is connected to all previous layers, and the input of each Resnet module is skipped with the result of the second convolution layer to avoid gradient disappearance during training. The Resnet residual module is used to extract image features. The next connected module is the LSTM module. LSTM implements three gate calculations, namely the input gate i t , forget gate f t and output gate o t , and a new cell state value vector C is created during the calculation process t The calculation process is as follows:
[0047] i t =σ(W xi *X t +W hi *H t-1 +W ci C t-1 +b i );
[0048] f t =σ(W xf *X t +W hf *H t-1 +W cf C t-1 +b f );
[0049] C t =f t C t-1 +i t tanh(W xc *Xt +W hc *H t-1 +b c );
[0050] o t =σ(W xo *X t +W ho *H t-1 +W co C t +b o );
[0051] H t =o t tanh(C t );
[0052] Where * represents the convolution operation, X t is the input of LSTM, which is the feature map generated by the previous layers of Resnet residual modules; C t It is the cell state that determines what information will be retained and what information will be forgotten; H t Represents the features output by the LSTM network.
[0053] Finally, a 3×3 convolution and ReLU activation function are used to obtain the attention map of this stage. The attention map is concatenated and fused with the input image with raindrops in the channel dimension and then sent to the neural network module of the next stage to continue feature extraction. After four rounds of feature extraction, the final more mature attention map can be obtained.
[0054] Based on the attention map obtained in the last round, it is concatenated and fused with the input raindrop image in the channel dimension, and then sent to the context autoencoder to generate the final raindrop-free image. The autoencoder consists of an encoder and a decoder, which contains a set of 5×5 convolution and ReLU activation functions and fifteen sets of 3×3 convolution and ReLU activation functions, and a jump connection mechanism is added. The features and structural information of three different scales output by the 1st, 3rd, and 5th layers from the bottom will be used for loss calculation, and the corresponding output image sizes are 1, 1 / 2, and 1 / 4 of the original input size, respectively.
[0055] The discriminant network consists of five groups of 5×5 convolutions and ReLU activation functions, one 5×5 convolution, two groups of 5×5 convolutions and ReLU activation functions, a fully connected layer and a Sigmoid activation function from top to bottom. The discriminant network is used to determine whether the generated image is real, and will be trained independently and synchronously with the generator network. For the input image, if it is an image generated by the generator, the discriminator will judge it as false, and if it is a true value image, the discriminator will judge it as true.
[0056] S400: Obtain the loss functions of the generator and the discriminator.
[0057] In this embodiment, generator and discriminator loss functions are designed.
[0058] The loss function expression of the generator is:
[0059] L G =10 -2 L GAN (O)+L ATT ({A},M)+L M ({S},{T})+L P (O,T)+L SSIM (O haar ,T haar );
[0060] Among them, because the discriminator judges the image generated by the generator as false,
[0061] L GAN (O) = log(1-D(O)).
[0062] The attention map obtained by each module of the generative-recurrent network will be used to calculate the mean square error (MSE) with the smoothed mask matrix M:
[0063]
[0064] Among them A t Represents the attention map generated by the t-th module, N=4, θ=0.8.
[0065] The multi-scale loss function extracts outputs of different scales from the penultimate 1st, 3rd, and 5th layers of the autoencoder, compares them with the true value images of the same scale, and calculates the mean square error MSE:
[0066]
[0067] Where S i represents the output of the i-th scale from the decoder side; T i Represents the true value image of the corresponding scale. i Represents the weights assigned by loss functions of different scales. The λ of the 1st, 3rd, and 5th layers from the bottom are set to 1, 0.8, and 0.6, respectively.
[0068] The perceptual loss function measures the difference between the global features of the generated image and the true image. This feature is extracted through a VGG16 network pre-trained on ImageNet, and the loss function is:
[0069] LP (O,T)=L MSE (VGG(O),VGG(T));
[0070] Where O is the generated image and T is the true image.
[0071] Finally, the SSIM is introduced as part of the loss function. The Haar wavelet transform is applied to the generated image and the true value image respectively, and the SSIM is calculated for the transformed features of both sides. The larger the SSIM value, the more similar the two images are, so the loss function expression is:
[0072] L SSIM (O haar ,T haar )=1-SSIM(O haar ,T haar );
[0073] Among them, haar is the Haar wavelet transform result of the generated image, T haar It is the Haar wavelet transform result of the true image.
[0074] The loss function of the discriminator is then calculated, and the discriminator classifies the images generated by the generator as fake and the true value images as real.
[0075] L D (O,R,A N )=-log(D(R))-log(1-D(O))+γL map (O,R,A N );
[0076] Among them, L map Represents the loss function of the attention maps extracted by the first few layers of the discriminator network and the attention maps generated by the generator:
[0077] L map (O,R,A N )=L MSE (D map (O),A N )+L MSE (D map (R),0);
[0078] Where D map represents the process of the discriminator generating the attention map, and γ is set to 0.05. R is a sample extracted from the true value image, and 0 represents an all-zero matrix. It shows that if the input is a raindrop-free image generated by the generator network, the attention map generated by the discriminator is ideally the attention map generated by the generator network. If the input image is a true value image, the attention map should be an all-zero matrix, that is, there is nothing to pay attention to.
[0079] S500: Train the raindrop removal network model to obtain the final model parameters.
[0080] In this implementation, the model is trained, and the gradient optimization method is used to minimize the loss function, thereby optimizing the network parameters and obtaining the final model parameters. This implementation is carried out under the Ubuntu 20.04 operating system and trained on a single NVIDIA GeForce RTX 3070 graphics card. The 1.10.0 version of the open source deep learning framework pytorch is used, and the CUDA version is 11.7. The gradient descent optimizer uses Adam, the learning rate is set to 0.0002, the batch size is set to 4, and 100 epochs are trained.
[0081] S600: Input the image with raindrops into the trained raindrop removal network model to obtain a clear image with raindrops removed.
[0082] In this embodiment, if Figure 4 As shown in Figure 2, the image with raindrops in the real scene is input into the generator to obtain a clear image with raindrops removed. The effect is as follows: Figure 4 (c) The test results of the improved Attentive GAN and the original version on the test_A and test_B test sets are shown in Table 1:
[0083]
[0084] Table 1
[0085] From Table 1, we can see that the improved Attentive GAN in this implementation has a significant improvement in both PSNR and SSIM indicators, indicating that the improved model proposed in this implementation has a better effect in removing raindrops.
[0086] What is disclosed above is only a preferred embodiment of the present invention, and it certainly cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.
Claims
1. An image raindrop removal method based on Attentive GAN, characterized in that: The steps include: Get the Attentive GAN dataset containing raindrop images and clear images; Preprocess the data in the data set and obtain a smooth mask matrix; Build a raindrop removal model based on an adversarial network; the adversarial network consists of a generation network and a discriminant network; Get the loss functions of the generator and discriminator; Train the raindrop removal network model to obtain the final model parameters; The image with raindrops is input into the trained raindrop removal network model to obtain a clear image without raindrops.
2. The image raindrop removal method based on Attentive GAN as claimed in claim 1, characterized in that: In the steps of obtaining the dataset containing raindrop images and clear images in Attentive GAN: Get the raindrop dataset in Attentive GAN, which contains a training set of 861 pairs of raindrop images and clear images, a test set test_A of 58 pairs of images, and a test set test_B of 249 pairs of images; the test_A test set is a subset of the test_B test set.
3. The image raindrop removal method based on Attentive GAN as claimed in claim 2, characterized in that: In the step of preprocessing the data in the dataset and obtaining a smoothed mask matrix: Zoom the image in the dataset, calculate the smooth mask matrix M based on the raindrop image and the corresponding clear image in the dataset, set the matrix value of the position with raindrops in the smooth mask matrix M to 1, and gradually transition from 1 to 0 near the raindrops; the calculation method is: When the sum of the differences between the RGB three-channel pixel values at the pixel x position of the image with raindrops and the image without raindrops is greater than 30, set M(x) = 1; When sums<15, let M(x)=0; when 15≤sums≤30, let M(x)=sums / 30.
4. The image raindrop removal method based on Attentive GAN as claimed in claim 3, characterized in that: In the step of constructing a raindrop removal model based on an adversarial network, wherein the adversarial network is composed of a generation network and a discrimination network: The generating network is used to generate an image without raindrops from an input image with raindrops, and the discriminating network is used to discriminate whether the image without raindrops is real.
5. The image raindrop removal method based on Attentive GAN as claimed in claim 4, characterized in that: In the step of obtaining the loss function of the generator and discriminator: The loss function of the generator is expressed as: m G inm D ax E R~pclean [log(D(R))]+E I~praindrop [log(1-D(G(I)))]; Where G is the generating network, D is the discriminating network, I is the input image with raindrops, and R is the clear true value image; The loss function of the discriminator is expressed as: L D (O,R,A N )=-log(D(R))-log(1-D(O))+γL map (O,R,A N ); Where O is the final output image of the generative network, R is the sample extracted from the true value image, and A N is the attention map generated by the Nth module.
6. The image raindrop removal method based on Attentive GAN as claimed in claim 1, characterized in that: In the step of training the raindrop removal network model and obtaining the final model parameters: The raindrop network model is trained, and the loss function is minimized using the gradient optimization method to obtain the final model parameters.
Citation Information
Patent Citations
Visual rain and fog removing method based on deep reinforcement learning
CN117196985A
Language-driven integrated image severe weather removal method
CN118115395A
Method for detection of a raindrop on the windscreen of a vehicle and driving assistance system
WO2013037403A1
