Image deraining network, image deraining network training method and device

Abstract

The present application discloses an image deraining network, a training method and a device for the image deraining network, wherein the image deraining network includes a first convolutional network and a second convolutional network, wherein the first convolutional network is used to predict the image to be derained based on an improved UNet network, and convolve the predicted first deraining filter kernel with the image to be derained pixel by pixel to obtain a first derained image; the second convolutional network is used to predict the first derained image based on the improved UNet network, and convolve the predicted second deraining filter kernel with the first derained image pixel by pixel to obtain a target derained image. By designing the image deraining network as two convolutional networks constructed based on the improved UNet network, and using the image deraining network to derain the image to be derained, the accuracy of image deraining can be effectively improved.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular, relates to an image deraining network, and a training method and device for the image deraining network. Background Art

[0002] In the scenario of using a panoramic imaging system to assist in parking, if there are raindrops on the surround-view camera, there will also be raindrops in the image captured by the surround-view camera, which will seriously interfere with the judgment of surrounding obstacles and markings by people and the vehicle computer, thereby affecting people's experience of the panoramic imaging system and increasing the difficulty of the vehicle computer to process downstream visual tasks.

[0003] At present, Generative Adversarial Network (GAN) is usually used to remove raindrops from images. However, this scheme has the problem that obvious residual traces are left after removing raindrops from a single frame image, and the rain removal accuracy is low. Summary of the invention

[0004] The embodiments of the present application provide an image deraining network, thereby being able to improve the accuracy of image deraining.

[0005] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0006] According to a first aspect of an embodiment of the present application, an image deraining network is provided, the image deraining network comprising a first convolutional network and a second convolutional network, wherein:

[0007] The first convolutional network is used to predict the image to be derained based on the improved UNet network, and perform pixel-by-pixel convolution on the predicted first deraining filter kernel and the image to be derained to obtain a first derained image;

[0008] The second convolutional network is used to predict the first derained image based on the improved UNet network, and perform pixel-by-pixel convolution on the predicted second derained filter kernel and the first derained image to obtain a target derained image.

[0009] In some embodiments of the present application, based on the foregoing scheme, the improved UNet network includes the first network layer to the thirteenth network layer in sequence, the first network layer to the fourth network layer, the fifth network layer, the seventh network layer, the ninth network layer and the eleventh network layer are all combination layers, the sixth network layer, the eighth network layer, the tenth network layer and the twelfth network layer are upsampling layers, the thirteenth network layer is a hole convolution layer, and the combination layer includes multiple of an average pooling layer, a convolution layer and an activation layer;

[0010] Among them, the expansion rates from the second network layer to the fourth network layer, the seventh network layer, the ninth network layer and the eleventh network layer are all greater than 1.

[0011] In some embodiments of the present application, based on the aforementioned scheme, the expansion rate of the first network layer to the fourth network layer, the fifth network layer, the seventh network layer, the ninth network layer and the eleventh network layer increases in a sawtooth shape.

[0012] In some embodiments of the present application, based on the aforementioned scheme, the second network layer is jump-connected to the tenth network layer, the third network layer is jump-connected to the eighth network layer, and the fourth network layer is jump-connected to the sixth network layer.

[0013] In some embodiments of the present application, based on the aforementioned scheme, the seventh network layer includes a channel attention module; the channel attention module includes a first input unit, a first average pooling unit, a second average pooling unit, a first fusion unit, a first convolution unit, a first activation unit, a second convolution unit, a second activation unit and a first inner product unit; wherein,

[0014] The first input unit is connected to the first average pooling unit, the second average pooling unit and the first inner product unit respectively;

[0015] The first average pooling unit and the second average pooling unit are both connected to the first fusion unit;

[0016] The first fusion unit is connected to the first inner product unit via the first convolution unit, the first activation unit, the second convolution unit, and the second activation unit in sequence.

[0017] In some embodiments of the present application, based on the aforementioned scheme, the eleventh network layer includes a spatial attention module; the spatial attention module includes a second input unit, a cross-channel average pooling unit, a cross-channel maximum pooling unit, a second fusion unit, a third convolution unit, a third activation unit, and a second inner product unit; wherein,

[0018] The second input unit is connected to the cross-channel average pooling unit, the cross-channel maximum pooling unit and the second inner product unit respectively;

[0019] The cross-channel average pooling unit and the cross-channel maximum pooling unit are both connected to the second fusion unit;

[0020] The second fusion unit is connected to the second inner product unit via the third convolution unit and the third activation unit in sequence.

[0021] According to a second aspect of an embodiment of the present application, a training method for an image deraining network is provided, the training method comprising: obtaining raindrop-free images and images to be derained collected by a vehicle at the same geographical location; inputting the images to be derained and the raindrop-free images into the above-mentioned image deraining network for training, to obtain a trained image deraining network.

[0022] In some embodiments of the present application, based on the aforementioned solution, the trained image deraining network is deployed on the hardware platform Nvidia Jetson Xavier, and the trained image deraining network is quantized to 8-bit integer.

[0023] According to a third aspect of an embodiment of the present application, a training device for an image deraining network is provided, the training device for the image deraining network comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps of the above method when executing the computer program.

[0024] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0025] In the present application, a first convolutional network and a second convolutional network are designed in an image deraining network, wherein the first convolutional network is used to predict the image to be derained based on an improved UNet network, and convolve the predicted first deraining filter kernel with the image to be derained pixel by pixel to obtain a first derained image; the second convolutional network is used to predict the first derained image based on the improved UNet network, and convolve the predicted second deraining filter kernel with the first derained image pixel by pixel to obtain a target derained image. By designing the image deraining network as two convolutional networks constructed based on an improved UNet network, and using the image deraining network to derain the image to be derained, the accuracy of image deraining can be effectively improved.

[0026] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0028] Figure 1A structural block diagram of an image deraining network in one embodiment;

[0029] Figure 2 for Figure 1 Schematic diagram of the structure of the image deraining network;

[0030] Figure 3 Schematic diagram of the structure of a channel attention module in one embodiment;

[0031] Figure 4 is a schematic diagram of the structure of a spatial attention module in one embodiment;

[0032] Figure 5 A schematic diagram of a flow chart of a training method for an image deraining network in one embodiment;

[0033] Figure 6 A schematic diagram of the effect of removing small raindrops from a test set image using a trained image deraining network in one embodiment;

[0034] Figure 7 A schematic diagram of the effect of removing large raindrops from a test set image using a trained image deraining network in one embodiment;

[0035] Figure 8 It is a structural block diagram of a training device for an image deraining network in one embodiment;

[0036] Fig. 9 FIG. 1 is a diagram showing the internal structure of a training device for an image deraining network in one embodiment. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0038] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0039] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0040] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0041] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the objects used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those shown or described.

[0042] Figure 1 FIG. 4 is a structural block diagram of an image deraining network in one embodiment.

[0043] like Figure 1 As shown, an embodiment of the present application provides an image deraining network, which includes a first convolutional network and a second convolutional network, wherein the first convolutional network is used to predict the image to be derained based on an improved UNet network, and perform pixel-by-pixel convolution of the predicted first deraining filter kernel with the image to be derained to obtain a first derained image; the second convolutional network is used to predict the first derained image based on the improved UNet network, and perform pixel-by-pixel convolution of the predicted second deraining filter kernel with the first derained image to obtain a target derained image.

[0044] It should be understood that since generative adversarial networks are widely used in image restoration, when removing raindrops from an image, the image to be derained is usually input into a trained generative adversarial network to obtain a derained image output by the generative adversarial network. However, the residual traces in the derained image obtained by the generative adversarial network are relatively obvious, and the deraining effect is poor. This embodiment uses an improved UNet network to construct two convolutional networks, and uses the two convolutional networks as image deraining networks. The first convolutional network performs rough deraining, and the second convolutional network performs precise deraining, which has higher accuracy than using a primary network to remove rain. In the convolutional network, the deraining filter kernel of each pixel of the input image is predicted by the UNet network, and then the deraining filter kernel is convolved with the input image pixel by pixel. Compared with the generative adversarial network, it can not only obtain more features of the raindrop area, but also make up for the information loss of the feature map transmitted from the previous layer when restoring the image, and has a better deraining effect.

[0045] The improved UNet network is a network based on the UNet network, which can extract the features of the raindrop area by downsampling, and then restore the image to its original size by upsampling, thereby realizing the feature extraction of the raindrop area of ​​the input image, that is, obtaining the rain removal filter kernel of each pixel of the input image. This embodiment does not limit the specific network structure of the improved UNet network.

[0046] Specifically, the first rain removal filter kernel performs pixel-by-pixel convolution with the image to be rained off, and a convolution operation is performed on each pixel of the image to be rained off and an element corresponding to each pixel in the first rain removal filter kernel. The second rain removal filter kernel performs pixel-by-pixel convolution with the first rain removal image in the same manner, which will not be repeated here.

[0047] In the specific implementation, raindrop image datasets of different scenes, different numbers and sizes, that is, different degrees of distortion can be collected, and the image datasets can be divided into training set, validation set and test set in the ratio of 3:1:1. The first two are used for training, hyperparameter adjustment and preliminary evaluation of the image deraining network, and the last one is evaluated after the image deraining network training is completed. After the image deraining network training is completed, the current raindrop image collected by the vehicle is input into the trained image deraining network, and the derained image output by the network can be obtained.

[0048] This embodiment designs a first convolutional network and a second convolutional network in the image deraining network, wherein the first convolutional network is used to predict the image to be derained based on the improved UNet network, and convolve the predicted first deraining filter kernel with the image to be derained pixel by pixel to obtain the first derained image; the second convolutional network is used to predict the first derained image based on the improved UNet network, and convolve the predicted second deraining filter kernel with the first derained image pixel by pixel to obtain the target derained image. By designing the image deraining network as two convolutional networks constructed based on the improved UNet network, and using the image deraining network to derain the image to be derained, the accuracy of image deraining can be effectively improved.

[0049] Figure 2 for Figure 1 Schematic diagram of the structure of the image deraining network.

[0050] The first convolutional network and the second convolutional network in the image deraining network use the same improved UNet network. For the convenience of distinction and description, Figure 2 Different numbers are used for the specific structures in the first convolutional network and the second convolutional network. The following focuses on the explanation of the improved UNet network in the first convolutional network. Figure 2 As shown, in one example, the improved UNet network includes the first network layer C1 to the thirteenth network layer D13 in sequence, the first network layer C1, the second network layer C2, the third network layer C3, the fourth network layer C4, the fifth network layer C5, the seventh network layer C7, the ninth network layer C9 and the eleventh network layer C11 are all combination layers, the sixth network layer S6, the eighth network layer S8, the tenth network layer S10 and the twelfth network layer S12 are upsampling layers, the thirteenth network layer D13 is a void convolution layer, and the combination layer includes multiple of an average pooling layer, a convolution layer and an activation layer.

[0051] It should be understood that after the derained image is operated through the first network layer C1 to the fifth network layer C5, the size of the feature map gradually decreases, and then after the seventh network layer C7 to the twelfth network layer S12, the size of the feature map gradually increases, that is, the original image size is restored by downsampling first and then upsampling; the second-level convolutional network takes the output of the first convolutional network as input, and performs pixel-by-pixel convolution with the input after the second predicted deraining filter kernel to obtain the final refined output, that is, the target derained image after the final raindrops are removed.

[0052] In this embodiment, the first network layer C1, the second network layer C2, the third network layer C3, the fourth network layer C4, the fifth network layer C5, the seventh network layer C7, the ninth network layer C9 and the eleventh network layer C11 are all designed with void convolutions, wherein the expansion rates of the second network layer C2, the third network layer C3, the fourth network layer C4, the seventh network layer C7, the ninth network layer C9 and the eleventh network layer C11 are all greater than 1.

[0053] The convolution kernel size and convolution step size of the first network layer C1, the second network layer C2, the third network layer C3, the fourth network layer C4, the fifth network layer C5, the seventh network layer C7, the ninth network layer C9 and the eleventh network layer C11 can be designed according to the specific situation. For example, the convolution kernel size can be 3 and the convolution step size can be 1. This embodiment does not impose any restrictions on this.

[0054] By designing dilated convolutions for the above network layers, the receptive field of the image deraining network is increased, and the ability to perceive the differentiated features of the raindrop pixel neighborhood area is improved.

[0055] In one embodiment, the dilation rates of the first network layer C1 to the fourth network layer C4, the fifth network layer C5, the seventh network layer C7, the ninth network layer C9 and the eleventh network layer C11 increase in a sawtooth shape.

[0056] For example, the expansion rates of the first network layer C1, the second network layer C2, the third network layer C3 and the fourth network layer C4 are 1, 2, 3, and 5 respectively, and the expansion rates of the fifth network layer C5, the seventh network layer C7, the ninth network layer C9 and the eleventh network layer C11 are 1, 2, 3, and 4 respectively.

[0057] In a specific implementation, the first network layer C1 may be composed of a dilation rate 1 hole convolution and a rectified linear unit (ReLU) layer, and the second network layer C2 to the fifth network layer C5 perform an average pooling operation on the basis of the first network layer C1.

[0058] The improved UNet network in the second convolutional network includes a fourteenth network layer C14 (corresponding to the first network layer C1), a fifteenth network layer C15 (corresponding to the second network layer C2), a sixteenth network layer C16 (corresponding to the third network layer C3), a seventeenth network layer C17 (corresponding to the fourth network layer C4), an eighteenth network layer C18 (corresponding to the fifth network layer C5), a nineteenth network layer S19 (corresponding to the sixth network layer S6), a twentieth network layer C20 (corresponding to the seventh network layer C7), a twenty-first network layer S21 (corresponding to the eighth network layer S8), a twenty-second network layer C22 (corresponding to the ninth network layer C9), a twenty-third network layer S23 (corresponding to the tenth network layer S10), a twenty-fourth network layer C24 (corresponding to the eleventh network layer C11), a twenty-fifth network layer S25 (corresponding to the twelfth network layer S12), and a twenty-sixth network layer D26 (corresponding to the thirteenth network layer D13). Its structural design can refer to the improved UNet network in the first convolutional network, and will not be repeated here.

[0059] It should be noted that the current convolutional neural networks (CNN) and GAN networks based on kernel prediction networks (KPN) are prone to "flashing" and unstable rain removal video images. This embodiment designs multi-layer dilated convolutions with different expansion rates so that the receptive fields of the multi-layer dilated convolutions are different, and multi-scale features corresponding to raindrops of different sizes can be obtained, so that the convolution network can better perceive the pixel differences between adjacent frames, so as to stably process the changes in raindrops and background pixels between adjacent frames. By designing the expansion rate into a sawtooth structure, the grid effect caused by the loss of too much information can be effectively avoided while increasing the receptive field.

[0060] In one embodiment, the second network layer C2 is jump-connected to the tenth network layer S10, the third network layer C3 is jump-connected to the eighth network layer S8, and the fourth network layer C4 is jump-connected to the sixth network layer S6.

[0061] Specifically, through the jump connection between the second network layer C2 and the tenth network layer S10, channel fusion can be achieved between the feature map output by the second network layer C2 and the feature map obtained by upsampling the output of the ninth network layer C9 (i.e., the feature map output by the tenth network layer S10); through the jump connection between the third network layer C3 and the eighth network layer S8, channel fusion can be achieved between the feature map output by the third network layer C3 and the feature map obtained by upsampling the output of the seventh network layer C7 (i.e., the feature map output by the eighth network layer S8); through the jump connection between the fourth network layer C4 and the sixth network layer S6, channel fusion can be achieved between the feature map output by the fourth network layer C4 and the feature map obtained by upsampling the output of the fifth network layer C5 (i.e., the feature map output by the sixth network layer S6).

[0062] By designing a skip connection structure in the process of predicting the deraining filter kernel through the first convolutional network and the second convolutional network, it is not only possible to prevent the gradient vanishing or exploding problem caused by deepening the network during training, but also to compensate for the information loss of the feature map transmitted from the previous layer during image restoration.

[0063] Figure 3 Schematic diagram of the structure of a channel attention module in one embodiment. Figure 2 and Figure 3 , the seventh network layer C7 includes a channel attention module; the channel attention module includes a first input unit, a first average pooling unit, a second average pooling unit, a first fusion unit, a first convolution unit, a first activation unit, a second convolution unit, a second activation unit and a first inner product unit; wherein the first input unit is respectively connected to the first average pooling unit, the second average pooling unit and the first inner product unit; the first average pooling unit and the second average pooling unit are both connected to the first fusion unit; the first fusion unit is sequentially connected to the first inner product unit via the first convolution unit, the first activation unit, the second convolution unit and the second activation unit.

[0064] It should be understood that the feature map output by the seventh network layer C7 is average pooled by the first average pooling unit and the second average pooling unit, and the processed feature map is sequentially subjected to channel fusion by the first fusion unit, convolution operation by the first convolution unit, ReLU activation function activation by the first activation unit, convolution operation by the second convolution unit, and Sigmoid activation function activation by the second activation unit, and then inner product processing is performed with the feature map output by the seventh network layer C7 by the first inner product unit, and finally the feature map is output.

[0065] Figure 4 FIG. 1 is a schematic diagram of the structure of a spatial attention module in one embodiment. Figure 2 and Figure 4, the eleventh network layer C11 includes a spatial attention module; the spatial attention module includes a second input unit, a cross-channel average pooling unit, a cross-channel maximum pooling unit, a second fusion unit, a third convolution unit, a third activation unit and a second inner product unit; wherein the second input unit is respectively connected to the cross-channel average pooling unit, the cross-channel maximum pooling unit and the second inner product unit; the cross-channel average pooling unit and the cross-channel maximum pooling unit are both connected to the second fusion unit; the second fusion unit is sequentially connected to the second inner product unit via the third convolution unit and the third activation unit.

[0066] It should be understood that the feature map output by the eleventh network layer C11 is pooled by a cross-channel average pooling unit and a cross-channel maximum pooling unit, respectively, and the processed feature map is sequentially subjected to channel fusion by the third fusion unit, convolution operation by the third convolution unit, and activation by the Sigmoid activation function by the third activation unit, and then inner product processing is performed with the feature map output by the eleventh network layer C11 by the second inner product unit, and finally the feature map is output.

[0067] The feature map output by the eleventh network layer C11 is upsampled and then subjected to a dilated convolution with a kernel size of 1, a stride of 1, a padding of 0, and a dilation rate of 2 to obtain the first rain removal filter kernel. The filter kernel is convolved pixel by pixel with the image to be rained off from the jump connection to obtain the first rain removal image.

[0068] It should be understood that, since the number of channels of the feature map input to the seventh network layer C7 is large during the jump connection process, and the size of the feature map input to the eleventh network layer C11 is large, in order to fully utilize the performance of the attention mechanism and reduce the amount of model calculation, the channel attention mechanism is only added to the seventh network layer C7, and the spatial attention mechanism is only added to the eleventh network layer C11. By adding the channel attention mechanism and the spatial attention mechanism to the improved UNet network, the image deraining network pays more attention to the characteristics of the raindrop area, increases the distinction between raindrops and background areas in the input image to be derained, and thus improves the accuracy of the model image deraining.

[0069] Figure 5 FIG. 1 is a flow chart of a training method for an image deraining network in one embodiment. Figure 5 As shown, a training method for an image deraining network is provided. This embodiment takes the method applied to a terminal as an example. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the training method for an image deraining network may include the following steps:

[0070] Step 501 : obtaining a raindrop-free image and an image to be rain-removed collected by a vehicle at the same geographical location.

[0071] Specifically, the vehicle collects raindrop image datasets of different scenes, different numbers and different sizes (i.e., different degrees of distortion). The specific method can be: in various scenes such as evening and cloudy outdoors, underground parking garage, sunny and daytime outdoors, and night outdoors, first take a raindrop-free photo when parking as the true value data, and then use a spray bottle with adjustable flow rate and use the minimum flow rate to manually spray water droplets on the vehicle's rearview lens. At this time, take a photo as small raindrop data; wipe off the raindrops on the lens, increase the flow rate by one gear and continue to spray water droplets on the lens, take a photo at this time as large raindrop data, and then wipe off the raindrops on the lens again; drive the vehicle to different locations and continue to repeat the above collection process until the data volume reaches the predetermined requirement.

[0072] The terminal obtains the collected data from the vehicle and pairs the distorted data (i.e., the image to be de-rained) and the corresponding true value data (i.e., the image without raindrops) of the same geographical location one by one to obtain a group of data sets. The terminal can divide the collected data sets into training sets, validation sets, and test sets in a ratio of 3:1:1. The first two are used for network training, hyperparameter adjustment, and preliminary evaluation, respectively, and the last one is used for evaluation of the trained network.

[0073] Step 502: input the image to be derained and the image without raindrops into the above-mentioned image deraining network for training to obtain a trained image deraining network.

[0074] Specifically, after the derained image is input into the image deraining network, it passes through the first convolutional network and the second convolutional network successively to obtain the target derained image after raindrops are removed. During training, the loss value of the target derained image and the input raindrop-free image is calculated, and the network parameters are continuously adjusted through training iterations; during inference, the target derained image is directly used as the final output image.

[0075] Refer to Figure 2 and Figure 5In one embodiment, the image shape value of the image to be derained is 368*640*3. After being input into the image deraining network, it is preprocessed to obtain a preprocessed image, wherein the preprocessing steps include converting BGR to RGB, normalizing to [0,1], subtracting the mean (0.306, 0.309, 0.300) and dividing the variance (0.150, 0.155, 0.162), etc., wherein the mean and variance values ​​are obtained by calculating on the training set. The second convolutional network reuses the first convolutional network. Only the first convolutional network is specifically described here. It uses an improved UNet network structure to predict the first deraining filter kernel, and then the filter kernel is convolved pixel by pixel with the image to be derained to obtain the first derained image. The first convolutional network adds channel and spatial attention mechanisms and multi-layer hole convolution designs with different expansion rates to enhance the model's ability to distinguish between raindrops and backgrounds and perceive the differentiated features of neighborhood pixels. The network structure of the first convolutional network is as follows: Figure 2As shown in the upper part, the first network layer C1, the second network layer C2, the third network layer C3, the fourth network layer C4, the fifth network layer C5, the seventh network layer C7, the ninth network layer C9 and the eleventh network layer C11 are all designed with dilated convolutions, and their kernel size is 3 and stride is 1. While increasing the receptive field, in order to avoid the grid effect caused by losing too much information, the dilation rate of the dilated convolution in these 8 network layers is designed to be a sawtooth structure, with values ​​of 1, 2, 3, and 5 respectively. The first network layer is composed of a dilated convolution with a dilation rate of 1 and a ReLU layer. The second network layer C2 to the fifth network layer C5 are first average pooled on the basis of the first network layer C1. The feature maps output by the second network layer C2, the third network layer C3 and the fourth network layer C4 are respectively fused with the feature maps obtained by upsampling the outputs of the ninth network layer C9, the seventh network layer C7 and the fifth network layer C5 through jump connections. Considering that the number of channels of the feature map input to the seventh network layer C7 is large and the size of the feature map input to the eleventh network layer C11 is large during the jump connection process, a channel attention mechanism is added to the seventh network layer C7, and a spatial attention mechanism is added only to the eleventh network layer C11. The feature map output by the eleventh network layer C11 is upsampled and then subjected to a hole convolution with a kernel size of 1, a stride of 1, a padding of 0, and a dilation rate of 2 to obtain the first deraining filter kernel, which is convolved pixel by pixel with the image to be derained from the jump connection to obtain the first deraining image. After the preprocessed image is operated from the first network layer C1 to the fifth network layer C5, the size of the feature map is gradually reduced, and then after the operation from the seventh network layer C7 to the twelfth network layer S12, the size of the feature map is gradually increased, that is, the original image size is restored by downsampling first and then upsampling. The second convolutional network takes the output of the first convolutional network as input, and the second deraining filter kernel predicted for the second time is convolved pixel by pixel with the input to obtain the final refined output, that is, the target deraining image. A pre-set loss function can be used during network training. During the iterative training of the network, the parameters are continuously updated through the back propagation of the loss value. The training is terminated when the training converges to obtain a trained image deraining network.

[0076] In the specific implementation, the loss function can be expressed by the following formula:

[0077] Loss=Loss SSIM +λLoss PSNR ;

[0078]

[0079] Loss SSIM =[l(x,y)]α [c(x,y)] β [s(x,y)] γ ;

[0080] in,

[0081]

[0082]

[0083]

[0084]

[0085] Among them, MAX I is the maximum pixel value of the target derained image, m and n are the width and height of the target derained image, I(i, j) is the pixel value of the raindrop-free image, K(i, j) is the pixel value of the image to be derained, μ x is the average value of the raindrop-free image, μ y is the average value of the image to be derained, σ x is the standard deviation of the raindrop-free image, σ y is the standard deviation of the image to be derained, σ xy is the covariance of the image to be derained, C1, C2, C3 and λ are constants. In a specific implementation, the following values ​​can be taken: λ = 0.03, α = β = γ = 1, C3 = C2 / 2.

[0086] The image deraining network trained by the above scheme can effectively solve the problem of obvious residual traces after removing raindrops from a single-frame image and the problem of "flashing" when removing raindrops from multiple consecutive frames of images when used for image deraining. It has the characteristics of high precision in single-frame image deraining and stable and smooth pictures after deraining multiple frames of images.

[0087] In one embodiment, the training method of the image deraining network may further include the following steps:

[0088] Step 503: deploy the trained image deraining network on the hardware platform Nvidia Jetson Xavier, and perform 8-bit integer quantization on the trained image deraining network.

[0089] Specifically, the trained image deraining network is a .pth format model, which can be converted to a .onnx format model, and then further quantized to a .trt format model with int8 data type through the tensorrt inference library on the hardware platform Nvidia Jetson Xavier. When tested on the test set, the model's inference speed is greater than 30 frames per second, and the peak signal to noise ratio (PSNR) and structural similarity (SSIM) results are shown in Table 1 below:

[0090]

[0091] Table 1

[0092] It can be seen from Table 1 that when the trained image deraining network is used to remove small or large raindrops in the test set image, the PSNR and SSIM values ​​are relatively high. It can be seen that the trained image deraining network has the advantage of high single-frame deraining accuracy.

[0093] Figure 6 This is a schematic diagram of the effect of removing small raindrops from a test set image using a trained image deraining network in one embodiment, wherein from left to right are an image without raindrops, an image to be derained with small raindrops in the test set, and a target derained image after removing raindrops. Figure 7 This is a schematic diagram of the effect of using a trained image deraining network to remove large raindrops from a test set image in one embodiment, wherein from left to right are an image without raindrops, an image to be derained with large raindrops in the test set, and a target derained image after removing raindrops.

[0094] like Figure 6 and Figure 7 As shown in the figure, when the trained image deraining network is used to remove small or large raindrops from the image to be derained, there is basically no "flickering" of the picture when dealing with changing backgrounds and raindrops, which has the advantage of stable and smooth derained pictures.

[0095] By deploying the trained image deraining network on the hardware platform Nvidia Jetson Xavier and performing 8-bit integer quantization on the trained image deraining network, the final model inference speed is faster, which can meet the real-time requirements of model deployment on Nvidia Jetson Xavier and greatly improve the user experience.

[0096] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0097] In one embodiment, Figure 8 As shown, a training device for an image deraining network is provided, comprising: a data acquisition unit 801 and a network training unit 802, wherein:

[0098] The data acquisition unit 801 is used to acquire raindrop-free images and rain-removed images collected by the vehicle at the same geographical location.

[0099] The network training unit 802 is used to input the image to be derained and the image without raindrops into the above-mentioned image deraining network for training to obtain a trained image deraining network.

[0100] In some embodiments of the present application, based on the aforementioned scheme, the training device of the image deraining network may further include a network deployment unit 803, which is used to deploy the trained image deraining network on the hardware platform Nvidia Jetson Xavier and perform 8-bit integer quantization on the trained image deraining network.

[0101] Each module in the training device of the above-mentioned image deraining network can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each of the above modules.

[0102] Based on the same inventive concept, an embodiment of the present application also provides a training device for an image deraining network. Fig. 9 FIG. 1 is an internal structure diagram of a training device for an image deraining network in one embodiment. Fig. 9 As shown, the training device of the image deraining network includes one or more memories 904, one or more processors 902, and at least one computer program (program code) stored in the memory 904 and executable on the processor 902. When the processor 902 executes the computer program, the training method of the image deraining network as described above is implemented.

[0103] Among them, Fig. 9 In the embodiment of the present invention, a bus architecture (represented by bus 900) is shown, which may include any number of interconnected buses and bridges, and bus 900 links various circuits including one or more processors represented by processor 902 and memory represented by memory 904. Bus 900 may also link various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. Bus interface 905 provides an interface between bus 900 and receiver 901 and transmitter 903. Receiver 901 and transmitter 903 may be the same element, namely a transceiver, providing a unit for communicating with various other devices over a transmission medium. Processor 902 is responsible for managing bus 900 and general processing, while memory 904 may be used to store data used by processor 902 when performing operations.

[0104] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the training device of the image deraining network to which the solution of the present application is applied. The specific training device of the image deraining network may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0105] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0106] Obtain raindrop-free images and rain-removing images collected by the vehicle at the same geographical location;

[0107] The image to be derained and the image without raindrops are input into the above-mentioned image deraining network for training to obtain a trained image deraining network.

[0108] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0109] The trained image deraining network is deployed on the hardware platform Nvidia Jetson Xavier, and the trained image deraining network is quantized to 8-bit integer.

[0110] The functions described herein may be implemented in hardware, software executed by a processor, firmware, or any combination thereof. If implemented in software executed by a processor, the functions may be stored as one or more instructions or codes on a computer-readable medium or transmitted via a computer-readable medium. Other examples and implementations are within the scope and spirit of the present application and the appended claims. For example, due to the nature of software, the functions described above may be implemented using software executed by a processor, hardware, firmware, hard wiring, or a combination of any of these. In addition, each functional unit may be integrated into a processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit.

[0111] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0112] The units described as separate components may or may not be physically separated, and the components of the control device may or may not be physical units, that is, they may be located in one place or distributed in multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0113] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0114] The above description is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of the claims of the present application.

Claims

1. An image deraining network, characterized in that: The image deraining network includes a first convolutional network and a second convolutional network, wherein: The first convolutional network is used to predict the image to be derained based on the improved UNet network, and perform pixel-by-pixel convolution on the predicted first deraining filter kernel and the image to be derained to obtain a first derained image; The second convolutional network is used to predict the first derained image based on the improved UNet network, and perform pixel-by-pixel convolution on the predicted second derained filter kernel and the first derained image to obtain a target derained image; The improved UNet network includes the first network layer to the thirteenth network layer in sequence, the first network layer to the fourth network layer, the fifth network layer, the seventh network layer, the ninth network layer and the eleventh network layer are all combination layers, the sixth network layer, the eighth network layer, the tenth network layer and the twelfth network layer are upsampling layers, the thirteenth network layer is a hole convolution layer, and the combination layer includes multiple of an average pooling layer, a convolution layer and an activation layer; Among them, the expansion rates from the second network layer to the fourth network layer, the seventh network layer, the ninth network layer and the eleventh network layer are all greater than 1.

2. The image deraining network according to claim 1, characterized in that: The expansion rates from the first network layer to the fourth network layer, the fifth network layer, the seventh network layer, the ninth network layer and the eleventh network layer increase in a sawtooth shape.

3. The image deraining network according to claim 1, characterized in that: The second network layer is jump-connected to the tenth network layer, the third network layer is jump-connected to the eighth network layer, and the fourth network layer is jump-connected to the sixth network layer.

4. The image deraining network according to claim 3, characterized in that: The seventh network layer includes a channel attention module; the channel attention module includes a first input unit, a first average pooling unit, a second average pooling unit, a first fusion unit, a first convolution unit, a first activation unit, a second convolution unit, a second activation unit and a first inner product unit; wherein, The first input unit is connected to the first average pooling unit, the second average pooling unit and the first inner product unit respectively; The first average pooling unit and the second average pooling unit are both connected to the first fusion unit; The first fusion unit is connected to the first inner product unit via the first convolution unit, the first activation unit, the second convolution unit, and the second activation unit in sequence.

5. The image deraining network according to claim 3, characterized in that: The eleventh network layer includes a spatial attention module; the spatial attention module includes a second input unit, a cross-channel average pooling unit, a cross-channel maximum pooling unit, a second fusion unit, a third convolution unit, a third activation unit, and a second inner product unit; wherein, The second input unit is connected to the cross-channel average pooling unit, the cross-channel maximum pooling unit and the second inner product unit respectively; The cross-channel average pooling unit and the cross-channel maximum pooling unit are both connected to the second fusion unit; The second fusion unit is connected to the second inner product unit via the third convolution unit and the third activation unit in sequence.

6. A training method for an image deraining network, characterized in that: The training method comprises: Obtain raindrop-free images and rain-removing images collected by the vehicle at the same geographical location; The image to be derained and the image without raindrops are input into the image deraining network according to any one of claims 1 to 5 for training to obtain a trained image deraining network.

7. The training method according to claim 6, characterized in that: The training method further comprises: The trained image deraining network is deployed on the hardware platform Nvidia Jetson Xavier, and the trained image deraining network is quantized to 8-bit integer.

8. A training device for an image deraining network, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 6 to 7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 6 to 7 are implemented.

Images