Image rain removal method and apparatus

Abstract

The application provides a kind of image rain removal method and device, belong to image processing field, method includes: S1: each original rain image and each clear rain image is carried out data enhancement processing respectively, obtains rain day training image and rain day clear training image;S2: initial rain removal model is constructed;Wherein, initial rain removal model includes image processing module and image discrimination module;S3: based on image processing module, each rain day training image and rain day clear training image are carried out image processing respectively to obtain reconstructed rain day image, rain stripe removal image and reconstructed clear image.The application solves the problem that fog-related feature extraction is not accurate and image generation quality is low, obtains clear image with richer image content and higher generation quality, and improves the accuracy of clear image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to an image deraining method and apparatus. Background Technology

[0002] Rain is a common weather phenomenon, and most current computer vision tasks assume a sunny environment. The presence of rain significantly affects the visibility and performance of many computer vision tasks, such as object detection and instance segmentation. Therefore, image deraining has become one of the important fundamental tasks in the fields of digital image processing and computer vision technology.

[0003] Existing rain removal methods use paired training sets and train networks using supervised learning. While these methods achieve good performance, they may fail when removing rain from real-world rain images. This is because rain streaks are uniformly distributed in synthetic data, but not in the real world. Furthermore, obtaining paired images in real-world scenes is difficult. Recently, some semi-supervised and unsupervised rain removal methods have been proposed to improve the model's generalization ability in real-world scenarios. However, these methods still suffer from inaccurate fog-related feature extraction and low image generation quality. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide an image deraining method and apparatus to address the shortcomings of the prior art.

[0005] The technical solution of the present invention to solve the above-mentioned technical problems is as follows: An image deraining method, comprising the following steps:

[0006] S1: Import multiple original rain images and multiple clear rain images that correspond one-to-one with the multiple original rain images, and perform data augmentation processing on each of the original rain images and each of the clear rain images to obtain rain training images of each of the original rain images and rain clear training images of each of the clear rain images.

[0007] S2: Construct an initial deraining model; wherein, the initial deraining model includes an image processing module and an image discrimination module;

[0008] S3: Based on the image processing module, image processing is performed on each of the rainy day training images and the rainy day clear training images of each of the clear rain images to obtain the reconstructed rainy day images, rain stripe removal images, and reconstructed clear images of each of the clear rain images.

[0009] S4: Perform loss function analysis based on each of the original rain images, each of the clear rain images, each of the reconstructed rain images from the rain training images, and each of the reconstructed clear images from the clear rain images to obtain the loss function for each of the original rain images.

[0010] S5: Based on the image discrimination module, the mean square error of each original rain image is calculated according to each original rain image, each clear rain image, each rain stripe removed image of each rain training image, and each reconstructed clear image of each clear rain image.

[0011] S6: Determine whether all the original rain images meet the condition that the loss function of the original rain image is less than or equal to a preset loss function and the root mean square error of the original rain image is less than or equal to a preset root mean square error; if not, update the parameters of the image processing module according to the loss function and root mean square error of all the original rain images respectively, and return to step S3 after parameter update to repeat until all the original rain images meet the condition; if yes, use the image processing module after the last parameter update as the target rain removal model and execute step S7;

[0012] S7: Import the rain image to be processed, and perform image deraining processing on the rain image to be processed through the target deraining model to obtain the image deraining result.

[0013] Based on the above-mentioned image deraining method, the present invention also provides an image deraining device.

[0014] Another technical solution of the present invention to solve the above-mentioned technical problems is as follows: An image deraining device, comprising:

[0015] The data augmentation module is used to import multiple original rain images and multiple clear rain images that correspond one-to-one with the multiple original rain images, and to perform data augmentation processing on each of the original rain images and each of the clear rain images to obtain rain training images of each of the original rain images and rain clear training images of each of the clear rain images.

[0016] A model building module is used to build an initial rain removal model; wherein, the initial rain removal model includes an image processing module and an image discrimination module;

[0017] The processing module is used to perform image processing based on the image processing module, according to each of the rainy day training images and the rainy day clear training images of each of the clear rain images, to obtain the reconstructed rainy day images, rain stripe removal images, and reconstructed clear images of each of the clear rain images.

[0018] The loss function analysis module is used to perform loss function analysis on each of the original rain images, each of the clear rain images, each of the reconstructed rain images from the rain training images, and each of the reconstructed clear images from the clear rain images, to obtain the loss function of each of the original rain images.

[0019] The mean square error calculation module is used to calculate the mean square error of each original rain image based on the image discrimination module, according to each original rain image, each clear rain image, each rain stripe removed image of each rain training image, and each reconstructed clear image of each clear rain image.

[0020] The judgment module is used to determine whether all the original rain images meet the following conditions: the loss function of the original rain image is less than or equal to a preset loss function, and the root mean square error of the original rain image is less than or equal to a preset root mean square error. If not, the image processing module updates its parameters according to the loss function and root mean square error of all the original rain images, and returns the updated parameters to the processing module until all the original rain images meet the conditions. If yes, the image processing module after the last parameter update is used as the target rain removal model.

[0021] The rain removal result acquisition module is used to import the rain image to be processed, and perform image rain removal processing on the rain image to be processed through the target rain removal model to obtain the image rain removal result.

[0022] Based on the above-mentioned image deraining method, the present invention also provides an image deraining system.

[0023] Another technical solution of the present invention to solve the above-mentioned technical problems is as follows: an image deraining system, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the image deraining method described above is implemented.

[0024] Based on the above-described image deraining method, the present invention also provides a computer-readable storage medium.

[0025] Another technical solution of the present invention to solve the above-mentioned technical problems is as follows: a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the image deraining method as described above.

[0026] The beneficial effects of this invention are as follows: Rainy day training images and clear rainy day training images are obtained through data augmentation processing of the original rainy images and the clear rainy day training images. Based on the image processing module, reconstructed rainy day images, rain stripe removal images, and reconstructed clear images are obtained through image processing of the rainy day training images and the clear rainy day training images. A loss function is obtained through loss function analysis of the original rainy images, clear rainy images, reconstructed rainy day images, and reconstructed clear images. A mean squared error is calculated based on the mean squared error of the original rainy images, clear rainy images, rain stripe removal images, and reconstructed clear images. It is then determined whether all original rainy images meet the conditions. If not, the parameters of the image processing module are updated based on the loss function and mean squared error of all original rainy images until all original rainy images meet the conditions. The image processing module with the last updated parameters is then used as the target deraining model. The target deraining model is used to process the rainy images to be processed to obtain the derained image result. This solves the problems of inaccurate extraction of fog-related features and low image generation quality, resulting in clear images with richer image content and higher generation quality, thus improving the accuracy of clear images. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating an image deraining method provided in an embodiment of the present invention;

[0028] Figure 2 This is a block diagram of an image deraining device provided in an embodiment of the present invention. Detailed Implementation

[0029] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0030] Figure 1 This is a flowchart illustrating an image deraining method provided in an embodiment of the present invention.

[0031] like Figure 1 As shown, an image deraining method includes the following steps:

[0032] S1: Import multiple original rain images and multiple clear rain images that correspond one-to-one with the multiple original rain images, and perform data augmentation processing on each of the original rain images and each of the clear rain images to obtain rain training images of each of the original rain images and rain clear training images of each of the clear rain images.

[0033] S2: Construct an initial deraining model; wherein, the initial deraining model includes an image processing module and an image discrimination module;

[0034] S3: Based on the image processing module, image processing is performed on each of the rainy day training images and the rainy day clear training images of each of the clear rain images to obtain the reconstructed rainy day images, rain stripe removal images, and reconstructed clear images of each of the clear rain images.

[0035] S4: Perform loss function analysis based on each of the original rain images, each of the clear rain images, each of the reconstructed rain images from the rain training images, and each of the reconstructed clear images from the clear rain images to obtain the loss function for each of the original rain images.

[0036] S5: Based on the image discrimination module, the mean square error of each original rain image is calculated according to each original rain image, each clear rain image, each rain stripe removed image of each rain training image, and each reconstructed clear image of each clear rain image.

[0037] S6: Determine whether all the original rain images meet the condition that the loss function of the original rain image is less than or equal to a preset loss function and the root mean square error of the original rain image is less than or equal to a preset root mean square error; if not, update the parameters of the image processing module according to the loss function and root mean square error of all the original rain images respectively, and return to step S3 after parameter update to repeat until all the original rain images meet the condition; if yes, use the image processing module after the last parameter update as the target rain removal model and execute step S7;

[0038] S7: Import the rain image to be processed, and perform image deraining processing on the rain image to be processed through the target deraining model to obtain the image deraining result.

[0039] Preferably, the preset mean square error can be 0.5.

[0040] It should be understood that data augmentation is performed on the original rain image and the clear image (i.e., the clear rain image) to obtain unpaired training samples. The training samples include unpaired rainy day image training samples (i.e., the rainy day training images) and clear image training samples (i.e., the rainy day clear training images). The training samples (i.e., the rainy day training images and the rainy day clear training images) are then input into an unsupervised learning-based rain removal model (i.e., the initial rain removal model) for training.

[0041] It should be understood that the original rain image and the clear image (i.e., the clear rain image) are randomly cropped to obtain training samples (i.e., the rainy day training image and the rainy day clear training image), and the training samples (i.e., the rainy day training image and the rainy day clear training image) are input into the rain removal model based on unsupervised learning (i.e., the initial rain removal model) for training.

[0042] Specifically, the original rainy day image and the clear image (i.e., the clear rainy image) are sampled, that is, the format, size, pixels, etc. of the samples are processed to obtain the training samples of the model (i.e. the rainy day training image and the clear rainy day training image), and the training samples (i.e. the rainy day training image and the clear rainy day training image) are input into the rain removal model based on unsupervised learning (i.e., the initial rain removal model) for training.

[0043] In the above embodiments, rain training images and clear rain training images are obtained by data augmentation processing of the original rain images and the clear rain images. Based on the image processing module, the rain training images, rain stripe removal images, and reconstructed clear images are obtained by image processing based on the rain training images and the clear rain training images. The loss function is obtained by loss function analysis based on the original rain images, clear rain images, reconstructed rain images, and reconstructed clear images. Based on the image discrimination module, the mean square error is calculated based on the mean square error of the original rain images, clear rain images, rain stripe removal images, and reconstructed clear images. It is determined whether all original rain images meet the conditions. If not, the parameters of the image processing module are updated according to the loss function and mean square error of all original rain images until all original rain images meet the conditions. Then, the image processing module with the last updated parameters is used as the target deraining model. The target deraining model is used to process the rain images to be processed to obtain the image deraining result. This solves the problems of inaccurate extraction of fog-related features and low image generation quality, and obtains clear images with richer image content and higher generation quality, thus improving the accuracy of clear images.

[0044] Optionally, as an embodiment of the present invention, the image processing module includes a rain stripe extraction unit, a rain stripe removal unit, and a rain image generation unit, and the process of step S3 includes:

[0045] Based on the rain stripe extraction unit, rain stripes are extracted from each of the rainy day training images to obtain rain stripe extraction images of each of the rainy day training images.

[0046] Based on the rain stripe removal unit, rain stripes are removed from each of the rainy day training images to obtain rain stripe removed images of each of the rainy day training images.

[0047] Image fusion is performed on the rain stripe extraction image and the rain stripe removal image of each of the rainy day training images to obtain the first fused image of each of the rainy day training images;

[0048] Image fusion is performed on the rain stripe extraction images of each of the rainy training images and the rainy clear training images of each of the clear rain images to obtain the second fused image of each of the clear rain images;

[0049] Based on the rain image generation unit, the rain stripe extraction image, the first fused image, and the second fused image of each of the rainy day training images are reconstructed to obtain the reconstructed rainy day images of each of the rainy day training images and the clear images to be removed from each of the clear rainy images.

[0050] Based on the rain stripe removal unit, rain stripes are removed from the clear images to be removed in each of the clear rain images to obtain the reconstructed clear images of each of the rainy day training images.

[0051] It should be understood that the rainy day image training samples (i.e., the rainy day training images) are respectively input into the rain stripe extraction module and the rain stripe removal module to obtain the output of the rain stripe extraction module (i.e., the rain stripe extracted image) and the output of the rain stripe removal module (i.e., the rain stripe removed image). The output of the rain stripe extraction module (i.e., the rain stripe extracted image) and the output of the rain stripe removal module (i.e., the rain stripe removed image) are then image-fused and input into the rain image generation module to obtain the output of the rain image generation module (i.e., the first fused image).

[0052] In the above embodiments, the image processing module obtains the reconstructed rainy image, the rain stripe removal image, and the reconstructed clear image based on the rainy training image and the clear training image on the rainy day. It can predict the shape and position of the rain stripes in the image and the clear image, thus obtaining a clear image with richer image content and higher generation quality, and improving the accuracy of the clear image.

[0053] Optionally, as an embodiment of the present invention, the rain stripe extraction unit includes a convolutional kernel and a sigmoid activation function layer. The process of extracting rain stripes from each of the rainy day training images based on the rain stripe extraction unit to obtain rain stripe extraction images for each of the rainy day training images includes:

[0054] Based on the convolution kernel, feature extraction is performed on each of the rainy day training images to obtain the feature-extracted images of each of the rainy day training images.

[0055] Based on the sigmoid activation function layer, normalization processing is performed on each of the feature-extracted images to obtain rain stripe extraction images of each of the rainy day training images.

[0056] It should be understood that, to adapt to images of rain of varying intensities, multiple parallel rain feature aggregation blocks are used to aggregate global rain-related information, and convolutional kernels of different sizes are simultaneously used to learn the multi-scale correlation of rain stripe features. By focusing on the most informative knowledge, the network's discriminative learning ability is improved to more accurately represent rain features.

[0057] Understandably, the rain feature aggregation block first uses convolutional kernels of different sizes to learn the multi-scale correlations of rain-related features. It further simulates the long-range dependency patterns and location patterns of rain stripe features, extracting global features. To refine relevant information for better fusion and representation, weights are assigned to the global features. This effectively highlights rain-related features in the spatial dimension. Finally, the sigmoid activation function is used to weight the output of the parallel rain feature aggregation block, adjusting the importance of information at different spatial locations in the original input.

[0058] Specifically, in the rain stripe extraction module, the input image (i.e., the rainy day training image) is first expanded to increase its channel count, resulting in high-dimensional features. Then, multiple parallel rain feature aggregation blocks are used to aggregate global rain stripe information at different feature scales. In the first layer of the rain feature aggregation block, 3×3 and 5×5 convolutional kernels are used to learn the multi-scale correlation of rain and fog-related features. Finally, a sigmoid activation function is used to weight the output of the parallel rain feature aggregation blocks to adjust the importance of information at different spatial locations in the original input.

[0059] Understandably, to adapt to images of rain of varying intensities, multiple parallel rain feature aggregation blocks are used to aggregate global rain-related information, while convolutional kernels of different sizes are used simultaneously to learn the multi-scale correlation of rain stripe features. By focusing on the knowledge with the richest information, the network's discriminative learning ability is improved, resulting in a more accurate representation of rain features.

[0060] Specifically, the rain feature aggregation block first uses convolutional kernels of different sizes to learn the multi-scale correlation of rain-related features. It further simulates the long-range dependency patterns and location patterns of rain stripe features, extracting global features. To refine relevant information for better fusion and representation, weights are assigned to the global features. To effectively highlight rain-related features in the spatial dimension, a sigmoid activation function is finally used to weight the output of the parallel rain feature aggregation block, adjusting the importance of information at different spatial locations in the original input.

[0061] In the above embodiments, a feature-extracted image is obtained based on the feature extraction of the training image on a rainy day using a convolutional kernel. A rain stripe extraction image is obtained based on the normalization processing of the feature-extracted image using a sigmoid activation function layer. This can learn the multi-scale correlation of rain and fog-related features, adjust the importance of information at different spatial locations in the original input, and improve the network's discriminative learning ability by focusing on the knowledge with the richest information, so as to more accurately express rain features. At the same time, it also effectively highlights rain-related features.

[0062] Optionally, as an embodiment of the present invention, the rain stripe removal unit includes an encoding block, a residual block, and a decoding block. The process of removing rain stripes from each of the rainy day training images based on the rain stripe removal unit to obtain rain stripe-removed images of each of the rainy day training images includes:

[0063] Based on the coding block, each of the rainy day training images is encoded to obtain the encoded images of each of the rainy day training images;

[0064] Based on the residual block, feature extraction is performed on each of the encoded images to obtain the feature-extracted encoded images of each of the rainy day training images. The encoded images and feature-extracted encoded images of each of the rainy day training images are then fused to obtain the fused images of each of the rainy day training images.

[0065] Based on the decoding block, each of the fused images is decoded to obtain rain stripe removal images of each of the rainy day training images.

[0066] It should be understood that the encoding block is used to encode and compress information in the input image; the residual block is used to prevent gradient vanishing and further refine the features of the output result of the encoding block; the decoding block is used to downsample the convolutional layer in the convolutional neural network model, extract the result obtained from the residual block, and obtain a clear image (i.e., the rain stripe removal image).

[0067] Specifically, in the rain stripe removal module, the input image (i.e., the rainy day training image) is first convolved to extract shallow features. Then, average pooling is used to downsample the shallow features to obtain the final encoded features. This process is repeated twice. To avoid gradient vanishing and overfitting problems caused by increasing network depth, a residual network is used to further refine the encoded features. Finally, during the decoding process, dense connections are used to fuse the outputs of different layers to preserve image content and rich details.

[0068] In the above embodiments, the rain stripe removal unit obtains the rain stripe removal image based on the rain stripe removal of the training image on a rainy day. It extracts shallow features, avoids the gradient vanishing and overfitting problems caused by the increase of network depth, and preserves the image content and rich details.

[0069] Optionally, as an embodiment of the present invention, the rain image generation unit includes multiple first convolutional layers and multiple second convolutional layers. The process of reconstructing the rain image based on the rain image generation unit according to the rain stripe extraction image, the first fused image, and the second fused image of each of the clear rain images of each rainy day training image to obtain the reconstructed rainy day image of each of the rainy day training images and the clear image to be removed of each of the clear rain images includes:

[0070] Based on multiple first convolutional layers, feature extraction is performed on the first fused image of each of the rainy day training images to obtain the first high-level semantic information of each of the rainy day training images.

[0071] Based on the first formula, the first affinity weights of each rainy day training image are calculated according to the first high-level semantic information and the rain stripe extraction image. The first formula is as follows:

[0072]

[0073] Where, ω a1 H1 is the first affinity weight, H1 is the first high-level semantic information, and F is the first affinity weight. rain Extract images for rain stripes, where K is the number of channels contained in the preset high-level semantic features;

[0074] The parameters of each first convolutional layer are updated according to the first affinity weights of all the rainy day training images to obtain the updated first convolutional layers of each first convolutional layer.

[0075] Based on multiple updated first convolutional layers, feature extraction is performed on the first fused image of each rainy day training image to obtain the reconstructed rainy day image of each rainy day training image;

[0076] Based on multiple second convolutional layers, feature extraction is performed on the second fused image of each of the rainy day training images to obtain the second high-level semantic information of each of the rainy day training images;

[0077] Based on the second formula, the second affinity weights are calculated according to the second high-level semantic information of each of the rainy day training images and the rain stripe extraction images, resulting in the second affinity weights of each of the rainy day training images. The second formula is:

[0078]

[0079] Where, ω a2 H2 is the second affinity weight, H2 is the second high-level semantic information, and F is the second affinity weight. rain Extract images for rain stripes, where K is the number of channels contained in the preset high-level semantic features;

[0080] The parameters of each second convolutional layer are updated according to the second affinity weights of all the rainy day training images to obtain the updated second convolutional layers of each second convolutional layer.

[0081] Based on multiple updated second convolutional layers, feature extraction is performed on the second fused image of each of the rainy day training images to obtain the clear image to be removed from each of the rainy day training images.

[0082] It should be understood that the rain image generation module is used to remove additional background information, further preserve rain stripe features, and generate more realistic rainy day images.

[0083] Specifically, in the rain image generation module, a convolutional layer (i.e., the first convolutional layer or the second convolutional layer) is first used to extract semantic information (i.e., the first high-level semantic information or the second high-level semantic information) from the input image. Then, the affinity weight (i.e., the first affinity weight or the second affinity weight) between the high-level semantic information (i.e., the first high-level semantic information or the second high-level semantic information) and the rain-related features (i.e., the rain stripe extraction image) is calculated using the following formula, while simultaneously guiding the low-level features to perform different feature completions to remove additional background information and further preserve the rain stripe features. The formula is as follows:

[0084]

[0085] Where H represents high-level semantic information (i.e., the first high-level semantic information or the second high-level semantic information), F rain K represents the rain-related features (i.e., the rain stripe extraction image), and K represents the number of channels contained in the high-level semantic features.

[0086] It should be understood that the affinity weights between high-level semantic information and the rain-related features guide the low-level features to perform different feature completions, in order to remove additional background information and further preserve the rain stripe features.

[0087] In the above embodiments, the rain image generation unit obtains a reconstructed rainy day image and a clear image to be removed based on the rain stripe extraction image, the first fused image, and the reconstruction of the second fused image. This can guide low-level features to complete different features, remove additional background information, and further preserve the rain stripe features.

[0088] Optionally, as an embodiment of the present invention, step S4 includes the following process:

[0089] Based on the third equation, the first loss function is calculated for each of the original rain images and the reconstructed rainy images from the original rain images, resulting in the first loss function for each of the original rain images. The third equation is:

[0090]

[0091] in, Let R be the first loss function. rec To reconstruct rainy day images, R gt The image is the original rain image, and ||1 represents a first-order paradigm.

[0092] Based on the fourth equation, the second loss function is calculated for each of the original rain images, based on the clear rain image and the reconstructed clear image, to obtain the second loss function for each of the original rain images. The fourth equation is:

[0093]

[0094] in, For the second loss function, C rec To reconstruct a clear image, C gt For a clear rain image, ||1 represents a first paradigm;

[0095] Specifically, the loss function of the original rain image includes the first loss function and the second loss function.

[0096] It should be understood that the formula for the first loss function is as follows:

[0097]

[0098] Where R rec R represents the reconstructed rainy day image (i.e., the reconstructed rainy day image). gt This represents the training samples of rainy day images (i.e., the original rain images).

[0099] The formula for the second loss function is as follows:

[0100]

[0101] Where R rec R represents the reconstructed sharp image (i.e., the reconstructed sharp image). gt The training samples represent clear images (i.e., the clear rain images).

[0102] In the above embodiments, the first loss function is calculated based on the first loss function of the original rain image and the reconstructed rain image using the third formula, and the second loss function is calculated based on the second loss function of the clear rain image and the reconstructed clear image using the fourth formula. This solves the problems of inaccurate extraction of fog-related features and low image generation quality, and obtains clear images with richer image content and higher generation quality, thereby improving the accuracy of clear images.

[0103] Optionally, as an embodiment of the present invention, the image discrimination module includes a rain image discrimination unit and a clear image discrimination unit, and the process of S5 includes:

[0104] Based on the clear image discrimination unit, the first mean square error is calculated for each of the clear rain images and the rain stripe removal images of each of the rain training images, to obtain the first mean square error of each of the original rain images.

[0105] Based on the rain image discrimination unit, the second mean square error is calculated for each of the original rain images and the reconstructed clear images of each of the clear rain images, to obtain the second mean square error of each of the original rain images;

[0106] The mean squared error of the original rain image includes the first mean squared error and the second mean squared error.

[0107] It should be understood that the rain image discrimination module and the clear image discrimination module are used to determine the authenticity of the results generated by the rain image generation module and the rain stripe removal module, respectively.

[0108] It should be understood that the clear image training sample (i.e., the clear rain image) and the clear image generated by the rain stripe removal module (i.e., the rain stripe removed image) are input together into the clear image discrimination module.

[0109] It should be understood that the rainy day image training samples (i.e., the original rainy images) and the rainy images generated by the rainy image generation module (i.e., the reconstructed clear images) are respectively input into the rainy image discrimination module.

[0110] In the above embodiments, the clear image discrimination unit calculates the first mean square error based on the first mean square error of the clear rain image and the rain stripe removal image, and the rain image discrimination unit calculates the second mean square error based on the second mean square error of the original rain image and the reconstructed clear image. This solves the problems of inaccurate extraction of fog-related features and low image generation quality, and obtains clear images with richer image content and higher generation quality, thereby improving the accuracy of clear images.

[0111] Optionally, as an embodiment of the present invention, the process of calculating the first mean square error of each original rain image based on the clear image discrimination unit, according to the rain stripe removal images of each of the clear rain images and each of the rainy day training images, includes:

[0112] Based on the fifth equation and the clear image discrimination unit, the first mean square error is calculated for each of the original rain images based on the clear rain image and the rain stripe removal image, respectively, to obtain the first mean square error of each of the original rain images. The fifth equation is:

[0113] mse c =|D rain (f fc )-D rain (C gt )|2,

[0114] Among them, mse c f is the first mean squared error. fc To remove rain streaks from an image, C gt For a clear rain image, D rain () represents the clear image discrimination module, and ||2 represents the second normal form.

[0115] It should be understood that the input images (i.e., the clear rain image and the rain stripe removed image) are converted into feature maps of 1 / 16 of the input size using four convolutional layers and activation functions to distinguish between network-generated images and real rainy day images, and the mean squared error mse between these two feature maps is calculated. c To update the entire network parameters.

[0116] Specifically, for the clear image discrimination module, four convolutional layers and a ReLU activation function are used to transform the input image into a feature map of 1 / 16th the size of the original image. Each layer has the same kernel size of 4×4 and a stride of 2, with filters of 64, 128, 256, and 512 from the first to the last layer. Finally, a convolution with a stride of 1 and a kernel size of 3×3 is used to learn to distinguish between real and fake images. For the rain image discrimination module, a new branch is added. This new branch contains a convolutional layer and a sigmoid function to distinguish whether the image contains rain or fog features. The last convolutional layer has a kernel size of 8×8 and a stride of 1.

[0117] It should be understood that the first mean squared error is calculated as follows:

[0118] mse c =|D rain (f fc )-D rain (f cgt )|2

[0119] Among them, f fc This represents the clear image generated by the rain stripe removal module (i.e., the rain stripe removed image), f cgt (i.e. C) gt ) represents a clear image training sample (i.e., the clear rain image), D c (*) indicates the clear image discrimination module.

[0120] In the above embodiments, based on the fifth formula and the clear image discrimination unit, the first mean square error is calculated according to the first mean square error of the clear rain image and the rain stripe removed image. This can distinguish between the network-generated image and the real rainy day image, and can distinguish between real and fake images through learning.

[0121] Optionally, as an embodiment of the present invention, the process of calculating the second mean square error of each original rain image based on the rain image discrimination unit, according to each original rain image and the reconstructed clear image of each clear rain image, to obtain the second mean square error of each original rain image includes:

[0122] Based on the sixth equation and the rain image discrimination unit, the second mean square error is calculated for each of the original rain images and the reconstructed clear images of each of the original rain images, to obtain the second mean square error of each of the original rain images. The sixth equation is:

[0123]

[0124] Among them, mse r C is the second mean square deviation. rec To reconstruct a clear image, R gt For the original rain image, D r () represents the rain image discrimination module. Let i be the i-th second normal form.

[0125] It should be understood that the conditions in step S6 are specifically as follows: the first loss function and the second loss function of the original rain image are both less than or equal to the preset loss function, and the first mean square error and the second mean square error of the original rain image are both less than or equal to the preset mean square error.

[0126] Specifically, the rain image discrimination module uses four convolutional layers to convert the input image (i.e., the original rain image and the reconstructed clear image) into two feature maps of 1 / 16 of the input size. Then, it uses three parallel convolutional layers plus a sigmoid activation function to discriminate the realism of the image across multiple dimensions. A total of six feature maps are obtained, and the mean squared error (MSE) among these six feature maps is calculated. r To update the entire network parameters.

[0127] It should be understood that the image discrimination module is similar to the clear image discrimination module, but uses multiple parallel convolutional layers on the last layer to discriminate the authenticity of the image in multiple dimensions.

[0128] It should be understood that the formula for the second mean squared error is as follows:

[0129]

[0130] Among them, f fr (i.e. C) rec ) represents the rain image generated by the rain image generation module (i.e., the reconstructed clear image), f rgt (i.e. R) gt ) represents the training samples of rainy day images (i.e., the original rain images), D r (*) indicates the rain image discrimination module.

[0131] It should be understood that i can be the number of convolutional layers and the number of Sigmoid activation functions.

[0132] In the above embodiments, based on the sixth formula and the rain image discrimination unit, the second mean square error is calculated according to the second mean square error of the original rain image and the reconstructed clear image. This can distinguish the authenticity of multiple dimensions of the image, obtain a clear image with richer image content and higher generation quality, and improve the accuracy of the clear image.

[0133] Optionally, as another embodiment of the present invention, the present invention is used to adaptively remove rain streaks from an image. The method includes: performing data augmentation on the original rain image and a clear image to obtain unpaired training samples, and inputting the training samples into an unsupervised rain removal model for training; firstly, inputting the rain image training samples into a rain streak extraction module and a rain streak removal module respectively to obtain a rain streak image and a clear image; fusing the obtained rain streak image and clear image into a rain image generation module to obtain a reconstructed rainy day image; then, fusing the clear image training samples with the rain streak image into the rain image generation module to generate a rainy day image corresponding to the clear image samples; inputting the obtained rainy day image into the rain streak removal module to generate a reconstructed clear image; and constraining the generated image through a rain image discrimination module and a clear image discrimination module.

[0134] Optionally, as another embodiment of the present invention, the rain removal model of the present invention includes two conversion units: a rain image to clear image conversion unit and a clear image to rain image conversion unit, which includes five modules: a rain stripe extraction module, a rain stripe removal module, a rain image generation module, a rain image discrimination module, and a clear image discrimination module;

[0135] For the rain image to clear image conversion unit, the rainy day image training samples are input into the rain stripe extraction module and the rain stripe removal module, respectively, to obtain the outputs of the rain stripe extraction module and the rain stripe removal module. The rain stripe extraction module is used to predict the shape and position of rain stripes in the image, and the rain stripe removal module is used to predict the clear image corresponding to the rainy day image. The outputs of the rain stripe extraction module and the rain stripe removal module are then fused and input into the rain image generation module to obtain the output of the rain image generation module. The rain image generation module is used to synthesize the rainy day image corresponding to the clear image.

[0136] For the clear image to rain image conversion unit, the clear image training samples and the output of the rain stripe extraction module are fused and input into the rain image generation module, and then the result is input into the rain stripe removal module to obtain the reconstructed clear image;

[0137] The rain image discrimination module and the clear image discrimination module are used to determine the authenticity of the results generated by the rain image generation module and the rain stripe removal module, respectively.

[0138] Optionally, as another embodiment of the present invention, the present invention fuses clear image training samples with rain stripe images and inputs them into a rain image generation module to generate a rainy day image corresponding to the clear image samples; and inputs the obtained rainy day image into a rain stripe removal module to generate a reconstructed clear image; and constrains the generated image through a rain image discrimination module and a clear image discrimination module.

[0139] Figure 2 This is a block diagram of an image deraining device provided in an embodiment of the present invention.

[0140] Alternatively, as another embodiment of the present invention, such as Figure 2 As shown, an image deraining device includes:

[0141] The data augmentation module is used to import multiple original rain images and multiple clear rain images that correspond one-to-one with the multiple original rain images, and to perform data augmentation processing on each of the original rain images and each of the clear rain images to obtain rain training images of each of the original rain images and rain clear training images of each of the clear rain images.

[0142] A model building module is used to build an initial rain removal model; wherein, the initial rain removal model includes an image processing module and an image discrimination module;

[0143] The processing module is used to perform image processing based on the image processing module, according to each of the rainy day training images and the rainy day clear training images of each of the clear rain images, to obtain the reconstructed rainy day images, rain stripe removal images, and reconstructed clear images of each of the clear rain images.

[0144] The loss function analysis module is used to perform loss function analysis on each of the original rain images, each of the clear rain images, each of the reconstructed rain images from the rain training images, and each of the reconstructed clear images from the clear rain images, to obtain the loss function of each of the original rain images.

[0145] The mean square error calculation module is used to calculate the mean square error of each original rain image based on the image discrimination module, according to each original rain image, each clear rain image, each rain stripe removed image of each rain training image, and each reconstructed clear image of each clear rain image.

[0146] The judgment module is used to determine whether all the original rain images meet the following conditions: the loss function of the original rain image is less than or equal to a preset loss function, and the root mean square error of the original rain image is less than or equal to a preset root mean square error. If not, the image processing module updates its parameters according to the loss function and root mean square error of all the original rain images, and returns the updated parameters to the processing module until all the original rain images meet the conditions. If yes, the image processing module after the last parameter update is used as the target rain removal model.

[0147] The rain removal result acquisition module is used to import the rain image to be processed, and perform image rain removal processing on the rain image to be processed through the target rain removal model to obtain the image rain removal result.

[0148] Optionally, another embodiment of the present invention provides an image deraining system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the image deraining method described above. This system can be a computer or similar system.

[0149] Optionally, another embodiment of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the image deraining method as described above.

[0150] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0151] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described apparatus and unit can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0152] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0153] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.

[0154] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0155] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0156] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An image deraining method, characterized in that, Includes the following steps: S1: Import multiple original rain images and multiple clear rain images that correspond one-to-one with the multiple original rain images, and perform data augmentation processing on each of the original rain images and each of the clear rain images to obtain rain training images of each of the original rain images and rain clear training images of each of the clear rain images. S2: Construct an initial deraining model; wherein, the initial deraining model includes an image processing module and an image discrimination module; S3: Based on the image processing module, image processing is performed on each of the rainy day training images and the rainy day clear training images of each of the clear rain images to obtain the reconstructed rainy day images, rain stripe removal images, and reconstructed clear images of each of the clear rain images; S4: Perform loss function analysis based on each of the original rain images, each of the clear rain images, each of the reconstructed rain images from the rain training images, and each of the reconstructed clear images from the clear rain images to obtain the loss function for each of the original rain images. S5: Based on the image discrimination module, the mean square error of each original rain image is calculated according to each original rain image, each clear rain image, each rain stripe removed image of each rain training image, and each reconstructed clear image of each clear rain image. S6: Determine whether all the original rain images meet the condition that the loss function of the original rain image is less than or equal to a preset loss function and the root mean square error of the original rain image is less than or equal to a preset root mean square error; if not, update the parameters of the image processing module according to the loss function and root mean square error of all the original rain images respectively, and return to step S3 after parameter update to repeat until all the original rain images meet the condition; if yes, use the image processing module after the last parameter update as the target rain removal model and execute step S7; S7: Import the rain image to be processed, and perform image deraining processing on the rain image to be processed through the target deraining model to obtain the image deraining result; The image processing module includes a rain stripe extraction unit, a rain stripe removal unit, and a rain image generation unit. Step S3 includes the following process: Based on the rain stripe extraction unit, rain stripes are extracted from each of the rainy day training images to obtain rain stripe extraction images of each of the rainy day training images. Based on the rain stripe removal unit, rain stripes are removed from each of the rainy day training images to obtain rain stripe removed images of each of the rainy day training images. Image fusion is performed on the rain stripe extraction image and the rain stripe removal image of each of the rainy day training images to obtain the first fused image of each of the rainy day training images; Image fusion is performed on the rain stripe extraction images of each of the rainy training images and the rainy clear training images of each of the clear rain images to obtain the second fused image of each of the clear rain images; Based on the rain image generation unit, the rain stripe extraction image, the first fused image, and the second fused image of each of the rainy day training images are reconstructed to obtain the reconstructed rainy day images of each of the rainy day training images and the clear images to be removed from each of the clear rainy images. Based on the rain stripe removal unit, rain stripes are removed from the clear images to be removed in each of the clear rain images to obtain the reconstructed clear images of each of the rainy day training images.

2. The image deraining method according to claim 1, characterized in that, The rain stripe extraction unit includes a convolutional kernel and a sigmoid activation function layer. The process of extracting rain stripes from each of the rainy day training images based on the rain stripe extraction unit to obtain rain stripe extraction images for each of the rainy day training images includes: Based on the convolution kernel, feature extraction is performed on each of the rainy day training images to obtain the feature-extracted images of each of the rainy day training images. Based on the sigmoid activation function layer, normalization processing is performed on each of the feature-extracted images to obtain rain stripe extraction images of each of the rainy day training images.

3. The image deraining method according to claim 1, characterized in that, The rain stripe removal unit includes an encoding block, a residual block, and a decoding block. The process of removing rain stripes from each of the rainy day training images based on the rain stripe removal unit to obtain rain stripe-removed images for each of the rainy day training images includes: Based on the coding block, each of the rainy day training images is encoded to obtain the encoded images of each of the rainy day training images; Based on the residual block, feature extraction is performed on each of the encoded images to obtain the feature-extracted encoded images of each of the rainy day training images. The encoded images and feature-extracted encoded images of each of the rainy day training images are then fused to obtain the fused images of each of the rainy day training images. Based on the decoding block, each of the fused images is decoded to obtain rain stripe removal images of each of the rainy day training images.

4. The image deraining method according to claim 1, characterized in that, The rain image generation unit includes multiple first convolutional layers and multiple second convolutional layers. The process of reconstructing the rain image based on the rain image generation unit, using the rain stripe extraction image, the first fused image, and the second fused image of each of the clear rain images from each rainy day training image, to obtain the reconstructed rainy day image of each of the rainy day training images and the clear image to be removed from each of the clear rain images includes: Based on multiple first convolutional layers, feature extraction is performed on the first fused image of each of the rainy day training images to obtain the first high-level semantic information of each of the rainy day training images. Based on the first formula, the first affinity weights of each rainy day training image are calculated according to the first high-level semantic information and the rain stripe extraction image. The first formula is as follows: ， in, The first affinity weight, This is the first level of semantic information. Extract images for rain streaks. The number of channels contained within the preset advanced semantic features; The parameters of each first convolutional layer are updated according to the first affinity weights of all the rainy day training images to obtain the updated first convolutional layers of each first convolutional layer. Based on multiple updated first convolutional layers, feature extraction is performed on the first fused image of each rainy day training image to obtain the reconstructed rainy day image of each rainy day training image; Based on multiple second convolutional layers, feature extraction is performed on the second fused image of each of the rainy day training images to obtain the second high-level semantic information of each of the rainy day training images; Based on the second formula, the second affinity weights are calculated according to the second high-level semantic information of each of the rainy day training images and the rain stripe extraction images, resulting in the second affinity weights of each of the rainy day training images. The second formula is: ， in, The second affinity weight, This is the second level of semantic information. Extract images for rain streaks. The number of channels contained within the preset advanced semantic features; The parameters of each second convolutional layer are updated according to the second affinity weights of all the rainy day training images to obtain the updated second convolutional layers of each second convolutional layer. Based on multiple updated second convolutional layers, feature extraction is performed on the second fused image of each of the rainy day training images to obtain the clear image to be removed from each of the rainy day training images.

5. The image deraining method according to claim 1, characterized in that, The process of step S4 includes: Based on the third equation, the first loss function is calculated for each of the original rain images and the reconstructed rainy images from the original rain images, resulting in the first loss function for each of the original rain images. The third equation is: ， in, For the first loss function, To reconstruct the rainy day image, The original rain image, It is a first normal form; Based on the fourth equation, the second loss function is calculated for each of the original rain images, based on the clear rain image and the reconstructed clear image, to obtain the second loss function for each of the original rain images. The fourth equation is: ， in, For the second loss function, To reconstruct a clear image, For clear rain images, It is a first normal form; Specifically, the loss function of the original rain image includes the first loss function and the second loss function.

6. The image deraining method according to claim 1, characterized in that, The image discrimination module includes a rain image discrimination unit and a clear image discrimination unit. The process in S5 includes: Based on the clear image discrimination unit, the first mean square error is calculated for each of the clear rain images and the rain stripe removal images of each of the rain training images, to obtain the first mean square error of each of the original rain images. Based on the rain image discrimination unit, the second mean square error is calculated for each of the original rain images and the reconstructed clear images of each of the clear rain images, to obtain the second mean square error of each of the original rain images; The mean squared error of the original rain image includes the first mean squared error and the second mean squared error.

7. The image deraining method according to claim 6, characterized in that, The process of calculating the first mean square error of each original rain image based on the clear image discrimination unit, according to the rain stripe removal images of each clear rain image and each rainy day training image, includes: Based on the fifth equation and the clear image discrimination unit, the first mean square error is calculated for each of the original rain images based on the clear rain image and the rain stripe removal image, respectively, to obtain the first mean square error of each of the original rain images. The fifth equation is: ， in, The first mean square deviation, Remove rain streaks from the image. For clear rain images, For the clear image discrimination module, It is in second normal form.

8. The image deraining method according to claim 6, characterized in that, The process by which the rain image discrimination unit calculates the second mean square error of each original rain image and the reconstructed clear image of each clear rain image to obtain the second mean square error of each original rain image includes: Based on the sixth equation and the rain image discrimination unit, the second mean square error is calculated for each of the original rain images and the reconstructed clear images of each of the original rain images, to obtain the second mean square error of each of the original rain images. The sixth equation is: ， in, The second mean square deviation, To reconstruct a clear image, The original rain image, For rain image discrimination module, For the first A second normal form.

9. An image deraining device, characterized in that, include: The data augmentation module is used to import multiple original rain images and multiple clear rain images that correspond one-to-one with the multiple original rain images, and to perform data augmentation processing on each of the original rain images and each of the clear rain images to obtain rain training images of each of the original rain images and rain clear training images of each of the clear rain images. A model building module is used to build an initial rain removal model; wherein, the initial rain removal model includes an image processing module and an image discrimination module; The processing module is used to perform image processing based on the image processing module, according to each of the rainy day training images and each of the clear rainy day training images, to obtain reconstructed rainy day images, rain stripe removed images, and reconstructed clear images of each of the clear rainy day training images; The loss function analysis module is used to perform loss function analysis on each of the original rain images, each of the clear rain images, each of the reconstructed rain images from the rain training images, and each of the reconstructed clear images from the clear rain images, to obtain the loss function of each of the original rain images. The mean square error calculation module is used to calculate the mean square error of each original rain image based on the image discrimination module, according to each original rain image, each clear rain image, each rain stripe removed image of each rain training image, and each reconstructed clear image of each clear rain image. The judgment module is used to determine whether all the original rain images meet the following conditions: the loss function of the original rain image is less than or equal to a preset loss function, and the root mean square error of the original rain image is less than or equal to a preset root mean square error. If not, the image processing module updates its parameters according to the loss function and root mean square error of all the original rain images, and returns the updated parameters to the processing module until all the original rain images meet the conditions. If yes, the image processing module after the last parameter update is used as the target rain removal model. The rain removal result acquisition module is used to import the rain image to be processed, perform image rain removal processing on the rain image to be processed through the target rain removal model, and obtain the image rain removal result. The image processing module includes a rain stripe extraction unit, a rain stripe removal unit, and a rain image generation unit. Specifically, the processing module is used for: Based on the rain stripe extraction unit, rain stripes are extracted from each of the rainy day training images to obtain rain stripe extraction images of each of the rainy day training images. Based on the rain stripe removal unit, rain stripes are removed from each of the rainy day training images to obtain rain stripe removed images of each of the rainy day training images. Image fusion is performed on the rain stripe extraction image and the rain stripe removal image of each of the rainy day training images to obtain the first fused image of each of the rainy day training images; Image fusion is performed on the rain stripe extraction images of each of the rainy training images and the rainy clear training images of each of the clear rain images to obtain the second fused image of each of the clear rain images; Based on the rain image generation unit, the rain stripe extraction image, the first fused image, and the second fused image of each of the rainy day training images are reconstructed to obtain the reconstructed rainy day images of each of the rainy day training images and the clear images to be removed from each of the clear rainy images. Based on the rain stripe removal unit, rain stripes are removed from the clear images to be removed in each of the clear rain images to obtain the reconstructed clear images of each of the rainy day training images.

Images