Image rain removal processing method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to an image rain removal processing method and device, computer equipment, a readable storage medium and a program product. The method comprises the steps of establishing an unsupervised rain removal framework, performing reconstruction processing on an original rain image through an end-to-end image generation network to obtain a clear image, performing feature extraction on the original rain image through a rain image generation network to obtain raindrop information, obtaining a reconstructed clear image according to the raindrop information and the original rain image, and obtaining the reconstructed clear image according to the reconstructed clear image. The method comprises the steps of obtaining a clear image and raindrop information, obtaining a reconstructed rain image according to the clear image and the raindrop information, establishing a loss constraint condition for an unsupervised rain removal frame based on the original rain image, the clear image, the reconstructed clear image and the reconstructed rain image, and optimizing the unsupervised rain removal frame according to the loss constraint condition. And carrying out image rain removal processing on the original rain image through the optimized unsupervised rain removal framework. By adopting the method, the capability of removing local tiny rainwater can be improved, so that the rain removal of the image is accurately realized.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image rain removal processing method, device, computer equipment, readable storage medium and program product. Background Art

[0002] In recent years, with the intensification of environmental pollution and severe weather conditions, the quality of images captured by optical sensors has frequently been challenged, especially in rainy weather, where images often suffer from problems such as color distortion, blur and low contrast.

[0003] In traditional methods, the problem of image deraining is mainly solved by using deep learning methods. The image deraining algorithm based on supervised learning is trained on a large amount of paired (rainy and rain-free) image data to minimize the error between the predicted image and the actual rain-free image to optimize the model parameters.

[0004] However, in practical applications, it is often very difficult to obtain these paired image data, resulting in low accuracy of image deraining. Summary of the invention

[0005] Based on this, it is necessary to provide an image deraining processing method, apparatus, computer equipment, readable storage medium and program product that can accurately implement image deraining in response to the above technical problems.

[0006] In a first aspect, the present application provides an image deraining processing method, comprising:

[0007] Establish an unsupervised rain removal framework; the unsupervised rain removal framework includes an end-to-end image generation network and a rain image generation network;

[0008] The original rainy image is reconstructed through an end-to-end image generation network to obtain a clear image, and the rain image generation network is used to extract features of the original rainy image to obtain raindrop information; the raindrop information is the product of the position information and density information of the rain in the original rainy image;

[0009] Obtain a reconstructed clear image based on the raindrop information and the original rainy image, and obtain a reconstructed rainy image based on the clear image and the raindrop information;

[0010] Based on the original rainy image, clear image, reconstructed clear image and reconstructed rainy image, loss constraints for the unsupervised rain removal framework are established; the loss constraints include global contrast loss constraints and local contrast loss constraints;

[0011] The unsupervised rain removal framework is optimized according to the loss constraint conditions, and the original rainy image is derained by the optimized unsupervised rain removal framework.

[0012] In one of the embodiments, the end-to-end image generation network includes a generator and a discriminator, the generator is used to reconstruct the original rainy image to obtain a clear image, and the discriminator is used to distinguish between the real image and the generated image; the rain image generation network includes a first subnet and a second subnet, the first subnet is used to obtain location information, and the second subnet is used to obtain density information.

[0013] In one embodiment, the steps of obtaining a reconstructed clear image based on raindrop information and an original rainy image, and obtaining a reconstructed rainy image based on the clear image and the raindrop information include:

[0014] Subtract the original rainy image and raindrop information to obtain a reconstructed clear image;

[0015] The clear image and raindrop information are added to obtain a reconstructed rainy image.

[0016] In one embodiment, the step of establishing a loss constraint condition for an unsupervised rain removal framework based on the original rainy image, the clear image, the reconstructed clear image, and the reconstructed rainy image comprises:

[0017] Establishing a first image content constraint based on a difference measure between a clear image and a reconstructed clear image, and a sum of difference measures between the clear image and the original rainy image and the reconstructed rainy image respectively;

[0018] Establishing a second image content constraint based on a difference measure between the reconstructed rainy image and the original rainy image, and a sum of difference measures between the reconstructed rainy image and the clear image and the reconstructed clear image, respectively;

[0019] Establishing a global contrast loss constraint based on the first image content constraint and the second image content constraint;

[0020] Establishing a first local feature constraint based on a difference measure between a patch block of a clear image and a patch block of a reconstructed clear image, and a sum of two difference measures between a patch block of a clear image and a patch block of an original rainy image and a patch block of a reconstructed rainy image;

[0021] A second local feature constraint is established based on a difference measure between a patch block of the reconstructed rainy image and a patch block of the original rainy image, and a sum of two difference measures between a patch block of the reconstructed rainy image and a patch block of a clear image and a patch block of the reconstructed clear image;

[0022] A local contrast loss constraint is established according to the first local feature constraint and the second local feature constraint.

[0023] In one embodiment, the loss constraint condition further includes an adversarial loss constraint, a content consistency loss constraint, and a clarity consistency loss constraint; and the step of establishing the loss constraint condition for the unsupervised deraining framework based on the original rainy image, the clear image, the reconstructed clear image, and the reconstructed rainy image includes:

[0024] Establish adversarial loss constraints based on the loss function of the discriminator and the loss function of the generator;

[0025] According to the pixel distance between the reconstructed rainy image and the original rainy image, a content consistency loss constraint is established;

[0026] According to the pixel distance between the clear image and the reconstructed clear image, a clear consistency loss constraint is established.

[0027] In one embodiment, the step of optimizing the unsupervised rain removal framework according to the loss constraint condition includes:

[0028] Establish the overall network loss function based on all loss constraints;

[0029] The unsupervised rain removal framework is optimized with the goal of minimizing the function value of the overall network loss function.

[0030] In a second aspect, the present application also provides an image rain removal processing device, comprising:

[0031] A framework building module is used to build an unsupervised rain removal framework; the unsupervised rain removal framework includes an end-to-end image generation network and a rain image generation network;

[0032] The image reconstruction module is used to reconstruct the original rainy image through an end-to-end image generation network to obtain a clear image, and extract features of the original rainy image through a rain image generation network to obtain raindrop information; the raindrop information is the product of the position information and density information of the rain in the original rainy image;

[0033] An image acquisition module, used to acquire a reconstructed clear image based on raindrop information and an original rainy image, and to acquire a reconstructed rainy image based on the clear image and raindrop information;

[0034] A constraint establishment module is used to establish loss constraints for the unsupervised rain removal framework based on the original rainy image, the clear image, the reconstructed clear image and the reconstructed rainy image; the loss constraints include global contrast loss constraints and local contrast loss constraints;

[0035] The rain removal processing module is used to optimize the unsupervised rain removal framework according to the loss constraint condition, and perform image rain removal processing on the original rainy image through the optimized unsupervised rain removal framework.

[0036] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method steps of any one of the first aspects are implemented.

[0037] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements any one of the method steps in the first aspect when the computer program is executed by a processor.

[0038] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which implements any one of the method steps in the first aspect when executed by a processor.

[0039] The above-mentioned image deraining processing method, device, computer equipment, readable storage medium and program product can accurately reconstruct the original rainy image through the end-to-end image generation network to generate a clear image, and effectively extract the characteristics of the original rainy image through the rain image generation network, accurately capture the position information and density information of raindrops, and then accurately obtain the reconstructed clear image and the reconstructed rainy image. Based on the original rainy image, the clear image, the reconstructed clear image and the reconstructed rainy image, loss constraints including global contrast loss constraints and local contrast loss constraints are established, and unsupervised bidirectional global and local contrast reconstruction constraints are realized between the rain image generation network and the end-to-end image generation network, thereby strengthening the network's constraint ability on the reconstructed image at both the global and local levels, and can improve the network's ability to remove local fine rain, making the deraining effect more delicate and realistic. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 A diagram showing an application environment of an image rain removal processing method in one embodiment;

[0042] Figure 2 Schematic diagram of a process flow of an image rain removal method in one embodiment;

[0043] Figure 3 is a schematic diagram of an overall network structure in one embodiment;

[0044] Figure 4 is a structural block diagram of an image rain removal processing device in one embodiment;

[0045] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0047] In recent years, with the intensification of environmental pollution and adverse weather conditions, optical sensors often face quality problems such as color distortion, blur and low contrast when capturing images, especially in rainy weather. These image quality issues not only seriously affect the overall visual effect of the image, but also bring great challenges to subsequent image analysis tasks. In practical application scenarios such as autonomous driving, drone inspection, security monitoring, etc., clear and high-quality images are the key factors to ensure the accuracy and efficiency of high-level visual tasks such as target detection, target tracking and image segmentation.

[0048] Among the deep learning-based rain removal methods, convolutional neural networks (CNNs) are widely used and have shown powerful rain removal capabilities. These methods are mainly divided into supervised rain removal and unsupervised rain removal. Traditional supervised image rain removal algorithms rely on a large number of paired rainy and rain-free images for training, but in practical applications, it is often challenging to obtain these paired data. Among the unsupervised rain removal methods, the main method is based on the Cycle Generative Adversarial Network (CycleGAN) framework. By exploring the intrinsic statistical characteristics and common characteristics of unpaired data, effective rain streaks removal and image restoration are achieved. However, during the training process of CycleGAN, the input of the generator includes two different data distributions of the dataset and the generated data, which may cause confusion in the generator learning, thereby affecting the rain removal effect and the stability of the model.

[0049] Based on this, in order to solve the problem of learning confusion of the generator in most current unsupervised deraining methods based on the CycleGAN framework, the input of the generator comes from data composed of different distributions during training, which leads to a decrease in the network's ability to remove rain, and ultimately leads to deviations in the color and texture details of the generated image, an embodiment of the present application provides an image region processing method, which establishes an unsupervised bidirectional global-local contrast reconstruction constraint between a rain image generation network and an end-to-end image generation network through a rain model, which can strengthen the network's constraints on the overall content from a global level, and improve the network's ability to remove local fine rain from the local deep feature space, and strengthen the network's constraint ability on the reconstructed image from both global and local aspects, thereby achieving high-quality image deraining results.

[0050] The image deraining processing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Among them, the server 104 is used to store the original rainy image. Among them, the terminal 102 is used to establish an unsupervised deraining framework, reconstruct the original rainy image through the end-to-end image generation network to obtain a clear image, and extract features of the original rainy image through the rain image generation network to obtain raindrop information, obtain a reconstructed clear image based on the raindrop information and the original rainy image, and obtain a reconstructed rainy image based on the clear image and the raindrop information. Based on the original rainy image, the clear image, the reconstructed clear image and the reconstructed rainy image, a loss constraint condition for the unsupervised deraining framework is established, the unsupervised deraining framework is optimized according to the loss constraint condition, and the original rainy image is derained by the optimized unsupervised deraining framework. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0051] In an exemplary embodiment, Figure 2 As shown, a method for image deraining is provided, and the method is applied to Figure 1 The terminal 102 in the example is used as an example to illustrate, including the following steps 202 to 210. Among them:

[0052] S202: Establish an unsupervised rain removal framework; the unsupervised rain removal framework includes an end-to-end image generation network and a rain image generation network.

[0053] Optionally, an unsupervised rain removal framework based on CycleGAN is established, including an end-to-end image generation network and a rain image generation network. The unsupervised rain removal framework does not require manual annotation of paired rainy images and rain-free images, which greatly reduces the cost and difficulty of data preparation. In practical applications, it is very difficult to obtain a large number of paired high-quality rainy and rain-free images, and the unsupervised rain removal framework can use a large number of unlabeled rainy images for training. Among them, the end-to-end image generation network is used to generate a clear image after rain removal based on the original rainy image, and the rain image generation network is used to extract rain features and generate rainy images that are closer to real rain patterns, thereby improving the rain removal effect of the end-to-end image generation network.

[0054] S204: reconstructing the original rainy image through an end-to-end image generation network to obtain a clear image, and extracting features of the original rainy image through a rain image generation network to obtain raindrop information; the raindrop information is the product of the position information and density information of the rain in the original rainy image.

[0055] Optionally, the end-to-end image generation network is a deep learning model that directly learns the mapping relationship from the input image (i.e., the original rainy image) to the output image (the target is a clear image) without manually designing intermediate processing steps. The end-to-end image generation network is trained to identify and remove the rain effect in the image, so as to reconstruct an image that is as clear as possible and not affected by rain. The rain image generation network is used to extract rain features from rainy images, including but not limited to the shape, size, direction, transparency, etc. of raindrops, and obtain the position information and density information of rain through these features, and then obtain raindrop information. Among them, the position information refers to the specific position of the raindrops in the image, and the density information reflects the density or intensity of the raindrops. It should be noted that the product of the position information and density information of the rain represents the degree of influence of the rain at each position. The distribution of rain in the image and its actual impact on the image quality are learned by the image generation network, so that the rain effect can be more effectively removed or reduced in the subsequent image processing or enhancement steps.

[0056] S206: Obtain a reconstructed clear image according to the raindrop information and the original rainy image, and obtain a reconstructed rainy image according to the clear image and the raindrop information.

[0057] Optionally, based on the extracted raindrop information and the original rainy image, an image reconstruction algorithm or a deep learning model is used to remove the raindrop effect, thereby generating an image that is as clear as possible and not affected by rain, so as to simulate the scene when rain is not captured. On the basis of the reconstructed clear image that has been obtained, the extracted raindrop information is reintroduced, and a rainy image is reconstructed by superimposing or synthesizing the raindrop information on the clear image.

[0058] S208: Based on the original rainy image, the clear image, the reconstructed clear image and the reconstructed rainy image, a loss constraint condition for the unsupervised rain removal framework is established; the loss constraint condition includes a global contrast loss constraint and a local contrast loss constraint.

[0059] Optionally, in the unsupervised rain removal framework, the goal is to optimize the model through a series of constraints so that the reconstructed clear image is as close to the real clear image as possible, and the reconstructed rainy image can also reasonably restore the features of the original rainy image. The loss constraint here plays the role of measuring the difference between the model output and the expected output. By minimizing these losses, the model can learn an effective rain removal strategy. Among them, the global contrast loss mainly focuses on the difference at the entire image level. It measures the difference in global attributes such as overall feature distribution, color, and brightness between the reconstructed clear image and the real clear image, and between the reconstructed rainy image and the original rainy image. By establishing the loss constraint between the reconstructed clear image and the clear image, the clarity of the image is ensured, and the reconstructed image can be as similar as possible to the original image in terms of overall visual effect. The local contrast loss focuses on the difference in the local area of ​​the image. Since rain may affect local details in the image, such as the edge and texture of the object, the local contrast loss can better capture the changes in these local features. By establishing a loss constraint between the reconstructed rainy image and the original rainy image, the image content information is ensured to be consistent, which enables the model to pay more attention to the details of the image. The model can learn a more sophisticated deraining method, thereby better restoring the local texture and details of the image.

[0060] S210: Optimizing the unsupervised rain removal framework according to the loss constraint condition, and performing image deraining processing on the original rainy image through the optimized unsupervised rain removal framework.

[0061] Optionally, the loss constraint here plays a role in measuring the difference between the output result of the unsupervised rain removal framework and the expected result. The loss constraint includes global contrast loss constraint and local contrast loss constraint, which focus on the difference between the reconstructed image and the original image from the overall and local levels respectively. These constraints can reflect the gap between the reconstructed image and the real clear image from different angles. After the current loss value is calculated through the loss constraint, if the loss value is large, it means that the output of the current framework does not meet expectations, and these parameters need to be adjusted so that the parameters change in the direction that can reduce the loss value. Finally, after processing within the framework, a real clear image that is as close as possible to the same scene without raindrop interference will be output. In terms of visual effect, the raindrops in the image are effectively removed, and the details, color, brightness, etc. of the object are well restored, which can more clearly show the true appearance of the original scene.

[0062] In the above-mentioned image deraining processing method, the original rainy image can be accurately reconstructed through the end-to-end image generation network to generate a clear image. The rain image generation network can effectively extract the features of the original rainy image, accurately capture the position information and density information of raindrops, and then accurately obtain the reconstructed clear image and the reconstructed rainy image. Based on the original rainy image, clear image, reconstructed clear image and reconstructed rainy image, loss constraints including global contrast loss constraints and local contrast loss constraints are established, and unsupervised bidirectional global-local contrast reconstruction constraints are realized between the rain image generation network and the end-to-end image generation network, thereby strengthening the network's constraint ability on the reconstructed image at both the global and local levels, and can improve the network's ability to remove local fine rain, making the deraining effect more delicate and realistic.

[0063] In an exemplary embodiment, an end-to-end image generation network includes a generator and a discriminator, the generator is used to reconstruct an original rainy image to obtain a clear image, and the discriminator is used to distinguish between a real image and a generated image; the rain image generation network includes a first subnet and a second subnet, the first subnet is used to obtain location information, and the second subnet is used to obtain density information.

[0064] Optionally, the end-to-end image generation network aims to generate the desired clear image directly from the input image without manual intervention in the intermediate process. It mainly consists of two parts: the generator and the discriminator. The overall function is to reconstruct the original rainy image and convert it into a clear image. Among them, the generator includes two parts: feature extraction and downsampling (i.e., the encoding layer) and image reconstruction and upsampling (i.e., the decoding layer). The original rainy image O first enters the encoding layer, and performs feature extraction and downsampling operations through multiple residual blocks. In the four-layer feature extraction process of the encoding layer, each layer will gradually extract more abstract and representative features of the image. For example, the first layer may capture some basic lines and edge information in the image. As the number of layers increases, the subsequent layers will extract more complex textures, shapes and other comprehensive features. At the same time, the size of the image is gradually reduced (downsampling), which helps to condense the image information and facilitate subsequent processing. After the feature extraction of the encoding layer, it enters the recovery module of the decoding layer. The decoding layer also has three layers. Through the network structure of deconvolution reconstruction and upsampling, the previously extracted features are gradually restored to the original image resolution, and a reconstructed clear image B is obtained.

[0065] Furthermore, the main function of the discriminator is to distinguish between the clear image B generated by the generator and the real unpaired clear image in the data set, and to judge whether the image generated by the generator is clear enough by learning the difference features between the two. If the discriminator can easily distinguish the reconstructed image B from the unpaired clear image, it means that the image generated by the generator may still have some features different from the clear image, that is, it is not clear enough. Then the generator needs to further adjust its parameters to generate a clearer image, so that it is difficult for the discriminator to distinguish between the clear image B and the real clear image, thereby ensuring that the image of the image generator is clear.

[0066] Furthermore, the goal of the rain image generation network is to obtain detailed information about rain. It contains two subnets, namely the first subnet and the second subnet, which are used to obtain different characteristic information of rain respectively, and finally synthesize accurate raindrop information for subsequent possible image-related rain processing operations. Among them, the first subnet also adopts a structure similar to the encoding layer and decoding layer of the end-to-end image generation network, extracts rain features through a series of residual convolution blocks, and outputs the position information R1 of raindrops after the four-layer feature extraction of the encoding layer and the three-layer recovery operation of the decoding layer. This position information can clarify the approximate distribution of raindrops in the image, such as which areas have denser raindrops and which areas have relatively fewer raindrops.

[0067] Furthermore, the structure and operation flow of the second subnet are similar to those of the first subnet, and it also undergoes four-layer feature extraction in the encoding layer and three-layer recovery operations in the decoding layer, and finally outputs the density information R2 of the raindrops, which can represent the density of raindrops in different areas of the image, such as the number of raindrops per unit area in a certain area. Finally, the position information R1 obtained by the first subnet and the density information R2 obtained by the second subnet are multiplied to obtain the accurate raindrop information R.

[0068] In this embodiment, the generator of the end-to-end image generation network can reconstruct the original rainy image and convert it into a clear image. The discriminator can further ensure that the image generated by the generator is clear. The first subnet and the second subnet of the rain image generation network can comprehensively consider the position and density of raindrops to obtain a more comprehensive and accurate description of the situation of raindrops in the image, thereby improving the accuracy of subsequent image deraining processing.

[0069] In an exemplary embodiment, a reconstructed clear image is obtained based on raindrop information and an original rainy image, and the step of obtaining the reconstructed rainy image based on the clear image and the raindrop information includes: subtracting the original rainy image and the raindrop information to obtain the reconstructed clear image; and adding the clear image and the raindrop information to obtain the reconstructed rainy image.

[0070] Optionally, raindrops appear as an interference factor on the image, which has specific characteristic information (such as position, density, etc.). If the raindrop information can be accurately obtained, the influence of raindrops can be removed from the original rainy image through corresponding mathematical operations, thereby reconstructing a clear image. Subtract the raindrop information R from the original rainy image O. The subtraction operation here is based on the pixel level, that is, for each pixel in the image, the value of the pixel in the original rainy image is subtracted from the value of the corresponding pixel in the raindrop information. Through such subtraction processing, it is equivalent to removing the influence of raindrops on the image, thereby reconstructing a clear image B'.

[0071] Similarly, the addition operation of the clear image B and the raindrop information R is also performed at the pixel level. For each pixel in the image, the value of the pixel in the clear image is added to the value of the corresponding pixel in the raindrop information. Through such addition processing, it is equivalent to adding the influence of raindrops back to the clear image, thereby reconstructing a rainy image O'.

[0072] In this embodiment, the reconstructed clear image is obtained by subtracting the raindrop information from the original rainy image, so that the reconstructed clear image is closer to the scene in the actual rainless state. The reconstructed rainy image is obtained by adding the clear image and the raindrop information, which can more realistically simulate the distribution of raindrops in the real scene and the impact on the image, so that the unsupervised rain removal framework can better learn the relationship between raindrops and clear images, thereby improving the rain removal effect.

[0073] In an exemplary embodiment, the step of establishing loss constraints for an unsupervised deraining framework based on an original rainy image, a clear image, a reconstructed clear image, and a reconstructed rainy image includes: establishing a first image content constraint based on a difference measure between the clear image and the reconstructed clear image, and a sum of difference measures between the clear image and the original rainy image and the reconstructed rainy image, respectively; establishing a second image content constraint based on a difference measure between the reconstructed rainy image and the original rainy image, and a sum of difference measures between the reconstructed rainy image and the clear image and the reconstructed clear image, respectively; establishing a global contrast loss constraint based on the first image content constraint and the second image content constraint; A first local feature constraint is established based on a difference measure between a patch block of a clear image and a patch block of a reconstructed clear image, and a sum of two difference measures between a patch block of the clear image and a patch block of an original rainy image and a patch block of the reconstructed rainy image; a second local feature constraint is established based on a difference measure between a patch block of a reconstructed rainy image and a patch block of an original rainy image, and a sum of two difference measures between a patch block of the reconstructed rainy image and a patch block of the clear image and a patch block of the reconstructed clear image; a local contrast loss constraint is established based on the first local feature constraint and the second local feature constraint.

[0074] Optionally, the first image content constraint includes a difference measure between a clear image and a reconstructed clear image, and a sum of difference measures between the clear image and the original rainy image and the reconstructed rainy image, wherein the difference measure between the clear image and the reconstructed clear image is used to measure the degree of proximity between the reconstructed clear image obtained by the unsupervised rain removal framework and the real clear image, and the sum of difference measures between the clear image and the original rainy image and the reconstructed rainy image is used to analyze the relationship between the reconstructed rainy image and the real clear image and the original rainy image. On the one hand, the reconstructed rainy image cannot be too different from the clear image, otherwise important image information may be lost in the reconstruction process; on the other hand, it should also have a certain similarity with the original rainy image to ensure the rationality of the reconstruction. By calculating the sum of these two difference measures, the performance of the reconstructed rainy image in different aspects can be comprehensively considered.

[0075] Optionally, the second image content constraint includes a difference measure between the reconstructed rainy image and the original rainy image, and a sum of difference measures between the reconstructed rainy image and the clear image and the reconstructed clear image, wherein the difference measure between the reconstructed rainy image and the original rainy image is used to ensure that the reconstructed rainy image can restore the features of the original rainy image as much as possible, and the sum of difference measures between the reconstructed rainy image and the clear image and the reconstructed clear image is used to ensure that the reconstructed rainy image is reasonable in relation to other images. The patch block refers to an image block after the image is cropped, and by calculating the local contrast loss for each small cropped image, the ability of the network to remove local fine rain can be improved from the local deep feature space.

[0076] Furthermore, by combining the first image content constraint and the second image content constraint, a global contrast loss constraint can be established, which considers the relationship between the clear image, the reconstructed clear image, the original rainy image and the reconstructed rainy image as a whole, so that the deraining framework not only pays attention to local details when processing images, but also grasps the characteristics of the image as a whole, so that the reconstructed clear image and rainy image are closer to the real situation as a whole.

[0077] For example, by constraining the overall content information of the clear image and the rain image, the clarity and overall color consistency of the generated image can be effectively improved. The specific formula is as follows:

[0078] The first image content constraint is expressed as:

[0079]

[0080] The numerator represents the clear image B and the reconstructed clear image B' in the feature V i The denominator represents the difference between the clear image B and the original rainy image O and the reconstructed image O' due to the feature V i The sum of the difference measures under .

[0081] Similarly, the second image content constraint is expressed as:

[0082]

[0083] In summary, the global contrast loss constraint is expressed as:

[0084]

[0085] in, and is a hyperparameter used to balance different losses.

[0086] Further, the first local feature constraint includes a difference measure between a patch block of a clear image and a patch block of a reconstructed clear image, and a sum of two difference measures between a patch block of a clear image and a patch block of an original rainy image and a patch block of a reconstructed rainy image, wherein the difference measure between a patch block of a clear image and a patch block of a reconstructed clear image is used to divide the image into a plurality of patches, and by comparing the difference between the corresponding patches of the clear image and the reconstructed clear image, the local details of the image can be paid more attention to. The sum of two difference measures between a patch block of a clear image and a patch block of an original rainy image and a patch block of a reconstructed rainy image is used to analyze the relationship between a patch block of a clear image and patches of other images, so as to ensure that the generation of a reconstructed rainy image and a clear image is reasonable at a local level. Similarly, the second local feature constraint includes a difference measure between a patch block of a reconstructed rainy image and a patch block of an original rainy image, and a sum of two difference measures between a patch block of a reconstructed rainy image and a patch block of a clear image and a patch block of a reconstructed clear image. Furthermore, a local contrast loss constraint is established based on the first local feature constraint and the second local feature constraint to analyze the local fine structure of the image, which can improve the ability of the deraining framework in restoring local details.

[0087] For example, by constructing local contrast loss for clear images and rainy images, each patch block in the image is constrained in the deep feature space, which improves the network's ability to restore local fine structures. The specific formula is as follows:

[0088] The first local feature constraint package is expressed as:

[0089]

[0090] Among them, the molecule represents the patch block B of the clear image B i Patch block B with reconstructed clear image B' i 'In feature V j The difference metric under the condition, the denominator represents the patch block B of the clear image B i Patch block O of the original rainy image O i and reconstruct the patch block O of the rainy image O' i 'The sum of the difference measures under the feature.

[0091] Similarly, the second local feature constraint package is expressed as:

[0092]

[0093] In summary, the local contrast loss constraint is expressed as:

[0094]

[0095] in, and is a hyperparameter used to balance different losses.

[0096] In this embodiment, by establishing global contrast loss constraints and local contrast loss constraints, the unsupervised rain removal framework is constrained from different angles, which enables the framework to better process images at both the overall and local levels, thereby improving the accuracy and quality of rain removal, making the reconstructed clear image closer to the real rain-free image, and the reconstructed rainy image is more reasonable.

[0097] In an exemplary embodiment, the loss constraints also include adversarial loss constraints, content consistency loss constraints and clarity consistency loss constraints; based on the original rainy image, the clear image, the reconstructed clear image and the reconstructed rainy image, the step of establishing loss constraints for the unsupervised rain removal framework includes: establishing an adversarial loss constraint based on the loss function of the discriminator and the loss function of the generator; establishing a content consistency loss constraint based on the pixel distance between the reconstructed rainy image and the original rainy image; establishing a clarity consistency loss constraint based on the pixel distance between the clear image and the reconstructed clear image.

[0098] Optionally, in an unsupervised rain removal framework, a discriminator and a generator are included, and the loss function of the discriminator and the loss function of the generator together constitute an adversarial loss constraint. The discriminator hopes to correctly distinguish between real images and generated images, so when it misjudges the generated image as a real image, it will incur a loss, while the generator hopes that the generated image can be mistaken for a real image by the discriminator, so when the discriminator judges its generated image as false, the generator will also incur a loss. By continuously optimizing these two loss functions, the discriminator and the generator can continuously improve their performance in the adversary, and ultimately the generator can generate high-quality reconstructed clear images and reconstructed rainy images.

[0099] For example, the adversarial loss is expressed as:

[0100]

[0101] in, represents the loss of the discriminator D, y represents the real clear image in the unpaired dataset, Denotes the generator G B The loss of , x is the rain image from the dataset.

[0102] Optionally, a content consistency loss constraint is established based on the pixel distance between the reconstructed rain image and the original rain image. Specifically, the difference between the reconstructed rain image and the original rain image at each pixel position is calculated. For example, the difference can be measured using an indicator such as mean square error (MSE) or structural similarity index (SSIM). The purpose of the content consistency loss constraint is to ensure that the reconstructed rain image is as similar as possible to the original rain image in terms of content. If the reconstructed rain image differs too much from the original rain image at the pixel level, it means that the reconstruction process may have lost important image information or introduced unreasonable changes.

[0103] Exemplarily, the content consistency constraint is achieved by reducing the L1 distance between the reconstructed O' and the input O pixel, which is expressed as:

[0104]

[0105] Among them, RM represents the rain model, GR represents the rain image generator, and G B Represents a clear image generator.

[0106] Optionally, similar to the content consistency loss constraint, the clarity consistency loss constraint is established based on the pixel distance between the clear image and the reconstructed clear image, and the pixel-level difference index can also be used to measure the similarity between the two. The purpose of the clarity consistency loss constraint is to ensure that the generated reconstructed clear image has a high consistency with the real clear image at the pixel level, avoiding the problems of blurring, distortion or loss of details in the generated clear image.

[0107] For example, by calculating the L1 distance between two clear images, the constraint of clear consistency is implemented to improve the network's rain removal performance, which is expressed as:

[0108]

[0109] In this embodiment, by establishing multiple loss constraints, the unsupervised rain removal framework can have more optimization targets during the training process, which can prevent the framework from falling into a local optimal solution. At the same time, by constraining the model from different angles through different loss constraints, they can complement each other, improve the stability and reliability of the framework, and thus improve the image rain removal effect.

[0110] In an exemplary embodiment, the step of optimizing the unsupervised rain removal framework according to the loss constraints includes: establishing an overall network loss function according to all the loss constraints; and optimizing the unsupervised rain removal framework with the goal of minimizing the function value of the overall network loss function.

[0111] Optionally, all loss constraints are integrated to establish an overall network loss function. After the overall network loss function is determined, the unsupervised rain removal framework is optimized with the goal of minimizing the function value of this function. During the optimization process, an optimization algorithm is used to adjust the parameters in the unsupervised rain removal framework. Specifically, the gradient of the overall network loss function to each parameter is calculated, and then the parameters are updated according to a certain step size based on the direction and size of the gradient. By continuously repeating this process, the value of the overall network loss function gradually decreases, that is, the output result of the unsupervised rain removal framework becomes closer and closer to the ideal state.

[0112] Exemplarily, the overall network loss function is expressed as:

[0113]

[0114] in, , , , ,as well as is a hyperparameter used to balance different losses, and its value can be: = =1, =0.52, =0.001, =0.008.

[0115] In this embodiment, by establishing an overall network loss function and integrating various loss constraints, multiple factors in the image deraining process can be comprehensively considered. Since different loss constraints measure the performance of the deraining framework from different perspectives, comprehensive consideration of these factors can optimize the deraining framework in all aspects, thereby improving the deraining effect.

[0116] In an exemplary embodiment, Figure 3 As shown, Figure 3 An example diagram of an overall network structure of the image deraining processing method provided in the embodiment of the present application is shown in FIG. Figure 3 As shown in , the image deraining processing method includes the following steps:

[0117] Step (1): Construct an end-to-end image generation network to generate a clear image end-to-end; at the same time, use the discriminator to distinguish the output of the network from the unpaired clear images in the dataset. Specifically, first extract features and downsample through multiple ResBlock blocks, then use the deconvolution reconstruction and upsampling network structure in the decoding stage to restore the original image resolution and obtain a reconstructed image B, then use the discriminator to distinguish this reconstructed image from the unpaired clear images in the dataset to ensure that the reconstructed image of the image generator is clear.

[0118] Step (2): Construct a rain image generation network, including two subnets. Subnet 1 and subnet 2 respectively use a series of residual convolution blocks to extract rain features. Subnet 1 obtains rain position information P after encoding and decoding operations, and subnet 2 obtains rain density information D after encoding and decoding operations. Then, these two features are multiplied to obtain accurate raindrop information R. Specifically, the structure of subnet 1 includes: the first layer of encoding layer feature extraction → the second layer of encoding layer feature extraction → the third layer of encoding layer feature extraction → the fourth layer of encoding layer feature extraction → the first layer of decoding layer recovery module → the second layer of decoding layer recovery module → the third layer of decoding layer recovery module → output raindrop position information R1. The structure of subnet 2 includes: the first layer of encoding layer feature extraction → the second layer of encoding layer feature extraction → the third layer of encoding layer feature extraction → the fourth layer of encoding layer feature extraction → the first layer of decoding layer recovery module → the second layer of decoding layer recovery module → the third layer of decoding layer recovery module → output raindrop position information R2. The final synthesized raindrop feature is expressed as: R1×R2 → obtain the final refined raindrop feature R.

[0119] Step (3): Reverse reconstruction of the clear image. Using the raindrop information R obtained by the rain image generation network, according to the physical model, the input image O is subtracted from R to reconstruct a clear image B'.

[0120] Step (4): Reverse reconstruction of the rainy image: the image B output by the end-to-end image generation network and the output R of the rain image generation network. Using the rain model, add B to R to get a reconstructed rainy image O'

[0121] Step (5): In this framework, the images output by the two networks are used to construct a bidirectional global-local contrast loss, and the two reversely reconstructed images are constrained for content consistency and clarity consistency to achieve better image reconstruction. The specific process includes: constructing a content consistency loss between O' and O → discriminant loss to ensure that the image generator generates a clear image → unsupervised rain removal framework → clarity consistency loss between B' and B → B and O' as anchor points, the images generated in the framework and the rain images in the dataset are used as positive and negative samples to construct a bidirectional global-local contrast constraint → unsupervised bidirectional global-local contrast reconstruction framework.

[0122] Exemplarily, in order to verify the deraining effect of the image deraining processing method provided in the embodiments of the present application, two public rain image datasets, Rain100L and Rain800, were used as training sets, and deraining comparison experiments were carried out with the cycle generative adversarial network (CycleGAN), the contrastive unsupervised transformation algorithm (CUT), the deraining cycle generative adversarial network (DerainCycleGAN), the generative adversarial network based on contrastive learning and dual learning (DCLGAN), and the dual discriminator generative adversarial network (DCD-GAN).

[0123] For example, as shown in Table 1 and Table 2, Table 1 is the comparison results on the Rain800 dataset, and Table 2 is the comparison results on the Rain100L dataset. It can be seen from Table 1 and Table 2 that compared with other algorithms, the unsupervised rain removal framework provided in the embodiment of the present application has better rain removal performance in various rainfall scenes, and can effectively reduce the impact of raindrops in the image, improve the clarity and details of the image, and restore a more realistic scene.

[0124] Table 1 Comparison results on the Rain800 dataset

[0125]

[0126] Table 2 Comparison results on the Rain100L dataset

[0127]

[0128] Furthermore, in order to verify the effectiveness of the proposed unsupervised rain removal framework, ablation experiments were carried out on the Rain100L dataset. Three experiments were designed based on the unsupervised rain removal framework: (1) Base: an unsupervised rain removal framework using unidirectional reconstruction of the rain model; (2) BCRF: a bidirectional global contrast reconstruction framework using the rain model; (3) BGCR: a bidirectional global local contrast reconstruction framework using the rain model.

[0129] For example, the experimental results are shown in Table 3. As can be seen from Table 3, experiments (1) and (2) show that the bidirectional reconstruction framework using the rain model improves the performance of the network, but since only the global contrast constraint is used, the constraint on the generated image is limited. Experiment (3) adopts a bidirectional global-local contrast reconstruction framework to improve the network reconstruction ability and strengthen the global and local constraints at the same time, thereby improving the network's deraining performance.

[0130] Table 3 Quantitative results of ablation experiments at each stage on the Rain100L dataset

[0131]

[0132] In this embodiment, an unsupervised rain removal network based on a bidirectional global-local contrast framework is established, including an image generator and a rain generator. The images obtained by the two generators are reversely reconstructed into a clear image and a rain image using a physical model. This can avoid the data confusion problem faced by the generator in the CycleGAN framework, which causes the generated image to have low quality. A bidirectional global-local contrast constraint is constructed through the reconstructed image to improve the network's ability to restore the details and color of the reconstructed image content, thereby effectively removing rain.

[0133] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0134] Based on the same inventive concept, the embodiment of the present application also provides an image deraining processing device for implementing the above-mentioned image deraining processing method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more image deraining processing device embodiments provided below can refer to the limitations of the image deraining processing method above, and will not be repeated here.

[0135] In an exemplary embodiment, Figure 4 As shown, an image deraining processing device is provided, comprising: a framework establishment module 10, an image reconstruction module 20, an image acquisition module 30, a constraint establishment module 40 and a deraining processing module 50, wherein:

[0136] The framework building module 10 is used to build an unsupervised rain removal framework; the unsupervised rain removal framework includes an end-to-end image generation network and a rain image generation network.

[0137] The image reconstruction module 20 is used to reconstruct the original rainy image through an end-to-end image generation network to obtain a clear image, and to extract features of the original rainy image through a rain image generation network to obtain raindrop information; the raindrop information is the product of the position information and density information of the rain in the original rainy image.

[0138] The image acquisition module 30 is used to acquire a reconstructed clear image based on the raindrop information and the original rainy image, and to acquire a reconstructed rainy image based on the clear image and the raindrop information.

[0139] The constraint establishment module 40 is used to establish loss constraints for the unsupervised rain removal framework based on the original rainy image, the clear image, the reconstructed clear image and the reconstructed rainy image; the loss constraints include global contrast loss constraints and local contrast loss constraints.

[0140] The rain removal processing module 50 is used to optimize the unsupervised rain removal framework according to the loss constraint condition, and perform image rain removal processing on the original rainy image through the optimized unsupervised rain removal framework.

[0141] In an exemplary embodiment, the end-to-end image generation network involved in the framework establishment module 10 includes a generator and a discriminator, the generator is used to reconstruct the original rainy image to obtain a clear image, and the discriminator is used to distinguish between real images and generated images; the rain image generation network includes a first subnet and a second subnet, the first subnet is used to obtain location information, and the second subnet is used to obtain density information.

[0142] In an exemplary embodiment, the image acquisition module 30 is further used to perform subtraction processing on the original rainy image and raindrop information to obtain a reconstructed clear image; and perform addition processing on the clear image and raindrop information to obtain a reconstructed rainy image.

[0143] In an exemplary embodiment, the constraint establishing module 40 is further used to establish a first image content constraint based on a difference measure between the clear image and the reconstructed clear image, and a sum of difference measures between the clear image and the original rainy image and the reconstructed rainy image, respectively; establish a second image content constraint based on a difference measure between the reconstructed rainy image and the original rainy image, and a sum of difference measures between the reconstructed rainy image and the clear image and the reconstructed clear image, respectively; establish a global contrast loss constraint based on the first image content constraint and the second image content constraint; establish a first local feature constraint based on a difference measure between a patch block of the clear image and a patch block of the reconstructed clear image, and a sum of two difference measures between a patch block of the clear image and a patch block of the original rainy image and a patch block of the reconstructed rainy image; establish a second local feature constraint based on a difference measure between a patch block of the reconstructed rainy image and a patch block of the original rainy image, and a sum of two difference measures between a patch block of the reconstructed rainy image and a patch block of the clear image and a patch block of the reconstructed clear image; establish a local contrast loss constraint based on the first local feature constraint and the second local feature constraint.

[0144] In an exemplary embodiment, the loss constraints also include adversarial loss constraints, content consistency loss constraints and clarity consistency loss constraints; the constraint establishment module 40 is also used to establish adversarial loss constraints based on the loss function of the discriminator and the loss function of the generator; establish content consistency loss constraints based on the pixel distance between the reconstructed rainy image and the original rainy image; establish clarity consistency loss constraints based on the pixel distance between the clear image and the reconstructed clear image.

[0145] In an exemplary embodiment, the rain removal processing module 50 is further used to establish an overall network loss function according to all loss constraints; and optimize the unsupervised rain removal framework with the goal of minimizing the function value of the overall network loss function.

[0146] Each module in the above-mentioned image deraining processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0147] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 5 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, an image deraining processing method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0148] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0149] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0150] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0151] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0152] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0153] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0154] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for removing rain from an image, characterized in that: The method comprises: Establishing an unsupervised rain removal framework; the unsupervised rain removal framework includes an end-to-end image generation network and a rain image generation network; The original rainy image is reconstructed by the end-to-end image generation network to obtain a clear image, and the rain image generation network is used to extract features of the original rainy image to obtain raindrop information; the raindrop information is the product of the position information and the density information of the rain in the original rainy image; Acquire a reconstructed clear image according to the raindrop information and the original rainy image, and acquire a reconstructed rainy image according to the clear image and the raindrop information; Based on the original rainy image, the clear image, the reconstructed clear image and the reconstructed rainy image, a loss constraint condition for the unsupervised rain removal framework is established; the loss constraint condition includes a global contrast loss constraint and a local contrast loss constraint; The unsupervised rain removal framework is optimized according to the loss constraint condition, and the original rainy image is subjected to image rain removal processing through the optimized unsupervised rain removal framework.

2. The method according to claim 1, characterized in that The end-to-end image generation network includes a generator and a discriminator, the generator is used to reconstruct the original rainy image to obtain a clear image, and the discriminator is used to distinguish between real images and generated images; the rain image generation network includes a first subnet and a second subnet, the first subnet is used to obtain the position information, and the second subnet is used to obtain the density information.

3. The method according to claim 1, characterized in that The step of obtaining a reconstructed clear image according to the raindrop information and the original rainy image, and obtaining a reconstructed rainy image according to the clear image and the raindrop information includes: Performing subtraction processing on the original rainy image and the raindrop information to obtain a reconstructed clear image; The clear image and the raindrop information are added to obtain a reconstructed rainy image.

4. The method according to claim 1, characterized in that: The establishing of loss constraints for the unsupervised rain removal framework based on the original rainy image, the clear image, the reconstructed clear image and the reconstructed rainy image includes: Establishing a first image content constraint based on a difference measure between the clear image and the reconstructed clear image, and a sum of difference measures between the clear image and the original rainy image, respectively, and the reconstructed rainy image; Establishing a second image content constraint based on a difference measure between the reconstructed image with rain and the original image with rain, and a sum of difference measures between the reconstructed image with rain and the clear image and the reconstructed clear image respectively; Establishing a global contrast loss constraint according to the first image content constraint and the second image content constraint; Establishing a first local feature constraint based on a difference measure between the patch block of the clear image and the patch block of the reconstructed clear image, and a sum of two difference measures between the patch block of the clear image and the patch block of the original rainy image and the patch block of the reconstructed rainy image; Establishing a second local feature constraint based on a difference measure between the patch block of the reconstructed rainy image and the patch block of the original rainy image, and a sum of two difference measures between the patch block of the reconstructed rainy image and the patch block of the clear image and the patch block of the reconstructed clear image; A local contrast loss constraint is established according to the first local feature constraint and the second local feature constraint.

5. The method according to claim 2, characterized in that: The loss constraint conditions also include adversarial loss constraints, content consistency loss constraints and clarity consistency loss constraints; the loss constraint conditions for the unsupervised deraining framework are established based on the original rainy image, the clear image, the reconstructed clear image and the reconstructed rainy image, including: Establishing an adversarial loss constraint according to the loss function of the discriminator and the loss function of the generator; Establishing a content consistency loss constraint according to a pixel distance between the reconstructed rainy image and the original rainy image; A clarity consistency loss constraint is established according to a pixel distance between the clear image and the reconstructed clear image.

6. The method according to claim 5, characterized in that Optimizing the unsupervised rain removal framework according to the loss constraint condition includes: Establish the overall network loss function based on all loss constraints; The unsupervised rain removal framework is optimized with the goal of minimizing the function value of the overall network loss function.

7. An image deraining processing device, characterized in that: The device comprises: A framework building module, used to build an unsupervised rain removal framework; the unsupervised rain removal framework includes an end-to-end image generation network and a rain image generation network; An image reconstruction module, configured to reconstruct the original rainy image through the end-to-end image generation network to obtain a clear image, and extract features of the original rainy image through the rain image generation network to obtain raindrop information; the raindrop information is the product of the position information and density information of the rain in the original rainy image; An image acquisition module, used to acquire a reconstructed clear image according to the raindrop information and the original rainy image, and to acquire a reconstructed rainy image according to the clear image and the raindrop information; A constraint establishment module, used to establish loss constraints for the unsupervised rain removal framework based on the original rainy image, the clear image, the reconstructed clear image and the reconstructed rainy image; the loss constraints include global contrast loss constraints and local contrast loss constraints; The rain removal processing module is used to optimize the unsupervised rain removal framework according to the loss constraint condition, and perform image rain removal processing on the original rainy image through the optimized unsupervised rain removal framework.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images