A single image rain removal method based on rain map decomposition

Abstract

The application discloses a single image rain removal method based on rain map decomposition. The application inputs a rain map into a rain streak feature prediction network to obtain a rain streak estimation map; then inputs the rain streak estimation map into a rain density perception classifier to obtain a rain density level label of the rain map; then inputs the rain streak estimation map and the rain density level label into a rain streak pixel kernel positioning network to obtain a pixel-by-pixel rain streak pixel kernel position vector; then inputs the rain streak pixel kernel position vector into an SNIP scale adaptive range predictor to generate a corresponding rain streak coverage area, and superimposes and synthesizes to obtain a rain streak overall coverage area; finally, uses a non-local mean filter to perform rain removal processing on the rain streak overall coverage area to obtain a rain-removed image. The application accurately decomposes the rain map into a rain streak area and a background area, can better solve the problem of loss of background details caused by excessive rain removal in the rain removal process, and effectively removes the rain streak and restores the background texture detail information.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and image processing, specifically a method for removing rain from a single image based on rain map decomposition. Background Technology

[0002] Image deraining is a key research task in computer vision, crucial for improving image quality and enhancing the performance of computer vision systems under extreme weather conditions. In practical applications, high-density rain streaks interfere with the visual quality of images, posing challenges to computer vision tasks such as object detection, visual tracking, road sign recognition, and autonomous driving. For example, in object detection, rain streaks or raindrops often obscure targets, preventing accurate separation and significantly reducing detection efficiency. Therefore, employing image deraining methods to process outdoor images and restore their clear visual quality is essential.

[0003] Mainstream image deraining methods can be divided into model-driven and data-driven methods. Model-driven methods rely on prior knowledge of the image, such as the direction, density, and size of rain streaks, to constrain the deraining problem and the deraining model. Then, an optimization algorithm is designed to solve the problem, resulting in a clean, rain-free image. Data-driven methods are based on deep learning algorithms. By constructing neural networks, they utilize paired rain streak labels and clean, rain-free images to learn a non-linear mapping from rainy to rainless conditions.

[0004] Current research on rain removal from single images primarily focuses on addressing the negative impact of dense rain streaks and their stacking on background detail recovery in rain images. Various methods are employed to progressively eliminate dense rain streaks while preserving background details. These existing methods are mainly limited to learning certain fixed features of rain streaks. They can use different feature extractors for rain images with uncertain rain densities, often prioritizing the model's general applicability across different rain images rather than specifically utilizing the rain density features. This approach often leads to over-removal of rain streaks in high-density rain images, making it difficult to handle the conflict between small background objects and high-density rain streaks, thus resulting in the loss of some background detail information. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the technical problem this invention aims to solve is to provide a method for rain removal from a single image based on rain map decomposition.

[0006] The technical solution of this invention to solve the aforementioned technical problem is to provide a single-image rain removal method based on rain map decomposition, characterized in that the method includes the following steps:

[0007] Step 1: Process the original rain map dataset to generate a rain map dataset B of uniform size. Total ; Rain map dataset B Total Divided into training set B train Validation set B vali and test set B test ;

[0008] Rain Map Dataset B Total It includes complete rainy images and their corresponding rainless images; the rainy images serve as input to the rain pattern feature prediction network, and the corresponding rainless images serve as reference labels used when calculating the loss function;

[0009] Step 2: Transfer training set B train Rain image B is input into the rain streak feature prediction network to generate a predicted rain streak image B containing background details for this iteration. J Then through validation set B vali The rain pattern feature prediction network is iteratively trained by constraining the loss function until the set number of iterations is reached, and the trained rain pattern feature prediction network is obtained.

[0010] The rain swastika feature prediction network consists of a residual transpose, a guided filter layer, and a smoothing convolution kernel connected in sequence; after the smoothing convolution kernel, a parallel fully convolutional network, a sub-network, and an autoencoder with skip connections are set up.

[0011] Step 3: Obtain the rain streak prediction map B from Step 2. J The data is input into the rain density sensing classifier to obtain the rain density level label B of the rain map B in this iteration. d Then through validation set B vali Constrain the loss function and iteratively train the rain density sensing classifier until the set number of iterations is reached to obtain the trained rain density sensing classifier.

[0012] Step 4: Test set B test Rain image B is input into a pre-trained rain pattern feature prediction network to generate a predicted rain pattern image B. J Then, project the rain pattern prediction map B. J The input is fed into a trained rain density sensing classifier to obtain the rain density level label B. d Then label the rain density level B. d The input is fed into a rain ripple pixel kernel localization network based on the UNet semantic segmentation network to obtain test set B. test The pixel-by-pixel rain ripple kernel position vector {p(B)} in the rain map B. S )};

[0013] Step 5: Calculate the rain ripple pixel kernel position vector {p(B} obtained in Step 4. SThe SNIP scale-adaptive range predictor is used to generate the rain pattern coverage area corresponding to each rain pattern pixel kernel; then all rain pattern coverage areas are superimposed to obtain the overall rain pattern coverage area.

[0014] The SNIP scale-adaptive range predictor consists of three-scale image pyramids, each consisting of an RPN candidate box generator, an RCN region classification discriminator, and a scale rescaling layer.

[0015] Step 6: Perform nonlocal mean filtering on the overall rain pattern coverage area obtained in Step 5 to obtain test set B. test The de-raining image of the moderate rain map B.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0017] (1) This invention inputs the rain image into a rain pattern feature prediction network to obtain a rain pattern prediction image; then, the rain pattern prediction image is input into a rain density perception classifier to obtain a rain density level label for the rain image; then, the rain pattern prediction image and the rain density level label are input into a rain pattern pixel kernel localization network to obtain a pixel-by-pixel rain pattern pixel kernel position vector; then, the rain pattern pixel kernel position vector is input into a SNIP scale adaptive range predictor to generate the corresponding rain pattern coverage area, which is then superimposed to obtain the overall rain pattern coverage area; finally, non-local mean filtering is used to de-rain the overall rain pattern coverage area to obtain a de-rained image. This invention accurately decomposes the rain image into rain pattern areas and background areas, which can better solve the problem of excessive de-raining leading to loss of background details during the de-raining process, thereby effectively removing rain patterns and restoring background texture details.

[0018] (2) This invention applies the idea of ​​rain map decomposition according to rain pattern density to the process of rain pattern feature prediction. By processing the high-frequency and low-frequency components of rain map decomposition with smooth convolution kernels of different scales, the accumulation effect of rain patterns of different scales and densities can be distinguished, reducing the complexity of the rain pattern prediction process.

[0019] (3) The present invention uses the estimated rain pattern image instead of the rain image itself to input into the rain density perception classifier to extract the rain density level label, which can reduce the conflict between rain pattern and background detail pixels, thereby more accurately identifying the edge pixel information of rain pattern and improving the classification accuracy of rain density level label.

[0020] (4) In the process of UNet skip connection for rain ripple pixel kernel location vector localization, the present invention uses a multi-scale high-order cooperative network for bidirectional feature mining, which can better transfer rain ripple feature information in shallow network to deep network and better learn large-scale rain ripple feature information.

[0021] (5) The present invention uses a scale-adaptive range predictor to adjust the conflict between large-scale rain patterns and small objects in the background details. It can adaptively generate range boxes for rain patterns of different scales and densities, reducing the process of manually adjusting model parameters.

[0022] (6) The present invention constructs a complete single-image rain removal process, which can adaptively process rain images with rain streaks of different scales and densities, and the rain removal effect is better than other methods. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the structure of the rain pattern feature prediction network of the present invention;

[0024] Figure 2 This is a schematic diagram of the structure of the rain density sensing classifier of the present invention;

[0025] Figure 3 This is a schematic diagram of the SNIP scale-adaptive range predictor of the present invention;

[0026] Figure 4 This is a rainy image from Embodiment 1 of the present invention;

[0027] Figure 5 This is a rainless image from Embodiment 1 of the present invention;

[0028] Figure 6 The rain pattern S obtained in Embodiment 1 of the present invention;

[0029] Figure 7 This is the global atmospheric illumination map A obtained in Embodiment 1 of the present invention;

[0030] Figure 8 The atmospheric transmission diagram T obtained in Embodiment 1 of the present invention;

[0031] Figure 9 Rain streak prediction image B obtained in Embodiment 1 of the present invention J ;

[0032] Figure 10 This is the derained image obtained in Embodiment 1 of the present invention. Detailed Implementation

[0033] Specific embodiments of the present invention are given below. These specific embodiments are only used to further illustrate the present invention in detail and do not limit the scope of protection of the claims of the present invention.

[0034] This invention provides a single-image rain removal method based on rain map decomposition (hereinafter referred to as the method), characterized by the following steps:

[0035] Step 1: Process the original rain map dataset to generate a rain map dataset B of uniform size.Total ; Rain map dataset B Total Divided into training set B train Validation set B vali and test set B test ;

[0036] Rain Map Dataset B Total It includes complete rainy images and their corresponding rainless images; the rainy images serve as the input to the rain pattern feature prediction network, and the corresponding rainless images serve as the reference labels; when calculating the loss function, the smaller the difference between the rainless image and the label, the smaller the loss function value.

[0037] Step 2: Transfer training set B train Rain image B is input into the rain streak feature prediction network (e.g., Figure 1 As shown in the figure, the rain streak prediction map B containing background detail information is generated for this iteration. J (Abbreviated as Rain Vein Prediction Chart B) J Then, through validation set B... vali The rain pattern feature prediction network is iteratively trained by constraining the loss function until the set number of iterations is reached, and the trained rain pattern feature prediction network is obtained.

[0038] The rain swastika feature prediction network consists of a residual transpose, a guided filter layer, and a smoothing convolution kernel connected in sequence; after the smoothing convolution kernel, a parallel fully convolutional network, a sub-network, and an autoencoder with skip connections are set up.

[0039] Preferably, in step 2, the fully convolutional network consists of p (12 in this embodiment) basic residual blocks; the sub-network consists of q (5 in this embodiment) convolutional blocks plus activation functions, plus 2 fully connected layers and 1 upsampling layer.

[0040] Preferably, in step 2, a rain streak prediction map B is generated. J The process is as follows:

[0041] S2.1 Rain map B is generated by residual transpose to form guiding rain map B. P Then combine rain map B and guiding rain map B. P The input is fed into a guided filter layer for low-pass smoothing to obtain the low-frequency component B of the rain map. L and high-frequency component B H B H =BB L ;

[0042] S2.2, Low-frequency component B L and high-frequency component B H After processing by a set of smooth convolution kernels K, a set of components {(B L ) i} and {(BH ) i};

[0043] Preferably, step S2.2 specifically involves: converting the low-frequency component B... L and high-frequency component B H Processed using a set of i smooth convolution kernels K, in each smooth convolution kernel channel, the low-frequency component B... L and high-frequency component B H The results obtained after processing by each smoothing convolution kernel K are merged and input into a convolution kernel (kernel size is 1×1) to obtain a set of components {(B L ) i} and {(B H ) i};

[0044] Preferably, in step S2.2, the size k of the smooth convolution kernel K is 2. i , i = 0, 1, ..., 6.

[0045] S2.3, Using a fully convolutional network from components {(B H ) i The rain pattern S is extracted from the component {(B}, and a subnetwork is used to extract the rain pattern S from the component {(B}). L ) i The global atmospheric illumination map A is obtained by predicting and learning from the component {(B}, and an autoencoder with skip connections is used to obtain the global atmospheric illumination map A from the component {(B}). L ) i} and {(B H ) i Atmospheric transmission map T is extracted from};

[0046] S2.4. Combine the rain strife map S, the global atmospheric illumination map A, and the atmospheric transmission map T to obtain the rain strife prediction map B. J .

[0047] Preferably, in step S2.1, the calculation formula for the residual transpose process is:

[0048]

[0049] In equation (1), c and d represent one of the three color channels r, g, and b, respectively; B c and B d This represents the values ​​of the c and d color channels at pixel x in the rain map B;

[0050] Preferably, in step S2.4, the rain streak prediction map B J The calculation formula is:

[0051]

[0052] In equation (2), O represents the clean image after removing rain streaks, v represents the rain intensity, and Γ represents the rain intensity. t The image shows the rain streaks at rainfall intensity t, where A represents the atmospheric illumination map and T represents the atmospheric transmittance map. This represents the matrix multiplication operation.

[0053] Preferably, in step S2.3, the extraction of the rain ripple map S, the global atmospheric illumination map A, and the atmospheric transmittance map T all use the mean squared error loss function, the calculation formula of which is:

[0054]

[0055] In equation (3), This represents the loss for extracting the feature map, where Ω represents one of the rain ripple map S, the global atmospheric illumination map A, and the atmospheric transmittance map T. gt The corresponding true rain map label is represented; MSE represents the formula for calculating the mean squared error loss function:

[0056]

[0057] In equation (4), m and n are the height and width of the image, respectively, and X ij and Y ij These represent the gray values ​​at pixel positions i and j in the reference image and the image to be measured, respectively.

[0058] Preferably, in step S2.4, the rain streak prediction map B is synthesized. J The formula for calculating the loss function used is:

[0059]

[0060]

[0061] In equation (5) λ S , λ A and λ T These represent their respective weights in the total loss function. Since all three use the same MSE (mean squared error) loss function at the same scale, and the weight of rain streaks is more important in the rain map, these three weights can be set to 1, 0.5, and 0.5, respectively.

[0062] Step 3: Obtain the rain streak prediction map B from Step 2. J Input to a rain density-sensing classifier (e.g.) Figure 2 As shown in the figure, the rain density level label B of the rain map B in this iteration is obtained. d Then through validation set B vali Constrain the loss function and iteratively train the rain density sensing classifier until the set number of iterations is reached to obtain the trained rain density sensing classifier.

[0063] Preferably, in step 3, the rain density sensing classifier (such as...) Figure 2 As shown, it is based on the ResNet-34 residual classifier and consists of three convolutional layers (Conv), one average pooling layer (AP), and two fully connected layers (FC) connected in sequence. The last layer consists of 5 output neurons, which are the rain density levels of the output image. The kernel size of the three convolutional layers is 3×3, namely Conv(3,24)-Conv(24,64)-Conv(64,24); the kernel size of the average pooling layer is 9×9; and the two fully connected layers are FC(127896,512)-FC(512,5).

[0064] Preferably, in step 3, the formula for calculating the loss function used to train the rain density-aware classifier is:

[0065] Γ=Γ E，r +Γ c (6)

[0067] In equation (6) Γ E，r Γ represents the per-pixel Euclidean loss used to estimate the residual components; c This represents the cross-entropy loss of the rain density-aware classifier.

[0068] Step 4: Test set B test Rain image B is input into a pre-trained rain pattern feature prediction network to generate a predicted rain pattern image B. J Then, project the rain pattern prediction map B. J The input is fed into a trained rain density sensing classifier to obtain the rain density level label B. d Then label the rain density level B. d The input is fed into a rain ripple pixel kernel localization network based on the UNet semantic segmentation network to obtain test set B. test The pixel-by-pixel rain ripple kernel position vector {p(B)} in the rain map B. S )};

[0069] Preferably, in step 4, the UNet semantic segmentation network is an encoder-decoder structure, characterized by a backpropagation structure for feature transfer between the encoder and decoder structures, i.e., fusing feature maps from the downsampling process during upsampling; this rain ripple pixel kernel localization network uses a multi-scale high-order cooperative network for bidirectional feature map mining and transfer during the skip-layer connection process, and its feature transfer calculation formula is:

[0070]

[0071] In Equation (7), X(i,j) represents the output of node X at position (i,j), where i is the downsampling layer along the encoder index and j is the convolutional layer along the skip connection dense block; function H represents a convolution operation followed by an activation function; functions D and U represent the downsampling layer and the upsampling layer; [] represents the concatenated layer; a node at level j = 0 receives only one input from the previous autoencoder layer; a node at level j = 1 receives two inputs from the encoder subnet, but at two consecutive levels; and a node at level j > 1 receives j+1 inputs, where the input of j is the output of the previous j nodes in the same skip connection, and the input of j+1 is the upsampling output from the lower skip connection. The dense convolutional block causes all previous feature maps to accumulate along the skip connection and reach the current node.

[0072] Step 5: Calculate the rain ripple pixel kernel position vector {p(B} obtained in Step 4. S Using the SNIP scale-adaptive range predictor (e.g.) Figure 3 (as shown) to generate the rain pattern coverage area corresponding to each rain pattern pixel kernel; then superimpose all the rain pattern coverage areas to obtain the overall rain pattern coverage area;

[0073] The SNIP scale-adaptive range predictor consists of three-scale image pyramids, each consisting of an RPN candidate box generator, an RCN region classification discriminator, and a scale rescaling layer.

[0074] Preferably, in step 5, the SNIP scale-adaptive range predictor has three image pyramids at different scales, each with its own RPN candidate box generator, and each predicts objects within a specified range. The RPN candidate box generator is responsible for generating multiple candidate boxes for each invalid ground truth region, and judging the validity of these candidate boxes by the RCN region classification discriminator, thereby deleting invalid boxes and retaining valid boxes. The specific steps of the generation and discrimination process are as follows:

[0075] (5.1) Generate candidate boxes and determine their positive or negative values, that is, assign + / - labels to the generated candidate boxes using all the real rain pattern data;

[0076] (5.2) Delete candidate boxes that are too large or too small, based on the effective range [S] at the i-th scale. i E i The system categorizes real rain pattern data into valid or invalid real data based on whether the data falls within the candidate box.

[0077] (5.3) Delete candidate boxes with inaccurate positioning and remove all candidate boxes with an overlap IoU (candidate box area / invalid real area) > 0.3;

[0078] The formula for calculating the overlap ratio IoU is:

[0079]

[0080] Step 6: Perform nonlocal mean filtering on the overall rain pattern coverage area obtained in Step 5 to obtain test set B. test The de-raining image of the moderate rain map B.

[0081] Preferably, in step 6, the calculation formula for nonlocal mean filtering is:

[0082]

[0083] In equation (9), v (i) This represents the gray value of pixel i in the contaminated image, and its gray value estimate obtained through filtering is NL(v)(i), with coefficient w. (i，j) This represents the weighting coefficient of the influence of pixel j on pixel i.

[0084] Preferably, the method further includes step 7: using the derained image obtained in step 6 and the corresponding test set B test The rain removal effect was evaluated by comparing rainless images in the image.

[0085] Preferably, in step 7, evaluating the rain removal effect specifically involves using test set B. test The rainy image in step 1 to step 6, followed by the de-rained image, is used as the test image, and the test set B is used as the test set. test The rainless image corresponding to the rainy image is used as a reference image. The effectiveness of the single-image rain removal process is determined by calculating the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) of the test image and the corresponding reference image. The larger the values ​​of PSNR and SSIM, the smaller the difference between the obtained test image and the reference image, indicating a better rain removal effect.

[0086] The formulas for calculating Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) are as follows:

[0087]

[0088] In equation (10), μ x and μ y δ represents the mean of the test image and the reference image, respectively. x 2 and δ y 2 δ represents the variance of the test image and the reference image, respectively. xy c1 and c2 are the covariance between the two images, while c1 and c2 are fixed constants chosen to avoid a denominator of 0.

[0089] Example 1

[0090] Step 1, Rain Map Dataset B Total Images were uniformly scaled to 320×320 pixels. The training set used 1800 pairs of rain images from the Rain100 dataset, the validation set used 100 pairs of rain images from the Rain100H dataset, and the test set used 2800 pairs of rain images from the Test2800 dataset.

[0091] The rain map dataset includes complete rainy images and their corresponding rainless images, where the rainy images (such as...) Figure 4 As shown in the image, the corresponding rainless image (e.g.) is used as the input to the rain streak feature prediction network. Figure 5 (As shown) is used as a reference label.

[0092] Step 2: Input the 320×320 rain map B into the residual transpose to obtain the guiding rain map B of the same scale. P Rain map B and guide rain map B P The input is fed into a guided filter layer for low-pass smoothing to obtain the low-frequency component B of the rain map at the same scale. L and high-frequency component B H B H =BB L The obtained low-frequency component B L and high-frequency component B H A set of smooth convolution kernels K was used for processing, with a size of k=2. i Given i = 0, 1, 2, 3, 4, with a stride of 2, padding = 0, and 3 channels; for the i-th smooth convolution kernel channel, the output {B L} and {B H The corresponding dimensions are 160×160×3, 80×80×3, 39×39×3, 16×16×3, and 1×1×3;

[0093] The fully convolutional network used to extract the rain ripple image S consists of 12 basic residual blocks. Each basic residual block consists of a 3×3×64 convolutional kernel and a corresponding ReLU activation function, and the corresponding output {B} H The features are replayed back to their original scale and then fused using mean fusion, resulting in a rain pattern image S with a size of 320×320 (e.g., Figure 6 (as shown);

[0094] The network used to extract the global atmospheric illumination map A has five convolutional blocks with kernel sizes of 3×3×16, 16×16×32, 32×32×64, 64×64×128, and 128×128×256, respectively. Each convolutional kernel is followed by a ReLU activation function. Two fully connected layers, FC(36864,256)-FC(256,16), and bilinear interpolation upsampling are added to recover the scale. The low-frequency component {B} of the rain map is extracted.L} to predict and learn to obtain a global atmospheric illumination map A with a size of 320×320 (e.g. Figure 7 (as shown);

[0095] The process of extracting atmospheric transmission maps T uses a basic convolutional autoencoder with skip connections, from {B L} and {B H The atmospheric transmission map T with a size of 320×320 was extracted from the data. Figure 8 (as shown);

[0096] Rain streak image B is obtained by combining rain streak image S, global atmospheric illumination image A, and atmospheric transmittance image T. J (like Figure 9 (As shown).

[0097] Step 3: Obtain the rain streak prediction map B from Step 2. J The input is fed into a rain density-aware classifier based on a ResNet-34 residual classifier to obtain the rain density level label B of rain map B. d The five neurons output by the last fully connected layer represent the rain density level of the output image.

[0098] Training set B train After processing by the rain strife feature prediction network trained in step 2, the resulting rain strife prediction map B is obtained. J The data is input into the rain density-aware classifier and passed through the validation set B. vali The loss function is constrained until the set number of training iterations are reached to obtain a trained rain density-aware classifier.

[0099] Step 4: Test set B test The rain pattern prediction network trained in step 2 is used to process the rain pattern prediction image B. J Then, the rain pattern prediction map B J The input is fed into a trained rain density sensing classifier to obtain the rain density level label B. d The input is fed into a rain ripple pixel kernel localization network based on the UNet semantic segmentation network to obtain test set B. test The pixel-by-pixel rain ripple kernel position vector {p(B)} in the medium rain image B S )};

[0100] Step 5: Test set B obtained in step 4 test The position vector p(B) of the rain streak pixel kernel in the medium rain image B S The SNIP scale-adaptive range predictor is used to generate the rain pattern coverage area corresponding to the rain pattern pixel kernel, and all rain pattern coverage areas are superimposed to obtain the overall rain pattern coverage area.

[0101] Step 6: Centralize the rain map data into test set B.test After performing a nonlocal mean filtering operation on the overall rain pattern coverage area obtained in step 5, the test set B is output. test Rainfall removal image of moderate rain map B (e.g.) Figure 10 (As shown).

[0102] Figure 6 and Figure 9 The comparison shows that, Figure 6 It only contains the rain streak information contained in the high-frequency components during the rain map decomposition process, and only has relatively simple rain streak features, while Figure 9 It not only includes rain pattern information from high-frequency components but also retains some background details from low-frequency components. This provides good global features for subsequent rain pattern pixel kernel localization and rain pattern region box generator, thereby improving the model's stability when handling large-scale, high-density rain patterns.

[0103] Depend on Figure 10 As can be seen, after processing with the single-image rain removal method, the rain stripes in the original image, which was covered with rain stripes, have been completely removed. Moreover, compared with the rainless reference image, the background details are still well preserved, thus verifying the effectiveness of the present invention in the rain removal task.

[0104] Other image deraining methods were compared with the method of this invention on the test set. The PSNR and SSIM evaluation values ​​used are shown in Table 1:

[0105] Table 1

[0106]

[0107] Progressive raining network PReNet, multi-stage progressive raining network MPRNet, rainstorm reconstruction network Restormer, multi-axis perceptron image processing network MAXIM, and mixed-layer image reconstruction network MHNet are the mainstream image deraining methods in the field of image deraining. As shown in Table 1, in the 2800 pairs of rain map dataset in the Test2800 test set, the PSNR and SSIM of this invention are both superior to other image deraining methods.

[0108] Any aspects not covered in this invention are applicable to existing technologies.

Claims

1. A single image deraining method based on rain map decomposition, characterized in that, The method comprises the following steps: Step 1, processing the original rain map data set to generate a uniform size rain map data set B Total ; dividing the rain map data set B Total into a training set B train , a validation set B vali and a test set B test ; Rain dataset B Total comprises complete rainy images and corresponding non-rainy images; the rainy images are taken as the input of the rain streak feature prediction network, and the corresponding non-rainy images are taken as the reference label, which is used when calculating the loss function; Step 2: Transfer training set B train Rain image B is input into the rain streak feature prediction network to generate a predicted rain streak image B containing background details for this iteration. J Then through validation set B vali The rain pattern feature prediction network is iteratively trained by constraining the loss function until the set number of iterations is reached, and the trained rain pattern feature prediction network is obtained. The rain streak feature prediction network comprises a residual transpose, a guided filtering layer and a smoothing convolution kernel connected in sequence; a parallel full convolution network, a subnetwork and a self-encoder with skip layer connection are arranged after the smoothing convolution kernel; Generating a rain streaks prediction map B J The process is as follows: S2.1, the rain map B is generated by residual transposition to generate a guided rain map B P ; the rain map B and the guided rain map B P are input into a guided filter layer to carry out a low-pass smoothing process, to obtain a low-frequency component B of the rain map L and a high-frequency component B H , wherein B H =B-B L ; S2.2, low frequency component B L and high frequency component B H After processing by a set of smoothing convolution kernels K, a set of components {(B L ) i} and {(B H ) i} are obtained; S2.3, extract rain streaks S from components {(B H ) i} using a fully convolutional network, estimate global atmospheric lighting A from components {(B L ) i} using a subnetwork, extract atmospheric transmission T from components {(B L ) i} and {(B H ) i} using an autoencoder with skip connections; S2.4, synthesize the rain streak map S, the global atmospheric lighting map A and the atmospheric transmission map T to obtain a rain streak estimation map B J ; Step 3: Obtain the rain streak prediction map B from Step 2. J The data is input into the rain density sensing classifier to obtain the rain density level label B for the rain map B in this iteration. d Then through validation set B vali The rain density sensing classifier is iteratively trained by constraining the loss function until the set number of iterations is reached, and a well-trained rain density sensing classifier is obtained. Step 4: Test set B test Rain image B is input into a pre-trained rain pattern feature prediction network to generate a predicted rain pattern image B. J Then, project the rain pattern prediction map B. J The input is fed into a trained rain density sensing classifier to obtain the rain density level label B. d Then label the rain density level B. d The input is fed into a rain ripple pixel kernel localization network based on the UNet semantic segmentation network to obtain test set B. test The pixel-by-pixel rain ripple kernel position vector {p(B)} in the rain map B. S )}; Step 5, using SNIP scale-adaptive range predictor to generate the rain streak coverage area corresponding to each rain streak pixel kernel based on the rain streak pixel kernel position vector {p(B S )} obtained in step 4; and superimposing all the rain streak coverage areas to obtain the total rain streak coverage area; The SNIP scale adaptive range predictor is composed of three scale image pyramids, each scale is composed of an RPN candidate box generator, an RCN region classification discriminator and a scale rescaling layer; Step 6, performing a non-local mean filtering operation on the rain streak overall coverage area obtained in step 5 to obtain a test set B test A rain-removed image of the rain map B.

2. The rain pattern decomposition based single image deraining method according to claim 1, characterized in that, In step 2, the full convolution network is composed of p basic residual blocks; the subnetwork is composed of q convolution blocks plus an activation function, plus two fully connected layers and one up-sampling layer.

3. The rain pattern decomposition based single image deraining method according to claim 1, characterized in that, Step S2.2 is specifically: processing the low-frequency component B L and the high-frequency component B H using a set of i smoothing convolution kernels K, in each smoothing convolution kernel channel, the low-frequency component B L and the high-frequency component B H are processed to obtain a set of components {(B L ) i} and {(B H ) i} through each smoothing convolution kernel K. The size k = 2 of the smoothing kernel K i .

4. The rain pattern decomposition based single image deraining method according to claim 1, characterized in that, In step S2.1, the calculation formula of the residual transpose process is: In formula (1), c and d each represent one of three color channels r, g, b; B c and B d represents the value of the c and d color channels of the rain map B at the pixel point x; In step S2.4, the rain pattern estimation map B J The calculation formula is: In formula (2), O denotes a clean image in which rain streaks are removed, v denotes a rain intensity, and Γ t denotes a rain streak layer at a rain intensity t, A denotes an atmospheric illumination map, T denotes an atmospheric transmission map, denotes a matrix multiplication operation; In step S2.3, the mean square error loss function is used for extracting the rain streak image S, the global atmospheric light image A and the atmospheric transmission image T, and the calculation formula is: In formula (3), represents the loss of extracting the feature map, Ω represents one of the rain streak image S, the global atmospheric illumination image A, and the atmospheric transmission image T, Ω gt represents the corresponding real rain image label; the formula of the MSE representing the mean square error loss function is: In formula (4), m and n are the height and width of the image, respectively, and X ij and Y ij respectively represent the gray values corresponding to the pixel point i, j of the reference image and the image to be measured. In step S2.4, the rain streaks prediction map B is synthesized J The loss function used is calculated by the formula: λ S , λ A , and λ T respectively represent the weight of each in the total loss function.

5. The rain pattern decomposition based single image deraining method according to claim 1, characterized in that, In step 3, the rain density perception classifier is based on a ResNet-34 residual classifier and is composed of three convolution layers, an average pooling layer and two fully connected layers connected in sequence, and the last layer has five neurons as the output of the rain density level of the output image; wherein the convolution kernel size of the three convolution layers is 3*3, which are Conv(3, 24)-Conv(24, 64)-Conv(64, 24); the convolution kernel size of the average pooling layer is 9*9; the two fully connected layers are FC(127896, 512)-FC(512, 5); In step 3, the calculation formula of the loss function used for training the rain density perception classifier is: Γ = Γ E，r + Γ c (6) Γ in equation (6) represents the per-pixel Euclidean loss for estimating the residual component; Γ E，r Γ in equation (6) represents the per-pixel Euclidean loss for estimating the residual component; Γ c Γ in equation (6) represents the per-pixel Euclidean loss for estimating the residual component; Γ 6. The rain pattern decomposition based single image deraining method according to claim 1, characterized in that, In step 4, the rain streak pixel kernel positioning network uses a multi-scale high-order collaborative network in the skip layer connection process to perform bidirectional mining and transmission of feature maps, and the feature transmission calculation formula is: In formula (7), X(i,j) represents the output of node X at position (i,j), wherein i is along the encoder index down-sampling layer, and j is along the convolution layer of the skip connection dense block; function H represents a convolution operation followed by an activation function; functions D and U represent down-sampling layers and up-sampling layers; [] represents a series connection layer; nodes of level j=0 only receive one input from the previous self-encoder layer; nodes of level j=1 receive two inputs from the encoder subnetwork, but on two consecutive levels; and nodes of level j>1 receive j+1 inputs, wherein the input of j is the output before the jth node in the same skip connection, and the j+1th input is the up-sampling output from the lower skip connection; the dense convolution block causes all previous feature maps to be accumulated along the skip layer connection and reach the current node.

7. The rain pattern decomposition based single image deraining method according to claim 1, characterized in that, In step 5, the SNIP scale adaptive range predictor has three scale image pyramids, each of which has its own RPN candidate box generator, and each predicts objects within a specified range; the RPN candidate box generator is responsible for generating multiple candidate boxes for each invalid real region, and the effectiveness of these candidate boxes is judged by the RCN region classification discriminator, so as to delete invalid boxes and retain valid boxes; the specific steps of the discrimination process are as follows: (5.1) Generate candidate boxes and judge positive and negative, that is, assign + / - labels to the generated candidate boxes with all real rain streak data; (5.2) delete the candidate box which is too large or too small, according to the effective range [S i ,E i ] under the i-th scale, whether the real rain streak data falls within the range of the candidate box is divided into effective real data or invalid real data; (5.3) delete the candidate boxes with inaccurate positioning, remove all candidate boxes with overlap degree IoU>0.3; The calculation formula of the overlap degree IoU is as follows:

8. The rain pattern decomposition based single image deraining method of claim 1, wherein, In step 6, the calculation formula of the non-local mean filtering is as follows: In formula (9), v (i) represents the gray value of the contaminated image at pixel point i, and the gray value estimated by the filtering operation is NL(v)(i). The coefficient w (i，j) represents the influence weight coefficient of pixel point j on pixel point i.

9. The rain pattern decomposition based single image deraining method of claim 1, wherein, The method also includes step 7: using the derained image obtained in step 6 and the corresponding test set B test The rain removal effect was evaluated by comparing rainless images in the dataset, specifically by using test set B. test The rainy image in step 1 to step 6, followed by the de-rained image, is used as the test image, and the test set B is used as the test set. test The rainless image corresponding to the rainy image is used as a reference image. The effectiveness of the single-image rain removal process is determined by calculating the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) of the test image and the corresponding reference image. The larger the values ​​of PSNR and SSIM, the smaller the difference between the obtained test image and the reference image, which proves that the rain removal effect is better. The calculation formula of the peak signal-to-noise ratio PSNR and the structural similarity SSIM is as follows: The calculation formula of the peak signal-to-noise ratio PSNR and the structural similarity SSIM is as follows: In formula (10), μ x and μ y respectively represent the mean of the test image and the reference image, δ x 2 and δ y 2 respectively represent the variance of the test image and the reference image, δ xy is the covariance between the two images, and c1 and c2 are fixed constants selected to avoid a denominator of zero.

Images