Region-aware iterative raindrop removal method

Abstract

This invention discloses a region-aware, iterative raindrop removal method, comprising: introducing multiple real raindrop-attached images, a corresponding clear image for each real raindrop-attached image, and a corresponding real raindrop mask image, and dividing them into a training set and a validation set; constructing a raindrop removal network model, which includes a cyclic raindrop region detection subnetwork and an image restoration encoding / decoding subnetwork; constructing a target loss function, training the raindrop removal network model using the training set, and validating it using the validation set to obtain the trained raindrop removal network model; inputting real-world raindrop-attached images into the above-mentioned raindrop removal network model, and sequentially passing them through the cyclic raindrop region detection subnetwork and the image restoration encoding / decoding subnetwork to obtain the predicted raindrop mask image and the raindrop removal result. This invention solves the problem in existing technologies of accurately locating and removing raindrops with large variations in size, shape, and transparency, and randomly distributed positions.

Description

Technical Field

[0001] This invention belongs to the technical field of deep learning image enhancement, specifically relating to a region-aware iterative raindrop removal method. Background Technology

[0002] Rain, as one of the most common severe weather conditions, can significantly reduce the performance of many outdoor vision systems originally designed for clear imaging. Raindrops adhering to camera lenses or vehicle windshields can cause severe blurring and occlusion of image background information. Single-image de-raining takes an image with raindrops attached as input and aims to remove the influence of raindrops on the image, restoring a clear background as much as possible.

[0003] Early image raindrop removal methods mostly focused on analyzing the physical properties of raindrops and the imaging process of light passing through them to detect and locate raindrops in images. Raindrops, typically elliptical in shape, can be viewed as fisheye lenses, blurring and magnifying the originally sharp background before image capture. Principal Component Analysis (PCA) and Discrete Wavelet Transform (DCT) were used to remove raindrops. These methods, based on different prior knowledge, remained at a low level in terms of image feature extraction and representation. Objects resembling raindrops in appearance were also mistakenly removed, limiting their applicability and robustness. In recent years, deep learning-based raindrop removal algorithms have been proposed, using convolutional neural networks to learn the end-to-end mapping relationship between raindrops attached to the image and their corresponding sharp background. In 2021, Zamir et al. pointed out that image restoration tasks need to ensure a balance between spatial detail information and high-level semantic information. From this perspective, they proposed the multi-stage progressive image restoration benchmark network MPRNet. However, this method lacks accurate localization of raindrop regions, and the raindrop removal results have some raindrop residue. It is difficult to handle raindrops with varying sizes and shapes and random locations in reality, and its generalization ability to real data is insufficient. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by providing a region-aware, iterative raindrop removal method. This method solves the problem in existing technologies of accurately locating and removing raindrops with large variations in size, shape, and transparency, and randomly distributed locations.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0006] A region-aware iterative raindrop removal method includes the following steps:

[0007] Step 1: Introduce multiple real raindrop attachment images, a real clear image corresponding to each real raindrop attachment image, and a real raindrop mask image corresponding to each real raindrop attachment image, and divide them into training set and validation set;

[0008] Step 2: Construct a de-raindrop network model, which includes a cyclic raindrop region detection sub-network and an image restoration encoding / decoding sub-network. The cyclic raindrop region detection sub-network is used to concatenate each real raindrop-attached image with the previous de-raindrop result to locate the raindrop degradation region, obtain the predicted raindrop mask image corresponding to each real raindrop-attached image, and generate a raindrop detection loss by combining the real raindrop mask image. The image restoration encoding / decoding sub-network is used to concatenate each real raindrop-attached image, the previous de-raindrop result, and the predicted raindrop mask image to perform de-raindrop processing, obtain the de-raindrop result corresponding to each real raindrop-attached image, and generate an image restoration loss by combining the real clear image.

[0009] Step 3: Construct a target loss function based on raindrop detection loss and image restoration loss. Use the training set from Step 1 to train the raindrop removal network model to optimize the target loss function and obtain the optimized model parameters. Validate the model using the validation set to finally obtain the trained raindrop removal network model.

[0010] Step 4: Input the real raindrop-attached image into the trained raindrop removal network model obtained in Step 3. The model will then pass through the cyclic raindrop region detection subnetwork and the image restoration encoding / decoding subnetwork in sequence, and perform forward propagation to obtain the predicted raindrop mask image and the raindrop removal result.

[0011] Further, the multiple real raindrop attachment images mentioned in step 1 are defined as:

[0012] {I k ,k∈[1,K]}

[0013] Among them, I k It is the kth real raindrop attachment image, where K represents the number of real raindrop attachment images;

[0014] The real, clear image corresponding to the multiple real raindrop attachment images in step 1 is defined as:

[0015]

[0016] in, It is the real, clear image corresponding to the kth real raindrop attachment image, where K represents the number of real raindrop attachment images;

[0017] The real raindrop mask image for each real raindrop attachment image in step 1 is defined as:

[0018]

[0019] in, It is the real raindrop mask image corresponding to the k-th real raindrop attached image, which is constructed by the absolute value of the residual between the real raindrop attached image and the corresponding real clear image, and K represents the number of synthesized rainy day images.

[0020] Furthermore, the recurrent raindrop region detection subnetwork includes a feature extraction layer, an SE channel attention residual module, an LSTM unit, and a convolutional layer;

[0021] Each real raindrop-attached image is concatenated with the previous de-raindrop result and input into the feature extraction layer to extract features and obtain a feature map. The feature map is then input into the SE channel attention residual module to reweight each feature channel according to its importance to obtain a new feature map, which is then input into the LSTM unit. The LSTM unit integrates the effective feature information and establishes long-term dependencies before sending it to the convolutional layer for prediction to obtain the predicted raindrop mask image corresponding to each real raindrop-attached image after the i-th iteration.

[0022] Furthermore, the SE channel attention residual module includes two regular convolutional layers, a global average pooling layer, and two fully connected layers. In the SE channel attention residual module, the input feature map first passes through two regular convolutional layers to extract features, and then passes through a global average pooling layer to integrate the feature maps of each channel into a value that reflects global feature information. Next, two fully connected layers perform channel reduction and channel expansion respectively, and generate corresponding importance weights for each feature channel. After obtaining the weight vector representing the correlation between channels, it is multiplied with the input feature map in the channel dimension. Finally, the features recalibrated by the SE channel attention residual module are added to the input features through a global residual connection to obtain the output feature map.

[0023] Furthermore, the method for constructing the raindrop detection loss function in step 2 is as follows:

[0024] Image I with real raindrops attached k Compared with the previous round of raindrop results The cascaded subnetwork is used as the input to the cyclic raindrop region detection subnetwork, where i∈[1,T] and T is the total number of iterations of the raindrop removal network. If there is no restoration result in the previous round in the initial iteration, the original raindrop attachment image is directly copied.

[0025] After iteration, the prediction result of the recurrent raindrop region detection subnetwork is the predicted raindrop mask image of the k-th real raindrop attachment image:

[0026]

[0027] Where K represents the number of composite rainy day images. It is the predicted raindrop mask image corresponding to the k-th real raindrop attachment image after the i-th iteration;

[0028] The raindrop detection loss function is:

[0029]

[0030] Where K represents the number of composite rainy day images. It is the predicted raindrop mask image corresponding to the k-th real raindrop attachment image after the i-th iteration. It is the real raindrop mask image corresponding to the k-th real raindrop attachment.

[0031] Furthermore, the image restoration encoding / decoding subnetwork includes a feature extraction module, a CBAM hybrid attention residual module, and a recurrent attention SAM module;

[0032] Each real raindrop-attached image, the previous round of raindrop removal results, and the predicted raindrop mask image are concatenated and input into the feature extraction module for feature extraction. The results are then input into the CBAM hybrid attention residual module. The CBAM hybrid attention residual module fuses the channel attention and spatial attention of the features and finally outputs a feature map. The recurrent attention SAM module takes the feature map as input to generate the current round of raindrop removal results and the recurrent attention feature map.

[0033] Furthermore, the CBAM residual module includes a convolutional layer, a channel attention module, and a spatial attention module. The features F extracted by the feature extraction module first pass through the convolutional layer, and then sequentially through the channel attention module and the spatial attention module. After passing through the channel attention module, a feature map F2 is obtained after being recalibrated in the channel dimension according to its importance, and it serves as the input to the subsequent spatial attention module. In the spatial attention module, the feature values ​​of the same pixel in each feature channel are integrated into a single value, and then multiplied point-to-point with the feature map F2 in the spatial dimension. Finally, the features recalibrated in both the channel dimension and the spatial dimension are added to the input feature F through a global residual connection to obtain the output feature map.

[0034] Furthermore, the image restoration loss is composed of a weighted sum of the sharp image reconstruction loss, the raindrop-attached image reconstruction loss, and the multi-scale structural similarity loss;

[0035] The loss for sharp image reconstruction is:

[0036]

[0037] The image reconstruction loss for raindrop attachment is:

[0038]

[0039] The multi-scale structural similarity loss is:

[0040]

[0041] The image restoration loss is:

[0042]

[0043] Where K represents the number of composite rainy day images. It is the predicted de-dropping result corresponding to the k-th real raindrop-attached image after the i-th iteration. This is the real, clear image corresponding to the k-th real raindrop attachment image. The result of predicting raindrops With predicted raindrop mask image The reconstructed image of raindrop attachment. L char MS_ represents Charbonnier loss, while MS_ represents multi-scale structural similarity calculation. c_rec , λ r_rec , λ MS_ These correspond to the weights for the clear image reconstruction loss, the raindrop-attached image reconstruction loss, and the multi-scale structural similarity loss, respectively.

[0044] Furthermore, in step 3, the target loss function loss total The result is obtained by weighted summation of the raindrop detection loss and image restoration loss in each iteration:

[0045]

[0046] Where, λ mask The weights corresponding to the raindrop detection loss are... It is the raindrop detection loss in the i-th iteration. It is the image restoration loss in the i-th iteration.

[0047] Furthermore, step 4, which involves using the trained de-raindrop network model to remove raindrops from the real raindrop-attached image, is as follows:

[0048] Input the real raindrop-attached image R into the de-raindrop network model trained in step 3 to predict the predicted raindrop mask image M corresponding to the real raindrop-attached image in each round. i C is the predicted de-dropping result for each round corresponding to the real raindrop attachment image. i :

[0049] M i =ε * (R,C i-1 )

[0050] C i =ρ * (R,Ci-1 M i )

[0051] Where i represents the i-th iteration, i∈[1,T], and T is the total number of iterations of the raindrop removal network. The predicted raindrop mask image M obtained from the final T-th iteration is... T And the predicted raindrop results C T As the result of raindrop detection and removal in the raindrop removal network.

[0052] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0053] This invention constructs a raindrop removal network model, comprising a cyclic raindrop region detection subnetwork and an image restoration encoding / decoding subnetwork. It progressively removes the influence of raindrops and restores a clear background through multiple iterative iterations. The cyclic raindrop region detection subnetwork employs an SE channel attention residual module and an LSTM unit, ensuring that each detection round is no longer independent. Important and effective features extracted from previous rounds are passed to the current round through hidden states. The image restoration encoding / decoding network uses a CBAM hybrid attention residual module and a cyclic attention SAM module, enabling the network to selectively utilize feature channels that are more effective in raindrop localization and removal, discarding unimportant features, and simultaneously emphasizing the original raindrops and regions rich in background detail in the pixel space dimension. This invention solves the problem in existing technologies of accurately locating and removing raindrops with varying shapes and sizes, large ranges of transparency, and random distribution, effectively restoring a clear and efficient background in complex and varied real-world raindrop-attached scenes. Attached Figure Description

[0054] Figure 1 This is a flowchart of the region-aware iterative raindrop removal method according to an embodiment of the present invention;

[0055] Figure 2 This is a structural diagram of the raindrop removal network model in an embodiment of the present invention;

[0056] Figure 3 This is a schematic diagram of the SE residual module structure in the cyclic raindrop detection subnetwork of this invention.

[0057] Figure 4 This is a schematic diagram of the CBAM residual module structure in the image restoration encoding / decoding subnetwork of this invention.

[0058] Figure 5 This is a schematic diagram of the recurrent attention (SAM) module structure in the image restoration encoding / decoding subnetwork of this invention.

[0059] Figure 6This is an example of raindrop detection masking results for a real raindrop attached image in an embodiment of the present invention, wherein (a) is a real raindrop attached image and (b) is the corresponding predicted raindrop masking image;

[0060] Figure 7 This is an example of the de-raindrop result for a real raindrop-attached image in an embodiment of the present invention. Detailed Implementation

[0061] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0062] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0063] The present invention will be further described below with reference to specific embodiments, but these are not intended to limit the scope of the invention.

[0064] like Figure 1 As shown, this embodiment of the invention provides a region-aware iterative raindrop removal method, characterized by comprising the following steps:

[0065] Step 1: Introduce multiple real raindrop attachment images, a real clear image corresponding to each real raindrop attachment image, and a real raindrop mask image corresponding to each real raindrop attachment image, and divide them into training set and validation set;

[0066] In this step, multiple images of real raindrop attachment can be defined as:

[0067] {I k ,k∈[1,K]};

[0068] Among them, I k It is the kth real raindrop attachment image, where K represents the number of real raindrop attachment images;

[0069] The true, clear image corresponding to multiple images of real raindrop attachment can be defined as:

[0070]

[0071] in, It is the real, clear image corresponding to the kth real raindrop attachment image, where K represents the number of real raindrop attachment images;

[0072] The real raindrop mask image for each real raindrop attachment image can be defined as:

[0073]

[0074] in, It is the real raindrop mask image corresponding to the k-th real raindrop attached image, which is constructed by the absolute value of the residual between the real raindrop attached image and the corresponding real clear image. Its pixel value indicates the severity of the raindrop degradation. The larger the value, the more severe the degradation in the area. K represents the number of synthesized rainy day images. In this embodiment, K can be 1722.

[0075] Step 2: Construct a raindrop removal network model, such as Figure 2 As shown, the de-raindrop network model includes a cyclic raindrop region detection subnetwork and an image restoration encoding / decoding subnetwork. The cyclic raindrop region detection subnetwork is used to predict the corresponding predicted raindrop mask image for each real raindrop-attached image and combine it with the real raindrop mask image to generate a raindrop detection loss. The image restoration encoding / decoding subnetwork is used to de-raindrop process each real raindrop-attached image to obtain the corresponding de-raindrop result and combine it with the real clear image to generate an image restoration loss.

[0076] In the recurrent raindrop region detection subnetwork, each real raindrop-attached image is concatenated with the previous de-dredging result and then input into the recurrent raindrop region detection network (if there is no restoration result from the previous round in the initial iteration, the original raindrop-attached image is directly copied). Raindrop degradation regions are located to obtain the predicted raindrop mask image corresponding to each real raindrop-attached image. The raindrop detection loss is constructed by combining the real raindrop mask images corresponding to each real raindrop-attached image. The recurrent raindrop region detection subnetwork includes a feature extraction layer, an SE channel attention residual module, an LSTM unit, and a convolutional layer. Each real raindrop-attached image is concatenated with the previous de-dredging result and then input into the feature extraction layer (a convolutional layer in this embodiment) for feature extraction to obtain a feature map, which is then input into the SE channel attention residual module. See [link to documentation]. Figure 3The SE channel attention residual module first extracts features from the input feature map through two regular convolutional layers, then integrates the feature maps of each channel into a single value that reflects global feature information through a global average pooling layer. Next, two fully connected layers perform channel reduction and expansion respectively, realizing channel rearrangement and correlation calculation, and generating corresponding importance weights for each feature channel. This weight vector, representing the correlation between channels, is then multiplied with the input feature map along the channel dimension. Finally, the features recalibrated by the SE channel attention residual module are added to the input features through a global residual connection to obtain the output feature map. The SE channel attention residual module integrates the SE channel attention mechanism and residual connections, thus inheriting the advantages of both network structures. It can alleviate gradient vanishing and model overfitting problems, and explicitly model the interdependence between feature channels, reweighting each feature channel according to its importance. This allows the network to selectively utilize feature channels that are more effective for raindrop localization and removal tasks.

[0077] Feature map F output by SE channel attention residual module t The hidden state h given by the LSTM unit in the previous iteration t-1 The channels are cascaded as inputs to the current LSTM unit, and the calculation is performed using the following formula:

[0078] i t =(Conv(F t ))

[0079] f t =(Conv(F t ))

[0080] g t =anh(Conv(F t ))

[0081] o t =(Conv(F t ))

[0082] c t = t ⊙ t-1 + t ⊙ t

[0083] h t = t ⊙(c t )

[0084]

[0085] The LSTM unit consists of an input gate, a forget gate, and an output gate. The input gate includes a feature extraction unit composed of convolutional layers and sigmoid activation layers, and a gating unit composed of convolutional layers and tanh activation layers, which respectively calculate the input feature i. t And the gating feature g that selectively controls the passage of input information. t The forget gate also consists of convolutional layers and sigmoid layers, providing a gating feature f for selectively retrieving and preserving cell state information. t The output gate structure is the same as the forget gate, yielding the output feature o. t The current cell state c is obtained by weighting the current input features and the cell state from the previous round using the corresponding gating features. t By activating the cell state through a ReLU layer and then weighting the output features, the hidden state h of the current LSTM unit can be obtained. t The cell state and hidden state integrate the effective feature information from all previous iterations, and are therefore passed to the LSTM unit in the next iteration to establish long-term dependencies.

[0086] The hidden layer information output from the LSTM unit is fed into the next two convolutional layers to obtain the prediction result of the recurrent raindrop region detection subnetwork, which is the raindrop mask image of the k-th real raindrop attachment image:

[0087]

[0088] In the formula, T is the total number of iterations of the raindrop removal network, and K represents the number of synthesized rainy day images;

[0089] Among them, the parameter to be optimized in the e-th convolutional layer of the cyclic raindrop region detection subnetwork is ε. e In this embodiment, e∈[1,17]; and the raindrop detection loss function for constructing the cyclic raindrop region detection subnetwork is:

[0090]

[0091] Where K represents the number of composite rainy day images. It is the predicted raindrop mask image corresponding to the k-th real raindrop attachment image after the i-th iteration. It is the real raindrop mask image corresponding to the k-th real raindrop attachment.

[0092] In the image restoration encoding / decoding sub-network, each real raindrop-attached image, the previous round's de-dredging result, and the predicted raindrop mask image are concatenated and input into the image restoration encoding / decoding network (in the initial iteration, if there is no restoration result from the previous round, the original raindrop-attached image is directly copied). De-dredging is performed to obtain the de-dredging result corresponding to each real raindrop-attached image. The image restoration loss is constructed by combining each real raindrop-attached image and its corresponding real clear image. The image restoration encoding / decoding sub-network includes a feature extraction module, a CBAM hybrid attention residual module, and a recurrent attention SAM module. Each real raindrop-attached image, the previous round's de-dredging result, and the predicted raindrop mask image are concatenated and input into the feature extraction module (in this embodiment, a convolutional layer) for feature extraction to obtain the feature map F. For example... Figure 4 As shown, in the CBAM hybrid attention residual module, the input feature map F first passes through two regular convolutional layers, then sequentially through the channel attention module and the spatial attention module. The channel attention module differs from the SE channel attention residual module in that it includes two branches: global average pooling and global max pooling, integrating global information from both the mean and maximum values ​​of the feature map. After passing through the channel attention module, a feature map F2, recalibrated according to importance in the channel dimension, is obtained and serves as the input to the subsequent spatial attention module. In the spatial attention module, the calibrated feature map F2 first passes through two branches: average pooling and max pooling in each channel, integrating the feature values ​​of the same pixel in each feature channel into a single value, resulting in two single-channel feature maps. These are then concatenated and multiplied point-to-point with feature map F2 in the spatial dimension. Finally, the recalibrated features in both the channel and spatial dimensions are added to the input feature F through a global residual connection to obtain the output feature map. The CBAM hybrid attention residual module is used to fuse channel attention and spatial attention, making the encoder focus more on feature extraction of raindrop regions and edge regions containing background details, while emphasizing feature channels that play a key role in the raindrop removal task.

[0093] See Figure 5The Recurrent Attention (SAM) module comprises two steps: generating the current round of raindrop removal results and generating a recurrent attention feature map. The feature map obtained after passing through the CBAM hybrid attention residual module is input into the convolutional layer of the SAM module, and finally, a global residual connection is performed with the real raindrop-attached image to obtain the current round of raindrop removal results. Within the SAM module, the current raindrop removal image is further processed through convolutional layers and a sigmoid activation layer to obtain an attention weight map. This attention weight map is used to recalibrate the feature map obtained after the CBAM residual module, resulting in an attention feature map used to guide the next round of raindrop removal iterations. The SAM module serves two purposes: 1) it uses the current raindrop removal result as the ground truth of the pseudo-sharp image, directly providing supervised explicit attention guidance for the next round of raindrop removal; 2) it generates an attention map to suppress features with less information and lower importance in the current stage, allowing only useful features to propagate to the next stage.

[0094] The prediction result of the image restoration encoding / decoding subnetwork is the de-raindrop result of the k-th real raindrop-attached image:

[0095]

[0096] Image I with real raindrops attached k The results of the previous round of raindrop analysis Predicting raindrop mask images The cascade is used as input, where i∈[1,T], T=3 represents the total number of iterations of the raindrop removal network, and the parameters to be optimized in the e-th convolutional layer of the image restoration encoding / decoding sub-network are ρ. e , e∈[1,30].

[0097] The image restoration loss is composed of a weighted sum of the sharp image reconstruction loss, the raindrop-attached image reconstruction loss, and the multi-scale structural similarity loss;

[0098] The sharp image reconstruction loss is defined as:

[0099]

[0100] The image reconstruction loss for raindrop attachment is defined as:

[0101]

[0102] Multi-scale structural similarity loss is defined as:

[0103]

[0104] Image restoration loss is defined as:

[0105]

[0106] In this embodiment, K = 1722 represents the number of synthesized rainy day images. It is the predicted de-dropping result corresponding to the k-th real raindrop-attached image after the i-th iteration. The result of predicting raindrops With predicted raindrop mask image The reconstructed image of raindrop attachment. L char The Charbonnier loss is often used to approximate L1 loss in image generation tasks, while MS_ represents multi-scale structural similarity calculation. c_rec =0.4, λ r_rec =0.1, λ MS_ =0.4 corresponds to the weights of the sharp image reconstruction loss, the raindrop attached image reconstruction loss, and the multi-scale structural similarity loss, respectively.

[0107] Step 3: Construct a target loss function based on raindrop detection loss and image restoration loss. Use the training set from Step 1 to train the raindrop removal network model to optimize the target loss function and obtain the optimized model parameters. Validate the model using the validation set to finally obtain the trained raindrop removal network model.

[0108] In this step, the raindrop detection loss and image restoration loss in each iteration are weighted and summed to obtain the target loss function. The network parameters are then updated via gradient backpropagation using the ADAM optimizer to obtain the trained raindrop removal network model. The construction of the overall objective function and network training of the raindrop removal network include:

[0109] The target loss function is obtained by weighted summation of the raindrop detection loss and image restoration loss in each iteration, and is defined as:

[0110]

[0111] Where, λ mask =0.2 is the weight corresponding to the raindrop detection loss. It is the raindrop detection loss in the i-th iteration. It is the image restoration loss in the i-th iteration, where i∈[1,T], and T=3 represents the total number of iterations of the raindrop removal network;

[0112] Using the real raindrop-attached images from the training set in Step 1 as input data, and combining the corresponding clear image and the real raindrop mask image for each real raindrop-attached image in the training set, the Adam optimization algorithm (i.e., stochastic gradient descent algorithm) is used to optimize the target loss function of the raindrop removal network model, resulting in the optimized parameters for the recurrent raindrop detection subnetwork. And optimized parameters of the image restoration encoding / decoding subnetwork The optimized parameters are then validated using the validation set from step 1 to obtain the trained raindrop removal network model.

[0113] Step 4: Input the real raindrop-attached image into the trained raindrop removal network model obtained in Step 3. The model will then pass through the recurrent raindrop region detection network and the image restoration encoding and decoding network in sequence, and perform forward propagation to obtain the predicted raindrop mask image and the raindrop removal result.

[0114] In this step, the process of removing raindrops from the real raindrop-attached image using the trained raindrop removal network model is as follows:

[0115] Input the real raindrop-attached image R into the de-raindrop network model trained in step 3 to predict the predicted raindrop mask image M for each round corresponding to the real raindrop-attached image. i C is the predicted de-dropping result for each round corresponding to the real raindrop attachment image. i :

[0116] M i =ε * (R,C i-1 )

[0117] C i =ρ * (R,C i-1 M i )

[0118] Where i represents the i-th iteration, i∈[1,T], T=3 (the value in this embodiment, but not limited to this), and T=3 represents the total number of iterations of the raindrop removal network. The predicted raindrop mask image M obtained from the final T-th iteration is then used. T And the predicted raindrop results C T As the result of raindrop detection and removal in the raindrop removal network.

[0119] To evaluate the effectiveness and generalization of the method in this embodiment for raindrop removal from single images, the raindrop removal performance of this embodiment is quantitatively compared with that of MPRNet, a representative algorithm for single-image restoration. Visual examples of raindrop detection and removal results on real data are provided. Experiments were conducted on a public test set and real raindrop-attached images. The public test set includes 58 pairs of test images. This embodiment uses two objective evaluation metrics, PSNR and SSIM, to compare and analyze the raindrop removal performance of this embodiment and the existing MPRNet method on the public test dataset. Table 1 shows the average PSNR and SSIM of the two algorithms. Compared with the existing MPRNet raindrop removal method, it can be seen that the PSNR of this embodiment is improved by 0.09 dB, and the SSIM is improved by 0.0015. Figure 6This embodiment demonstrates a raindrop detection mask image on a real raindrop attachment image. Figure 7 This demonstrates the visualization results of raindrop removal on a real raindrop-attached image in this embodiment. It can be seen that the method in this embodiment can effectively detect areas with raindrops in the image and restore background details. The color fidelity is high and it is very close to the real clear image. It can accurately locate and remove raindrops with large size range and random distribution. It has good generalization and robustness in real scenes.

[0120] Table 1 Comparison of raindrop removal effects between this embodiment and MPRNet

[0121] PSNR SSIM MPRNet 27.47 0.8571 Method of this embodiment 27.56 0.8586

[0122] In summary, this embodiment addresses the problem of raindrop removal from single images by accurately locating raindrop degradation regions to guide image restoration. It proposes a region-aware, iterative image raindrop removal algorithm, which introduces a visual attention mechanism to enable the network to selectively process different feature channels and spatial regions. Through multiple iterations, it accurately detects and effectively removes raindrops with varying shapes and sizes and random locations, demonstrating good robustness and generalization on real data.

[0123] The above are merely preferred embodiments of the present invention and are not intended to limit the implementation methods and protection scope of the present invention. Those skilled in the art should recognize that any equivalent substitutions and obvious changes made based on the content of this specification should be included within the protection scope of the present invention.

Claims

1. A region-aware iterative raindrop removal method, characterized in that, Includes the following steps: Step 1: Introduce multiple real raindrop attachment images, a real clear image corresponding to each real raindrop attachment image, and a real raindrop mask image corresponding to each real raindrop attachment image, and divide them into training set and validation set; Step 2: Construct a de-raindrop network model, which includes a cyclic raindrop region detection sub-network and an image restoration encoding / decoding sub-network. The cyclic raindrop region detection sub-network is used to concatenate each real raindrop-attached image with the previous de-raindrop result to locate the raindrop degradation region, obtain the predicted raindrop mask image corresponding to each real raindrop-attached image, and generate a raindrop detection loss by combining the real raindrop mask image. The image restoration encoding / decoding sub-network is used to concatenate each real raindrop-attached image, the previous de-raindrop result, and the predicted raindrop mask image to perform de-raindrop processing, obtain the de-raindrop result corresponding to each real raindrop-attached image, and generate an image restoration loss by combining the real clear image. Step 3: Construct a target loss function based on raindrop detection loss and image restoration loss. Use the training set from Step 1 to train the raindrop removal network model to optimize the target loss function and obtain the optimized model parameters. Validate the model using the validation set to finally obtain the trained raindrop removal network model. The target loss function The result is obtained by weighted summation of the raindrop detection loss and image restoration loss in each iteration: in, The weights corresponding to the raindrop detection loss are... It is the raindrop detection loss in the i-th iteration. It is the image restoration loss in the i-th iteration; Step 4: Input the real-collected raindrop-attached image into the trained de-raindrop network model obtained in Step 3. The model passes sequentially through the recurrent raindrop region detection sub-network and the image restoration encoding / decoding sub-network, performing forward propagation to obtain the predicted raindrop mask image and the de-raindrop result, including: Attach real raindrops to the image Input the de-raindrop network model trained in step 3 to predict the predicted raindrop mask image for each round corresponding to the real raindrop attachment image. The predicted deraining results for each round corresponding to the real raindrop attachment image. : Where i represents the i-th iteration, i∈[1,T], and T is the total number of iterations of the raindrop removal network. The predicted raindrop mask image obtained in the final T-th iteration is used. And predicting raindrop results As the result of raindrop detection and removal in the raindrop removal network The optimized parameters for the cyclic raindrop region detection subnetwork are as follows: These are the optimized parameters for the image restoration encoding / decoding subnetwork.

2. The region-aware iterative raindrop removal method according to claim 1, characterized in that, The multiple real raindrop attachment images mentioned in step 1 are defined as follows: in, It is the kth real raindrop attachment image, where K represents the number of real raindrop attachment images; The real, clear image corresponding to the multiple real raindrop attachment images in step 1 is defined as: in, It is the real, clear image corresponding to the kth real raindrop attachment image, where K represents the number of real raindrop attachment images; The real raindrop mask image for each real raindrop attachment image in step 1 is defined as: in, It is the real raindrop mask image corresponding to the k-th real raindrop attached image, which is constructed by the absolute value of the residual between the real raindrop attached image and the corresponding real clear image, and K represents the number of synthesized rainy day images.

3. The region-aware iterative raindrop removal method according to claim 1, characterized in that, The recurring raindrop region detection subnetwork includes a feature extraction layer, an SE channel attention residual module, an LSTM unit, and a convolutional layer; Each real raindrop-attached image is concatenated with the previous de-raindrop result and input into the feature extraction layer to extract features and obtain a feature map. The feature map is then input into the SE channel attention residual module to reweight each feature channel according to its importance to obtain a new feature map, which is then input into the LSTM unit. The LSTM unit integrates the effective feature information and establishes long-term dependencies before sending it to the convolutional layer for prediction to obtain the predicted raindrop mask image corresponding to each real raindrop-attached image after the i-th iteration.

4. The region-aware iterative raindrop removal method according to claim 3, characterized in that, The SE channel attention residual module consists of two regular convolutional layers, a global average pooling layer, and two fully connected layers. In the SE channel attention residual module, the input feature map first passes through two regular convolutional layers to extract features, and then passes through a global average pooling layer to integrate the feature maps of each channel into a numerical value that reflects global feature information. Next, two fully connected layers perform channel reduction and channel expansion respectively, and generate corresponding importance weights for each feature channel. After obtaining the weight vector representing the correlation between channels, it is multiplied with the input feature map in the channel dimension. Finally, the features recalibrated by the SE channel attention residual module are added to the input features through a global residual connection to obtain the output feature map.

5. The region-aware iterative raindrop removal method according to claim 1, characterized in that, The method for constructing the raindrop detection loss function in step 2 is as follows: Images with real raindrops attached Compared with the previous round of raindrop results The cascaded subnetwork is used as the input to the cyclic raindrop region detection subnetwork, where i∈[1,T] and T is the total number of iterations of the raindrop removal network. If there is no restoration result in the previous round in the initial iteration, the original raindrop attachment image is directly copied. After iteration, the prediction result of the recurrent raindrop region detection subnetwork is the predicted raindrop mask image of the k-th real raindrop attachment image: , ； Where K represents the number of composite rainy day images. It is the predicted raindrop mask image corresponding to the k-th real raindrop attachment image after the i-th iteration; The raindrop detection loss function is: Where K represents the number of composite rainy day images. It is the predicted raindrop mask image corresponding to the k-th real raindrop attachment image after the i-th iteration. It is the real raindrop mask image corresponding to the k-th real raindrop attachment.

6. The region-aware iterative raindrop removal method according to claim 1, characterized in that, The image restoration encoding / decoding subnetwork includes a feature extraction module, a CBAM hybrid attention residual module, and a recurrent attention SAM module; Each real raindrop-attached image, the previous round of raindrop removal results, and the predicted raindrop mask image are concatenated and input into the feature extraction module for feature extraction. The results are then input into the CBAM hybrid attention residual module. The CBAM hybrid attention residual module fuses the channel attention and spatial attention of the features and finally outputs a feature map. The recurrent attention SAM module uses the feature map as input to generate the current round of raindrop removal results and the recurrent attention feature map.

7. The region-aware iterative raindrop removal method according to claim 6, characterized in that, The CBAM residual module includes convolutional layers, channel attention modules, and spatial attention modules. The features F extracted by the feature extraction module first pass through the convolutional layer, and then sequentially through the channel attention module and the spatial attention module. After passing through the channel attention module, a feature map F2 is obtained after being recalibrated according to the importance of the channels, and it is used as the input to the subsequent spatial attention module. In the spatial attention module, the feature values ​​of the same pixel in each feature channel are integrated into a single value, and then multiplied point-to-point with the feature map F2 in the spatial dimension. Finally, the features recalibrated in both the channel and spatial dimensions are added to the input features F through a global residual connection to obtain the output feature map.

8. The region-aware iterative raindrop removal method according to claim 1, characterized in that, The image restoration loss is composed of a weighted sum of the sharp image reconstruction loss, the raindrop-attached image reconstruction loss, and the multi-scale structural similarity loss; The loss for sharp image reconstruction is: The image reconstruction loss for raindrop attachment is: The multi-scale structural similarity loss is: The image restoration loss is: Where K represents the number of composite rainy day images. It is the predicted de-dropping result corresponding to the k-th real raindrop-attached image after the i-th iteration. This is the real, clear image corresponding to the k-th real raindrop attachment image. This is the k-th real image of raindrop attachment. The result of predicting raindrops With predicted raindrop mask image The reconstructed image of raindrop attachment. , Indicates Charbonnier loss. This indicates multi-scale structural similarity calculation. , , These represent the weights corresponding to the clear image reconstruction loss, the raindrop-attached image reconstruction loss, and the multi-scale structural similarity loss, respectively.

Images