Image rain removal restoration method

Abstract

The invention discloses an image rain removal and restoration method, which comprises a rain imprint enhancement module and a rain removal and restoration module which are connected in series, and comprises the following steps: carrying out initialization processing on an input rain picture Input and then carrying out multiple iterations, and restoring each iteration through one rain imprint enhancement module and one rain removal and restoration module, for the estimation of the background layer in each iteration, the input rain picture Input is used for subtracting the rain imprint information R (S) updated in the iteration, and the background picture of the iteration is preliminarily recovered; and taking the preliminarily recovered background picture as the input of the rain removal and restoration module for processing again, and finally outputting the final restored picture of the iteration. According to the method, the robustness aiming at different rain imprint changes can be enhanced, the stability of the processing effect is improved, and artifacts and blurring phenomena on the edge part are avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of image rain removal, and in particular to an image rain removal and restoration method. Background Art

[0002] Image rain removal methods are divided into methods based on prior knowledge and methods based on deep learning. The former uses traditional statistical methods, relies on the morphological features of rain streaks (such as direction, intensity, and density) for modeling, and separates rain streaks and background information through filters, sparse representation, or low-rank matrix decomposition, thereby achieving rain removal. The latter uses deep learning algorithms to learn the complex features of rain streaks and backgrounds from large-scale data, and predicts rain-free images through training networks or directly learns the features of rain streaks for removal.

[0003] The relatively classic image rain removal methods proposed based on prior knowledge are mainly implemented based on sparse coding. The basic idea is to decompose a rainy image into a rain streak layer and a background layer, and identify and remove rain streaks through dictionary learning or sparse representation. However, for this rain removal method that does not rely on a large amount of training data, obvious rain streaks usually remain in the restoration results. Recently, image rain removal methods based on deep learning have achieved relatively advanced results. Some scholars have proposed a new deep network architecture based on deep convolutional neural networks and recurrent neural networks, which decomposes the rain removal process into multiple stages for processing. Its disadvantage is that a single recursive structure is difficult to completely eliminate all rain streaks. Other scholars have proposed a multi-scale progressive fusion strategy, which uses recursive calculations to capture global textures, constructs a multi-scale pyramid structure, and introduces an attention mechanism to guide the fusion of relevant information. Its disadvantage is that it may misinterpret some background details as rain streaks and remove them, resulting in background blur. Some scholars have also proposed a multi-stage architecture, which uses an encoder-decoder structure to learn context features, introduces a pixel-by-pixel adaptive design to reweight local features, and has advantages in multi-task learning. However, its disadvantage is that the processing accuracy for a single task is relatively low.

[0004] In traditional methods, due to the limitations of their network structures, the generalization ability of the model may be insufficient under some extreme weather conditions or special types of rain streaks. Secondly, single convolutions are mostly used in the network structure, and they cannot extract feature information well, especially for object shadows and light refractions that already exist in the image. Therefore, further research is still needed in terms of detail restoration.

[0005] At present, the performance of many applications in the field of computer vision, such as object detection, image segmentation, and face recognition, significantly degrades in rainy days. De-raining techniques can improve the accuracy of these applications. For example, in autonomous driving and driverless systems, sensors and cameras are often affected by rainy days. Through de-raining techniques, the accuracy of environmental perception can be improved to ensure safe driving. Intelligent monitoring systems may encounter problems such as low visibility and blurred image quality in rainy days. De-raining techniques can help improve the quality of monitoring images and enhance the ability to detect and identify events. Therefore, it is of great significance to design an image de-raining algorithm for the above situation. Summary of the Invention

[0006] To solve the problems existing in the prior art, the object of the present invention is to provide an image de-raining and restoration method, which can enhance the robustness against different rain streak changes, improve the stability of the processing effect, and avoid artifacts and blurring at the edge part.

[0007] To achieve the above object, the technical solution adopted by the present invention is: an image de-raining and restoration method, including a rain streak enhancement module and a de-raining and restoration module connected in series. The method includes the following steps:

[0008] Step 1: The input rainy image Input first performs an initialization operation, and filters out the rain streak information Z through a convolution operation with a predefined rain kernel C. 00 , and then Input and Z 00 are concatenated in channels and input into the de-raining and restoration module. The de-raining and restoration module learns the features of the rain streak information Z 00 , re-integrates the channels of the concatenated image, and first separates the background information B 0 and the rain streak information Z 0 . The background information B 0 contains the extracted RGB channel information, and the remaining channel information is used as the rain streak information Z 0 . After the initialization operation, the obtained Z 0 information has richer rain streak feature information compared to Z 00 ;

[0009] Step 2: The input rainy picture Input is iterated multiple times after the initialization operation, and each iteration passes through a rain streak enhancement module and a de-raining and restoration module for restoration. For each iteration, the estimation of the background layer uses the input rainy picture Input minus the updated rain streak information R (S) in this iteration to preliminarily restore the background picture of this iteration;

[0010] Step 3: Use the preliminarily restored background image as the input of the rain removal and restoration module for further processing, and finally output the final restored image of this iteration. Determine whether to perform the next iteration by judging whether the final restored image of the iteration converges (i.e., achieves the optimal effect) during the training process of this dataset. After the training effect converges, output the restoration result Output.

[0011] As a further improvement of the present invention, step 2 specifically includes the following steps:

[0012] Step 2.1: Regard the input rainy image Input as a combination of a rain-free image B and rain streaks R_hat, and their relationship is expressed as shown in formula (1):

[0013] R_hat (S) = Input - B (S-1) (1)

[0014] Among them, R_hat (S) represents the rain streak information separated from the input rainy image Input, S represents the number of iterations, and B (S-1) represents the background restored after S - 1 iterations; here, R_hat (S) is different from Z 0 , the number of channels of R_hat (S) is the same as that of Input and B (S-1) , all being the three RGB color channels; while Z 0 is obtained through the predefined rain kernel C and the rain removal and restoration module transformation, and its number of channels is related to the rain kernel C, containing multi-channel high-dimensional information;

[0015] Step 2.2: Since there is still minute detailed information similar to rain streaks in the background in R_hat (S) , the obvious rain streak information is highlighted through sparsification processing, and the similar minute background information is filtered out. The deconvolution operation DeConv_C adjusts the feature morphology of the sparsification result. The sparsification processing and the deconvolution operation are only used in the first iteration, and subsequent iterations directly enter the rain streak enhancement module for processing;

[0016] Step 2.3: The main function of the rain streak enhancement module is to extract features of the rain streaks, reorganize the channels, and obtain more accurate multi-dimensional rain streak information M (S); The rain streak enhancement module is implemented through residual attention. First, two sets of 3×3 convolutions and ReLU activation functions are applied to understand the feature information and gradually extract more abstract and complex features. Second, redundant features are removed through global average pooling, and a fully connected layer (FC layer) is introduced to fuse local features into global features and process the relationships between complex features. Finally, four sets of multi-scale convolution enhancement modules are applied to effectively extract rain streak features of different scales and resolutions, enabling the rain streak enhancement module to flexibly process diverse details and structural information of rain streaks;

[0017] Step 2.4, M (S) Then, it is adjusted through convolution operation Conv_C by the rain core C, and finally the enhanced rain streak information R after dimensionality reduction is obtained (S) 。

[0018] As a further improvement of the present invention, step 3 specifically includes the following steps:

[0019] Step 3.1, Obtain the rain streak information R through the rain streak enhancement module (S) , and remove R from the input rainy image Input (S) The image background B_hat can be separated (S) , as shown in formula (2)

[0020] B_hat (S) =Input - R (S) (2)

[0021] As the number of iterations increases, the separated background B_hat (S) will become cleaner and cleaner;

[0022] Step 3.2, Weightedly fuse the repaired background B (S-1) from the previous iteration and the background information B_hat separated in this iteration (S) through formula (3)

[0023] B_mid (S) =η 1 B_hat (S) +(1 - η 1 )B (S-1) (3)

[0024] Among them, the weight parameter η 1 is automatically updated during the training process, and its goal is to retain as many details as possible in the two pictures;

[0025] Step 3.3, Combine the fusion result B_mid (S) with the rain streak information Z separated in the previous iteration (S-1)Perform splicing as the input of the rain removal and restoration module for processing, and separate the restored background B after this iteration (S) and the updated rain streak information Z (S) .

[0026] The beneficial effects of the present invention are as follows:

[0027] The present invention provides an image rain removal and restoration method, including a rain streak enhancement module and a rain removal and restoration module connected in series, which can effectively remove rain streaks in the input rainy picture and at the same time propose an enhanced edge loss for the rain removal and restoration module, paying special attention to those edge regions that affect visual perception, enhancing the adaptability of the model to different environmental conditions, and by paying attention to the edge information of the image, the robustness of the model against different rain streak changes can be enhanced, the stability of the processing effect can be improved, and artifacts and blurring phenomena can be avoided at the edge part; the present invention conducts a generalization experiment, and when performing crack removal experiments on its own Thangka and mural crack and mildew dot datasets, the present invention also achieves good results. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is the overall network architecture diagram of the embodiment of the present invention;

[0029] Figure 2 is the initialization process architecture diagram in the embodiment of the present invention;

[0030] Figure 3 is the rain streak enhancement process architecture diagram in the embodiment of the present invention;

[0031] Figure 4 is the residual attention structure diagram in the embodiment of the present invention;

[0032] Figure 5 is the rain removal and restoration process structure diagram in the embodiment of the present invention;

[0033] Figure 6 is the visual comparison diagram of Rain100L and Rain100H in the embodiment of the present invention;

[0034] Figure 7 is the visual comparison diagram of the real-world dataset Internet-Data in the embodiment of the present invention;

[0035] Figure 8 is the visual comparison diagram of the Thangka and mural crack and mildew dot datasets in the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0036] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0037] Embodiment

[0038] As Figure 1As shown, an image de-raining and restoration method includes:

[0039] Step 1: Select the input rainy picture Input;

[0040] Step 2: First, perform an initialization operation on Input. Filter out the rain streak information Z through convolution operation with the predefined rain kernel C. 00 , and then Input and Z 00 After channel concatenation, input it into the de-raining and restoration module. The de-raining and restoration module learns the features of the rain streak information Z 00 , re-integrate the channels of the concatenated image, and first separate the background information B 0 and the rain streak information Z 0 . The background information B 0 contains the extracted RGB channel information, and the remaining channel information is used as the rain streak information Z 0 . After the initialization operation, the obtained Z 0 information has richer rain streak feature information compared to Z 00 ;

[0041] Step 3: The input rainy picture Input can be regarded as a combination of a rain-free image B and a rain streak R_hat. Their relationship can be expressed as shown in formula (1).

[0042] R_hat (S) = Input - B (S-1) (1)

[0043] where R_hat (S) represents the rain streak information separated from the input rainy picture Input, S represents the number of iterations, and B (S-1) represents the background restored after S - 1 iterations. The background B (S-1) in the first iteration is the background information B separated by the initialization operation 0 . Here, R_hat (S) is different from Z 0 . The number of channels of R_hat (S) is the same as that of Input and B (S-1) , all being the three RGB color channels; while Z 0 is obtained through transformation by the predefined rain kernel C and the de-raining and restoration module. Its number of channels is related to the rain kernel C and contains multi-channel high-dimensional information;

[0044] Step 4: Since R_hat (S)It still contains the tiny detail information similar to rain streaks in the background. Therefore, through sparsification processing, the obvious rain streak information is highlighted, and the similar tiny background information is filtered out. The deconvolution operation DeConv_C adjusts the feature morphology of the sparsification result. The sparsification processing and the deconvolution operation are only used in the first iteration, and subsequent iterations directly enter the rain streak enhancement module for processing;

[0045] Step 5: The main function of the rain streak enhancement module is to extract features of rain streaks, reorganize the channels, and obtain more accurate multi-dimensional rain streak information M as the number of iterations increases (S) . The rain streak enhancement module is implemented through residual attention. First, two groups of 3×3 convolutions and ReLU activation functions are applied to understand the feature information and gradually extract more abstract and complex features. Secondly, global average pooling is used to remove redundant features, and a fully connected layer (FC layer) is introduced to fuse local features into global features and handle the relationships between complex features. Finally, four groups of multi-scale convolution enhancement modules are applied to effectively extract rain streak features of different scales and resolutions, enabling the rain streak enhancement module to flexibly handle diverse details and structural information of rain streaks. Since the network proposed in this paper needs to be iterated multiple times, introducing residual connections multiple times in residual attention can solve problems such as gradient disappearance, gradient explosion, and information loss that may occur during the training process;

[0046] Step 6: M (S) Then, it is adjusted by the convolution operation Conv_C of the rain kernel C, and finally the enhanced rain streak information R after dimensionality reduction is obtained (S) ;

[0047] Step 7: After the rain streak enhancement module, the rain streak information R is obtained (S) , and R is removed from the input rainy image Input (S) to separate the image background B_hat (S) , as shown in formula (2)

[0048] B_hat (S) = Input - R (S) (2)

[0049] As the number of iterations increases, the separated background B_hat (S) will become cleaner and cleaner;

[0050] Step 8: The repaired background B of the previous iteration (S-1) and the background information B_hat separated in this iteration (S) are weighted and fused through formula (3)

[0051] B_mid (S) = η 1 B_hat (S)+(1 - η 1 )B (S-1) (3)

[0052] where the weight parameter η 1 is automatically updated during the training process, and its goal is to retain as much detailed information as possible in the two pictures;

[0053] Step 9: Concatenate the fusion result B_mid (S) with the rain streak information Z separated in the previous iteration (S-1) as the input of the rain removal and restoration module for processing, and separate the restored background B (S) and the updated rain streak information Z (S) ;

[0054] Step 10: Determine whether to perform the next iteration by judging whether the final restored picture B (S) reaches convergence, that is, achieves the optimal effect during the training process on this dataset, and output the restoration result Output after the training effect converges

[0055] Rain streak enhancement module: It is implemented through residual attention. First, apply two groups of 3×3 convolutions and ReLU activation functions to understand the feature information and gradually extract more abstract and complex features. Secondly, remove redundant features through global average pooling, and introduce a fully connected layer (FC layer) to fuse local features into global features to handle the relationships between complex features. Finally, apply 4 groups of multi-scale convolution enhancement modules to effectively extract rain streak features of different scales and resolutions, so that the rain streak enhancement module can flexibly handle the diverse details and structural information of rain streaks. Since the network proposed in this embodiment needs to be iterated multiple times, introducing residual connections multiple times in residual attention can solve problems such as gradient disappearance, gradient explosion, and information loss that may occur during the training process.

[0056] Rain Removal and Restoration Module: In addition to residual attention, multi-scale parallel depthwise separable convolution and hybrid attention mechanism are introduced. The multi-scale parallel depthwise separable convolution uses convolutional kernels with sizes of 3, 5, and 7 respectively. The purpose of choosing such all-odd sizes is to ensure the alignment of the feature space. The 3×3 convolution is used to capture local details and fine features, aiming at processing small objects, edges, and textures in images, such as human facial features and facial contours. The 7×7 convolution is used to capture global features, aiming at processing complex backgrounds or large-scale objects, such as the sky and numerous trees and flowers on both sides of the road. The 5×5 convolution captures medium features (such as cars and roads) while integrating different information captured by the three convolutions. Finally, during parallel processing, multi-scale feature fusion is performed. Here, the feature maps of each scale are concatenated together in the channel dimension, which can retain the features of each scale for further processing. The hybrid attention mechanism mainly focuses on more subtle details in the image based on the restoration of multi-scale parallel depthwise separable convolution, such as the delicate textures of skin and clothing, shadows on objects, and reflections of light, etc.

[0057] Enhanced Edge Loss: Proposed for the rain removal and restoration module, it helps the rain removal and restoration module to specifically focus on those edge regions that affect visual perception, enhances the model's adaptability to different environmental conditions, and by focusing on the edge information of the image, it can enhance the model's robustness to different rain streak changes, improve the stability of the processing effect, and avoid artifacts and blurring in the edge part.

[0058] The experimental results of evaluating the image quality using two evaluation criteria, Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM), are shown in Table 1 as follows:

[0059]

[0060] Table 1 Comparison of Quantitative Results of Rain100L and Rain100H Datasets

[0061] Visual Comparison and Generalization Experiment of Rain100L and Rain100H - Visual Comparison Diagrams of Thangka and Mural Crack and Mildew Datasets and Real World Dataset Internet-Data are as Figures 6 - 8 shown.

[0062] The above-described embodiments only represent the specific implementation manners of the present invention. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the patent of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all belong to the protection scope of the present invention.

Claims

1. An image deraining and restoration method, characterized in that: The method comprises a rain mark enhancement module and a rain removal and repair module connected in series, wherein the method comprises the following steps: Step 1: Input rain image Input first performs convolution operation through the predefined rain kernel C to filter out the rain mark information Z 00 , and then Input and Z 00 After channel splicing, the module is input into the rain removal and restoration module, which learns the rain mark information Z 00 The features of the spliced ​​image are re-integrated to separate the background information B0 and the rain mark information Z0 for the first time. The background information B0 contains the extracted RGB channel information, and the remaining channel information is used as the rain mark information Z0. After the initialization operation, the obtained Z0 information is compared with Z 00 It has richer rain streak feature information; Step 2: After the rain image Input is initialized, it is iterated multiple times, and each iteration is repaired by a rain mark enhancement module and a rain removal repair module. The background layer is estimated in each iteration by using the rain image Input minus the rain mark information R updated in this iteration. (S) , preliminarily restore the background image of this iteration; Step 3: The initially restored background image is processed again as the input of the rain removal and restoration module, and finally the final restoration image of this iteration is output. The decision on whether to proceed to the next round of iteration is made by judging whether the final restoration image of the iteration has reached convergence, that is, whether the optimal effect has been achieved during the training process of the data set. After the training effect converges, the restoration result Output is output.

2. The image deraining and restoration method according to claim 1, characterized in that: The step 2 specifically includes the following steps: Step 2.1: The input rainy picture Input is regarded as the combination of the rain-free image B and the rain mark R_hat. The relationship between them is expressed as shown in formula (1): R_hat (S) =Input-B (S-1) (1) Among them, R_hat (S) represents the rain streak information separated from the input rainy picture Input, S represents the number of iterations, and B (S-1) represents the background repaired after iteration S-1 times; here, R_hat (S) Unlike Z0, R_hat (S) The number of channels is related to Input and B (S-1) The same, both are RGB three color channels; while Z0 is obtained by transforming the predefined rain core C and the rain removal and restoration module. Its number of channels is related to the rain core C and contains multi-channel high-dimensional information; Step 2.2, due to R_hat (S) The image still contains tiny details similar to rain marks in the background, so the obvious rain marks are highlighted through sparse processing, and similar tiny background information is filtered out. The deconvolution operation DeConv_C adjusts the feature morphology of the sparse results. The sparse processing and deconvolution operation are only used in the first iteration, and the subsequent iterations directly enter the rain mark enhancement module for processing; Step 2.3: The main function of the rain mark enhancement module is to extract the features of the rain marks and re-regulate the channels. As the number of iterations increases, more accurate multi-dimensional rain mark information M is obtained. (S) ; The rain mark enhancement module is implemented through residual attention; first, two groups of 3×3 convolution and ReLU activation functions are applied to understand the feature information and gradually extract more abstract and complex features; second, redundant features are removed through global average pooling, and a fully connected layer is introduced to fuse local features into global features to handle the relationship between complex features; finally, four groups of multi-scale convolution enhancement modules are applied to effectively extract rain mark features of different scales and resolutions, so that the rain mark enhancement module can flexibly process the diverse details and structural information of rain marks; Step 2.4, M (S) Then the rain kernel C is adjusted by convolution operation Conv_C, and finally the enhanced rain mark information R after dimensionality reduction is obtained. (S) .

3. The image deraining and restoration method according to claim 2, characterized in that: The step 3 specifically includes the following steps: Step 3.1: Obtain rain streak information R through the rain streak enhancement module (S) , remove R from the input rainy image Input (S) Can separate the image background B_hat (S) , as shown in formula (2), B_hat (S) =Input-R (S) (2) As the number of iterations increases, the separated background B_hat (S) It will become cleaner and cleaner; Step 3.2: Use formula (3) to replace the repaired background B of the previous iteration (S-1) And the background information B_hat separated in this iteration (S) Perform weighted fusion. B_mid (S) =η1B_hat (S) +(1-η1)B (S-1) (3) Among them, the weight parameter η1 is automatically updated during the training process, and its goal is to retain as much detail information as possible in the two pictures; Step 3.3: The fusion result B_mid (S) The rain streak information Z separated from the previous iteration (S-1) The splicing is used as the input of the rain removal and restoration module to separate the restoration background B after this iteration. (S) And the updated rain streak information Z (S) .

Images