RCP-based multi-scale fusion iterative update rain removal image processing method

Abstract

The invention discloses a multi-scale fusion iteration updating rain removal image processing method based on RCP, and belongs to the technical field of computer vision. Due to different densities and directions of rain fringes, image rain removal is directly performed on a real rain image, a pyramid feature extraction module LPB is correspondingly provided, kernels of different sizes are fused during convolution operation, multi-scale features are obtained based on the convolution kernels of different sizes, rich features are captured in a multi-scale mode, and the method has the advantages of being high in robustness and high in robustness. The problem of detail recovery in the image reconstruction process is solved. The invention also provides a method for carrying out iterative updating on the RCP image; and a multi-scale interactive fusion module IMFB is provided, so that the problem of feature interference possibly caused by model decentralization is solved, the features of the RCP and the image are better fused, the features are gradually combined together, all rain stripes are effectively removed under complex conditions, the structural information of the image is protected, and a high-quality rain-free image is reconstructed.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a multi-scale fusion iterative updating deraining image processing method based on residual channel prior RCP. Background Art

[0002] When shooting outdoor scenes, weather factors often have a significant impact on image quality. Restoring images obscured by rain bands is critical for many computer vision applications in outdoor scenes. If the target in the background overlaps with the rain bands in the image, it is difficult to keep the background details intact while removing the rain bands. Especially in heavy rain, due to the severe reduction in visibility, the image becomes blurred and distorted, affecting the extraction of image features and reducing the accuracy of subsequent target recognition and behavior detection.

[0003] At present, among the deep learning-based image deraining algorithms, algorithms based on convolutional neural networks (CNNs) have achieved great success. Fu et al. proposed the first model for removing rain streaks based on convolutional neural networks. Later, they also proposed a deeper network based on deep residual networks and used image domain knowledge to remove rain streaks. Yang et al. proposed a recurrent deep network for joint rain detection and removal to gradually remove raindrops. Li et al. proposed a recurrent excitation context aggregation network to make full use of context information. Deng et al. proposed a detail restoration image detraining network, which consists of a rain residual network and a detail restoration network. Jiang et al. designed a coarse-to-fine progressive fusion network, and Wang et al. proposed a cross-scale fusion network based on intrascale connection blocks. In addition, Ren et al. proposed a progressive recurrent network (PReNet) by repeatedly unfolding a shallow ResNet. Wang et al. introduced a rain convolution dictionary network that continuously optimizes the model to obtain better rain streaks and rain-free images.

[0004] Although these methods bring great performance improvements, they are difficult to remove all rain streaks and restore the structural information of images in complex scenes. The density of rain streaks varies, resulting in excessive or insufficient removal of rain streaks, resulting in incomplete structural information of reconstructed images. Existing methods focus on learning the structure of rain streaks, pay less attention to the structure of objects, and ignore the importance of image priors. Therefore, it is crucial to develop a method that can effectively remove rain streaks while preserving the structure of objects. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a multi-scale fusion iterative update rain removal image processing method based on RCP, which can protect the image prior of the image structure and introduce the prior into the model to guide high-quality image reconstruction, and use a progressive reconstruction method to extract more accurate RCP in the intermediate reconstruction stage to further guide image reconstruction, so as to solve the technical problems that the current image rain removal algorithm cannot effectively remove all rain streaks and is difficult to protect the structural information of the image, and cannot reconstruct a high-quality rain-free image.

[0006] The multi-scale fusion iterative update rain removal image processing method based on RCP includes the following steps, and the following steps are performed in sequence:

[0007] Step 1: obtain multiple sets of real rainy images with rain streaks and corresponding clear rainless images, normalize and unify the image sizes to form a data set, and divide the data set into a training set and a test set;

[0008] Step 2: Establish a network model, which includes a rain streak feature extraction module and a rain streak removal module:

[0009] 1) The rain streak feature extraction module uses a convolution operation to extract rain streak features and obtain a rain streak feature image;

[0010] In the convolution operation, kernels of different sizes are fused to obtain multi-scale features, and a group convolution method is used to reduce the amount of calculation and model capacity. In order to prevent the structured sparse characteristics of group convolution, the lack of information exchange between channels of different groups, and the lack of information exchange between features generated by kernels of different sizes, a pyramid feature extraction module LPB is used to perform point-by-point convolution. The pyramid feature extraction module LPB sets convolution kernels of different sizes and uses a zero-filling method to keep the input feature map and the output feature map consistent. At the same time, the number of channels is consistent with the aforementioned convolution to ensure the lowest memory access consumption.

[0011] 2) The rain streak removal module includes an RCP feature extraction module, a multi-scale interactive fusion module IFMB, and an iterative update module; the RCP feature extraction module uses a residual channel prior RCP to directly generate an RCP image on a real rainy image. The RCP image has no rain streaks and only contains background object image information. It is the residual result of the maximum channel value and the minimum channel value of the real rainy image; the multi-scale interactive fusion module performs multi-scale interactive fusion IMFB on the generated RCP image and the rain streak feature image; the iterative update block uses a dynamic reasoning mechanism to iteratively update the fused RCP image of the RCP image and the background object image information as a new RCP image, gradually improve the accuracy of the RCP image, and obtain a clear rain-removed image in a progressive path;

[0012] Step 3: Use the training set to train the network model, set the initial parameters, and set the loss function. Through training and continuous updating, the optimal goal is achieved, the optimal network weights are obtained, and the trained network model is obtained.

[0013] Step 4: Use the test set to test the network model and obtain the final RCP-based multi-scale fusion iterative update deraining image processing model;

[0014] Step 5: Use the final RCP-based multi-scale fusion iterative update rain removal image processing model to perform rain removal processing on rainy images.

[0015] The convolution kernels set in the pyramid feature extraction module LPB are 5×5, 7×7, and 9×9, respectively, and the number of channels is 3.

[0016] The loss function uses L2 loss as the objective function. In the case of iterative update, the network model outputs LPB n Output characteristics Get the rain removal result from Get a clearer prior Therefore, the comprehensive loss function of the network model is It is expressed as:

[0017]

[0018] Among them, B represents a rain-free image, B i Representing as well as The output results of these three different stages are:

[0019] Among them, the loss function of the rain streak removal module is:

[0020] σ=σ E,r +σ E,b +γ F σ F ;

[0021] Among them, σ E,r represents the Euclidean loss per pixel of the rain streak component, σ E,b represents the per-pixel Euclidean loss of the output derained image, γ F represents the loss of extracting RCP image features, σ F The feature-based loss for derained images is defined as:

[0022]

[0023] Among them, C represents the feature map channel, W represents the feature map width, H represents the feature map height, and F represents the nonlinear convolution transformation. is the output image, x represents the input image, c, w, h represent the dimensions of the feature map, and is a tensor with c channels, w width, and h height dimensions.

[0024] Through the above design scheme, the present invention can bring the following beneficial effects:

[0025] Since the density and direction of rain streaks are different, the use of multi-scale features can further improve the deraining performance. Therefore, the present invention proposes a pyramid feature extraction module LPB, which fuses kernels of different sizes during convolution operations, and has multi-scale features based on convolution kernels of different sizes to capture rich features in a multi-scale manner, solving the problem of detail recovery in the image reconstruction process. The present invention also proposes an iterative update on the RCP image; and proposes a multi-scale interactive fusion module IMFB, which solves the problem of feature interference that may be caused by model decentralization, thereby achieving better fusion of RCP and image features, and gradually combining features together, thereby effectively removing all rain streaks in complex situations, and at the same time protecting the structural information of the image, and reconstructing a high-quality rain-free image. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The present invention is further described below with reference to the accompanying drawings and specific embodiments:

[0027] Figure 1 It is a flowchart of the multi-scale fusion iterative update rain removal image processing method based on RCP of the present invention;

[0028] Figure 2 It is a structural schematic diagram of a pyramid feature extraction module LPB in the multi-scale fusion iterative update rain removal image processing method based on RCP of the present invention;

[0029] Figure 3 It is a structural schematic diagram of a multi-scale interactive fusion module IFMB in the multi-scale fusion iterative update deraining image processing method based on RCP of the present invention;

[0030] Figure 4 It is a schematic diagram of the image effect of removing rain streaks using the traditional DDN method in the multi-scale fusion iterative update rain removal image processing method based on RCP of the present invention;

[0031] Figure 5 It is a schematic diagram of the image effect of removing rain streaks using the traditional method RESCAN in the multi-scale fusion iterative update rain removal image processing method based on RCP of the present invention;

[0032] Figure 6 It is a schematic diagram of the image effect of rain streak removal in the multi-scale fusion iterative update rain removal image processing method based on RCP of the present invention;

[0033] Figure 7It is a schematic diagram of a rain image in the multi-scale fusion iterative update rain removal image processing method based on RCP of the present invention;

[0034] Figure 8 It is a schematic diagram of an RCP image of a rain image in the RCP-based multi-scale fusion iterative update rain removal image processing method of the present invention. DETAILED DESCRIPTION

[0035] Starting from the rain removal model, the present invention designs a multi-scale fusion iterative update rain removal image processing method based on RCP. Figure 1 As shown, the specific steps are as follows:

[0036] Step 1: Dataset preparation.

[0037] The dataset includes a training set and a test set, each of which contains real rainy images and corresponding clear rainless images. Before image preprocessing, the pixel value range of the paired rainy images and clean rainless images needs to be planned to be between 0 and 1 to facilitate subsequent processing. This preprocessing method can be achieved by normalizing the image, dividing the pixel value by 255 so that the pixel value range is between 0 and 1. This ensures that the pixel value of the input image is within the appropriate range for subsequent processing and training.

[0038] Step 2: Establish a network model, which includes a rain streak feature extraction module and a rain streak removal module.

[0039] In the prior art, most deraining convolutional networks are composed of regular 3×3 kernel stacks that only extract a single feature in a specific receptive field. Considering the different densities and directions of rain streaks, these networks explicitly ignore the complementary correlation of multi-scale features, resulting in limited information integration and circulation when parsing rain scenes.

[0040] Therefore, in terms of multi-scale feature extraction, the rain streak feature extraction module proposes to fuse kernels of different sizes during convolution operations to obtain multi-scale features. The convolution kernels are set to 3×3, 5×5, and 7×7 respectively. When extracting useful information from feature maps of different scales, for example, large-size kernels tend to extract texture detail information, while small-size kernels pay more attention to edge structure information. The introduction of large-size kernels will inevitably expand the model capacity. In order to reduce the amount of calculation and model capacity, the rain streak feature extraction module adopts a grouped convolution method to optimize the model capacity for the problem of large convolution kernel parameters. In order to prevent the structured sparse characteristics of grouped convolution, there is no information exchange between channels of different groups and the lack of information exchange between features generated by kernels of different sizes. The rain streak feature extraction module also uses a pyramid feature extraction module LPB to fuse different features and reduce the information separation phenomenon, and then performs point-by-point convolution. Point-by-point convolution is a 1x1 ordinary convolution that fuses different information generated by kernels of different sizes to obtain a rain streak feature image.

[0041] like Figure 2 As shown, the convolution kernels of the pyramid feature extraction module LPB are set to 5×5, 7×7, and 9×9 respectively, and a zero-filling strategy is adopted to ensure that the input and output feature maps are consistent. At the same time, the number of channels is consistent to minimize memory access consumption. The number of channels here is 3.

[0042] The rain streak removal module includes an RCP feature extraction module, a multi-scale interactive fusion module IFMB and an iterative update module.

[0043] The RCP feature extraction module designs a residual channel prior (Residue Channel Prior RCP) to directly generate an RCP image on a real rain image, and its convolution kernels are also set to 3×3, 5×5, and 7×7 respectively.

[0044] like Figure 3 As shown, the multi-scale interactive fusion module IFMB performs multi-scale interactive fusion IMFB on the RCP image and the rain streak feature image for RCP extraction and guidance.

[0045] The iterative update module adopts a dynamic reasoning mechanism to perform iterative updates on the RCP image, and performs different iterative update times processing operations on images with different rainfall degrees and different background feature complexities.

[0046] The iterative update module iteratively updates the RCP image that is a fusion of the RCP image and the background object image information as a new RCP image, gradually improves the accuracy of the RCP image, and obtains a clear derained image in a progressive path;

[0047] The introduced RCP feature extraction module is used to protect the details of background objects, so as to achieve structural preservation and rain removal of background objects, and make full use of RCP image information to reconstruct high-quality rain-free images. Specifically, convolution is first used to obtain a limited number of initial feature maps. In addition, in order to reduce the noise in the initial features and enrich the semantic information of the features, SRiR in the residual block is used to extract the deep features of RCP.

[0048] The extracted RCP image has no rain streaks and only contains background object image information. It is the residual result of the maximum channel value and the minimum channel value of the real rainy image, such as Figure 7 Rain Image to Figure 8 Generate the RCP image as shown below. Given a rainfall image image The residual channel prior It can be defined as:

[0049]

[0050] In the formula, Express all color channels r, g and b, output The maximum value of Indicates the most prominent color component; Express all color channels r, g and b, output The minimum value of Indicates the weakest color component; r, g, b represent red, green, and blue respectively;

[0051] RCP still works without adopting color constancy. Due to the dominant gray atmospheric light produced by cloudy skies, the appearance of rain streaks is already colorless in most cases. Based on this observation, the RCP feature extraction module can extract a more complete and accurate object structure and can extract high-dimensional features of RCP images.

[0052] Although RCP images have clearer structural information than rainy day images, how to make full use of the features of RCP to guide the network model is still a challenging task. The existing algorithm directly fuses the RCP image features with the rain streak feature image. However, this cannot effectively guide the model to be decentralized and may cause feature interference. In order to solve this problem, the present invention proposes a multi-scale interactive fusion module IMFB to achieve better fusion of RCP images and rain streak feature images to gradually combine the features together. First, the 3×3 convolution kernel of the rain streak feature image is fused with the 3×3 convolution kernel of the RCP image, and then the 5×5 convolution kernel of the rain streak feature image is fused with the 5×5 convolution kernel of the RCP image. Then, the 7×7 convolution kernel of the rain streak feature image is fused with the 7×7 convolution kernel of the RCP image to obtain a new 3×3, 5×5, 7×7 convolution kernel, and then these three convolution kernels are subjected to a multi-scale interactive fusion to obtain an image with more complete features.

[0053] As the image quality improves, the extracted RCP image also shows better results. Based on this observation, an iterative guidance strategy is proposed to obtain a clearer RCP image by iteratively updating the module and replacing the RCP image of the rain image from LPB. n Output characteristics You can get the rain removal result from A clearer prior can be obtained The structure is expressed in the following formula:

[0054]

[0055] In the formula, n represents the number of LPBs, and n is set to 3; It is LPB n In order to enable the model to learn richer features and produce better results, the idea of ​​ensemble learning is used to adopt the concat operation to fuse the output features of the previous LPB with the output features of the current LPB.

[0056] Step 3: Network training phase: Use the training set and test set for training and testing, constrain the results through the loss function, and use L2 loss as the objective function. In the case of iterative update, the network model outputs LPB n Output characteristics Get the rain removal result from Get a clearer prior Therefore, the comprehensive loss function of the proposed model can be expressed as:

[0057]

[0058] Among them, B represents a rain-free image, B i Representing as well as The output results of these three different stages.

[0059] Among them, the loss function of the rain streak removal module is:

[0060] σ=σ E,r +σ E,b +γ F σ F ;

[0061] Among them, σ E,r represents the Euclidean loss per pixel of the rain streak component, σ E,b represents the per-pixel Euclidean loss of the output derained image, γ F represents the loss of extracting RCP image features, σ F The feature-based loss for derained images is defined as:

[0062]

[0063] Among them, C represents the feature map channel, W represents the feature map width, H represents the feature map height, and F represents the nonlinear convolution transformation. is the output image, x represents the input image, c, w, h represent the dimensions of the feature map, and is a tensor with c channels, w width, and h height dimensions.

[0064] Last passed Figure 4 As shown in the figure, when using the traditional DDN method for rain removal, it can be seen that the rain streaks are not removed sufficiently, the background structure is blurred, and the final rain-free image is incomplete. Figure 5 As shown in Figure 2, when using the traditional RESCAN method for rain removal, it can be seen that the rain streaks can be removed, but after obtaining the rain-free image, the background is still blurred and unclear, and the rain-free image is not complete. Figure 6 As shown, when the patented method is used for rain removal, it can be seen that in the obtained rain-free image, not only the rain streaks are removed, but also the background image is intact, thereby obtaining a clear rain-free image. Therefore, through the comparison of the images, it can be seen that the method of the patent is more applicable.

Claims

1. A multi-scale fusion iterative update deraining image processing method based on RCP, characterized by: The method comprises the following steps, and the following steps are performed in sequence: Step 1: obtain multiple sets of real rainy images with rain streaks and corresponding clear rain-free images, normalize and unify the image sizes to form a data set, and divide the data set into a training set and a test set; Step 2: Establish a network model, which includes a rain streak feature extraction module and a rain streak removal module: 1) The rain streak feature extraction module uses a convolution operation to extract rain streak features and obtain a rain streak feature image; In the convolution operation, kernels of different sizes are fused to obtain multi-scale features, and a group convolution method is used to reduce the amount of calculation and model capacity. In order to prevent the structured sparse characteristics of group convolution, the lack of information exchange between channels of different groups, and the lack of information exchange between features generated by kernels of different sizes, a pyramid feature extraction module LPB is used to perform point-by-point convolution. The pyramid feature extraction module LPB sets convolution kernels of different sizes and uses a zero-filling method to keep the input feature map and the output feature map consistent. At the same time, the number of channels is consistent with the aforementioned convolution to ensure the lowest memory access consumption. 2) The rain streak removal module includes an RCP feature extraction module, a multi-scale interactive fusion module IFMB, and an iterative update module; the RCP feature extraction module uses a residual channel prior RCP to directly generate an RCP image on a real rainy image. The RCP image has no rain streaks and only contains background object image information. It is the residual result of the maximum channel value and the minimum channel value of the real rainy image; the multi-scale interactive fusion module performs multi-scale interactive fusion IMFB on the generated RCP image and the rain streak feature image; the iterative update block uses a dynamic reasoning mechanism to iteratively update the fused RCP image of the RCP image and the background object image information as a new RCP image, gradually improve the accuracy of the RCP image, and obtain a clear rain-free image in a progressive path; Step 3: Use the training set to train the network model, set the initial parameters, and set the loss function. Through training and continuous updating, the optimal goal is achieved, the optimal network weights are obtained, and the trained network model is obtained. Step 4: Use the test set to test the network model and obtain the final RCP-based multi-scale fusion iterative update deraining image processing model; Step 5: Use the final RCP-based multi-scale fusion iterative update rain removal image processing model to perform rain removal processing on rainy images.

2. The multi-scale fusion iterative update deraining image processing method based on RCP according to claim 1 is characterized in that: The convolution kernels set in the pyramid feature extraction module LPB are 5×5, 7×7, and 9×9, respectively, and the number of channels is 3.

3. The multi-scale fusion iterative update deraining image processing method based on RCP according to claim 1 is characterized by: The loss function uses L2 loss as the objective function. In the case of iterative update, the network model outputs LPB n Output characteristics Get the rain removal result from Get a clearer prior Therefore, the comprehensive loss function of the network model is It is expressed as: Among them, B represents a rain-free image, B i Representing as well as The output results of these three different stages are: Among them, the loss function of the rain streak removal module is: s = s E,r +s E,b +g F s F ; Among them, σ E,r represents the Euclidean loss per pixel of the rain streak component, σ E,b represents the per-pixel Euclidean loss of the output derained image, γ F represents the loss of extracting RCP image features, σ F The feature-based loss for derained images is defined as: Among them, C represents the feature map channel, W represents the feature map width, H represents the feature map height, and F represents the nonlinear convolution transformation. is the output image, x represents the input image, c, w, h represent the dimensions of the feature map, and is a tensor with c channels, w width, and h height dimensions.

Images