Parallel single image rain removal method based on residual prior attention mechanism

Abstract

The application discloses a parallel single image rain removal method based on a residual prior attention mechanism, and comprises the following steps: 1) obtaining an image to be removed; 2) extracting picture detail feature information through an image detail extraction network; 3) extracting picture rain line feature information through a rain line feature extraction network; 4) fusing the detail feature information and the rain line feature information through a feature fusion network; and 5) completing rain removal. The application strengthens effective feature information and weakens irrelevant items through a residual prior attention module, thereby improving the rain removal performance of the model; and the rain removal performance of the model is improved through reasonable fusion and fine-tuning of the detail feature and the rain line feature.

Description

TECHNICAL FIELD

[0001] The present application relates to a parallel single image rain removal method based on residual prior attention mechanism. BACKGROUND

[0002] Severe weather conditions can cause image acquisition devices such as cameras to obtain low-quality images, which often lose details and signal distortion, but input high-quality clean images are the premise for many computer vision applications. Rainy environment is a typical severe weather, and the rain lines in the image make the image blurred and lose a lot of information. Rainy conditions can affect the quality of image generation, affecting vision while reducing the performance of computer vision algorithms. Single image rain removal is still a challenging task, and existing algorithms can cause loss of background detail texture while removing rain lines.

[0003] (1) Residual network: Residual network is a very effective network to alleviate the problem of gradient disappearance, which greatly deepens the network at that time after being proposed. The residual unit can be realized in the form of skip-layer connection, that is, the input of the unit is directly added to the unit output and then activated. Therefore, the residual network can be easily implemented using the mainstream automatic differentiation deep learning framework, and the gradient of the loss to a low-level output is directly updated using the BP algorithm.

[0004] (2) Attention mechanism: Attention mechanism is similar in nature to human visual selective attention mechanism, which selects more critical information from numerous information for the current task target, obtains the difference in importance of each feature map through means, suppresses useless information, and highlights useful information (focuses attention on important areas in the image). It is widely used in natural language processing, image recognition and language recognition.

[0005] (3) Image enhancement: Image enhancement is to purposefully emphasize the overall or local characteristics of the image, such as improving the color, brightness and contrast of the image, etc., to make the original unclear image clear or to emphasize certain features of interest, to enlarge the difference between different object features in the image, to suppress features not of interest, and to improve the visual effect of the image. In recent years, convolutional neural networks have made great breakthroughs in many low-level computer vision tasks, including image super-resolution, deblurring, rain removal, denoising, image enhancement, etc.

[0006] Current existing rain removal methods focus on improving the parameter quantity and depth of the model to remove rain lines, although the rain line removal effect is good, but forget that image details belong to low-dimensional features and do not need very deep network to extract. The purpose of rain removal is not only to remove rain lines, but also to preserve and restore the details of the image. In order to facilitate subsequent high-level image processing tasks. SUMMARY

[0007] To address the problems of the existing technologies, this invention proposes a parallel single-image deraining method based on a residual prior attention mechanism. By using the attention mechanism, more critical information for the deraining task is selected to improve the deraining effect. At the same time, a low-dimensional feature extraction network is designed, and multiple simplified Unet models are connected in parallel to preserve and restore image detail information.

[0008] This invention can be achieved through the following technical solutions:

[0009] A parallel single-image rain removal method based on residual prior attention mechanism includes the following steps:

[0010] 1) Obtain the image to be de-rained;

[0011] 2) Extract detailed feature information from images using an image detail extraction network;

[0012] 3) Extract rain line feature information from the image using a rain line feature extraction network;

[0013] 4) The detailed feature information and rain line feature information are fused through a feature fusion network.

[0014] 5) Complete the rain removal.

[0015] Further, in step 1), the model training images are first obtained. The images consist of clear images without rain and images to be rained off. The images to be rained off are obtained by superimposing simulated rain lines and raindrops on the clear images. The images to be rained off and the clear images without rain correspond one-to-one. All images are divided into training set and test set according to a 9:1 ratio.

[0016] Furthermore, the image detail extraction network is composed of multiple Unet modules connected in parallel, as shown in the following formula:

[0017]

[0018] In the formula, This represents a detailed feature map, with a size of (Nx16)x128x128. `Cat` represents the array concatenation operation. i Representing the Unet module, x is the image to be derained, and includes an encoder and a decoder. The encoder and decoder are mainly composed of convolutional modules, activation modules, upsampling and downsampling modules, and feature layers of the same size between the encoder and decoder use skip connections.

[0019] Further, the rain line feature extraction network extracts image feature information from shallow to deep by a plurality of feature extraction modules connected by a residual prior attention module (RSAM), wherein the feature extraction module is composed of four convolution modules in series, each convolution module is composed of 3x3 convolution (Conv), excitation compression module (SE), batch normalization (BN) and activation function Leaky Relu, the channel number of each layer of convolution is not fixed, and is determined according to the task difficulty and the inference frame rate; the excitation compression module is placed after the convolution operation, learns the correlation between channels in the convolution process, and assigns a weight to each channel in the input information matrix by fusing global information in each feature.

[0020] Further, the input of the residual prior attention module is divided into three parts, the first part is the output feature map of the previous feature extraction module, the second part is the input feature map of the previous feature extraction module, and the third part is the RGB image of the picture to be removed. The original data of the first part is used three times, the first time is added to the third part after convolution to obtain a feature map containing prior information, the feature map is subjected to 1x1 convolution and sigmod operation to obtain an attention map, the second time the original data of the first part is used to multiply the attention map to obtain a feature map with assigned weights, which highlights the feature information related to the rain removal task. Figure One In order to prevent information loss and gradient vanishing or explosion and ensure the stability of the model, the residual idea is adopted, and the first part and the second part of the input feature map from the previous feature extraction module are convolved and added to the feature map with assigned weights as the output of the residual prior attention module.

[0021] Further, the input of the rain line feature extraction network is the image to be removed, and the output is a rain line feature map. The rain line feature extraction network first converts the image to be removed with a channel number of 3 into a feature map that can be used for subsequent calculation by a 3x3 convolution operation, and the number of layers of the feature map is determined by the feature extraction module. Subsequently, the image feature information is extracted from shallow to deep by a plurality of serially connected feature extraction modules, and each feature extraction module is connected by a residual prior attention module (RSAM), as shown in the following formula:

[0022]

[0023] In the formula, is the output feature map of the i-th residual prior attention module, F RSAM is the residual prior attention module, and δ is the feature extraction module.

[0024] Further, the feature fusion fine-tuning network first splices the feature maps from the rain line feature extraction network and the image detail extraction network together through a channel splicing operation, then completes the fusion of the features through three convolution-batch normalization-activation modules, finally converts the output dimension to 3 through an output convolution operation, controls the data in the positive value through a relu activation function to complete the final output, and obtains the final rain-removed image.

[0025] Further, the rain-removed image and the rain-free image are equal in each pixel point value, and a mean square error (MSE) loss is used as a first loss function:

[0026]

[0027] In the formula, H, W and C represent the height, width and channel number of the output picture, represents the rain-removed image, B x，y，z represents the clean rain-free image.

[0028] Structural similarity (SSIM) is an index for measuring the similarity of two images, and is used to evaluate the similarity of two images by using three different standards of brightness, contrast and structure. The closer the two images are, the closer the SSIM index is to 1. The second loss function is obtained by subtracting the SSIM index of the rain-removed image and the clean image from 1:

[0029]

[0030] In combination with the two loss functions given above, the final model loss function is:

[0031] L=L MSE +L SSIM .

[0032] Through the above technical scheme, the present application has the following beneficial effects:

[0033] 1) By obtaining image detail feature information, the background details and contour information of the rain-removed image are retained;

[0034] 2) The residual prior attention module strengthens the effective feature information and weakens the irrelevant items, thereby improving the rain-removed performance of the model;

[0035] 3) The reasonable fusion and fine-tuning network of the detail features and the rain line features improve the rain-removed performance of the model. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 is a rain-removed flowchart of the present application;

[0037] Figure 2 is a complete model structure diagram of the present application;

[0038] Figure 3 For the simplified version of Unet network diagram;

[0039] Figure 4 For the feature extraction module;

[0040] Figure 5 Residual prior attention module;

[0041] Figure 6 The performance of the existing methods DID-MDN, RESCAN and the method of the present application on the synthetic data set Rain100H. DETAILED DESCRIPTION

[0042] The present application will be described in more detail by specific embodiments, and those skilled in the art can easily understand other advantages and effects of the present application from the disclosure.

[0043] The present application proposes a parallel single image rain removal method based on residual prior attention mechanism, as shown in Figure 1 .

[0044] First, the model training picture is obtained, which is composed of a clear image without rain and a to-be-removed-rain image. The to-be-removed-rain image is obtained by superimposing simulated rain lines and raindrops on the clear image. The to-be-removed-rain image and the clear image without rain are one-to-one corresponding. All images are divided into training set and test set according to the ratio of 9:1.

[0045] After obtaining the to-be-removed-rain image, the present method first performs image order shuffling and random cropping. A 128x128 image is randomly cropped in each image, and then the image is sent to two parallel feature extraction networks, namely image detail feature extraction network and rain line removal network. The two parts are mutually related and have their own focuses. The feature maps extracted by the two parts are removed through the designed feature fusion fine-tuning network.

[0046] The complete structure diagram of the model is shown in Figure 2 , and the two network parts are framed in the figure. The upper part is the image detail feature extraction network, and the lower part is the image rain line feature extraction network. The remaining part is the feature fusion fine-tuning network. As shown in the following formula:

[0047]

[0048] In the formula, F fuse is the feature fusion fine-tuning network, is the detail feature map, is the rain line feature map.

[0049] (1) Image detail extraction network

[0050] The input of the image detail extraction network is a to-be-rain-removed image, and the output is an image detail feature map. The detail information of the image belongs to low-dimensional information, and does not need a deep network to extract features, so the image detail extraction network is composed of N lightweight simplified Unet modules in parallel. As shown in the following formula:

[0051]

[0052] In the formula, represents a detail feature map, the feature map size is (Nx16)x128x128, Cat represents an array splicing operation, u i represents a Unet module, and x is a to-be-rain-removed image.

[0053] Considering that the number of Unet modules used is large, the parameter amount of a single module is small, and the number of channels of each layer of the original version is reduced to one-fourth of the original, as shown in detail in Figure 3 The input of the image detail extraction network is a to-be-rain-removed image, and the image channel number is 3. First, it will be sent to the encoder for encoding operation. The encoder will perform feature extraction through a convolution module and an activation module every time it goes deeper, and will convert the height and width of the feature map to half of the original through downsampling method. After four times of downsampling, the channel numbers of each layer are 16, 32, 64, 128 and 256 respectively. The decoder is opposite to the encoder, and will perform upsampling operation every time a convolution module and an activation module are passed. The encoding and decoding module is mainly composed of a convolution module, an activation module, an upsampling and a downsampling module, and the same size feature layers in the encoding and decoding are connected by a jump connection.

[0054] After the to-be-rain-removed image passes through N lightweight simplified Unet modules in parallel, n feature maps with a channel number of 16 will be obtained, and then through splicing operation, the output feature map information of each channel is spliced together to obtain a final feature information with a channel number of Nx16 as the output of the image detail extraction network.

[0055] (2) Rain line feature extraction network

[0056] The input of the rain line feature extraction network is a to-be-rain-removed image, and the output is a rain line feature map. The rain line feature extraction network first converts the to-be-rain-removed image with a channel number of 3 into a feature map that can be used for subsequent calculation through a 3x3 convolution operation, and the number of feature map layers is determined by the feature extraction module. Subsequently, the image feature information is extracted by a plurality of serially connected feature extraction modules from shallow to deep, and each feature extraction module is connected by a residual prior attention module (RSAM). As shown in the following formula:

[0057]

[0058] In the formula, is the output feature map of the i-th residual prior attention module, F RSAM is the residual prior attention module, and δ is the feature extraction module.

[0059] The feature extraction module is composed of four convolution modules in series, and each convolution module is composed of a 3x3 convolution (Conv), an excitation compression module (SE), a batch normalization (BN), and an activation function Leaky Relu, as shown in Figure 4 . The input feature matrix is first further extracted deep feature information through 3x3 convolution operation, and the channel number of each layer of convolution is not fixed, which can be determined according to the task difficulty and the inference frame rate. Then, the excitation compression module is passed through, and the input dimension of the excitation compression module is the same as the channel number of the feature information, and the dimension of the hidden layer is 2 times the input dimension. The excitation compression module is placed after the convolution operation, which learns the correlation between the channels in the convolution process, and assigns a weight to each channel in the input information matrix by fusing the global information in each feature. This method not only solves the shortcoming of traditional convolutional neural networks that cannot use information outside the receptive field, but also highlights the most relevant feature information for the rain removal task and weakens other redundant interference information. The feature information after the weight redistribution will be passed through the batch normalization layer next, which normalizes and linearly transforms the input data of each layer of network to make the mean and variance of the input data of each layer of network within a certain range, while accelerating the convergence speed and stability of the model. Finally, the activation function Leaky Relu is used. The difference between Leaky Relu and the activation function relu is that the negative value information of the data is retained, and a non-zero slope is assigned to all negative values to prevent information loss.

[0060] The input of the residual prior attention module (RSAM) is divided into three parts, as shown in Figure 5 . The first part is the output feature map of the previous feature extraction module, the second part is the input feature map of the previous feature extraction module, and the third part is the RGB image of the picture to be removed. The original data of the first part is used three times. As shown in the following formula,

[0061]

[0062]

[0063] In the formula, is the point-to-point multiplication, Conv i is the convolution operation, and sigmod is the activation function, F att is the attention map.

[0064] The channel number of the original data is converted to 3 after the first use of the 1x1 convolution, which is the same as the channel number of the third part of the rain-removed picture, and then added to the third part to obtain a feature map containing prior information, and the feature map is subjected to 1x1 convolution and sigmod operation to obtain an attention map F att , the second use of the first part of the original data is multiplied by the attention Figure One The feature map after the weight distribution is highlighted, and in order to prevent information loss and prevent gradient vanishing or explosion, the residual idea is adopted, and the second part of the feature map after the convolution of the first part and the input feature map from the previous feature extraction module is added to the feature map after the weight distribution as the output of the residual prior attention module.

[0065] (3) Feature fusion fine-tuning network

[0066] The feature fusion fine-tuning network fuses the output feature maps of the rain line feature extraction network and the image detail extraction network to obtain a rain-removed picture. The network first splices the feature maps from the rain line feature extraction network and the image detail extraction network through a channel splicing operation, and then completes the fusion of the features through three 3x3 convolution-batch normalization-activation modules LeakRelu. The channel number of the three convolution features will gradually decrease, and finally an output 3x3 convolution operation is used to convert the output dimension to 3. The relu activation function controls the data in the positive value to complete the final output, and the final rain-removed image is obtained.

[0067] (4) Loss function design:

[0068] Obviously, the rain-removed rain-free image should be completely the same as the corresponding rain-free image at the pixel level, that is, the pixel point values of the rain-removed image and the rain-free image are equal. Based on this, the present application adopts mean square error (MSE) loss as the first loss function:

[0069]

[0070] In the formula, H, W and C represent the height, width and channel number of the output picture. The rain-removed image is represented by B x，y，z The rain-free image is represented by B

[0071] Structural similarity (SSIM) is an index for measuring the similarity between two images, which evaluates the similarity between two images based on three different standards of brightness, contrast and structure. The closer the two images are, the closer the SSIM index is to 1. Therefore, the present application subtracts the SSIM index of the rain-removed image from the clean image as the second loss function:

[0072]

[0073] Combining the two loss functions given above, the final model loss function is:

[0074] L=L MSE +L SSIM .

[0075] The experiment is based on the pytorch1.10.0 framework, and is run on the ubuntu22.04 system. The model training is accelerated by GPU, and is calculated in the mobile version of NVIDIA3060 graphics card, CUDA11.2 environment, and the memory is 6g. A total of 7 parallel simplified Unet networks are used in the image detail extraction network, and the convolution layer channel number of the rain line feature extraction network is 24, and a total of 6 RSAM networks are used. In order to show that the model is easy to train and reduces the requirement of the model on the hyperparameter, and reduces the training cost, the Adam optimizer is used as the optimizer of the model in the present application. The Adam optimizer comprehensively considers the first moment and the second moment of the gradient, calculates the update step, can adaptively optimize the learning rate, and reduces the training cost. Compared with other optimizers, it is not sensitive to the initial learning rate, and is widely used in model training. The initial learning rate of the present application is set to 0.0005, and a total of 2000 epochs are trained. The training picture size is 128x128.

[0076] The present application compares the performance of the proposed model with existing models on three synthetic data sets. The most commonly used SSIM and PSNR are used as performance indicators, and the experimental results are placed in Table 1. It can be seen that the method proposed in the present application has greatly improved the SSIM and PSNR indicators compared with the current method. Specifically, the PSNR indicators on Rain100L, Rain100H and Rain800 achieve 37.80db, 32.24db and 32.86db, and the SSIM indicators achieve 0.982, 0.937 and 0.926. Both indicators exceed the other models RESCAN, JORDER and DID-MDN compared.

[0077] Table 1 Comparison of model performance on different synthetic data sets

[0078]

[0079] In order to intuitively show the performance of the method proposed in the present application, Figure 6Some of the rain removal results on the Rain100H test set are shown. The leftmost column is the original rainy image, and the other columns from left to right are the results of DID-MDN, RESCAN and the method of the present application, respectively. It is clear that the method of the present application has the best rain removal effect, which can effectively remove the rain streaks while effectively preserving the texture and structural details. For example, from the horizontal comparison of the first column of images, it can be seen that the method of the present application removes the curves while restoring the details of the animal's hair in the image, and the background is also more harmonious than other results, without the residual streaks of rain lines.

[0080] The above description is merely preferred embodiments of the present application, and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall fall within the scope of protection of the present application.

Claims

1. A parallel single-image rain removal method based on residual prior attention mechanism, characterized in that, Includes the following steps: 1) Obtain the image to be de-rained; 2) Extract detailed feature information from images using an image detail extraction network; 3) Extract rain line feature information from the image using a rain line feature extraction network; 4) The detailed feature information and rain line feature information are fused using a feature fusion network; 5) Complete the rain removal; The rain line feature extraction network takes the image to be de-rained as input and outputs a rain line feature map. The network first transforms the image with 3 channels into a feature map usable for subsequent calculations through a 3x3 convolution operation. The number of feature layers is determined by the feature extraction module. Then, multiple cascaded feature extraction modules are fed into the network to extract image features from shallow to deep layers. Each feature extraction module is connected by a Residual Prior Attention (RSAM) module, as shown in the following formula: In the formula, Let F be the output feature map of the i-th residual prior attention module. RSAM δ represents the residual prior attention module, and δ represents the feature extraction module.

2. The parallel single-image rain removal method based on residual prior attention mechanism according to claim 1, characterized in that, In step 1), the model training images are first obtained. The images consist of clear images without rain and images to be removed from the rain. The images to be removed from the rain are obtained by superimposing simulated rain lines and raindrops on the clear images. The images to be removed from the rain and the clear images without rain correspond one-to-one. All images are divided into training set and test set according to a 9:1 ratio.

3. The parallel single-image rain removal method based on residual prior attention mechanism according to claim 1, characterized in that, The image detail extraction network consists of multiple Unet modules connected in parallel, as shown in the following formula: In the formula, This represents a detailed feature map, with a size of (Nx16)x128x128. `Cat` represents the array concatenation operation. i Representing the Unet module, x is the image to be derained, and includes an encoder and a decoder. The encoder and decoder are mainly composed of convolutional modules, activation modules, upsampling and downsampling modules, and feature layers of the same size between the encoder and decoder use skip connections.

4. The parallel single-image rain removal method based on residual prior attention mechanism according to claim 3, characterized in that, The input to the image detail extraction network is the image to be derained, which has 3 channels. It is first sent to the encoder for encoding. As the network progresses, the encoder performs feature extraction by downsampling the height and width of the feature map by half each time it passes through a convolutional module and an activation module. This downsampling is done four times in total, with the number of channels in each layer being 16, 32, 64, 128, and 256, respectively. The decoder performs the opposite operation, upsampling once after each convolutional module and activation module.

5. The parallel single-image rain removal method based on residual prior attention mechanism according to claim 4, characterized in that, The image to be de-rained is passed through N parallel lightweight simplified versions of Unet, resulting in n feature maps with 16 channels each. Then, through a concatenation operation, the output feature map information is concatenated together along the channel dimension to obtain a final feature information with Nx16 channels, which is used as the output of the image detail extraction network.

6. The parallel single-image rain removal method based on residual prior attention mechanism according to claim 5, characterized in that, The feature extraction module consists of four convolutional modules connected in series. Each convolutional module consists of a 3x3 convolution, an activation compression module, batch normalization, and the Leaky ReLU activation function. The number of channels in each convolutional layer is not fixed and is determined according to the task difficulty and the inference frame rate. The compression activation module is placed after the convolution operation. It learns the correlation between channels during the convolution process and assigns a weight to each channel in the input information matrix by fusing global information within each feature.

7. The parallel single-image rain removal method based on residual prior attention mechanism according to claim 3, characterized in that, The input to the residual prior attention module is divided into three parts: the first part is the output feature map of the previous feature extraction module, the second part is the input feature map of the previous feature extraction module, and the third part is the RGB image of the image to be derained. The original data of the first part is used three times. The first time, after convolution, it is added to the third part to obtain a feature map containing prior information. The feature map is then subjected to 1x1 convolution and sigmoid operation to obtain the attention map. The original data of the first part is used a second time and multiplied one by one with the attention map to obtain the feature map after weighting, which highlights the feature information related to the deraining task. At the same time, in order to prevent information loss and gradient vanishing or exploding, and to ensure the stability of the model, the residual idea is adopted. The first part and the second part of the feature map from the input feature map of the previous feature extraction module are convolved and added to the feature map after weighting, which is used as the output of the residual prior attention module.

8. The parallel single-image rain removal method based on residual prior attention mechanism according to claim 1, characterized in that, The feature fusion fine-tuning network first concatenates the feature maps from the rain line feature extraction network and the image detail extraction network together through a channel concatenation operation. Then, it completes the feature fusion through three convolution-batch normalization-activation modules. Finally, it transforms the output dimension to 3 through an output convolution operation, and the ReLU activation function controls the data to a positive value to complete the final output, resulting in the final derained image.

9. The parallel single-image rain removal method based on residual prior attention mechanism according to claim 8, characterized in that, The values ​​of each pixel in the rain-removed image and the rain-free image are equal, and the mean squared error loss is used as the first loss function: In the formula, H, W, and C represent the height, width, and number of channels of the output image. B represents the image after the rain. x，y，z A clean, rainless image; Structural similarity (SSIM) is a metric that measures the similarity between two images. It evaluates the similarity of two images using three different criteria: brightness, contrast, and structure. The more similar the two images are, the closer the SSIM value is to 1. The second loss function is obtained by subtracting the SSIM values ​​of the rain-removed image and the clean image from 1. Combining the two loss functions given above, the final model loss function is: L=L MSE +L SSIM 。

Images