Single image rain removal method based on intra- and extra-cycle dense connection and attention enhancement

Abstract

The application discloses a single image rain removal method based on cyclic inner and outer dense connection and attention enhancement, comprising the following steps: constructing a cyclic inner and outer dense connection and attention enhancement network for single image rain removal, wherein the network comprises a cyclic subnetwork with multiple shared parameters; determining a loss function, wherein the loss function comprises a structural similarity and a perception loss; inputting training data; and training network parameters to obtain a cyclic rain removal model. The application establishes dense connection in the cycle, expands the channel of network information transmission, makes the shallow layer features containing more background detail information fully utilized by the deep layer and the subsequent convolution layer of the network through information transmission, and enables the network to retain more background details in the cyclic progressive rain removal process.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a method for removing rain from a single image based on dense connections inside and outside the loop and attention enhancement. Background Technology

[0002] In recent years, with the continuous development of chip technology and the increasing computing power of smart devices, some deep learning algorithms have been widely applied, such as object recognition algorithms, semantic segmentation algorithms, and autonomous driving algorithms. Their inputs are mostly single images or continuous videos, where video can be viewed as a collection of related single images along a timeline. The quality of the input image greatly affects the performance of these algorithms. Images acquired in harsh environments (rain, fog, or snow) will lead to poor algorithm performance. Rain is a common weather phenomenon, and rainy images are unavoidable inputs; how to process rainy images has become a concern in the industry. A common solution is to use deraining algorithms to process rainy images, thereby obtaining high-quality rain-free images. Single-image deraining algorithms, because there is no correlation between the input single rainy images, have less usable information and are more difficult to implement compared to video deraining algorithms.

[0003] Traditional single-image deraining algorithms treat the image deraining problem as an image decomposition problem. They model rainy images as a linear superposition of a rain-free background layer and a rain stripe layer, and use various priors to model rain stripes with different directions and scales, such as Gaussian mixture model-based priors, sparsity-based priors, and high-frequency component-based priors. Although these traditional algorithms based on physical or subjective priors show good performance in certain situations, they cannot remove complex rain stripes in real-world scenes where the priors do not hold. In recent years, deep learning techniques, represented by deep convolutional neural networks, have achieved great success in single-image deraining tasks. This is thanks to researchers developing various effective convolutional neural network architectures, including recurrent convolutional neural networks, multi-scale convolutional neural networks, and generative adversarial networks. Although these network models far surpass traditional image deraining algorithms in the quality of the output derained images, they are also prone to under-deraining problems during the deraining process, i.e., failing to fully remove rain stripes from the image, or past-raining problems, i.e., smoothing out background textures during the deraining process. Summary of the Invention

[0004] In order to overcome the above-mentioned shortcomings and deficiencies of the prior art, the purpose of this invention is to provide a single image rain removal method based on dense connections inside and outside the loop and attention enhancement.

[0005] The objective of this invention is achieved through the following technical solution:

[0006] A single-image rain removal method based on dense connections inside and outside the loop and attention enhancement includes:

[0007] A recurrent, densely connected, and attention-enhanced network is constructed for rain removal from a single image. This network includes recurrent subnetworks that share parameters multiple times, specifically:

[0008] The output of a single-cycle sub-network is a residual image, which is added pixel-by-pixel to the input rainy image to obtain the de-rained image output in this cycle.

[0009] The rain-free image output from this loop is concatenated with the rain-input image along the channel dimension as the input for the next loop sub-network, and then passed through N densely connected blocks in sequence;

[0010] Determine the loss function, which includes structural similarity and perceptual loss;

[0011] Input training data and train the network parameters to obtain a cyclic rain removal model.

[0012] Furthermore, the dense connection block includes a convolutional block, an LSRM module, and a basic module.

[0013] Furthermore, the processing procedure for the densely connected block is as follows:

[0014]

[0015] Where [;] indicates that features are concatenated along the channel dimension. Represents pixel-level addition, where t is the circular index, n is the densely connected block index, and B t,n denoted by t, the output of the nth densely connected block in the t-th iteration, BM(·) represents the processing of the basic module, LSTM(·) represents the processing of the long short-term memory operation, Conv(·) represents the convolution operation, and ReLU(·) represents the ReLU activation function.

[0016] Furthermore, the input of the basic module includes the output features of all the densely connected blocks preceding it after processing by the Long Short-Term Memory operation, as well as the input features of the first densely connected block. These features are concatenated along the channel dimension and then input into the convolutional layer. The number of channels is reduced to the number of concatenated features through the convolution operation. t-1 The values ​​are consistent, and then input into the basic module.

[0017] Furthermore, the basic module includes convolutional layers, attention operations, and activation layers.

[0018] Furthermore, the attention operation is a channel attention operation.

[0019] Furthermore, the loss function is expressed by the following formula:

[0020]

[0021] Among them, A T The image represents the derained image output by the last iteration of the recurrent neural network, and GT represents the target rain-free image. SSIM(·) represents the computation of perceptual loss, and SSIM(·) represents the computation of structural similarity.

[0022] Furthermore, the network is trained using backpropagation and gradient descent.

[0023] Furthermore, the initial learning rate of the network is set to 0.001. When the training epochs reach 30, 50 and 80, the learning rate is multiplied by 0.2. Training stops when the epoch reaches 100.

[0024] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0025] 1. This invention incorporates attention operations into the basic modules, enhancing the feature representation capabilities of the network's basic modules;

[0026] 2. This invention establishes dense connections within a loop, enabling deep convolutional layers within a loop to directly obtain features from shallow convolutional layers that contain more background details. This allows the network to retain more background details during rain removal within a loop, effectively alleviating the problems of past rain and under-rain removal.

[0027] 3. This invention embeds multiple LSTM modules at different depths in the network, enabling features at the same depth to transmit information across cycles, establishing dense connections outside the cycles, expanding the channels for network information transmission, and providing more information support for removing rain stripes and preserving background details in the network.

[0028] 4. This invention does not use downsampling operations during feature processing, thus avoiding the loss of some information and changes in scale due to downsampling. At the same time, it is beneficial for features to be directly transmitted at different network depths.

[0029] 5. This invention shares parameters among sub-networks in each iteration, thereby greatly reducing the size of network training parameters and enabling deployment on resource-constrained devices. Attached Figure Description

[0030] Figure 1 This is a flowchart of the process of this invention;

[0031] Figure 2 This is an overall structural diagram of the densely connected inner and outer loops and attention-enhancing network of the present invention.

[0032] Figure 3 This is a structural block diagram of the LSTM module in an embodiment of the present invention.

[0033] Figure 4 This is a structural block diagram of the basic module in an embodiment of the present invention.

[0034] Figure 5 This is a structural block diagram of the attention module in an embodiment of the present invention. Detailed Implementation

[0035] The present invention will be further described in detail below with reference to the embodiments, but the implementation of the present invention is not limited thereto.

[0036] like Figure 1 As shown, this invention establishes dense connections inside and outside each loop of the recurrent neural network and embeds attention operations within the loop, enabling the network to retain more background details during the progressive rain removal process. This invention does not use downsampling operations during feature processing, avoiding the loss of feature information caused by downsampling, while maintaining the feature scale, which is beneficial for the direct transfer of features across different network depths. Furthermore, to address the problem of excessively large neural network parameters caused by multiple loops, this invention shares parameters among the sub-networks in each loop, thereby significantly reducing the size of the network training parameters.

[0037] Specifically, the implementation steps of the method of the present invention are as follows:

[0038] S1. Construct a recurrent, densely connected, and attention-enhanced network for rain removal from a single image. The overall network architecture mainly consists of recurrent subnetworks that share parameters multiple times, and the rain removal is achieved through multiple iterations.

[0039] S2. The output of the single-cycle sub-network is a residual image, which is added pixel-by-pixel to the input rainy image to obtain the rain-free image output in this cycle;

[0040] S3. In the recurrent sub-network, the rain-removed image output by the previous recurrent sub-network is concatenated with the input rainy image along the channel dimension as the input of the next recurrent sub-network;

[0041] S4. Determine the loss function, which consists of structural similarity and perceptual loss;

[0042] S5. Input rainy images and corresponding rainless images as training data, train the network parameters, and obtain a cyclic rain removal model.

[0043] In step S1, the processing procedure of the recurrent sub-network is as follows: the input rainy image is set as I, and the de-raining image output by the previous recurrent sub-network is set as A. t-1 Then the single-cycle subnetwork will connect I and A t-1 The input is concatenated along the channel dimension and then passed through N densely connected blocks sequentially. The processing of each densely connected block is as follows:

[0044]

[0045] Where [;] indicates that features are concatenated along the channel dimension. Represents pixel-level addition, where t is the circular index, n is the densely connected block index, and B t,n denoted by t, the output of the nth densely connected block in the t-th iteration, BM(·) represents the processing of the basic module, LSTM(·) represents the processing of the long short-term memory operation, Conv(·) represents the convolution operation, and ReLU(·) represents the ReLU activation function.

[0046] As shown in the formula, the input of the basic module includes the output features of all the densely connected blocks preceding it after processing by the Long Short-Term Memory (LSTM) operation, as well as the input features of the first densely connected block. These features are concatenated along the channel dimension and then fed into the convolutional layer. The convolution operation reduces the number of channels to the number of concatenated features. [I The values ​​are consistent with [At-1], and then input into the base module.

[0047] The long and short memory operations are as follows Figure 3 As shown, the Long Short-Term Memory (LSTM) operation fuses the current input features with the features retained from the previous loop, and outputs the fused features for this loop. The specific formula is as follows:

[0048]

[0049]

[0050] Where B represents the input of the long short-term memory operation in the current loop, and pre_h and pre_c represent the output of the long short-term memory operation in the previous loop. If it is the first loop, it is a blank tensor with the same scale as B, and h and c represent the output of the long short-term memory operation in the current loop. This indicates pixel-level multiplication. Tanh(·) represents the Tanh activation function, and Sigmoid(·) represents the Sigmoid activation function.

[0051] The basic modules are as follows Figure 4 As shown, it consists of convolutional layers, attention operations, and activation layers, and its input feature f in The output feature f is obtained by passing the data through a convolutional layer, attention operation, and activation function twice in sequence. out Specifically, it can be expressed by the formula:

[0052] f out =LR(CA(conv(LR(CA(Conv(f)) in ))))))

[0053] Where LR(·) represents the LeakyReLU activation function, and CA(·) represents the attention operation process.

[0054] The attention operation is as follows Figure 5 The diagram shows the channel attention operation, specifically expressed by the following formula:

[0055]

[0056] Where x represents the input of the attention operation, y represents the output of the attention operation, and Avg(·) represents the average pooling operation performed on the features of each channel.

[0057] In step S2, as Figure 2 As shown, in the t-th iteration, the feature B output by the last densely connected block is... t,N After passing through a convolutional layer, the image size is restored to match the input rainy image. Then, the image is added pixel-by-pixel to the input rainy image to obtain the de-rained image output in the t-th iteration. The specific formula is as follows:

[0058]

[0059] Among them, A t B represents the derained image output by the t-th recurrent subnetwork. t,N This represents the characteristic output of the Nth densely connected block in the t-th iteration.

[0060] In step S3, the input to the recurrent subnetwork is as follows: For the input single rainy image I, it is concatenated with its copy along the channel dimension during the first loop and used as the input for the first loop. In subsequent loops, I is combined with the rain-free image A output from the previous loop of the network. t-1 The data is concatenated along the channel dimension as input for the next iteration, as expressed by the following formula:

[0061]

[0062] in, This represents the input of the t-th cyclic subnetwork.

[0063] The network is trained in a loop for T iterations, or other stopping conditions depending on specific project requirements. The rain-removed image A is output by the recurrent neural network in the last iteration. T This is the final result of the rain-removal process.

[0064] In step S4, the loss function consists of structural similarity and perceptual loss, and is expressed by the following formula:

[0065]

[0066] Among them, AT The image represents the derained image output by the last iteration of the recurrent neural network, and GT represents the target rain-free image. SSIM(·) represents the computation of perceptual loss, and SSIM(·) represents the computation of structural similarity.

[0067] The perceptual loss is obtained by extracting A using a pre-trained VGG16 neural network. T For features of ground truth (GT) at different depths in the network, calculate the weighted sum of L2 distances between the features at different depths of the two networks, as expressed by the following formula:

[0068]

[0069] Where, φ i (·) represents the pre-trained VGG16 model. The method of this invention extracts features from layers 4, 9, 16, and 23 of VGG16. α i The weights for the L2 distance between different depth features of an image are set as follows in the method of this invention: And 1.0, ||·||2 indicates the calculation of L2 distance.

[0070] In step S5, the method of the present invention uses backpropagation and gradient descent to train the network. The initial learning rate of the network is set to 0.001. When the training epochs reach 30, 50 and 80, the learning rate is multiplied by 0.2. When the epochs reach 100, training is stopped, and the trained model is obtained.

[0071] To address the issue of insufficient rain removal, this invention employs a recurrent neural network (RNN) to more thoroughly remove rain stripes through multiple iterations. However, this process can lead to the loss of background details and the problem of past rain. Therefore, this invention establishes dense connections within and outside the loops, expanding the information transmission channels within the recurrent network. This allows shallow features containing more background details to be fully utilized during information transmission, alleviating the past rain problem. Furthermore, downsampling is avoided during rain removal, preventing the loss of feature information and scale changes caused by downsampling, which facilitates direct feature transmission across different network depths. In addition, considering that the network model's storage size may hinder deployment and application in resource-constrained scenarios, the RNN can reduce the network parameter size through a recurrent parameter sharing strategy, making it easier to deploy on mobile devices with limited storage space.

[0072] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the embodiments described above. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A single-image rain removal method based on dense connections inside and outside the loop and attention enhancement, characterized in that, include: A recurrent, densely connected, and attention-enhanced network is constructed for rain removal from a single image. This network includes recurrent subnetworks that share parameters multiple times, specifically: The output of a single-cycle sub-network is a residual image, which is added pixel-by-pixel to the input rainy image to obtain the de-rained image output in this cycle. The rain-free image output from this loop is concatenated with the rain-input image along the channel dimension as the input for the next loop sub-network, and then passed through N densely connected blocks in sequence; Determine the loss function, which includes structural similarity and perceptual loss; Input training data, train network parameters to obtain a cyclic deraining model, which is used for deraining single images; The dense connection block includes a convolutional block, an LSTM module, and a basic module; The processing procedure for the densely connected block is as follows: in, This indicates that features are concatenated along the channel dimension. Represents pixel-level addition. For circular index, For densely joined block indexes, Indicates the first The second cycle Features of the output of a densely connected block This indicates the processing procedure of the basic module. This represents the processing steps involved in long and short-term memory operations. This represents the convolution operation. Represents the ReLU activation function; A t-1 Indicates the first -1 times the output of the rain-free image from the recurrent subnetwork, where I is the input single rainy image; The input to the basic module includes the output features of all the densely connected blocks preceding it after Long Short-Term Memory (LSTM) operations, as well as the input features of the first densely connected block. These features are concatenated along the channel dimension and then input into the convolutional layer. The convolution operation reduces the number of channels to the number of concatenated features. Consistent, then input into the basic module; The basic module includes convolutional layers, attention operations, and activation layers; its input features The output features are obtained by passing the data through a convolutional layer, attention operation, and activation function twice in sequence. Specifically, it can be expressed by the formula: in, This represents the LeakyReLU activation function. This describes the processing steps involved in attention operations. The Long Short-Term Memory (LSTM) operation fuses the current input features with the features retained from the previous loop, outputting the fused features for the current loop. The specific formula is as follows: in, This represents the input for the current cycle of long short-term memory operations. This represents the output of the Long Short-Term Memory operation in the previous loop; if it's the first loop, it's related to... Uniformly sized blank tensors This represents the output of the long short-term memory operation in this iteration. This indicates pixel-level multiplication. This represents the Tanh activation function. This represents the Sigmoid activation function; No downsampling operation is used during feature processing.

2. The single-image rain removal method according to claim 1, characterized in that, The attention operation is a channel attention operation.

3. The single-image rain removal method according to claim 1, characterized in that, The loss function is expressed by the following formula: in, This represents the derained image output by the recurrent neural network in its last iteration. This indicates a rainless image of the target. This indicates the calculation of perceived loss. This indicates the calculation of structural similarity.

4. The single-image rain removal method according to claim 1, characterized in that, The network was trained using backpropagation and gradient descent.

5. The single-image rain removal method according to claim 4, characterized in that, The initial learning rate of the network is set to 0.

001. When the training epochs reach 30, 50 and 80, the learning rate is multiplied by 0.

2. Training stops when the epoch reaches 100.

Images