Single image rain removal method based on multi-level entropy module

Abstract

The application provides a single image rain removal method based on a multi-level entropy module, comprising the following steps: S1, constructing a rain removal network model: composed of a preprocessing module, a plurality of multi-level entropy modules and an image reconstruction module; S2, designing a loss function, and using the loss function to constrain the rain removal network model; S3, using public data to train the rain removal network model of step S1, and obtaining model parameters of the rain removal network model; S4, importing the model parameters trained in step S3 into the network model, inputting a rainy image, and outputting a rain-removed image. The application regards the process of recovering a clean image from a rainy image as an entropy-increasing process, removes the increased image entropy information through modeling of the front and rear features, and makes the attention modeling process not only depend on the current feature, thereby improving the effect of single image rain removal.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a single image rain removal method based on a multi-level entropy module. Background Art

[0002] Severe weather conditions can cause cameras and other image acquisition devices to capture low-quality images, often suffering from loss of detail and signal distortion. However, high-quality, clean images are a prerequisite for many computer vision applications. Rainy conditions are a typical example of inclement weather. Rain streaks blur the image and cause significant loss of information. Rainy conditions can affect image quality, impacting visual perception and reducing the performance of computer vision algorithms. Deraining a single image remains a challenging task.

[0003] The self-attention mechanism has been widely used in research fields such as single-image rain removal. However, existing self-attention mechanisms usually rely on the current feature as input and use a 1×1 convolution module to act on the current feature to generate attention features. Although the effect of rain line removal is good, it cannot use features at other levels to extract relevant features, which leads to certain limitations. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a single image deraining method based on a multi-level entropy module. The process of recovering a clean image from a rainy image can be regarded as an entropy increase process. By modeling the previous and next features to remove the increased image entropy information, the attention modeling process is made to not only rely on the current features, thereby improving the effect of single image deraining.

[0005] To solve the above technical problems, an embodiment of the present invention provides a single image rain removal method based on a multi-level entropy module, comprising the following steps:

[0006] S1. Build a rain removal network model: It consists of a preprocessing module, several multi-level entropy modules, and an image reconstruction module;

[0007] S2. Design a loss function and use it to constrain the rain removal network model;

[0008] S3. Using public data to train the rain removal network model in step S1 to obtain model parameters of the rain removal network model;

[0009] S4. Import the model parameters trained in step S3 into the network model, input the rainy image, and output the rain-free image.

[0010] Furthermore, in step S1, one branch of the input image is connected to the preprocessing module, and the other branch is added with the features processed by several multi-level entropy modules to improve the ability to express the detailed features of the image.

[0011] Furthermore, the preprocessing module and the image reconstruction module are both composed of a 3×3 convolution module. The input of the preprocessing module is the image to be derained, the number of channels of the image is 3, and the output is N-channel features; the output of the image reconstruction module is the derained image, the number of channels of the image is 3.

[0012] Furthermore, the multi-level entropy module is composed of a 3×3 convolutional layer, a ReLU activation function, a 3×3 convolutional layer, and a multi-level entropy modeling component, which are arranged in sequence. The steps are as follows:

[0013] (1) Input feature x i , through a 3×3 convolutional layer, using the ReLU activation function, the output feature f1 is obtained:

[0014] f1=conv(θ(conv(x i ))),

[0015] Where conv represents a 3×3 convolutional layer and θ is the ReLU activation function;

[0016] (2) x i Adding it to f1, we get:

[0017] f2=x i +f1;

[0018] (3) Input f2 to the multi-level entropy modeling component to obtain f3, and output f2+f3.

[0019] Furthermore, the multi-level entropy modeling component divides the input features into multiple layers of sub-features from the channel level, establishes an entropy-based network model for each layer of sub-features, and then merges the processed sub-features of each layer. The specific steps are as follows:

[0020] (1) Input feature x j , the number of channels is C, from the channel level x j Divide into K sub-blocks and obtain C / K sub-features

[0021] (2) For each sub-feature Perform entropy modeling and obtain

[0022]

[0023] in, Represents a shallow feature extraction network, X0 and X are Input and output;

[0024] (3) For all Perform channel superposition and 1×1 convolution operations in sequence to obtain x' j , input feature x j and x' j Add, output x j +x' j .

[0025] Furthermore, the loss function in step S2 is defined as follows:

[0026]

[0027] Among them, F s is a smooth function, defined as: Represents the pixel value of channel i of the true image, represents the derained image obtained after being processed by the deraining network model constructed in step S1, express The pixel value of channel i is , and N is the number of image pairs.

[0028] Furthermore, the steps for training the rain removal network model are as follows:

[0029] (1) Select a sample set of rainy and non-rainy image pairs, 12,000 each, and 1,200 pairs of test images;

[0030] (2) The optimization function is Adam, with parameters β1 = 0.9 and β2 = 0.99, and the learning rate is set to 0.001;

[0031] (3) Train for 200 rounds and test the results every 20 rounds. The test results are measured using peak signal-to-noise ratio and structural similarity. The best result is selected to save the model parameters.

[0032] The beneficial effects of the above technical solution of the present invention are as follows:

[0033] The present invention regards the process of restoring a clean image from a rainy image as an entropy increase process. By modeling the previous and next features, the increased image entropy information is removed, so that the attention modeling process does not only rely on the current features, thereby improving the effect of single image deraining. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a diagram of the rain removal network model in the present invention;

[0035] Figure 2 This is a network model diagram of the multi-level entropy module in the present invention;

[0036] Figure 3 A network model diagram of a multi-level entropy modeling component in the present invention;

[0037] Figure 4 This is a comparison diagram of an image before and after rain removal in Example 1 of the present invention;

[0038] Figure 5 This is a comparison diagram of an image before and after rain removal in Example 2 of the present invention. DETAILED DESCRIPTION

[0039] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0040] An embodiment of the present invention provides a single image rain removal method based on a multi-level entropy module, comprising the following steps:

[0041] S1. Build a rain removal network model: It consists of a preprocessing module, several multi-level entropy modules and an image reconstruction module (such as Figure 1 shown);

[0042] S2. Design a loss function and use it to constrain the rain removal network model;

[0043] S3. Using public data to train the rain removal network model in step S1 to obtain model parameters of the rain removal network model;

[0044] S4. Import the model parameters trained in step S3 into the network model, input the rainy image, and output the rain-free image.

[0045] In step S1, one branch of the input image is connected to the preprocessing module, and the other branch is added with the features processed by several multi-level entropy modules to improve the ability to express the detailed features of the image.

[0046] The preprocessing module and the image reconstruction module are both composed of a 3×3 convolution module. The input of the preprocessing module is the image to be derained, the number of channels of the image is 3, and the output is N-channel features; the output of the image reconstruction module is the derained image, the number of channels of the image is 3.

[0047] like Figure 2 As shown in the figure, the multi-level entropy module is composed of a 3×3 convolutional layer, a ReLU activation function, a 3×3 convolutional layer, and a multi-level entropy modeling component, which are set in sequence. The steps are as follows:

[0048] (1) Input feature x i , through a 3×3 convolutional layer, using the ReLU activation function, the output feature f1 is obtained:

[0049] f1=conv(θ(conv(xi ))),

[0050] Where conv represents a 3×3 convolutional layer and θ is the ReLU activation function;

[0051] (2) x i Adding it to f1, we get:

[0052] f2=x i +f1;

[0053] (4) Input f2 to the multi-level entropy modeling component to obtain f3, and output f2+f3.

[0054] The multi-level entropy modeling component performs entropy modeling on the feature layers separately. The specific steps are as follows:

[0055] (1) Input feature x j , the number of channels is C, from the channel level x j Divide into K sub-blocks and obtain C / K sub-features

[0056] (2) For each sub-feature Perform entropy modeling and obtain

[0057]

[0058] in, Represents a shallow feature extraction network, X0 and X are Input and output;

[0059] (4) For all Perform channel superposition and 1×1 convolution operations in sequence to obtain x' j , input feature x j and x' j Add, output x j +x' j .

[0060] In step S2, the model is constrained using a loss function, which is defined as follows:

[0061]

[0062] Among them, F s is a smooth function, defined as: Represents the pixel value of channel i of the true image, represents the derained image obtained after being processed by the deraining network model constructed in step S1, express The pixel value of channel i is , and N is the number of image pairs.

[0063] The steps for training are:

[0064] (1) Select a sample set of "rainy-no-rainy" image pairs, 12,000 each, and 1,200 pairs of test images;

[0065] (2) The optimization function is Adam, with parameters β1 = 0.9 and β2 = 0.99, and the learning rate is set to 0.001;

[0066] (3) Train for 200 rounds and test the results every 20 rounds. The test results are measured using peak signal-to-noise ratio (PSNR) and structural similarity (SSIM). The best result is selected to save the model parameters.

[0067] In this embodiment, the process of restoring a rainy image to a clean image is regarded as an entropy increase process. According to Shannon's information theory, the entropy of the variable Y is defined as:

[0068]

[0069] where y i is the probability of Y occurring, Is the probability density function. For the continuous feature variables in the neural network Its probability density function is p(x), then The entropy of can be defined as:

[0070]

[0071] for The previous level characteristic variable The relevant information entropy between them is:

[0072]

[0073] and It can be seen as from Subtract therefore pass Such entropy modeling operations can obtain feature variables with lower entropy. By modeling the previous and next features to remove the increased image entropy information, the attention modeling process is not only dependent on the current features, but also has a highly ordered and compact feature representation, thereby improving the effect of single image deraining.

[0074] In order to verify the innovation of this invention, tests were conducted. The test results are as follows: using Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) as the metrics, PSNR=36.62, SSIM=0.9901 on the RESIDE indoor test set, and PSNR=34.71, SSIM=0.9904 on the outdoor test set, both of which are better than the existing methods. Figure 4 、 Figure 5 shown.

[0075] in, Figure 4 a and Figure 5 a is the image before rain, Figure 4 b and Figure 5 b is the image after rain removal.

[0076] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A single image rain removal method based on a multi-level entropy module, characterized in that: The following steps are involved: S1. Build a rain removal network model: It consists of a preprocessing module, several multi-level entropy modules, and an image reconstruction module; S2. Design a loss function and use it to constrain the rain removal network model; S3. Using public data to train the rain removal network model in step S1 to obtain model parameters of the rain removal network model; S4. Import the model parameters trained in step S3 into the network model, input the rainy image, and output the rain-free image; The multi-level entropy module is composed of a 3×3 convolutional layer, a ReLU activation function, a 3×3 convolutional layer, and a multi-level entropy modeling component. The steps are as follows: (1) Input feature x i , through a 3×3 convolutional layer, using the ReLU activation function, the output feature f1 is obtained: f1=conv(θ(conv(x i ))), Where conv represents a 3×3 convolutional layer and θ is the ReLU activation function; (2) x i Adding it to f1, we get: f2=x i +f1; (3) Input f2 to the multi-level entropy modeling component to obtain f3, and output f2+f3; the multi-level entropy modeling component divides the input features into multiple layers of sub-features from the channel level, and establishes an entropy-based network model for the sub-features of each level, and then merges the processed sub-features of each level. The specific steps are as follows: (1) Input feature x j , the number of channels is C, from the channel level x j Divide into K sub-blocks and obtain C / K sub-features (2) For each sub-feature Perform entropy modeling and obtain in, Represents a shallow feature extraction network, X0 and X are Input and output; (3) For all Perform channel superposition and 1×1 convolution operations in sequence to obtain x' j , output x j +x' j .

2. The single image rain removal method based on a multi-level entropy module according to claim 1, characterized in that: In step S1, one branch of the input image is connected to the preprocessing module, and the other branch is added with the features processed by several multi-level entropy modules.

3. The single image rain removal method based on multi-level entropy modules according to claim 1, characterized in that: The preprocessing module and the image reconstruction module are both composed of a 3×3 convolution module.

4. The single image rain removal method based on multi-level entropy modules according to claim 1, characterized in that: The loss function in step S2 is defined as follows: Among them, F s is a smooth function, defined as: Represents the pixel value of channel i of the true image, represents the derained image obtained after being processed by the deraining network model constructed in step S1, express The pixel value of channel i is , and N is the number of image pairs.

5. The single image rain removal method based on multi-level entropy modules according to claim 1, characterized in that: The steps for training the rain removal network model are: (1) Select a sample set of rainy and non-rainy image pairs, 12,000 each, and 1,200 pairs of test images; (2) The optimization function is Adam, with parameters β1 = 0.9 and β2 = 0.99, and the learning rate is set to 0.001; (3) Train for 200 rounds and test the results every 20 rounds. The test results are measured using peak signal-to-noise ratio and structural similarity. The best result is selected to save the model parameters.

Images