An uncertainty minimization rain removal method based on self-calibration network

Abstract

This invention discloses a rain removal method based on a self-calibration network that minimizes uncertainty. The method includes: selecting a training set from a dataset to obtain m pairs of images as training samples; constructing a self-calibration network, which includes an input layer, multiple identical self-calibration modules, and a corresponding output layer; constructing an uncertainty loss function to minimize rain removal uncertainty, and jointly optimizing the self-calibration network using a rain residual loss function and a recovery loss function; inputting the training samples into the optimized self-calibration network to obtain the optimal rain removal model; selecting a test set from the dataset, and inputting the test samples into the optimal rain removal model to obtain the final rain-removed image. This method addresses the problem of existing multi-scale rain removal networks easily losing image spatial information.

Description

Technical Field

[0001] This invention relates to the fields of image processing and computer vision, and in particular to a method for minimizing uncertainty and removing rain from a self-calibrating network. Background Technology

[0002] Inclement outdoor weather conditions, such as rain, snow, and fog, can degrade image quality, affecting not only visual appeal but also various low-level visual tasks. In recent years, the restoration of rainy images has received increasing attention, particularly in autonomous driving and crime detection. However, rainy images contain rain streaks of varying sizes and orientations, and the linear occlusion of these streaks, along with the lack of texture detail in occluded areas, leads to severe image degradation. Therefore, image deraining remains a challenging problem.

[0003] To address this problem, numerous technologies are under development, and these algorithms can be categorized into two types. One type is video-based deraining algorithms, which utilize inter-frame information to extract rain stripe information or occluded background information. The other type is single-image-based deraining algorithms. Traditional single-image-based deraining algorithms primarily attempt to acquire prior knowledge to remove rain from a single image. This prior knowledge mainly includes sparsity and high-frequency information. However, the ability to utilize prior knowledge is limited; it cannot simulate the diverse rain stripes found in nature, easily leading to under-deraining or the removal of high-frequency background information. With the development of deep learning, the field of deraining has also achieved significant breakthroughs. Many researchers have begun to use neural networks to learn the rain stripe features or background features of input rainy images to achieve single-image deraining.

[0004] However, existing deep learning-based single-image rain removal algorithms are trained on synthetic datasets, and the rain streaks in the synthetic rain images are all in the same direction. In reality, the state of rain streaks is related to the scene's depth density. The area closer to the lens is dominated by rain streaks, but in areas farther from the lens, the stacking of rain streaks of different directions and sizes creates a fog effect. Therefore, how to completely remove foggy rain streaks and restore the rain-obscured background remains a major challenge that needs to be addressed and solved in the field of rain removal. Summary of the Invention

[0005] In order to overcome the above-mentioned shortcomings and deficiencies of the prior art, the purpose of this invention is to provide a rain removal method based on uncertainty minimization using a self-calibrating network.

[0006] This method continuously expands the receptive field of the network through multiple self-calibration modules and forms a self-calibration network to perform self-calibration processing on the input features to obtain more refined rain stripe information. In addition, an uncertainty loss function is used to reduce the uncertainty error of each self-calibration module in rain removal, so as to achieve a good rain removal effect.

[0007] The objective of this invention is achieved through the following technical solution:

[0008] A rain removal method based on uncertainty minimization using a self-calibration network includes:

[0009] Select the training set from the dataset, scale the rainy images and the corresponding rainless images in the training set to size h×w, and obtain m pairs of images as training samples.

[0010] A self-calibration network is constructed, which includes an input layer, multiple self-calibration modules with identical structures, and a corresponding output layer.

[0011] An uncertainty loss function is constructed to minimize the uncertainty of rain removal, and the self-calibration network is optimized by combining the rain residual loss function and the recovery loss function.

[0012] The training samples are input into the optimized self-calibration network to obtain the optimal rain removal model;

[0013] Select the test set from the dataset, scale the original rain image and the corresponding rainless image to size h×w as test samples; input the test samples into the optimal rain removal model to obtain the final rain-removed image.

[0014] Furthermore, the multiple identical self-calibration modules are cascaded in sequence, and each self-calibration module corresponds to an output layer.

[0015] Furthermore, the self-calibration module is defined as follows:

[0016]

[0017]

[0018] in, M represents the (i-1)th self-calibration module. i-1 The output characteristics, F i This represents the output features of two convolutional layers with a kernel size of 3 in the self-calibration module. Sigmoid represents the Sigmoid activation function, SA represents spatial attention, and CA represents channel attention. This represents pixel-level summation. This represents pixel-level product.

[0019] Furthermore, the spatial attention mechanism generates a 0 / 1 weight mask for each location and then performs weighted summation to enhance the target region of interest while weakening irrelevant background regions. The specific implementation is as follows:

[0020]

[0021] Where c represents the number of channels of the input feature F, F c (i, j) represents the position of feature F in channel c. F represents the sum of values ​​at the same position in different channels. SA This represents the output characteristics of the spatial attention mechanism.

[0022] Furthermore, the channel attention mechanism aims to model the correlation between different channels, obtain the different importance levels of each channel, and then assign different weight coefficients to each channel to strengthen important features and suppress unimportant features. The specific implementation is as follows:

[0023]

[0024] Where GAP represents global average pooling, FC represents a fully connected layer, and F... CA This represents the output characteristics of the channel attention mechanism.

[0025] Furthermore, an uncertainty loss function is constructed to minimize rain removal uncertainty. Simultaneously, the rain residual loss function and the recovery loss function are combined to optimize the self-calibration network, specifically:

[0026] Establish a joint total loss function, which includes: uncertainty loss function, residual loss function, and recovery loss function, as shown in the following formula:

[0027]

[0028] in, Represents the uncertainty loss function. Represents the residual loss function for rain. This represents the rain removal loss function. This represents the total loss function of the rain removal method based on minimizing uncertainty using a self-calibrating network.

[0029] further,

[0030] Constructing an uncertainty loss function to minimize rain removal uncertainty is as follows:

[0031] The original input I of the network is compared with the rain residual image R output by the corresponding output layer of each self-calibration module. i Subtraction yields the rain-removed image O iNext, the variance is calculated and minimized to effectively reduce the rain-induced uncertainty error of each self-calibration module, as defined below:

[0032] O i =IR i

[0033]

[0034] Among them, R i For the rain residual output by the i-th self-calibration module, O i Let be the derained image output by the i-th self-calibration module, and n be the number of self-calibration modules in the network.

[0035] Furthermore, the final clean background image is recovered using the recovery loss function, resulting in the final derained image O. n Calculate using the following formula:

[0036]

[0037] Where MSE represents mean squared error.

[0038] Furthermore, the rain residual loss function is used to supervise the network to learn more refined rain stripes.

[0039] A computer storage medium storing a program that, when executed by a processor, implements the uncertainty minimization rain removal method.

[0040] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0041] This invention provides an uncertainty-minimizing deraining method based on a self-calibrating network, aiming to address the problem of existing multi-scale deraining networks easily losing image spatial information. Compared with existing technologies, the beneficial effects are that the algorithm of this invention proposes a self-calibrating image deraining network with uncertainty minimization. It expands the receptive field of the self-calibrating network by superimposing multiple self-calibrating modules, and each module can perform self-calibration processing on input features to obtain more refined rain stripe information. Simultaneously, it proposes an uncertainty loss function to minimize the deraining uncertainty, thereby reducing the uncertainty error of each self-calibrating module. Attached Figure Description

[0042] Figure 1 This is a structural diagram of the present invention;

[0043] Figure 2 yes Figure 1 A structural diagram of the self-calibration module in the system;

[0044] Figure 3 yes Figure 1 Structural diagram of the mid-space attention mechanism SA;

[0045] Figure 4 yes Figure 1 Structural diagram of the mid-channel attention mechanism CA;

[0046] Figure 5 This is a flowchart of the process of the present invention. Detailed Implementation

[0047] The present invention will be further described in detail below with reference to the embodiments, but the implementation of the present invention is not limited thereto.

[0048] like Figure 1 or Figure 5 As shown, this invention provides a method for rain removal based on uncertainty minimization using a self-calibrating network, comprising the following steps:

[0049] Step A: Select the training set in the dataset, scale the rainy images and the corresponding rainless images in the training set to size h×w, and obtain m pairs of images as training samples.

[0050] Step B: Construct a self-calibration network, which mainly includes an input layer and six structurally identical self-calibration modules (SCMs). i And the corresponding output layer composition;

[0051] Step C: Construct an uncertainty loss function to minimize rain removal uncertainty, and combine the rain residual loss function and the recovery loss function to jointly optimize the self-calibration network;

[0052] Step D: Input the training samples into the self-calibration network. The input layer extracts shallow information from the input image; this is achieved by cascading six self-calibration modules (SCM). i To continuously expand the network's receptive field and calibrate the features learned by the network, in order to capture more accurate rain stripe information, obtain the optimal rain removal model, and achieve a good rain removal effect;

[0053] Step E: Select the test set from the dataset, scale the rainy images and the corresponding rainless images to h×w size as test samples; input the test samples into the trained self-calibration network to obtain the final rain-free images.

[0054] Further, step B includes the following steps:

[0055] Step B1: To extract shallow feature information, the input layer is defined as follows:

[0056] F swa =ReLU(Bn(Conv) 7 (I)))

[0057] Where I represents the input rainy image, Conv 7 This indicates a convolutional layer with a kernel size of 7, Bn indicates a BatchNorm layer, ReLU indicates the ReLU activation function, and F... swa This represents the output features of the input layer.

[0058] Step B2: By cascading six self-calibration modules, the receptive field of the network is continuously expanded, and the learned intermediate features are self-calibrated to obtain more refined rain stripe information. Each self-calibration module (SCM) i The definition is as follows:

[0059]

[0060]

[0061] in, M represents the (i-1)th self-calibration module. i-1 The output characteristics, F i This represents the output features of two convolutional layers with a kernel size of 3 in the self-calibration module. Sigmoid represents the Sigmoid activation function, SA represents spatial attention, and CA represents channel attention. This represents pixel-level summation. This represents pixel-level product.

[0062] Step B3: Each self-calibration module corresponds to an output layer. Define the rain residual image output by the output layer as R. i The definition is as follows:

[0063]

[0064] Further, step B2 includes the following steps:

[0065] Step B21: The spatial attention mechanism generates a 0 / 1 weight mask for each location and performs weighted summation, thereby enhancing the target region of interest while weakening irrelevant background regions. The specific implementation is as follows:

[0066]

[0067] Where c represents the number of channels of the input feature F, F c (i, j) represents the position of feature F in channel c. F represents the sum of values ​​at the same position in different channels. SA This represents the output characteristics of the spatial attention mechanism.

[0068] Step B22: The channel attention mechanism aims to model the correlation between different channels, obtain the different importance levels of each channel, and then assign different weight coefficients to each channel to strengthen important features and suppress unimportant features. The specific implementation is as follows:

[0069]

[0070] Where GAP represents global average pooling, FC represents a fully connected layer, and F... CA This represents the output characteristics of the channel attention mechanism.

[0071] Further, step C includes the following steps:

[0072] Based on the sparsity of rain stripes and variance, a joint total loss function is proposed for network optimization. The total loss function mainly includes: an uncertainty loss function, a rain residual loss function, and a rain removal loss function. It is calculated using the following formula:

[0073]

[0074] in, Represents the uncertainty loss function. Represents the residual loss function for rain. This represents the rain removal loss function. This represents the total loss function of the rain removal method based on minimizing uncertainty using a self-calibrating network.

[0075] Specifically, the uncertainty loss function is used to calculate an uncertainty value for the final rain removal result. This uncertainty value is the variance mentioned in the formula of the uncertainty loss function. The rain residual loss function and the rain removal loss function are used to supervise the network to learn more refined rain stripes and restore the final clean background image.

[0076] Construct an uncertainty loss function to minimize the uncertainty of rain removal, as follows:

[0077] The original input I of the network is compared with the rain residual image R output by the corresponding output layer of each self-calibration module. i Subtraction yields the rain-removed image O i Next, the variance is calculated and minimized to effectively reduce the rain-induced uncertainty error of each self-calibration module, as defined below:

[0078] O i =IR i

[0079]

[0080] Among them, R iFor the rain residual output by the i-th self-calibration module, O i This is the derained image output by the i-th self-calibration module. In this implementation example, the number of self-calibration modules in the network is 6.

[0081] The rain residual loss function calculates the rain stripe information learned at each stage, using the following formula:

[0082]

[0083] Where R6 represents the rain stripe image of the output layer corresponding to the 6th self-calibration module, R I This represents the rain stripe image corresponding to the original rainy image.

[0084] The recovery loss function is used to calculate the final derained image O6 of the network, using the following formula:

[0085]

[0086] Where MSE represents mean squared error.

[0087] The self-calibration module in this embodiment is as follows: Figure 2 As shown, the input features are first passed into two convolutional layers with batch normalization (BatchNorm) and ReLU activation functions to obtain an output feature. This output feature is then passed through a convolutional layer with a kernel size of 3 and a convolutional layer with a kernel size of 5, and then added together. The added feature is then normalized to between 0 and 1 by an activation function (Sigmoid). The normalized feature is then multiplied by the output feature and added together. The added feature is then input into the spatial attention module (SA). The feature output by SA is then input into the channel attention module (CA). The feature output by CA is then added to the input of the current self-calibration module to obtain the final output feature.

[0088] like Figure 3 and Figure 4 As shown, in this embodiment, the spatial attention mechanism is set after a convolutional layer with a kernel size of 3 and a convolutional layer with a kernel size of 5 in the self-calibration module. The channel attention mechanism is set after the spatial attention mechanism. The spatial attention mechanism and the channel attention mechanism are also cascaded.

[0089] Another embodiment of the present invention provides a computer storage medium storing a program that, when executed by a processor, implements the uncertainty minimization rain removal method described above.

[0090] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the embodiments described above. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A rain removal method based on uncertainty minimization using a self-calibrating network, characterized in that, include: Select the training set from the dataset, scale the rainy images and the corresponding rainless images in the training set to size h×w, and obtain m pairs of images as training samples. A self-calibration network is constructed, which includes an input layer, multiple self-calibration modules with identical structures, and a corresponding output layer. An uncertainty loss function is constructed to minimize the uncertainty of rain removal, and the self-calibration network is optimized by combining the rain residual loss function and the recovery loss function. The training samples are input into the optimized self-calibration network to obtain the optimal rain removal model; Select the test set from the dataset, and scale the original rain image and the corresponding rainless image to h×w size as the test sample. The test samples are input into the optimal rain removal model to obtain the final rain-removed image; The multiple self-calibration modules with the same structure are cascaded in sequence, and each self-calibration module corresponds to an output layer; The self-calibration module is defined as follows: in, This represents the (i-1)th self-calibration module. The output characteristics, This represents the output features of two convolutional layers with a kernel size of 3 in the self-calibration module. This represents the Sigmoid activation function. This represents the spatial attention mechanism. This indicates the channel attention mechanism. This represents pixel-level summation. Represents pixel-level product; The spatial attention mechanism generates a 0 or 1 weight mask at each location and then performs weighted summation to enhance the target region of interest while weakening irrelevant background regions. The specific implementation is as follows: in, F represents the number of channels of the input feature F. c (p, q) represents the position of feature F in channel c. This represents the sum of values ​​at the same position in different channels. Represents the output features of the spatial attention mechanism; The channel attention mechanism aims to model the correlation between different channels, obtain the different importance levels of each channel, and then assign different weight coefficients to each channel to strengthen important features and suppress unimportant features. The specific implementation is as follows: in, This indicates global average pooling, and FC indicates a fully connected layer. This represents the output characteristics of the channel attention mechanism.

2. The uncertainty minimization rain removal method according to claim 1, characterized in that, An uncertainty loss function is constructed to minimize the uncertainty of rain removal. Simultaneously, the rain residual loss function and the rain removal loss function are combined to optimize the self-calibration network. Specifically: A joint total loss function is established, which includes: the uncertainty loss function, the rain residual loss function, and the rain removal loss function, as shown in the following formula: in, Represents the uncertainty loss function. Represents the residual loss function for rain. This represents the rain removal loss function. This represents the total loss function of the rain removal method based on minimizing uncertainty using a self-calibrating network.

3. The uncertainty minimization rain removal method according to claim 2, characterized in that, Constructing an uncertainty loss function to minimize rain removal uncertainty is as follows: The original input I of the network is compared with the rain residual image output from the corresponding output layer of each self-calibration module. Subtraction yields the rain-removed image. Next, the variance is calculated and minimized to effectively reduce the rain-induced uncertainty error of each self-calibration module, as defined below: in, The rain residual output by the i-th self-calibration module. Let be the derained image output by the i-th self-calibration module, and n be the number of self-calibration modules in the network.

4. The uncertainty minimization rain removal method according to claim 2, characterized in that, The final clean background image is recovered using a rain removal loss function, resulting in the final rain-removed image. Calculate using the following formula: Where MSE represents mean squared error.

5. The uncertainty minimization rain removal method according to claim 2, characterized in that, The rain residual loss function is used to supervise the network to learn more refined rain stripes.

6. A computer storage medium, characterized in that, The storage medium stores a program that, when executed by a processor, implements the uncertainty minimization rain removal method as described in any one of claims 1 to 5.

Images