Image raindrop removing method based on uncertainty perception sparse Transform network

Abstract

The invention discloses an image raindrop removing method based on an uncertainty perception sparse Transform network, and the method comprises the steps: firstly, capturing the global sparse features of a raindrop-containing image through employing a sparse Transform block and a Top-k sparse attention mechanism, and inhibiting the redundant and irrelevant feature representation; secondly, providing an uncertainty branch module which is used for generating an uncertainty graph representing the image degradation degree, and adopting an uncertainty driving loss function to enable a network to effectively remove raindrops and restore background information of a scene; and finally, a pyramid feature refining module is provided, local features in a multi-scale perception domain are fully mined, and effective expression of deepest layer features is stimulated by using a residual channel space attention mechanism. Experimental results show that raindrops in the image can be effectively removed, rich background detail information can be restored, and the method has high applicability in actual scenes.

Description

Technical Field

[0001] This invention relates to the fields of digital image processing and surveillance video processing, and in particular to an image raindrop removal method based on uncertainty-aware sparse Transformer networks. Background Technology

[0002] In rainy weather, raindrops adhering to camera lenses often cause severe content loss in images captured by cameras, such as edges, textures, colors, and background details. This not only reduces the visual quality of the image but also adversely affects various advanced semantic perception tasks, such as object detection, semantic segmentation, and autonomous driving. Image deraining aims to recover a clear image from images containing raindrops, and has high application value and research significance.

[0003] In recent years, deep learning has been widely applied in fields such as computer vision and digital image processing. With the surge in deep learning, image raindrop removal technology has developed rapidly. Currently, deep learning-based image raindrop removal methods mainly rely on three mainstream models: convolutional neural network (CNN)-based methods, generative adversarial network (GAN)-based methods, and Transformer-based methods. CNN-based methods often achieve effective raindrop removal by designing complex convolutional modules and attention mechanisms. GAN-based methods employ adversarial game theory, controlling the texture details of the generated images by adjusting the parameters of the generator and discriminator, thus restoring more natural and detailed images. However, both of these methods rely on ordinary convolutions, whose local receptive fields significantly limit their information representation capabilities, making it difficult to effectively model global feature dependencies between pixels. Since the Transformer network architecture's self-attention mechanism excels in modeling non-local feature representations, it overcomes the limitations of the two aforementioned methods, resulting in good restoration effects for raindrop removal methods.

[0004] While the three methods mentioned above have achieved some success in image deraining, none of them fully consider the sparse distribution of raindrops and the different degrees of degradation in the raindrop-covered areas. First, raindrops often exhibit a sparse distribution in images, but existing methods treat pixels equally, frequently processing every pixel across the entire image. This can lead to the destruction of content in raindrop-free areas, while the restoration quality of raindrop areas is unsatisfactory. Second, existing methods cannot effectively model the different degrees of degradation in raindrop areas, thus failing to adaptively restore raindrop images based on the degree of degradation in each raindrop region.

[0005] In summary, there is currently no image deraining method that possesses the ability to perceive raindrop degradation and distribution, strong local / global feature dependency extraction capabilities, and good robustness. Summary of the Invention

[0006] The present invention aims to solve the aforementioned technical problems existing in the prior art by providing an image raindrop removal method based on an uncertainty-aware sparse Transformer network.

[0007] The technical solution of this invention is: an image raindrop removal method based on an uncertainty-aware sparse Transformer network, which is carried out according to the following steps:

[0008] Step 1. Input pairs of images with / without raindrops from the training set;

[0009] Step 2. Build a deep convolutional neural network N for raindrop removal from images. derain It consists of one convolutional layer (Conv_in) and one sparse Transformers subnetwork (N) with a U-shaped structure for encoding and decoding features. ST One uncertainty estimator subnetwork N for predicting uncertainty graphs. NDE and one feature reconstructor subnetwork N for generating rain-reduced images. FRC composition;

[0010] Step 2.1 The input image is convolved using a convolutional layer Conv_in to obtain a feature map with dimension HWC, where H represents the height of the feature map, W represents the width of the feature map, and C represents the number of feature channels of the feature map.

[0011] Step 2.2 Establish subnetwork N ST It consists of one encoder, two pyramid feature refinement modules (PFR), four residual channel spatial attention modules (RCSA), and one decoder. The encoder and decoder are each composed of nine sparse Transformer blocks.

[0012] Step 2.2.1 Create a sparse Transformer block, consisting of one layer normalization module M. LN1 1 Top-k sparse attention module M TSA 1 layer normalization module M LN2 M is composed of one cross-feedforward network module. CFFN For the input feature map at the (j-1)th scale Its dimension is H / 2 j-1 W / 2 j-1 C / 2 j-1 The sparse Transformer block is then processed according to the following steps. Process it;

[0013] Step 2.2.1.1 Utilize the layer normalization module M LN1 For feature maps The feature map is obtained through processing. Its dimension remains H / 2 j-1 W / 2 j-1 C / 2 j-1 ;

[0014] Step 2.2.1.2 Utilize Top-k sparse attention module M TSA First, the feature map The feature maps are split into Q, K, and V with dimensions L′d, and then processed according to formulas (1) and (2) to obtain the feature maps.

[0015]

[0016] The L = H / 2 j-1 ′W / 2 j-1 d = C / 2 j-1 , This represents the temperature factor, with the superscript T indicating the transpose operation. k (A) represents the function for performing a Top-k selection operation on feature map A, where A = QK T / t,s j Let represent the k-th largest value in the j-th row of A, and Softmax(g) represent the Softmax function;

[0017] Step 2.2.1.3 According to formula (3), the feature map is... With feature map The feature maps are obtained by fusion.

[0018]

[0019] Step 2.2.1.4 Utilize the layer normalization module M LN2 For feature maps The feature map is obtained through processing.

[0020] Step 2.2.1.5 Utilize the cross-feedforward network module M CFFN According to formulas (4) to (9), the feature map is... The feature map is obtained through processing.

[0021]

[0022] The d(g) represents the ReLU activation function, Conv 1′1 (g) represents a 1×1 convolution operation, DWConv 3′3 (g) represents a depthwise convolution operation with a kernel size of 3×3, DWConv 5′5 (g) represents a depthwise convolution operation with a kernel size of 5×5, and · represents a concatenation operation. M represents CFFN Intermediate feature maps generated during the computation process;

[0023] Step 2.2.1.6 According to formula (10), the feature map is... With feature map The feature maps are obtained by fusion.

[0024]

[0025] Step 2.2.2 Establish the pyramid feature refinement module PFR, which includes four parallel convolutional modules PConv1, PConv2, PConv3, and PConv4, and spatial attention modules SA1, SA2, SA3, and SA4. For the input feature map F at the j-th scale... j The pyramid feature refinement module PFR refines F according to the following steps. j Process it;

[0026] Step 2.2.2.1 Using the convolutional module PConv1 and the spatial attention module SA1, the feature map F is processed according to formula (11). j Processing yields F1. j ;

[0027]

[0028] SA(g) represents the spatial attention function, and BN(g) represents the batch normalization function. This represents a dilated convolution operation with a kernel size of 3×3 and a dilation factor of 0.

[0029] Step 2.2.2.2 Using the convolutional module PConv2 and the spatial attention module SA2, the feature map F is processed according to formula (12). j Processing is performed to obtain

[0030]

[0031] The This represents a dilated convolution operation with a kernel size of 3×3 and a dilation factor of 1.

[0032] Step 2.2.2.3 Using the convolution module PConv3 and the spatial attention module SA3, according to formula (13)

[0033] For feature map F j Processing is performed to obtain

[0034]

[0035] The This represents a dilated convolution operation with a kernel size of 3×3 and a dilation factor of 2.

[0036] Step 2.2.2.4 Using the convolutional module PConv4 and the spatial attention module SA4, the feature map F is processed according to formula (14). j Processing is performed to obtain

[0037]

[0038] The This represents a dilated convolution operation with a kernel size of 3×3 and a dilation factor of 3.

[0039] Step 2.2.2.5 Calculate the feature map using a 1×1 convolutional layer according to formula (15).

[0040]

[0041] The Conv 1′1 (g) represents a 1×1 convolution operation;

[0042] Step 2.2.3 Establish the residual channel spatial attention module RCSA, which includes one channel attention module CA1 and one spatial attention module SA5. For the input feature map E at the j-th scale... j The pyramid feature refinement module RCSA processes the data according to the following steps;

[0043] Step 2.2.3.1 Apply formula (16) to the feature map E j The feature map is obtained through processing.

[0044]

[0045] The Conv 3′3 (g) represents a convolution operation with a kernel size of 3×3;

[0046] Step 2.2.3.2: Use the channel attention module CA1 to process the feature map. The process is performed, and the feature map is calculated according to formula (17).

[0047]

[0048] AP(g) represents the average pooling function, MP(g) represents the max pooling function, and MLP(g) represents the multilayer perceptron function. represents element-wise multiplication, and s(g) represents the sigmoid function;

[0049] Step 2.2.3.3 Utilize the spatial attention module SA5 to process the feature map The process is performed, and the feature map is calculated according to formula (18).

[0050]

[0051] The Conv 7′7 (g) represents a convolution operation with a kernel size of 7×7;

[0052] Step 2.2.3.4 Calculate the feature map according to formula (19)

[0053]

[0054] Step 2.3 Establish the uncertainty estimator subnetwork N NDE It contains 3 Conv-ELU modules;

[0055] Step 2.4 Establish the feature reconstructor subnetwork N FRC It includes the Conv-ReLU module;

[0056] Step 3. Establish the total loss function L over ;

[0057] Step 3.1 Let the uncertainty feature map generated by the uncertainty estimator be S. i Based on formula (20), an uncertainty-driven loss function L is established. U ;

[0058]

[0059] N represents the uncertainty feature map S. i The number of pixels, y i Let represent the i-th pixel of the clean truth image y. Indicates the generated clean image For the i-th pixel, exp(g) represents an exponential function with the natural constant e as the base;

[0060] Step 3.2 Establish the content loss function L according to formula (21)C ;

[0061]

[0062] The ||g||1 represents the L1 norm;

[0063] Step 3.3 Establish the content loss function L based on formula (22) P ;

[0064]

[0065] The ||g‖2 represents the L2 norm. and Representing image y and image respectively The feature map on the q-th convolutional layer of the VGG-19 network pre-trained with the ImageNet dataset, where q is a preset constant;

[0066] Step 3.3 Establish the total loss function L according to formula (23) over ;

[0067] L over =l C L C +l P L P +l U L U (twenty three)

[0068] The l C l P and l U All are preset constants;

[0069] Step 4. Apply the deep convolutional neural network N derain Perform initialization and training;

[0070] Step 4.1 Use a random initialization method to initialize N. derain Initialize the parameters;

[0071] Step 4.2 Input the images from the training set into N derain Train the network and optimize the model using the Adam optimizer;

[0072] Step 4.3 During network training, after completing the set number of iterations, the learning rate is reduced to zero using linear decay to obtain the trained network model.

[0073] Compared with existing technologies, this invention has the following advantages: First, addressing the sparsity of raindrop distribution, this invention introduces a Top-k sparse attention mechanism in the sparse Transformers module, which can effectively capture the global sparse features of raindrop-containing images while suppressing redundant and irrelevant feature representations. Second, this invention utilizes uncertainty branches to generate an uncertainty map regarding the degree of image degradation, accurately modeling the diversity of degradation in raindrop regions. Guided by this uncertainty map, an uncertainty-driven loss function is then used to guide the neural network to remove raindrops more effectively, ensuring the quality of background information restoration in raindrop-containing images. Third, this invention places the pyramid feature refinement module between the deepest encoder and decoder layers of the network, fully mining local features within the multi-scale receptive field. Furthermore, four residual channel spatial attention modules are added between the two pyramid feature refinement modules, which further enhances the effective expression of the deepest features. In summary, this invention not only captures the global sparse representation of raindrop images but also models the degree of degradation in raindrop regions and adaptively adjusts the penalty level on the network accordingly, thereby effectively removing raindrops and restoring rich background details. Therefore, the present invention has strong applicability in practical scenarios. Attached Figure Description

[0074] Figure 1 This is a diagram illustrating the structure and workflow of a deep convolutional neural network according to an embodiment of the present invention.

[0075] Figure 2 This is a diagram illustrating the structure and workflow of a sparse Transformer block according to an embodiment of the present invention.

[0076] Figure 3 This is a diagram illustrating the structure and workflow of the pyramid feature refinement module in an embodiment of the present invention.

[0077] Figure 4 This is a flowchart of the residual channel spatial attention module according to an embodiment of the present invention.

[0078] Figure 5 These are uncertainty maps generated by the uncertainty estimator in this embodiment of the invention and their corresponding raindrop images.

[0079] Figure 6 This is a qualitative comparison result diagram of the raindrop removal results in an embodiment of the present invention.

[0080] Figure 7 It is a qualitative comparison of the raindrop removal results of different methods and a visualization of the pixel distribution of the generated image on the blue line.

[0081] Figure 8 This is a semantic segmentation result image before and after raindrop removal, obtained based on the pre-trained Segmenter. Detailed Implementation

[0082] The technical solution of this invention is: an image raindrop removal method based on an uncertainty-aware sparse Transformer network, which is carried out according to the following steps:

[0083] Step 1. Input the dataset and split it;

[0084] Step 1.1 Input the Deraindrop dataset, which has 1110 pairs of images with / without raindrops, consisting of 861 training pairs, 58 test A pairs, and 249 test B pairs, with test A being a subset of test B.

[0085] Step 1.2 Use the 861 pairs of images with / without raindrops in the Deraindrop dataset as the training set for the neural network, and use the 58 pairs of images in the test set A as the test set for the neural network;

[0086] Step 2. Establish as follows Figure 1 The deep convolutional neural network N shown is used for raindrop removal from images. derain It consists of one convolutional layer (Conv_in) and one sparse Transformers subnetwork (N) with a U-shaped structure for encoding and decoding features. ST (a) One uncertainty estimator subnetwork N for predicting uncertainty graphs NDE (b) and a feature reconstruction subnetwork N for generating rain-reduced images. FRC (c) Composition;

[0087] Step 2.1 The input image is convolved using a convolutional layer Conv_in to obtain a feature map with dimension HWC, where H represents the height of the feature map, W represents the width of the feature map, and C represents the number of feature channels of the feature map.

[0088] Step 2.2 Establish subnetwork N ST It consists of one encoder, two pyramid feature refinement modules (PFR), four residual channel spatial attention modules (RCSA), and one decoder. The encoder and decoder are each composed of nine sparse Transformer blocks.

[0089] Step 2.2.1 Create a sparse Transformer block, such as Figure 2 The diagram shows a sequence consisting of one layer normalization module M. LN1 1 Top-k sparse attention module M TSA 1 layer normalization module M LN2 M is composed of one cross-feedforward network module. CFFNFor the input feature map at the (j-1)th scale Its dimension is H / 2 j-1 W / 2 j-1 C / 2 j-1 The sparse Transformer block is then processed according to the following steps. Process it;

[0090] Step 2.2.1.1 Utilize the layer normalization module M LN1 For feature maps The feature map is obtained through processing. Its dimension remains H / 2 j-1 W / 2 j-1 C / 2 j-1 ;

[0091] Step 2.2.1.2 Utilize Top-k sparse attention module M TSA First, the feature map The feature maps are split into Q, K, and V with dimensions L′d, and then processed according to formulas (1) and (2) to obtain the feature maps.

[0092]

[0093]

[0094] The L = H / 2 j-1 ′W / 2 j-1 d = C / 2 j-1 , This represents the temperature factor, with the superscript T indicating the transpose operation. k (A) represents the function for performing a Top-k selection operation on feature map A, where A = QK T / t,s j Let represent the k-th largest value in the j-th row of A, and Softmax(g) represent the Softmax function;

[0095] Step 2.2.1.3 According to formula (3), the feature map is... With feature map The feature maps are obtained by fusion.

[0096]

[0097] Step 2.2.1.4 Utilize the layer normalization module M LN2 For feature maps The feature map is obtained through processing.

[0098] Step 2.2.1.5 Utilize the cross-feedforward network module M CFFN According to formulas (4) to (9), the feature map is... The feature map is obtained through processing.

[0099]

[0100] The d(g) represents the ReLU activation function, Conv 1′1 (g) represents a 1×1 convolution operation, DWConv 3′3 (g) represents a depthwise convolution operation with a kernel size of 3×3, DWConv 5′5 (g) represents a depthwise convolution operation with a kernel size of 5×5, and · represents a concatenation operation. M represents CFFN Intermediate feature maps generated during the computation process;

[0101] Step 2.2.1.6 According to formula (10), the feature map is... With feature map The feature maps are obtained by fusion.

[0102]

[0103] Step 2.2.2 Establish the Pyramid Feature Refinement Module (PFR), such as... Figure 3 The diagram shows four parallel convolutional modules (PConv1, PConv2, PConv3, PConv4) and spatial attention modules (SA1, SA2, SA3, SA4). For the input feature map F at the j-th scale... j The pyramid feature refinement module PFR refines F according to the following steps. j Process it;

[0104] Step 2.2.2.1 Using the convolutional module PConv1 and the spatial attention module SA1, the feature map F is processed according to formula (11). j Processing yields F1. j ;

[0105]

[0106] SA(g) represents the spatial attention function, and BN(g) represents the batch normalization function. This represents a dilated convolution operation with a kernel size of 3×3 and a dilation factor of 0.

[0107] Step 2.2.2.2 Using the convolutional module PConv2 and the spatial attention module SA2, the feature map F is processed according to formula (12). j Processing is performed to obtain

[0108]

[0109] The This represents a dilated convolution operation with a kernel size of 3×3 and a dilation factor of 1.

[0110] Step 2.2.2.3 Using the convolutional module PConv3 and the spatial attention module SA3, the feature map F is processed according to formula (13). j Processing is performed to obtain

[0111]

[0112] The This represents a dilated convolution operation with a kernel size of 3×3 and a dilation factor of 2.

[0113] Step 2.2.2.4 Using the convolutional module PConv4 and the spatial attention module SA4, the feature map F is processed according to formula (14). j Processing is performed to obtain

[0114]

[0115] The This represents a dilated convolution operation with a kernel size of 3×3 and a dilation factor of 3.

[0116] Step 2.2.2.5 Calculate the feature map using a 1×1 convolutional layer according to formula (15).

[0117]

[0118] The Conv 1′1 (g) represents a 1×1 convolution operation;

[0119] Step 2.2.3 Establish the Residual Channel Spatial Attention Module (RCSA), such as... Figure 4 The diagram shows one channel attention module CA1 and one spatial attention module SA5, for the input feature map E at the j-th scale. j The pyramid feature refinement module RCSA processes the data according to the following steps;

[0120] Step 2.2.3.1 Apply formula (16) to the feature map E j The feature map is obtained through processing.

[0121]

[0122] The Conv 3′3 (g) represents a convolution operation with a kernel size of 3×3;

[0123] Step 2.2.3.2: Use the channel attention module CA1 to process the feature map. The process is performed, and the feature map is calculated according to formula (17).

[0124]

[0125] AP(g) represents the average pooling function, MP(g) represents the max pooling function, and MLP(g) represents the multilayer perceptron function. represents element-wise multiplication, and s(g) represents the sigmoid function;

[0126] Step 2.2.3.3 Utilize the spatial attention module SA5 to process the feature map The process is performed, and the feature map is calculated according to formula (18).

[0127]

[0128] The Conv 7′7 (g) represents a convolution operation with a kernel size of 7×7;

[0129] Step 2.2.3.4 Calculate the feature map according to formula (19)

[0130]

[0131] Step 2.3 Establish the uncertainty estimator subnetwork N NDE It contains 3 Conv-ELU modules;

[0132] Step 2.4 Establish the feature reconstructor subnetwork N FRC It includes the Conv-ReLU module;

[0133] Step 3. Establish the total loss function L over ;

[0134] Step 3.1 Let the uncertainty feature map generated by the uncertainty estimator be S. i Based on formula (20), an uncertainty-driven loss function L is established. U ;

[0135]

[0136] N represents the uncertainty feature map S. i The number of pixels, y iLet represent the i-th pixel of the clean truth image y. Indicates the generated clean image For the i-th pixel, exp(g) represents an exponential function with the natural constant e as the base;

[0137] Step 3.2 Establish the content loss function L according to formula (21) C ;

[0138]

[0139] The ||g||1 represents the L1 norm;

[0140] Step 3.3 Establish the content loss function L based on formula (22) P ;

[0141]

[0142] The ||g||2 represents the L2 norm. and Representing image y and image respectively The feature map on the q-th convolutional layer of the VGG-19 network pre-trained with the ImageNet dataset, where q is a preset constant, and in this embodiment of the invention, q = 15;

[0143] Step 3.3 Establish the total loss function L according to formula (23) over ;

[0144] L over =l C L C +l P L P +l U L U (twenty three)

[0145] The l C l P and l U All are preset constants;

[0146] Step 4. Apply the deep convolutional neural network N derain Perform initialization and training;

[0147] Step 4.1 Use a random initialization method to initialize N. derain Initialize the parameters;

[0148] Step 4.2 Input the 861 pairs of images with / without raindrops from the training set into N. derain The network was trained, and the model was optimized using the Adam optimizer. The parameters of the Adam optimizer were set to b1 = 0.5 and b2 = 0.999, and the initial learning rate was set to 1 × 10⁻⁶.-4 Batch size is 1, patch size is 256;

[0149] Step 4.3 During network training, let epoch = 300. After completing the first 150 epochs, use linear decay to reduce the learning rate to zero, thus obtaining the trained network model.

[0150] test:

[0151] The 58 pairs of test images from test set A are input into the trained network model. In the process, the resulting image after rain removal is obtained, where the uncertainty map generated by the uncertainty estimator and its corresponding raindrop image are shown in the figure. Figure 5 As shown.

[0152] The objective evaluation metrics, including SSIM, PSNR, FID, LPIPS, FSIM, NIQE, PIQE, and BRISQUE, are used to evaluate the resulting images, and the objective evaluation results for each image are output.

[0153] Table 1 presents a quantitative comparison of the raindrop removal results of the embodiments of the present invention with those of AttenGAN, Quan's, DuRN, BPP, TKLMR, MCW, and DA-CLIP on the test set A. The evaluation metrics include SSIM, PSNR, FID, LPIPS, FSIM, NIQE, PIQE, and BRISQUE.

[0154] Table 1. Quantitative comparison of raindrop removal results using different methods

[0155]

[0156] Figure 6 This invention provides a qualitative comparison of the raindrop removal effects of embodiments thereof with those of AttenGAN, Quan's, DuRN, BPP, TKLMR, MCW, and DA-CLIP methods. Figure 6 (a) is the original image containing raindrops. Figure 6 (b) shows the raindrop removal results using the AttenGAN method. Figure 6 (c) shows the raindrop removal results of Quan's method. Figure 6 (d) shows the raindrop removal results of the DuRN method. Figure 6 (e) shows the raindrop removal results of the BPP method. Figure 6 (f) shows the raindrop removal results of the TKLMR method. Figure 6 (g) shows the raindrop removal results using the MCW method. Figure 6 (h) shows the raindrop removal results of the DA-CLIP method. Figure 6(i) is the raindrop removal result of an embodiment of the present invention. Figure 6 (j) is the baseline image. Figure 7 Furthermore, a magnified view of the raindrop-removed image from an embodiment of the present invention is provided, along with the pixel distribution of a certain column in the generated image. This is compared with the results of AttenGAN, Quan's, DuRN, BPP, TKLMR, MCW, and DA-CLIP methods. Figure 7 (a) is the original image containing raindrops. Figure 7 (b) shows the results of the AttenGAN method. Figure 7 (c) is the result of Quan's method. Figure 7 (d) shows the result of the DuRN method. Figure 7 (e) shows the result of the BPP method. Figure 7 (f) shows the results of the TKLMR method. Figure 7 (g) represents the result of the MCW method. Figure 7 (h) represents the result of the DA-CLIP method. Figure 7 (i) represents the result of an embodiment of the present invention. Figure 7 (j) is the baseline image. From Figure 6 and Figure 7 As can be seen, the embodiments of the present invention can remove raindrops while preserving the original edge details and textures in the image.

[0157] Considering the negative impact of raindrop occlusion on semantic segmentation, Figure 8 The present invention provides a comparison of semantic segmentation results with AttenGAN, Quan's, DuRN, BPP, TKLMR, MCW, and DA-CLIP methods using a pre-trained Segmenter on the ADE20K dataset, thereby evaluating the performance of various raindrop removal methods from a semantic segmentation perspective. Figure 8 (a) is the original image containing raindrops. Figure 8 (b) shows the results of the AttenGAN method. Figure 8 (c) is the result of Quan's method. Figure 8 (d) shows the result of the DuRN method. Figure 8 (e) shows the result of the BPP method. Figure 8 (f) shows the results of the TKLMR method. Figure 8 (g) represents the result of the MCW method. Figure 8 (h) represents the result of the DA-CLIP method. Figure 8 (i) represents the result of an embodiment of the present invention. Figure 8 (j) is the baseline image.

Claims

1. An image raindrop removal method based on uncertainty-aware sparse Transformer network, characterized in that Follow these steps: Step 1. Input the image pairs with / without raindrops in the training set; Step 2. Establish a deep convolutional neural network N for image raindrop removal derain , which consists of a convolutional layer Conv_in, a sparse Transformers sub-network N of a U-shaped structure for encoding and decoding features. ST , 1 uncertainty estimator subnetwork N for predicting the uncertainty map NDE and a feature reconstructor subnetwork N for generating derained images FRC composition; Step 2.1 uses the convolution layer Conv_in to perform a convolution operation on the input image to obtain a feature map with a dimension of HWC, where H represents the height of the feature map, W represents the width of the feature map, and C represents the number of feature channels of the feature map; Step 2.2 Establish subnetwork N ST , which consists of 1 encoder, 2 pyramid feature refinement modules PFR, 4 residual channel spatial attention modules RCSA and 1 decoder, where the encoder and decoder are composed of 9 sparse Transformer blocks respectively; Step 2.2.1 Create a sparse Transformer block, which consists of a layer normalization module M LN1 , 1 Top-k sparse attention module M TSA , 1 layer normalization module M LN2 and 1 cross feedforward network module to form M CFFN , for the input feature map at the (j-1)th scale Its dimension is H / 2 j-1 W / 2 j-1 C / 2 j-1 , then the sparse Transformer block performs the following steps to to process; Step 2.2.1.1 Using layer normalization module M LN1 For feature maps Processing to obtain feature map Its dimension is still H / 2 j-1 W / 2 j-1 C / 2 j-1 ; Step 2.2.1.2 Using Top-k sparse attention module M TSA First, the feature map Split into Q, K and V with dimension L′d, and then process the feature maps Q, K and V according to formula (1) and formula (2) to obtain the feature maps L = H / 2 j-1 ′W / 2 j-1 , d = C / 2 j-1 , represents the temperature factor, the superscript T represents the transposition operation, T k (A) represents the function of performing Top-k selection operation on feature map A, A = QK T / t,s j represents the kth largest value in the jth row of A, and Softmax(g) represents the Softmax function; Step 2.2.1.3 According to formula (3), the feature map With feature map Fusion to obtain feature map Step 2.2.1.4 Use layer normalization module M LN2 For the feature map F in j it Processing to obtain feature map Step 2.2.1.5 Using the cross-feedforward network module M CFFN , according to formula (4) to formula (9), the characteristic graph Processing to obtain feature map The d(g) represents the ReLU activation function, Conv 1′1 (g) represents a 1×1 convolution operation, DWConv 3′3 (g) represents the depth convolution operation with a convolution kernel size of 3×3, DWConv 5′5 (g) represents a depthwise convolution operation with a convolution kernel size of 5×5, · represents a cascade operation, Indicates M CFFN Intermediate feature maps generated during the operation; Step 2.2.1.6 According to formula (10), the feature map With the feature map F L j NCFFN Fusion to obtain feature map Step 2.2.2 Establish the pyramid feature refinement module PFR, which includes four parallel convolution modules PConv1, PConv2, PConv3, PConv4 and spatial attention modules SA1, SA2, SA3, SA4. For the input feature map F at the jth scale j The pyramid feature refinement module PFR performs the following steps on F j to process; Step 2.2.2.1 Use the convolution module PConv1 and the spatial attention module SA1 to perform feature map F according to formula (11): j Process and get F1 j ; The SA(g) represents the spatial attention function, and BN(g) represents the batch normalization function. Indicates a dilated convolution operation with a convolution kernel size of 3×3 and a dilation factor of 0; Step 2.2.2.2 Use the convolution module PConv2 and the spatial attention module SA2 to perform feature map F according to formula (12): j Process and obtain Said Indicates a dilated convolution operation with a convolution kernel size of 3×3 and a dilation factor of 1; Step 2.2.2.3 Use the convolution module PConv3 and the spatial attention module SA3 to perform feature map F according to formula (13) j Process and obtain Said Indicates a dilated convolution operation with a convolution kernel size of 3×3 and a dilation factor of 2; Step 2.2.2.4 Use the convolution module PConv4 and the spatial attention module SA4 to perform feature map F according to formula (14): j Process and obtain Said Indicates a dilated convolution operation with a convolution kernel size of 3×3 and a dilation factor of 3; Step 2.2.2.5 According to formula (15), use 1×1 convolution layer to calculate the feature map The Conv 1′1 (g) represents a 1×1 convolution operation; Step 2.2.3 establishes a residual channel spatial attention module RCSA, which includes a channel attention module CA1 and a spatial attention module SA5. For the input feature map E at the jth scale j ,The pyramid feature refinement module RCSA is processed according to the following steps; Step 2.2.3.1 According to formula (16), the feature map E j Processing to obtain feature map The Conv 3′3 (g) represents the convolution operation with a convolution kernel size of 3×3; Step 2.2.3.2 Use channel attention module CA1 to focus on feature maps Processing is performed and the feature map is calculated according to formula (17): The AP(g) represents the average pooling function, the MP(g) represents the maximum pooling function, and the MLP(g) represents the multi-layer perceptron function. represents element-by-element multiplication operation, s(g) represents the sigmoid function; Step 2.2.3.3 Use the spatial attention module SA5 to focus on the feature map Processing is performed and the feature map is calculated according to formula (18) The Conv 7′7 (g) represents the convolution operation with a convolution kernel size of 7×7; Step 2.2.3.4 Calculate the characteristic map according to formula (19) Step 2.3 Establish the uncertainty estimator subnetwork N NDE , including 3 Conv-ELU modules; Step 2.4 Establish feature reconstruction subnetwork N FRC , including the Conv-ReLU module; Step 3. Establish the total loss function L over ; Step 3.1 Let the uncertainty feature map generated by the uncertainty estimator be S i , according to formula (20), the uncertainty driven loss function L is established U ; The N represents the uncertainty characteristic map S i The number of pixels, y i represents the i-th pixel of the clean truth image y, Represents the generated clean image The i-th pixel of , exp(g) represents the exponential function with the natural constant e as the base; Step 3.2 Establish the content loss function L according to formula (21) C ; The ‖g‖1 represents the L1 norm; Step 3.3 Establish the content loss function L according to formula (22) P ; The ‖g‖2 represents the L2 norm, and Represent image y, image Feature map of the qth convolutional layer of the VGG-19 network pre-trained with the ImageNet dataset, where q is a preset constant; Step 3.3 Establish the total loss function L according to formula (23) over ; L over l C L C +l P L P +l U L U (23) The l C , l P and l U All are preset constants; Step 4. Deep convolutional neural network N derain Perform initialization and training; Step 4.1 Use random initialization method to initialize N derain Initialize the parameters; Step 4.2 Input the images of the training set into N derain Perform network training and use the Adam optimizer to optimize the model; Step 4.3 During the network training process, after completing the set number of times, use linear decay to reduce the learning rate to zero to obtain a trained network model

Images