Image rain removal method and device based on hybrid network

Abstract

The present application belongs to the field of computer vision, and particularly relates to a kind of image rain removal method and equipment based on hybrid network;The method of the present application comprises the following steps: step 1, constructing a self-attention module;Step 2, constructing a transformer module based on self-attention;Step 3, constructing a reversible neural network module;Step 4, constructing a hybrid network module based on the transformer module and the reversible neural network module;Step 5, constructing an end-to-end neural network;The dataset is input into the end-to-end neural network in step 5 for training;After training, input the real image to be removed into the trained model, and obtain the rain-removed image.The transformer network structure based on self-attention and the reversible neural network structure can consider global features and local details, thereby achieving good rain removal effect.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to an image deraining method and device based on hybrid networks. Background Technology

[0002] In open application environments such as autonomous driving, smart cities, and deep-sea and deep-space exploration, the acquisition of digital images is constrained by hardware and software limitations and is affected by unfavorable shooting environments. Rainwater produces low-quality image data, posing a serious challenge to image processing and semantic understanding tasks. In recent years, based on large-scale, high-quality image datasets, deep learning-based methods for processing low-quality images have developed rapidly. A convolutional neural network-based image rain removal method was proposed in patent CN107909556B. This method is similar to those proposed by Wang et al. (H. Wang, Q. Xie, Q. Zhao, and D. Meng, “A Model-Driven Deep Neural Network for Single Image Rain Removal,” in 2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA: IEEE, Jun. 2020, pp. 3100–3109.) and Fu et al. (X. Fu, J. Huang, D. Zeng, Y. Huang, X. Ding, and J. Paisley, “Removing Rain from Single Images via a Deep Detail Network,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition). The Recognition (CVPR), Honolulu, HI: IEEE, Jul. 2017, pp. 1715–1723, also proposed a method based on convolutional neural networks; Zamir et al. proposed an image rain removal method based on a transformer structure (SWZamir, A. Arora, S. Khan, M. Hayat, F.S. Khan, and M.-H. Yang, “Restormer: Efficient Transformer for High-Resolution Image Restoration.” in 2022 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA: IEEE, Jun. 2022, pp. 5718-5729).However, convolutional neural networks are lacking in global feature extraction. The transformer structure cannot take into account local image features and has a large number of parameters. Reversible neural networks can extract local image details better, but ignore global features. Existing methods can extract global features better, but lose some image details. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of existing technologies and propose an image deraining method and device based on a hybrid network. This method features a hybrid network module that integrates transformers and invertible neural networks, capable of balancing global features and local details. The self-attention structure utilizes a shared convolutional kernel to achieve multi-scale contextual features, resulting in superior image deraining performance. The technical solution adopted in this invention is as follows:

[0004] A hybrid network-based image deraining method includes:

[0005] Step 1: Construct a self-attention module, defining the input of the self-attention module as A. in The channel dimension is C in ;

[0006] Step 2: Construct a self-attention-based transformer module, defining the input of the transformer module as B. in ;

[0007] Step 3: Construct a reversible neural network module, defining the input of the reversible neural network module as S. in ;

[0008] Step 4: Construct a hybrid network module, defining the input of the hybrid network module as M. in ;

[0009] Step 5: Construct an end-to-end neural network, defining the input image of the end-to-end neural network as... Where W and H represent the width and height of the image, respectively;

[0010] The image dataset is input into the end-to-end neural network in step 5 for training. After training is completed, the real image to be de-rained is input into the trained model to obtain the de-rained image.

[0011] Furthermore, step 1 includes:

[0012] Step 1-1, for A in Performing a linear transformation yields A1, as shown in the following equation:

[0013] A1=A in W1+b1

[0014] Where W1 is the weight parameter that the self-attention module model needs to learn, and is the bias parameter that the self-attention module model needs to learn;

[0015] Step 1-2: Calculate Q, Q = sum(ds_conv_i(A1)), where ds_conv_i represents dilated convolution and sum represents summation. Here, three types of dilated convolution are used with dilation parameters of 1, 3, and 5 respectively.

[0016] Steps 1-3, for A in Performing a linear transformation yields A2, as shown in the following equation:

[0017] A2 = A in W2+b2

[0018] Where W2 represents the weight parameters that the self-attention module model needs to learn, b2 represents the bias parameters that the self-attention module model needs to learn, and A2 has a channel dimension of 2C. in ; Divide A2 into two tensors, K and V, at the channel level, where the channel dimension of K and V is C. in ;

[0019] Steps 1-4: Calculate the similarity between Q and K. Then, the softmax activation function is used to normalize s. Represents matrix multiplication operation

[0020] Steps 1-5, Calculation A linear transformation is performed on o to obtain the output att of the self-attention module, as shown in the following equation:

[0021] att = oW3 + b3

[0022] Where W3 is the weight parameter that the self-attention module model needs to learn, and b3 is the bias parameter that the self-attention module model needs to learn.

[0023] Furthermore, step 2 includes:

[0024] Step 2-1, B in After layer normalization, the data is fed into the self-attention module of step 1 to obtain the attention map att_b, and B1 = B is calculated. in +att_b;

[0025] Step 2-2: After performing layer normalization on B1, we obtain B2. Then, we perform a linear transformation on B2 to obtain B3, as shown in the following equation:

[0026] B3 = B2W4 + b4

[0027] Where W4 represents the weight parameters that the transformer module model needs to learn, and b4 represents the bias parameters that the transformer module model needs to learn.

[0028] Steps 2-3: Process B3 using the Gaussian error linear unit activation function to obtain B4, as shown in the following equation:

[0029]

[0030] Steps 2-4: Perform a linear transformation on B4 to obtain B5, as shown in the following equation:

[0031] B5 = B4W5 + b5

[0032] Where W5 is the weight parameter that the transformer module model needs to learn, and b5 is the bias parameter that the transformer module model needs to learn.

[0033] Steps 2-5: Calculate the output out of the transformer module, where out = B3 + B5.

[0034] Furthermore, step 3 includes:

[0035] Step 3-1: Input S to the reversible neural network module in Preprocessing is performed on S at the channel level. in Normalization

[0036] S1=αS in +β

[0037] Where α and β are the trainable parameters of the network;

[0038] Step 3-2: Multiply S1 by the invertible 1×1 convolution to obtain... The convolution kernel is a reversible 1×1 convolution, and its parameters are obtained through network training. S2 is divided into two parts at the channel level, namely S... a and S b ;

[0039] Step 3-3: Construct a shallow neural network NN. NN is a neural network with an arbitrary number of layers and consists of three convolutional layers. (The rest of the text appears to be a separate, unrelated section about S.) a Input NN, get S3;

[0040] Steps 3-4: Divide S3 into two equal parts at the channel level, namely f1 and f2, and use f1 and f2 to process S... a Performing an affine transformation, we obtain F, as shown in the following equation:

[0041]

[0042] in This represents matrix multiplication, where active(·) represents the activation function, defined as follows:

[0043]

[0044] Where φ represents the input to the activation function;

[0045] Step 3-5: Combine F obtained in step 3-4 with S obtained in step 3-2. b Cascaded, the output of the invertible neural network is obtained as out. inn As shown below:

[0046] out inn =concat(F,S) b )

[0047] Here, concat represents a cascading operation at the channel level.

[0048] Furthermore, step 4 includes:

[0049] The output of the hybrid network module is M out As shown in the following formula:

[0050] M out =INN(M in )+Trans(M in )

[0051] Where Trans represents the self-attention-based transformer module in step 2, and INN represents the reversible neural network module in step 3.

[0052] Furthermore, step 5 includes:

[0053] Step 5-1, I in Input a convolutional layer and get Where C represents I in The number of channels for the feature is obtained after processing;

[0054] Step 5-2: Calculate I2 = MIX_Blocks(I1), where MIX_Blocks represents several hybrid network modules described in step 4;

[0055] Step 5-3: Apply a convolutional layer to transform the I2 dimension to 3, resulting in...

[0056] Step 5-4: Calculate network output I out , where I out =I in +I3.

[0057] Furthermore, the objective function during the training process is as follows:

[0058] loss=||J-φ(I,w)||1

[0059] Where I represents the input rainy image, J represents its corresponding rainless image, φ(I,w) represents the rainless image obtained after network processing, w represents the model parameters, and ||J-φ(I,w)||1 represents the calculation of the absolute value of the difference between the two.

[0060] The present invention also proposes an electronic device comprising a memory, a processor, and program instructions stored in the memory that are executable by the processor, the processor executing the program instructions to implement the steps of the method described in the present invention.

[0061] Compared with the prior art, the present invention has the following beneficial effects:

[0062] 1. This invention combines a self-attention-based transformer network structure and a reversible neural network structure, which can take into account both global features and local details, thereby achieving a better rain removal effect.

[0063] 2. In this invention, the parameters of multiple dilated convolution kernels in the self-attention module are shared, which achieves better feature description without adding more parameters, thus realizing the lightweighting of the network. Attached Figure Description

[0064] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0065] Figure 1 This is a flowchart of an image deraining method based on a hybrid network proposed in this invention. Detailed Implementation

[0066] The present invention will be further explained in detail below with reference to the accompanying drawings, so that those skilled in the art can better understand and implement the present invention. However, the following examples are only used to explain the present invention and are not intended to limit the present invention.

[0067] like Figure 1 As shown, an image deraining method based on a hybrid network includes:

[0068] Step 1: Construct a self-attention module, defining the input of the self-attention module as A. in The channel dimension is C in ;

[0069] Step 2: Construct a self-attention-based transformer module, defining the input of the transformer module as B. in ;

[0070] Step 3: Construct a reversible neural network module, defining the input of the reversible neural network module as S. in ;

[0071] Step 4: Construct a hybrid network module, defining the input of the hybrid network module as M. in ;

[0072] Step 5: Construct an end-to-end neural network, defining the input image of the end-to-end neural network as... Where W and H represent the width and height of the image, respectively;

[0073] The image dataset is input into the end-to-end neural network in step 5 for training. After training, the real images to be de-rained are input into the trained model to obtain the de-rained images. The objective function during the training process is as follows:

[0074] loss=||J-φ(I,w)||1

[0075] Where I represents the input rainy image, J represents its corresponding rainless image, φ(I,w) represents the rainless image obtained after network processing, w represents the model parameters, and ||J-φ(I,w)||1 represents calculating the absolute value of the difference between the two.

[0076] Step 1 also includes the following steps:

[0077] Step 1-1, for A in Performing a linear transformation yields A1, as shown in the following equation:

[0078] A1=A in W1+b1

[0079] Where W1 is the weight parameter that the self-attention module model needs to learn, and is the bias parameter that the self-attention module model needs to learn;

[0080] Step 1-2: Calculate Q, Q = sum(ds_conv_i(A1)), where ds_conv_i represents dilated convolution and sum represents summation. Here, three types of dilated convolution are used with dilation parameters of 1, 3, and 5 respectively.

[0081] Steps 1-3, for A in Performing a linear transformation yields A2, as shown in the following equation:

[0082] A2 = A in W2+b2

[0083] Where W2 represents the weight parameters that the self-attention module model needs to learn, b2 represents the bias parameters that the self-attention module model needs to learn, and A2 has a channel dimension of 2C. in ; Divide A2 into two tensors, K and V, at the channel level, where the channel dimension of K and V is C. in ;

[0084] Steps 1-4: Calculate the similarity between Q and K. Then, the softmax activation function is used to normalize s. Represents matrix multiplication operation

[0085] Steps 1-5, Calculation A linear transformation is performed on o to obtain the output att of the self-attention module, as shown in the following equation:

[0086] att = oW3 + b3

[0087] Where W3 is the weight parameter that the self-attention module model needs to learn, and b3 is the bias parameter that the self-attention module model needs to learn.

[0088] Step 2 also includes the following steps:

[0089] Step 2-1, B in After layer normalization, the data is fed into the self-attention module of step 1 to obtain the attention map att_b, and B1 = B is calculated. in +att_b;

[0090] Step 2-2: After performing layer normalization on B1, we obtain B2. Then, we perform a linear transformation on B2 to obtain B3, as shown in the following equation:

[0091] B3 = B2W4 + b4

[0092] Where W4 represents the weight parameters that the transformer module model needs to learn, and b4 represents the bias parameters that the transformer module model needs to learn.

[0093] Steps 2-3: Process B3 using the Gaussian error linear unit activation function to obtain B4, as shown in the following equation:

[0094]

[0095] Steps 2-4: Perform a linear transformation on B4 to obtain B5, as shown in the following equation:

[0096] B5 = B4W5 + b5

[0097] Where W5 is the weight parameter that the transformer module model needs to learn, and b5 is the bias parameter that the transformer module model needs to learn.

[0098] Steps 2-5: Calculate the output out of the transformer module, where out = B3 + B5.

[0099] Step 3 also includes the following steps:

[0100] Step 3-1: Input S to the reversible neural network module in Preprocessing is performed on S at the channel level. in Normalization

[0101] S1=αS in +β

[0102] Where α and β are the trainable parameters of the network;

[0103] Step 3-2: Multiply S1 by the invertible 1×1 convolution to obtain... The convolution kernel is a reversible 1×1 convolution, and its parameters are obtained through network training. S2 is divided into two parts at the channel level, namely S... a and S b ;

[0104] Step 3-3: Construct a shallow neural network NN. NN is a neural network with an arbitrary number of layers and consists of three convolutional layers. (The rest of the text appears to be a separate, unrelated section about S.) a Input NN, get S3;

[0105] Steps 3-4: Divide S3 into two equal parts at the channel level, namely f1 and f2, and use f1 and f2 to process S... a Performing an affine transformation, we obtain F, as shown in the following equation:

[0106]

[0107] in This represents matrix multiplication, where active(·) represents the activation function, defined as follows:

[0108]

[0109] Where φ represents the input to the activation function;

[0110] Step 3-5: Combine F obtained in step 3-4 with S obtained in step 3-2. b Cascaded, the output of the invertible neural network is obtained as out. inn As shown below:

[0111] outinn =concat(F,S) b )

[0112] Here, concat represents a cascading operation at the channel level.

[0113] Step 4 includes the following steps:

[0114] The output of the hybrid network module is M out As shown in the following formula:

[0115] M out =INN(M in )+Trans(M in )

[0116] , where Trans represents the self-attention-based transformer module in step 2, and INN represents the reversible neural network module in step 3.

[0117] Step 5 also includes the following steps:

[0118] Step 5-1, I in Input a convolutional layer and get Where C represents I in The number of channels for the feature is obtained after processing;

[0119] Step 5-2: Calculate I2 = MIX_Blocks(I1), where MIX_Blocks represents several hybrid network modules described in step 4;

[0120] Step 5-3: Apply a convolutional layer to transform the I2 dimension to 3, resulting in...

[0121] Step 5-4: Calculate network output I out , where I out =I in +I3.

[0122] The following experimental data further illustrates the beneficial effects of the present invention:

[0123] Peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are the mainstream comparison criteria for current rain removal algorithms. The method of this invention was trained and tested on three public datasets: Rain100H, Test2800, and Test1200. As shown in Table 1, the PSNR and SSIM of the method of this invention are compared with those of mainstream methods such as RESCAN, PreNet, MPRNet, DualGCN, SPAIR, and Restomer. As can be seen from Table 1, the PSNR and SSIM of the method of this invention are higher than those of other mainstream methods on the three public datasets of Rain100H, Test2800, and Test1200, which further verifies that the image rain removal method of this invention has a better effect.

[0124] Table 1 compares the PSNR and SSIM of the method of this invention with those of mainstream methods.

[0125]

[0126] It should be noted that the description of the apparatus in the embodiments of the present invention is similar to that of the method embodiments described above, and has similar beneficial effects as the method embodiments, so it will not be repeated.

[0127] Finally, the present invention proposes an electronic device comprising a memory, a processor, and program instructions stored in the memory that are executable by the processor. The processor executes the program instructions to implement the various steps of the video anomaly detection method based on scene classification proposed in this invention.

[0128] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0129] The specific implementation schemes described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific implementation schemes of the present invention and are not intended to limit the scope of the present invention. Any equivalent changes and modifications made by those skilled in the art without departing from the concept and principles of the present invention should fall within the scope of protection of the present invention.

Claims

1. An image deraining method based on hybrid networks, characterized in that, Include: Step 1: Construct a self-attention module, defining the input of the self-attention module as... Channel dimension is ; Step 2: Construct a self-attention-based transformer module, defining the input of the transformer module as... ; Step 3: Construct a reversible neural network module, defining the input of the reversible neural network module as... ; Step 4: Construct the hybrid network module and define the input of the hybrid network module as... ; Step 5: Construct an end-to-end neural network, defining the input image of the end-to-end neural network as... , where W and H represent the width and height of the image, respectively; The image dataset is input into the end-to-end neural network in step 5 for training. After training is completed, the real image to be de-rained is input into the trained model to obtain the de-rained image. Step 1 includes: Step 1-1, for Perform a linear transformation to obtain As shown in the following formula: ； in These are the weight parameters that the self-attention module model needs to learn. These are the bias parameters that the self-attention module model needs to learn; Steps 1-2, Calculation , ,in This represents dilated convolution, and `sum` represents the summation operation. Three types of dilated convolution are used here, with dilation parameters of 1, 3, and 5 respectively. Steps 1-3, for Perform a linear transformation to obtain As shown in the following formula: ； in These are the weight parameters that the self-attention module model needs to learn. These are the bias parameters that the self-attention module model needs to learn. Channel dimension is ;Will Divided at the channel level and Two tensors, and The channel dimensions are all ; Steps 1-4, Calculation and similarity And use the softmax activation function to Normalization is performed. This represents matrix multiplication. Steps 1-5, Calculation ,right Perform a linear transformation to obtain the output of the self-attention module. As shown in the following formula: ； in These are the weight parameters that the self-attention module model needs to learn. These are the bias parameters that the self-attention module model needs to learn; Step 2 includes: Step 2-1, After layer normalization, the data is fed into the self-attention module of step 1 to obtain the attention map. Calculations yielded ; Step 2-2, After performing layer normalization, we get ,Will Perform a linear transformation to obtain As shown in the following formula: ； in These are the weight parameters that the transformer module model needs to learn. These are the bias parameters that the transformer module model needs to learn; Steps 2-3: Apply Gaussian error linear unit activation function to... Processing is performed to obtain As shown in the following formula: ； Steps 2-4, Perform a linear transformation to obtain As shown in the following formula: ； in These are the weight parameters that the transformer module model needs to learn. These are the bias parameters that the transformer module model needs to learn; Steps 2-5: Calculate the output of the transformer module ,in ; Step 3 includes: Step 3-1: Input to the reversible neural network module Preprocessing is performed at the channel level. Normalization ； in and These are the trainable parameters of the network; Step 3-2, and reversible Convolution multiplication yields , Reversible Convolution, its kernel parameters are obtained through network training; Dividing it into two parts at the channel level, respectively and ; Step 3-3: Construct a shallow neural network NN. NN is a neural network with any number of layers and consists of three convolutional layers. Input NN, get ; Steps 3-4, Dividing it into two parts at the channel level, respectively and , use and right Perform an affine transformation to obtain As shown in the following formula: ； in This represents matrix multiplication, where The activation function is defined as follows: ； in This represents the input to the activation function; Step 3-5: Obtain the following from step 3-4 and in step 3-2 Cascaded, the output of the reversible neural network is obtained. As shown below: ； in This indicates a cascading operation at the channel level; Step 4 includes: The output of the hybrid network module is As shown in the following formula: ； in This refers to the self-attention-based transformer module in step 2. This refers to the reversible neural network module in step 3.

2. The image deraining method based on hybrid networks according to claim 1, characterized in that, Step 5 includes: Step 5-1, Input a convolutional layer and get Where C represents The number of channels for the feature is obtained after processing; Step 5-2, Calculation ,in This represents several hybrid network modules described in step 4; Step 5-3: Apply a convolutional layer to... Dimensional transformation to 3 yields ; Step 5-4: Calculate the network output ,in .

3. The image deraining method based on hybrid networks according to claim 1, characterized in that, During the training process, the objective function is as follows: ； in This indicates that the input image contains rain. This represents the corresponding rainless image. This represents the derained image obtained after network processing. Represents the parameters of the model. This indicates that the absolute value of the difference between the two is calculated.

4. An electronic device comprising a memory, a processor, and program instructions stored in the memory and executable by the processor, characterized in that, The processor executes the program instructions to implement the steps of the method according to any one of claims 1-3.

Images