Image deraining method based on multi-scale staggered Transformer

Abstract

The present invention belongs to the field of image processing technology, and relates to an image deraining method based on a multi-scale interlaced Transformer, wherein the training images in a data set are preprocessed to obtain rainy images and corresponding rain-free images; a multi-scale interlaced Transformer network is designed; rainy images and rain-free images are input into the multi-scale interlaced Transformer network for training; rainy images are input into the trained multi-scale interlaced Transformer network for testing, and rain-free images are output; and image deraining is performed using the multi-scale interlaced Transformer network that meets the accuracy requirements in the test. The present invention effectively enhances the ability to restore background details and achieves a clearer and less disturbing image deraining effect.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an image deraining method based on a multi-scale staggered Transformer. Background Art

[0002] Rainy days will affect the quality of outdoor captured images, which will cause the performance of outdoor visual tasks to deteriorate. Therefore, it is crucial to restore images degraded by rain, which is beneficial to computer vision applications such as autonomous driving, target detection, and video surveillance. In recent years, image deraining research has attracted widespread attention.

[0003] The vigorous development of deep learning technology has brought significant breakthroughs and progress to the image deraining task. Unlike traditional methods, deep learning models can automatically learn the characteristics and distribution patterns of rain by building deep neural networks, thereby more accurately removing traces of rain from images. This method not only improves the deraining effect, but also greatly reduces the need for manual intervention and parameter adjustment. In addition, deep learning models also show stronger robustness and adaptability when dealing with image deraining tasks in complex scenes and changing weather conditions. Therefore, deep learning has become the mainstream technology in the current image deraining field and is expected to continue to promote progress and development in this field in the future.

[0004] Among the image deraining methods based on deep learning, convolutional neural network (CNN) is undoubtedly the most contributing. With its powerful feature extraction and representation learning capabilities, CNN provides an efficient and accurate solution for image deraining tasks. By constructing multi-level convolutional layers, the network can gradually extract rain features in the image and perform feature conversion and enhancement through nonlinear activation functions. The application of CNN in the field of image deraining not only improves the deraining effect, but also provides high-quality input data for subsequent image processing and analysis. However, CNN still has some limitations. First, the receptive field of CNN is limited, which limits the ability of CNN to process large-scale spatial information. Second, most of the existing CNN-based image deraining models adopt a single-path flow design. Although this design can enrich feature representation, it often ignores the effective fusion between low-level contextual information and high-level features. This one-way information flow processing method leads to deficiencies in feature extraction and representation of the network, making it difficult to fully capture the rain details and contextual relationships in the image, which in turn affects the final image deraining effect.

[0005] In order to overcome the problem of limited receptive field of CNN, researchers proposed Transformer network. Transformer network is based on self-attention mechanism and can capture global dependencies. Therefore, Transformer network can effectively expand the receptive field. Through multi-layer self-attention stacking, Transformer network can fully understand and integrate the contextual information in the image, providing a more comprehensive and accurate feature representation for the deraining task. In addition, Transformer network also has the advantage of parallel computing, which can significantly improve the processing speed and meet the needs of real-time image deraining. Compared with the traditional CNN model, Transformer network not only improves the deraining effect in the image deraining task, but also shows stronger generalization ability and robustness. Although Transformer network has shown significant performance improvement in the field of image restoration, there are still some challenges and problems in terms of computational complexity and local detail recovery ability. Summary of the invention

[0006] In order to overcome the shortcomings of the Transformer network in image deraining, the present invention proposes an image deraining method based on a multi-scale staggered Transformer. The present invention reconstructs high-quality images from information in different states to better capture the size and distribution of raindrops, and uses multi-scale feature information to improve the deraining effect.

[0007] The present invention is implemented by the following technical solution: An image deraining method based on a multi-scale staggered Transformer, the steps are as follows:

[0008] Step 1: Preprocess the training images in the data set to obtain rainy images and corresponding rain-free images;

[0009] Step 2: Design a multi-scale interleaved Transformer network;

[0010] Step 3: Input the rainy images and rainless images into the multi-scale interleaved Transformer network for training;

[0011] Step 4: Input the rainy image into the trained multi-scale interleaved Transformer network for testing and output the rain-free image;

[0012] Step 5: Use the multi-scale interleaved Transformer network that meets the accuracy requirements to perform image deraining;

[0013] The multi-scale interleaved Transformer network includes an encoder consisting of N layers of encoders and a decoder consisting of N layers of decoders. Given an input image, the multi-scale interleaved Transformer network first applies convolution to extract shallow features. Then, the shallow features are converted into deep features through an N-layer symmetrical encoder-decoder structure. The encoder uses a downsampling operation from the top layer to the bottom layer, and the decoder uses an upsampling operation from the bottom layer to the top layer. Each layer of the encoder contains B Transformer modules, and each layer of the decoder contains D Transformer modules. Between adjacent branches, the Transformer modules are interleaved horizontally and vertically to gradually merge multi-level contexts from different states.

[0014] Further optimization, input shallow features , the output of the first layer of the encoder is expressed as:

[0015] ;

[0016] In the formula, is the bth Transformer module in the first layer of the encoder, is the b-1th Transformer module in the first layer of the encoder, is the output feature of the bth Transformer module in the first layer of the encoder, and the range of b is [1, B];

[0017] The operation of encoder layer 2 to encoder layer N is expressed as:

[0018] ;

[0019] In the formula, represents the operation result of the b-th Transformer module in the n-th layer of the encoder, represents the operation result of the b-th Transformer module in the n-1-th layer of the encoder, represents the operation result of the b-1th Transformer module in the nth layer of the encoder, is the operation result of the Bth Transformer module in the n-1th layer of the encoder, is the operation result of the first Transformer module in the encoder layer n-1, is the b-th Transformer module in the n-th layer of the encoder; n is in the range [2, N];

[0020] The outputs of different Transformer modules in the last layer of the encoder are concatenated together and used as the input of the first layer of the decoder after 1×1 convolution. :

[0021] );

[0022] Among them, Conv stands for convolution and Concat stands for concatenation operation. is the result of the operation of the first Transformer module in the Nth layer of the encoder, is the result of the operation of the second Transformer module in the Nth layer encoder, is the result of the Bth Transformer module in the Nth layer encoder;

[0023] Further preferably, the calculation process of the first layer of the decoder is expressed as:

[0024] ;

[0025] in, represents the operation of the dth Transformer module in the first layer of the decoder, is the result of the Bth Transformer module in the Nth layer of the encoder; represents the operation result of the dth Transformer module in the first layer of the decoder, Represents the operation result of the d-1th Transformer module in the first layer of the decoder;

[0026] The calculation process of decoder layer 2 to decoder layer N is as follows:

[0027] ;

[0028] in, represents the operation of the dth Transformer module in the nth layer of the decoder; represents the output of the Bth Transformer module in the N-n+1th layer of the decoder, Represents the operation output result of the d-th Transformer module in the n-1-th layer of the decoder; Represents the operation output result of the d-th Transformer module in the n-th layer of the decoder, Represents the operation output result of the d-1th Transformer module in the nth layer of the decoder;

[0029] The outputs of each Transformer module of the Nth layer decoder are concatenated and fused to generate deep features ; Finally, Apply 1×1 convolution to obtain the residual image R;

[0030] ;

[0031] ;

[0032] Among them, ConCat represents the concatenation operation, and Conv represents the convolution operation;

[0033] Finally, the residual image is compared with the input image Combined to obtain the restored image.

[0034] Further preferably, the Transformer module in the first layer of the encoder is composed of a multi-head attention module and a gated feedforward network module; given the input feature ,but It first passes through the multi-head attention module, and then directly passes through the gated feedforward network module to obtain the output features.

[0035] Further preferably, the calculation definition of the multi-head attention module is as follows:

[0036] ;

[0037] ;

[0038] ;

[0039] ;

[0040] ;

[0041] In the formula, Q represents query, K represents key, and V represents value. is the input feature After the output features of the multi-head attention module, is the query 3×3 depthwise convolution, is a 3×3 depthwise convolution of the key, is a 3×3 depthwise convolution of values, is the 1×1 point-wise convolution of the query, is the 1×1 pointwise convolution of the key, is the 1×1 point-wise convolution of the values, represents the query after reshaping, represents the reshaped key, represents the reshaped value, Attention represents the attention operation, Softmax is the activation function, represents 1×1 convolution, is the scaling parameter.

[0042] Further preferably, the calculation definition of the gated feedforward network module is as follows:

[0043] ;

[0044] ;

[0045] In the formula, represents the operation of the gating mechanism, is the output feature of the gated feedforward network module, represents 1×1 convolution, represents 3×3 convolution, ⊙ represents element-by-element multiplication, represents GELU nonlinearity and LN is layer normalization.

[0046] Further preferably, the Transformer modules other than the first layer of the encoder adopt a hybrid cross-attention Transformer module. The hybrid cross-attention Transformer module introduces a cross-attention module and a parallel two-dimensional attention module based on the multi-head attention module and the gated feedforward network module. Given two input features and input features , where the input feature First pass through the multi-head attention module and input the features After downsampling, they pass through a parallel two-dimensional attention module, and then the two pass through a cross attention module for data fusion, and finally the output features are obtained through a gated feedforward network module.

[0047] Further preferably, the calculation definition of the multi-head attention module of the hybrid cross-attention Transformer module is as follows:

[0048] ;

[0049] ;

[0050] ;

[0051] ;

[0052] ;

[0053] In the formula, Q represents query, K represents key, and V represents value. is the input feature After the output features of the multi-head attention module, is the query 3×3 depthwise convolution, is a 3×3 depthwise convolution of the key, is a 3×3 depthwise convolution of values, is the 1×1 point-wise convolution of the query, is the 1×1 pointwise convolution of the key, is the 1×1 point-wise convolution of the values, represents the query after reshaping, represents the reshaped key, represents the reshaped value, Attention represents the attention operation, Softmax is the activation function, represents 1×1 convolution, is the scaling parameter.

[0054] Further preferably, the computational definition of the parallel two-dimensional attention module of the hybrid cross-attention Transformer module is as follows:

[0055] = ;

[0056] ;

[0057] ;

[0058] ;

[0059] In the formula, is the input feature Output features after parallel two-dimensional attention module; Represents 3×3 convolution; Represents the features after 3×3 convolution; represents a 1×1 convolution in the spatial dimension, represents a 3×3 convolution in the spatial dimension, Represents a 1×1 convolution on the channel dimension; represents the global average pooling operation, ConCat represents the concatenation operation, represents the channel attention vector, represents the spatial attention vector.

[0060] Further preferably, the computational definition of the criss-cross attention module of the hybrid criss-cross attention Transformer module is as follows:

[0061] );

[0062] ;

[0063] ;

[0064] ;

[0065] ;

[0066] In the formula, is the query of the criss-cross attention module, is the key of the criss-cross attention module, is the value of the criss-cross attention module, is the input feature After the output features of the multi-head attention module, is the input feature After the output features of the parallel two-dimensional attention module, is the output feature of the cross-attention module; is The result obtained by reshaping based on is The result obtained by reshaping based on is The result obtained by reshaping based on.

[0067] Further preferably, the computational definition of the gated feedforward network module of the hybrid cross-attention Transformer module is as follows:

[0068] ;

[0069] ;

[0070] In the formula, represents the operation of the gating mechanism, is the output feature of the gated feedforward network module, represents 1×1 convolution, represents 3×3 convolution, ⊙ represents element-by-element multiplication, represents GELU nonlinearity and LN is layer normalization.

[0071] The multi-scale interleaved Transformer network provided by the present invention adopts a multi-scale, multi-path and multi-branch framework to fully utilize the information hierarchy characteristics, wherein each branch is composed of multiple Transformer modules to achieve better rain removal effect; each stage is an asymmetric encoder-decoder network, and the Transformer module is a core component, which has a powerful ability to capture global context information, which helps to better remove raindrops and improve the rain removal effect.

[0072] The present invention introduces a cross attention module in the Transformer module to replace the traditional data fusion module, so as to better utilize multi-scale feature information and improve the deraining effect.

[0073] A parallel dual-dimensional attention module (PDAM) is used on the input of the first layer of the network before the cross-attention module to enhance the feature extraction capability of the downsampling stage. The information of the channel and spatial dimensions is captured and modeled simultaneously through parallel channel attention and spatial attention, thereby making more comprehensive use of the degraded features.

[0074] Compared with existing image deraining algorithms, the present invention can obtain higher quality rain-free images on rainy images of different types (raindrops and rain streaks). The present invention effectively enhances the ability to restore background details and achieves a clearer and less disturbing image deraining effect. The present invention has achieved good results in the two key indicators for quantitatively evaluating deraining performance - peak signal-to-noise ratio (PSNR) and structural similarity (SSIM), demonstrating its outstanding advantages in image clarity and structure preservation. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 It is a flowchart of the image deraining method based on the multi-scale interlaced Transformer of the present invention.

[0076] Figure 2 It is a structural schematic diagram of the image deraining method based on the multi-scale interlaced Transformer of the present invention.

[0077] Figure 3 Schematic diagram of the hybrid cross-attention Transformer module structure of the present invention.

[0078] Figure 4 Schematic diagram of the structure of the parallel two-dimensional attention module of the present invention. DETAILED DESCRIPTION

[0079] The present invention will be further described in detail below with reference to the accompanying drawings.

[0080] Reference Figure 1 , the image deraining method based on multi-scale interlaced Transformer, the steps are as follows:

[0081] Step 1: Preprocess the training images in the data set to obtain rainy images and corresponding rain-free images;

[0082] Step 2: Design a multi-scale interleaved Transformer network;

[0083] Step 3: Input the rainy images and rainless images into the multi-scale interleaved Transformer network for training;

[0084] Step 4: Input the rainy image into the trained multi-scale interleaved Transformer network for testing and output the rain-free image;

[0085] Step 5: Use the multi-scale interleaved Transformer network that meets the accuracy requirements to perform image deraining.

[0086] like Figure 2 As shown in Figure 1, the multi-scale interleaved Transformer network includes an encoder and a decoder, each of which contains N layers. Given an input image , the multi-scale interleaved Transformer network first applies convolution to extract shallow features ;in, Represents the dimension, is the spatial dimension, is the number of channels. Then, the shallow features obtained by 1×1 convolution of the input image are After N layers of symmetrical encoder-decoder structure, it is transformed into deep features The encoder uses downsampling operations from the top layer to the bottom layer, and the decoder uses upsampling operations from the bottom layer to the top layer. In addition, each layer of the encoder contains B Transformer modules, and each layer of the decoder contains D Transformer modules; between adjacent branches, the Transformer modules are interleaved horizontally and vertically to gradually merge multi-level contexts from different states.

[0087] Enter a shallow feature , the output of the first layer of the encoder is expressed as:

[0088] ;

[0089] In the formula, is the bth Transformer module in the first layer of the encoder, is the b-1th Transformer module in the first layer of the encoder, is the output feature of the bth Transformer module in the encoder layer 1, and b is in the range of [1, B]. Then, the encoder layer 1 is copied to get the encoder layer 2 and the encoder layer 3. The subnetwork structure of the decoder and encoder is the same. Therefore, in the multi-scale interleaved Transformer network, each branch has the same structure but does not share weight parameters.

[0090] In the encoder, unlike the encoder layer 1, the sub-networks in the encoder layer 2 and encoder layer 3 can be regarded as a refinement process that can fuse features of different states. Figure 2As shown in the figure, the output of the Transformer module in the encoder layer 1 will be given as input to the corresponding Transformer module in the encoder layer 2. The encoder layer 2 can incorporate the underlying context from the encoder layer 1 into the current information flow. The same is true for the operation of the encoder layer 3. The operation of the encoder layer 2-encoder layer N is expressed as:

[0091] ;

[0092] In the formula, represents the operation result of the b-th Transformer module in the n-th layer of the encoder, represents the operation result of the b-th Transformer module in the n-1-th layer of the encoder, represents the operation result of the b-1th Transformer module in the nth layer of the encoder, is the operation result of the Bth Transformer module in the n-1th layer of the encoder, is the operation result of the first Transformer module in the encoder layer n-1, is the b-th Transformer module in the n-th layer of the encoder; n and b are in the range [2,N] and [1,B] respectively. In this part, the initial input of the encoder layer 2 is the first and last output of the encoder layer 1. The same is true for the encoder layer 3.

[0093] exist Figure 2 As can be seen in Figure 1, the outputs of different Transformer modules in the last layer of the encoder are concatenated together and processed by 1×1 convolution as the input of the first layer of the decoder. :

[0094] );

[0095] Among them, Conv represents convolution, Concat represents concatenation operation, is the result of the operation of the first Transformer module in the Nth layer encoder, is the result of the operation of the second Transformer module in the Nth layer encoder, is the result of the Bth Transformer module in the Nth layer encoder;

[0096] The decoder layer 1 calculation can be expressed as:

[0097] ;

[0098] in, represents the operation of the dth Transformer module in the first layer of the decoder, is the result of the Bth Transformer module in the Nth layer of the encoder; represents the operation result of the dth Transformer module in the first layer of the decoder, Represents the operation result of the d-1th Transformer module in the first layer of the decoder.

[0099] The calculation process of decoder layer 2 to decoder layer N is as follows:

[0100] ;

[0101] in, represents the operation of the dth Transformer module in the nth layer of the decoder; represents the output of the Bth Transformer module in the N-n+1th layer of the decoder, Represents the operation output result of the d-th Transformer module in the n-1-th layer of the decoder; Represents the operation output result of the d-th Transformer module in the n-th layer of the decoder, Represents the output result of the operation of the d-1th Transformer module in the nth layer of the decoder; where n and d are in the range of [2,N] and [1,D] respectively.

[0102] The outputs of each Transformer module in the Nth layer of the decoder are concatenated and fused to generate deep features ; Finally, Apply 1×1 convolution to get the residual image .

[0103] ;

[0104] ;

[0105] Finally, the residual image is compared with the input image Combined to obtain the restored image.

[0106] ;

[0107] in, is the restored image.

[0108] The Transformer module is a sequence processing model whose core lies in the self-attention mechanism. This mechanism allows the model to autonomously compare each element in the input sequence with other elements in the sequence when processing sequence data, thereby ensuring that each element can be properly processed in different contexts.

[0109] The Transformer module in the first layer of the encoder is composed of a multi-head attention module and a gated feedforward network module; given the input features ,but It first passes through the multi-head attention module, and then directly passes through the gated feedforward network module to obtain the output features.

[0110] The core components of the Transformer module are the multi-head attention module and the gated feedforward network module. Between these two modules, this embodiment innovatively adds a cross-attention module. The cross-attention module can be used to replace the traditional data fusion module. The cross-attention module can better fuse the information of different branches. In addition, the feature extraction capability of the downsampling stage of the previous branch is enhanced by the parallel two-dimensional attention module. And through this operation, the query (Q), key (K) and value (V) of the cross-attention are distinguished so that they all come from different input features. Since each head extracts information from the input independently, they can focus on different aspects of the input sequence. This design enables the Transformer module to capture richer contextual information and improves the expressive power of the Transformer module.

[0111] After improvement, we obtained Figure 3 The hybrid cross-attention Transformer module shown in Figure 2 It can be seen that except for the first branch of the encoder, the operations of other branches have two inputs, so first give two input features and input features , where the input feature First pass through the multi-head attention module and input the features After downsampling, they pass through a parallel two-dimensional attention module, and then the two pass through a cross attention module for data fusion, and finally pass through a gated feedforward network module.

[0112] (1) Multi-head attention module. The self-attention layer in the Transformer module faces huge challenges in practical applications due to the sharp increase in computational complexity when processing high-resolution images. In order to reduce the computational complexity, a cross-channel self-attention calculation method is adopted. Because this calculation method is not calculated in the spatial dimension, the complexity can be reduced. In addition, deep convolution is used to emphasize the processing of local context, so as to more effectively generate a global attention map. Given the input feature , the calculation definition of the multi-head attention module is as follows:

[0113] ;

[0114] ;

[0115] ;

[0116] ;

[0117] ;

[0118] In the formula, is the input feature After the output features of the multi-head attention module, is the query 3×3 depthwise convolution, is a 3×3 depthwise convolution of the key, is a 3×3 depthwise convolution of values, is the 1×1 point-wise convolution of the query, is the 1×1 pointwise convolution of the key, is the 1×1 point-wise convolution of the values. All in Based on the reshaping, represents the query after reshaping, represents the reshaped key, represents the reshaped value, Attention represents the attention operation, Softmax is the activation function, represents 1×1 convolution. In order to control and The size of the dot product takes a learnable scaling parameter .

[0119] (2) Parallel two-dimensional attention module. This embodiment proposes an innovative parallel two-dimensional attention module (PDAM), which aims to address the limitations of vanilla convolution in feature extraction, especially multi-scale feature processing. Traditional channel attention and spatial attention methods can only focus on the feature relationship of a certain dimension respectively, and cannot fully capture and model feature information. To this end, we designed a parallel two-dimensional attention module, which can use channel attention and spatial attention mechanisms in parallel to extract and integrate key information of these two dimensions. The core of the parallel two-dimensional attention module lies in its parallel processing mechanism, which enables it to make full use of important features in the encoding stage and effectively filter out useless information. Unlike previous studies, the parallel two-dimensional attention module does not simply adopt a single or serialized attention method, but runs channel attention and spatial attention in parallel, and connects the feature information in the channel dimension, thereby achieving a more comprehensive utilization of features. Considering that the multi-scale interleaved Transformer network needs to downsample the feature information of the previous branch and then fuse it with the current branch through the cross attention module, by introducing the parallel two-dimensional attention module, the feature extraction capability of the downsampling stage can be significantly enhanced, thereby improving the performance of the model.

[0120] like Figure 4 As shown, given an input feature First, the input features are processed by 3×3 convolution and Relu activation function to generate features Then the feature After two branches of operations, one branch first passes through the global average pooling layer in the channel dimension, then passes through two 1×1 convolutions (the first 1×1 convolution is followed by the Relu activation function, and the second 1×1 convolution is followed by the Sigmoid activation function), and then is combined with the feature Multiply and use residual connection to generate channel attention vector ; The other branch undergoes a 3 × 3 convolution (using Relu activation function) and a 1 × 1 convolution (using Sigmoid activation function) and then is combined with feature X 0 Multiply and use residual connection to get the spatial attention vector Finally, and The concatenation is performed in the channel dimension. After 3×3 convolution, the final result is obtained through residual connection. The calculation definition of the parallel two-dimensional attention module is as follows:

[0121] = ;

[0122] ;

[0123] ;

[0124] ;

[0125] In the formula, is the input feature Output features after parallel two-dimensional attention module; Represents 3×3 convolution; represents a 1×1 convolution in the spatial dimension, represents a 3×3 convolution in the spatial dimension, Represents a 1×1 convolution on the channel dimension; represents the global average pooling operation, and ConCat represents the concatenation operation.

[0126] (3) Cross-attention module. The previous article mentioned the characteristics of the multi-scale interleaved Transformer network, that is, the need to fuse feature information from different branches. However, traditional data fusion methods often simply superimpose or mix image information of different scales or features, but ignore the deep connections between them. In order to deeply explore these connections, the cross-attention mechanism is introduced as an alternative. In the Transformer module, a special cross-attention module is designed. This cross-attention module can divide the input image into multiple regions and capture the contextual information within each region through the self-attention mechanism. At the same time, with the help of the cross-attention mechanism, the multi-scale interleaved Transformer network can focus on the mutual correlation between different regions, thereby more comprehensively understanding the entire image. This improvement helps the multi-scale interleaved Transformer network to more accurately capture the complex relationships and features in the image and improve the performance of image processing. Unlike multi-head attention, the cross-attention module are from different inputs. The output features from the multi-head attention module, Derived from input features , Derived from input features The output features of the concatenation operation with the results after the parallel two-dimensional attention module. The calculation is defined as follows:

[0127] );

[0128] ;

[0129] ;

[0130] ;

[0131] ;

[0132] In the formula, is the query of the criss-cross attention module, is the key of the criss-cross attention module, is the value of the criss-cross attention module, is the input feature After the output features of the multi-head attention module, is the input feature After the output features of the parallel two-dimensional attention module, is the output feature of the cross-attention module; is The result obtained by reshaping based on is The result obtained by reshaping based on is The result obtained by reshaping based on.

[0133] (4) Gated feedforward network module. In order to enhance the ability of the feedforward network in capturing local features of the image, convolution operations are introduced into the feedforward network. Considering the parameters and computational efficiency, depthwise separable convolution is selected, which is more efficient than standard convolution. After the linear operation of the feedforward network, the feature labels are first reshaped into a two-dimensional feature map, and then a 3×3 depthwise separable convolution is used to extract the local context information of the image. Afterwards, these features are flattened back into the labeled form and further processed by another linear layer. In addition, the GELU activation function is used after the convolution layer to enhance the nonlinear expression ability of the gated feedforward network. The calculation definition of the gated feedforward network module is as follows:

[0134] ;

[0135] ;

[0136] In the formula, represents the operation of the gating mechanism, is the output feature of the gated feedforward network module, represents 1×1 convolution, represents 3×3 convolution, ⊙ represents element-by-element multiplication, represents GELU nonlinearity and LN is layer normalization.

[0137] In order to verify the effectiveness of this method, raindrop removal experiments and rain line removal experiments were carried out, and the results are shown in Tables 1 and 2.

[0138] Table 1. Raindrop removal experimental results

[0139]

[0140] Table 2. Rain line removal test results

[0141]

[0142] First, for raindrop removal, our method is compared with several image restoration methods, including classic methods swinIR, SDE and CMFNet, as well as cutting-edge techniques for removing raindrops such as DACLIP, FocalNet, NeRd and WaveDM. We use the raindrop dataset, which contains 861 pairs of images for training. Each pair of images includes two pictures with the same background: one picture is affected by raindrops and appears blurred, while the other remains clear and unobstructed. This setting can accurately depict the occlusion effect of raindrops on the scene, which is convenient for comparison and analysis. After training, the model is evaluated on two test sets of raindrops, where the test set test a Contains 58 images, test set test b Contains 249 images. As shown in Table 1, this method performs well in the test set test a The peak signal-to-noise ratio (PSNR) on the test set is 0.06 dB higher than the previous best method, and the structural similarity (SSIM) is 0.0007 higher than the previous best method. b The peak signal-to-noise ratio (PSNR) also exceeds the previous best method by 0.05 dB, and the structural similarity (SSIM) is 0.0029 higher than the previous best method.

[0143] Secondly, the proposed method is compared with several CNN-based methods, including DerainNet, SEMI, DIDMDN, UMRL, RESCAN, SPANet, PReNet, and MSPFN, when evaluating the performance of rain line removal. In addition, the proposed method is compared with advanced rain line removal techniques, such as PCNet, IDLIR, Uformer-B, SwinDerain, and DAWN. In order to obtain more accurate training results, multiple datasets, including Rain14000, Rain1800, Rain800, and Rain12, are jointly trained in this study. This joint dataset is named Rain13K and contains a total of 13,712 pairs of images, each of which includes a rain scene image and a clear image. Rain14000 is based on 1,000 clear images without rain marks, and 14 images with different rain mark patterns are synthesized. Rain14000 contains 11,200 pairs of images for training and 2,800 pairs of images for testing (Test2800). Rain1800 focuses on heavy rain scenes. Rain800 contains 700 pairs of images for training and 100 pairs of images for testing (Test100). Rain12 contains 12 pairs of images for training. After training, it was evaluated on Test100, Rain100H, Rain100L, and Test2800. As shown in Table 2, the proposed method performs well and stably on four different datasets (Test100, Rain100H, Rain100L, Test2800). The average value of the peak signal-to-noise ratio (PSNR) is 0.98dB higher than the previous state-of-the-art method, and the average value of the structural similarity (SSIM) is 0.006 higher than the previous state-of-the-art method. The results highlight that the proposed method has a significant and stable performance improvement over existing methods, thus verifying its high efficiency and reliability.

[0144] The invention described above only expresses the implementation methods of the embodiments of the present invention, and cannot be understood as limiting the scope of the invention patent, nor does it impose any form of limitation on the structure of the embodiments of the present invention. It should be pointed out that for ordinary technicians in this field, several changes and improvements can be made without departing from the concept of the embodiments of the present invention, which all belong to the protection scope of the embodiments of the present invention.

Claims

1. Image deraining method based on multi-scale interlaced Transformer, characterized by: Here are the steps: Step 1: Preprocess the training images in the data set to obtain rainy images and corresponding rain-free images; Step 2: Design a multi-scale interleaved Transformer network; Step 3: Input the rainy images and rainless images into the multi-scale interleaved Transformer network for training; Step 4: Input the rainy image into the trained multi-scale interleaved Transformer network for testing and output the rain-free image; Step 5: Use the multi-scale interleaved Transformer network that meets the accuracy requirements to perform image deraining; The multi-scale interleaved Transformer network includes an encoder and a decoder, each of which includes N layers. Given an input image, the multi-scale interleaved Transformer network first applies convolution to extract shallow features. Then, the shallow features are transformed into deep features through an N-layer symmetrical encoder-decoder structure; The encoder uses downsampling operations from the top to the bottom layer, and the decoder uses upsampling operations from the bottom to the top layer; each layer of the encoder contains B Transformer modules, and each layer of the decoder contains D Transformer modules; between adjacent branches, Transformer modules are interleaved horizontally and vertically to gradually merge multi-level contexts from different states; Input shallow features , the output of the first layer encoder is expressed as: ； In the formula, is the bth Transformer module in the first layer of the encoder, is the b-1th Transformer module in the first layer of the encoder, is the output feature of the bth Transformer module in the first layer of the encoder, and the range of b is [1, B]; The operation of encoder layer 2 to encoder layer N is expressed as: ； In the formula, represents the operation result of the b-th Transformer module in the n-th layer of the encoder, represents the operation result of the b-th Transformer module in the n-1-th layer of the encoder, represents the operation result of the b-1th Transformer module in the nth layer of the encoder, is the operation result of the Bth Transformer module in the n-1th layer of the encoder, is the operation result of the first Transformer module in the encoder layer n-1, is the b-th Transformer module in the n-th layer of the encoder; n is in the range [2, N]; The outputs of different Transformer modules in the last layer of the encoder are concatenated together and used as the input of the first layer of the decoder after 1×1 convolution. : )； Among them, Conv represents convolution, Concat represents concatenation operation, is the result of the operation of the first Transformer module in the Nth layer of the encoder, is the result of the operation of the second Transformer module in the Nth layer of the encoder, is the result of the Bth Transformer module in the Nth layer of the encoder; The computation process of the first layer of the decoder is expressed as: ； in, represents the operation of the dth Transformer module in the first layer of the decoder, is the result of the Bth Transformer module in the Nth layer of the encoder; represents the operation result of the dth Transformer module in the first layer of the decoder, Represents the operation result of the d-1th Transformer module in the first layer of the decoder; The calculation process of decoder layer 2 to decoder layer N is as follows: ； in, represents the operation of the dth Transformer module in the nth layer of the decoder; represents the output of the Bth Transformer module in the N-n+1th layer of the decoder, Represents the operation output result of the d-th Transformer module in the n-1-th layer of the decoder; Represents the operation output result of the d-th Transformer module in the n-th layer of the decoder, Represents the operation output result of the d-1th Transformer module in the nth layer of the decoder; The outputs of each Transformer module of the Nth layer decoder are concatenated and fused to generate deep features ; Finally, Apply 1×1 convolution to obtain the residual image R; ； ； Among them, ConCat represents the concatenation operation, and Conv represents the convolution operation; Finally, the residual image is compared with the input image Combined to obtain the restored image.

2. The image deraining method based on multi-scale interleaved Transformer according to claim 1, characterized in that: The Transformer module of the first layer of the encoder includes a multi-head attention module and a gated feedforward network module; the Transformer modules other than the first layer of the encoder use a hybrid cross-attention Transformer module, which includes a multi-head attention module, a cross-attention module, a parallel two-dimensional attention module, and a gated feedforward network module. Given two input features and input features , where the input feature First pass through the multi-head attention module and input the features After downsampling, they pass through a parallel two-dimensional attention module, and then the two pass through a cross attention module for data fusion, and finally the output features are obtained through a gated feedforward network module.

3. The image deraining method based on multi-scale interleaved Transformer according to claim 2, characterized in that: The computational definition of the multi-head attention module of the hybrid cross-attention Transformer module is as follows: ； ； ； ； ； In the formula, Q represents query, K represents key, and V represents value. is the input feature After the output features of the multi-head attention module, is the query 3×3 depthwise convolution, is a 3×3 depthwise convolution of the key, is a 3×3 depthwise convolution of values, is the 1×1 point-wise convolution of the query, is the 1×1 pointwise convolution of the key, is the 1×1 point-wise convolution of the values, represents the query after reshaping, represents the reshaped key, represents the reshaped value, Attention represents the attention operation, Softmax is the activation function, represents 1×1 convolution, is the scaling parameter.

4. The image deraining method based on multi-scale interleaved Transformer according to claim 2, characterized in that: The computation of the parallel two-dimensional attention module is defined as follows: = ； ； ； ； In the formula, is the input feature Output features after parallel two-dimensional attention module; Represents 3×3 convolution; Represents the features after 3×3 convolution; represents a 1×1 convolution in the spatial dimension, represents a 3×3 convolution in the spatial dimension, Represents a 1×1 convolution on the channel dimension; represents the global average pooling operation, ConCat represents the concatenation operation, represents the channel attention vector, represents the spatial attention vector.

5. The image deraining method based on multi-scale interleaved Transformer according to claim 3, characterized in that: The computation of the criss-cross attention module is defined as follows: )； ； ； ； ； In the formula, is the query of the criss-cross attention module, is the key of the criss-cross attention module, is the value of the criss-cross attention module, is the input feature After the output features of the multi-head attention module, is the input feature After the output features of the parallel two-dimensional attention module, is the output feature of the cross-attention module; is The result obtained by reshaping based on is The result obtained by reshaping based on is The result obtained by reshaping based on.

6. The image deraining method based on multi-scale interleaved Transformer according to claim 2, characterized in that: The calculation of the gated feedforward network module is defined as follows: ； ； In the formula, represents the operation of the gating mechanism, is the output feature of the gated feedforward network module, represents 1×1 convolution, represents 3×3 convolution, ⊙ represents element-by-element multiplication, ϕ represents GELU nonlinearity, and LN is layer normalization.

Images