Multi-scale Transform image rain removal method based on residual channel prior guidance

Abstract

The invention discloses a multi-scale Transform image rain removal method based on residual channel prior guidance, and the method comprises the steps: obtaining a rain image, inputting the rain image into a trained image rain removal processing network, and obtaining a rain-removed image. The training process of the image rain removal processing network comprises the following steps: acquiring a rain image sample and a corresponding real rain-free image; extracting a prior feature map of the rain image sample by using a residual channel prior extraction module; extracting features of the rain image sample by using a context feature extraction module to obtain shallow layer features of rain fringes on different scales; performing feature fusion to obtain a first comprehensive feature map; processing the first comprehensive feature map by using L recursive Transform extraction modules (RSTMs) to obtain an image after rain removal; and calculating the loss between the image after rain removal and a real rain-free image corresponding to the input rain image sample, and optimizing network parameters through back propagation of the loss. According to the invention, rain stripes can be removed from different directions and scales, so that the rain-removed image is more consistent with a real image.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a multi-scale Transformer image deraining method based on residual channel prior guidance. Background Art

[0002] Images taken outdoors on rainy days are easily disturbed by rain streak density, shape, and direction, which affects their visual quality and causes degradation of some advanced computer vision tasks, such as object detection, image classification, and image segmentation. How to effectively remove rain streaks from rainy images while retaining the image's detailed information has always been a research hotspot in the field of rain removal.

[0003] The existing single image deraining algorithms are mainly divided into two categories: model-driven and data-driven. In the model-driven algorithms, traditional filters, dictionary learning, sparse coding and Gaussian mixture models cannot adapt to the diversity of rain marks, resulting in a large number of rain marks remaining in the image after deraining.

[0004] Most of the recent state-of-the-art methods are based on convolutional networks, which have achieved impressive results. However, in complex scenes, it is difficult for them to remove all rain streaks and restore the structural information of the image. This is because: most CNN-based methods learn the mapping relationship between rain-free images and rainy images, and implement and obtain rain-free images under the constraints of loss functions. However, due to the variation of rain streak density, excessive or insufficient rain streaks are removed, resulting in incomplete structural information of the reconstructed image; these methods focus on learning the structure of rain streaks, but pay less attention to learning the structure of objects, ignoring the importance of image priors. At the same time, convolution has a limited receptive field, and CNN-based methods find it difficult to effectively utilize the global information of rain images and the long-distance dependencies between pixels, resulting in the loss of some details and structural information of the derained image, and cannot obtain satisfactory deraining results. In particular, the density, shape, position and size of rain streaks are uncertain, making it difficult for the network to recover a clean image from various degradations.

[0005] Transformer has shown great potential in the field of artificial intelligence applications. Initially, researchers applied Transformer to the field of Natural Language Processing (NLP) and achieved excellent results in this field. Inspired by it, researchers extended it to computer vision tasks, and the Vision Transformer (ViT) appeared, and made breakthroughs in visual research such as object detection and image deblurring. However, as the spatial resolution of the input image increases, the computational complexity of the original vision transformer increases quadratically. If it is used in visual tasks such as image deraining, its efficiency will be greatly reduced and cannot meet the needs of the actual industry.

[0006] Standard Transformer usually aggregates features using all attention relations based on query-key pairs. Since tokens from the key are not always related to tokens from the query, the native intensive computation mode of self-attention amplifies relatively small similarity weights, making feature interactions and aggregation processes susceptible to implicit noise. This leads to the introduction of corresponding redundant or irrelevant representations when modeling global feature dependencies. Therefore, these findings motivate researchers to continuously explore how to resolve irrelevant redundant information in order to make full use of these features for better image deraining. Summary of the invention

[0007] In order to solve the above technical problems, the present invention proposes a multi-scale Transformer image deraining method based on residual channel prior guidance, the method comprising: obtaining a rainy image, inputting the image into a trained image deraining processing network, and obtaining a derained image;

[0008] The training process of the image deraining processing network includes:

[0009] S1: Obtain rainy image samples and their corresponding true rain-free images;

[0010] S2: extracting the prior feature map of rainy image samples using the residual channel prior extraction module;

[0011] S3: Use the context feature extraction module to extract features from rainy image samples and obtain shallow features of rain streaks at different scales;

[0012] S4: using a feature fusion module to fuse the prior feature map with shallow features of rain streaks at different scales to obtain a first comprehensive feature map;

[0013] S5: Using L recursive Transformer extraction modules RSTM to process the first comprehensive feature map to obtain a rain-free image;

[0014] S6: Calculate the loss between the derained image and the real rain-free image corresponding to the input rainy image sample, and optimize the network parameters by backpropagating the loss until the loss function converges and then stop the optimization.

[0015] Beneficial effects of the present invention: The present invention introduces a residual based on the L2 norm through the residual channel prior extraction module, focuses on the local differences between channels, and enhances the blurred or low-contrast rain streaks, thereby effectively removing these interferences in the rain removal process and guiding high-quality image reconstruction; at the same time, considering the differences in rain lines of different scales and densities, a context aggregation module is introduced to adapt the algorithm to the diversity of rain streak distribution. Introducing SwinTransformer to process images can capture global information and long-distance dependencies between pixels, reduce the computational overhead brought by global self-attention, and effectively capture the features of local areas. However, similar or repeated information may still exist in each local window, and redundancy still exists in the channel. PConv operations are introduced in the recursive Transformer extraction module to enhance the information elements in the features, and deep separable convolution is introduced to reduce the processing burden of redundant information. The present invention can remove rain streaks from different directions and scales with fewer parameters and fewer computing resources, so that the rain removal image is more consistent with the effect of the real image. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 Flow chart of the training steps of the image deraining processing network in an embodiment of the present invention;

[0017] Figure 2 Schematic diagram of the structure of an image rain removal processing network in an embodiment of the present invention;

[0018] Figure 3 Schematic diagram of the structure of a residual channel priori extraction module in an embodiment of the present invention;

[0019] Figure 4 This is a schematic diagram of the structure of a context aggregation module in an embodiment of the present invention;

[0020] Figure 5 It is a structural diagram of a feature fusion module in an embodiment of the present invention;

[0021] Figure 6 Schematic diagram of the structure of a recursive transformer extraction module RSTM in an embodiment of the present invention;

[0022] Figure 7 Schematic diagram of the structure of a multi-scale feature refinement feed-forward network MFRFN in an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The terms "first", "second", "third", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances. This is just a way of distinguishing objects with the same attributes when describing the embodiments of this application.

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] The embodiment of the present invention proposes a multi-scale Transformer image deraining method based on residual channel prior guidance, which includes: obtaining a rainy image, inputting the image into a trained image deraining processing network, and obtaining a derained image.

[0026] Figure 1 The flowchart is a training step of the image deraining processing network in an embodiment of the present invention.

[0027] Figure 2 Schematic diagram of the structure of the image deraining processing network in an embodiment of the present invention. Figure 2 In the example, the input image is a rainy image sample, and after being processed by the image deraining processing network, a derained image is output. The image deraining processing network includes a parallel residual channel prior extraction module and a context feature extraction module, a feature fusion module, a Patch Embedding, several RSTM modules and a Patch Unembed. The prior feature map of the input rainy image is extracted by the residual channel prior extraction module; the shallow features of rain streaks at different scales are obtained by extracting features from the input rainy image by the context feature extraction module; the first comprehensive feature map is obtained by performing feature fusion on the prior feature map and the shallow features of rain streaks at different scales by the feature fusion module; the first comprehensive feature map is divided into multiple small patches by Patch Embedding, and then each small patch is converted into a vector representation of a fixed dimension, that is, the image is mapped from the pixel space to an embedding space; deraining is performed by several RSTM modules (i.e., recursive Transformer extraction modules); image reconstruction is completed by Patch Unembed (whose function is to convert the features processed by several RSTM modules back to the image) to obtain the derained image.

[0028] Reference Figure 1 As shown, the training process of the image deraining processing network includes:

[0029] S1: Obtain rainy image samples and their corresponding true rain-free images.

[0030] The synthetic data set used in the examples of the present invention includes multiple pairs of rainy images and corresponding real rain-free images, which are one-to-one corresponding, such as 1input corresponds to 1target, 2input corresponds to 2target, 1input represents the rainy image numbered 1, and 1target represents the real rain-free image corresponding to the rainy image numbered 1. If the output image obtained by the image deraining network is named 1output, then the two images 1output and 1target are compared. The loss of the image deraining network during training is calculated by calculating the loss between the output image of the network (such as 1output) and its corresponding real rain-free image (such as 1target), and the training is stopped by making the loss converge.

[0031] S2: Use the residual channel prior extraction module to extract the prior feature map of the rainy image samples.

[0032] Figure 3 Schematic diagram of the structure of the residual channel prior extraction module in an embodiment of the present invention. Figure 3 In the figure, the residual channel prior extraction module includes a residual layer based on the L2 norm, the first convolutional layer Conv, three stacked channel attention networks ECANet, and the second convolutional layer Conv. Mean Channel represents the mean value in the channel dimension.

[0033] In some preferred embodiments, the residual channel prior extraction module is used to extract the prior feature map of the rainy image sample, and the specific process includes:

[0034] S201: Calculate the color image intensity of the input rain image sample (including rain streaks), which is specifically expressed as:

[0035]

[0036] in, represents the color vector of the color intensity in the input rainy image, x represents the pixel of the input image, t represents the time when a raindrop passes through pixel x, and β rs (x) represents the refraction coefficient, which is composed of the refraction coefficient, specular reflection coefficient and internal reflection coefficient of the raindrop; B stands for brightness, R represents the background reflection, α represents the chromaticity of B, π represents the chromaticity of R, is the exposure time, r represents red, g represents green, and b represents blue.

[0037] In the formula, the first term is the rain mark term, the second term For background items.

[0038] S202: Based on the color image intensity, the input rain image is normalized, and the specific normalization formula is:

[0039]

[0040] Where O(x) represents the normalized image, x represents the pixel of the input image, α is estimated by using the existing color constant algorithm, and O rs (x) represents the rain streak term, O bg (x) represents the background term, Vector division is element-wise division.

[0041] When the input rainy image is normalized, the light chromaticity will be cancelled and the color effect of the spectral sensitivity will also be cancelled. Therefore, introducing the residual channel prior P, which only contains a transformed background detail, can achieve the effect of maintaining the structure of the rain-free image.

[0042] S203: Based on the normalized rainy image, a residual channel prior P is introduced to obtain a background image.

[0043] The residual channel prior P is defined as:

[0044]

[0045] In the formula, P(x) represents the residual value of the input image based on L2 in the channel dimension, that is, the mean of all pixels on each channel (R, G, B) is calculated, R represents red, G represents green, B represents blue, and x represents the pixel of the input image. represents the pixel value of (i, j) in the channel dimension, μ(x) represents the mean in the channel dimension, H represents the height of the feature map, i represents the position index of the height of the feature map, W represents the width of the feature map, and j represents the position index of the width of the feature map.

[0046] S204: The background image is sequentially passed through the first convolutional layer Conv to obtain an initial feature map, and the initial feature map is input into three layers of sequentially stacked channel attention network ECANet and the second convolutional layer Conv for processing to obtain a priori feature map of the rainy image sample.

[0047] Specifically, the residual prior is introduced into the model to guide image deraining. The background image of the rainy image is obtained through the residual prior, and the initial feature map is obtained through convolution in sequence. Then, the noise in the initial feature is reduced, the semantic information of the feature is enriched, and the deep features of this prior are extracted using the channel attention mechanism.

[0048] S3: Use the context feature extraction module to extract features from rainy image samples and obtain shallow features of rain streaks at different scales.

[0049] Figure 4 Schematic diagram of the structure of the context aggregation module in an embodiment of the present invention. Figure 4 In , the context aggregation module includes the channel attention module, the pixel attention module and the multi-scale feature extraction module (MFEM). pooling represents the pooling layer, Conv represents the convolution layer, Relu represents the Relu activation function, Sigmoid represents the Sigmoid activation function, and Leaky Relu represents the activation function.

[0050] In some preferred embodiments, reference Figure 4 As shown, the context aggregation module includes a channel attention module, a pixel attention module and a multi-scale feature extraction module MFEM. The channel attention module and the pixel attention module are connected in series in a product manner and then connected in parallel with the multi-scale feature extraction module MFEM. The context aggregation module introduces a pyramid structure to extract rain streak distribution information of different scales to solve the problem of insufficient feature extraction at a single scale, in which multi-scale learning is a key step.

[0051] In some preferred embodiments, reference Figure 4 As shown in FIG. 1 , the multi-scale feature extraction module (MFEM) is used to process the rain strip features in the input rainy image. The specific process is as follows:

[0052] S301: For a given rain strip feature f in To downsample at a rate of 2, we get features of different scales, specifically:

[0053] R o =LeakyReLU(conv 3×3 (f in ))

[0054] R n =ResBlock3(R o )

[0055] R m =MaxPool m (R n )

[0056] In the formula, R o It represents a feature obtained after the rain strip feature passes through convolution and activation function, f in Represents the given rain strip feature, conv 3×3 (f in ) performs a 3×3 convolution operation on the input rain strip features, LeakyReLU(·) represents the activation function, R n Represents the use of residual blocks for R o The features obtained by processing, ResBlock3(·) represents the residual block operation, m represents different scales, MaxPool m (·) indicates maximum pooling, using MaxPool to R n Downsampling, R m Represents the features of different scales obtained after downsampling, where the scale m = 1, 2, 4, 8.

[0057] S302: Different scale features R m (R 1 ,R 2 ,R 4 ,R 8 ) is convolved with 4 convolutional layers activated by LeakyReLU to obtain a feature output of different scales Specifically:

[0058]

[0059] in, Represents a feature output of different scales obtained after multiple layers of convolution and activation function, m = 1, 2, 4, 8.

[0060] S303: Outputting the features of different scales After three times of upsampling by a factor of 2, the output features of the multi-scale feature extraction module are obtained.

[0061] Specifically, Upsample by 2 times to the previous adjacent scale, and then After concatenation, a feature output is obtained, which is upsampled by 2 times to the previous adjacent scale and then combined with The output is then upsampled by a factor of 2 to the previous adjacent scale and then The concatenation results in an output, which is convolved by 1×1 to obtain the output features of the multi-scale feature extraction module, which are as follows:

[0062]

[0063] in, Indicates that four different scale features are gradually concatenated (for two adjacent scales, the small scale feature is first upsampled, and its output is concatenated with the latter until it is restored to its original size). Up(·) is a linear interpolation operation, Concat(·) is a concatenation operation, and R out represents the final output of the entire MFEM module, represents the output obtained by the original multiple, represents the output obtained by downsampling by 2 times, represents the output obtained by downsampling 4 times, Represents the output features obtained by downsampling 8 times.

[0064] The channel attention module mainly focuses on the completely different weighted information contained in different channel features, extracts important features of rainy images and assigns different weighted information.

[0065] In some preferred embodiments, reference Figure 4 As shown in the figure, the channel attention module is used to extract features of the input rainy image and assign different weighted information. The specific process is as follows:

[0066] S311: Use the global adaptive average pooling function to convert the global spatial information of the channel in the input rainy image into a channel descriptor:

[0067]

[0068] Among them, G C represents the output of adaptive global average pooling, H P (·) represents the adaptive global average pooling function, f in represents the given rain strip feature, H represents the height of the feature map, W represents the width of the feature map, and X C (i, j) represents the value of the Cth channel at the (i, j) position, i represents the height position index of the image, and j represents the width position index of the image.

[0069] S312: Based on the channel descriptor G C , the input rainy image passes through two convolutional layers, Sigmoid and ReLu activation functions, and the weights of different channels are obtained, which are specifically:

[0070] W CA =Sigmoid(Conv(ReLu(Conv(G C )))

[0071] Where W CArepresents the weight of the channel, Conv(·) represents the convolution operation, ReLu(·) represents the ReLu activation function, and Sigmoid(·) represents the Sigmoid activation function.

[0072] S313: The rain stripe feature f in the input rainy image in and the channel weight W CA Multiply the corresponding elements to get the output features of the entire channel attention, which are:

[0073]

[0074] In the formula, F CA Represents the output features of the entire channel attention, Represents element-wise multiplication.

[0075] Taking into account the uneven distribution of rain streaks on different image pixels, the pixel attention module is used to make the network pay more attention to the information characteristics of rain streaks and high-frequency areas in rainy images, and pay more attention to retaining the texture detail information of objects in the input image.

[0076] The output feature F of the entire channel attention CA The input is fed into two convolutional layers, including ReLu and Sigmoid activation functions, and the output features of the pixel attention module are obtained after processing, which are as follows:

[0077] F PA =Sigmoid(Conv(ReLu(Conv(F CA )))

[0078]

[0079] In the formula, F PA Represents the output features of the pixel attention module, for the input H CA With H PA Using element-wise multiplication, F CPA Represents the output features of the pixel attention and channel attention modules in series.

[0080] The input rainy image is processed by the channel attention module and the pixel attention module to obtain the first feature map F CPA (·), the input rainy image is processed by the multi-scale feature extraction module MFEM to obtain the second feature map MFEM(·), the first feature map F CPA (·) is connected in parallel with the second feature map MFEM(·) by multiplication, and the result of the connection is connected in parallel with the rain stripe features in the rainy image to obtain the final output of the context aggregation module, that is, the feature map finally output by the module, which includes the shallow features of rain stripes at different scales extracted by the module.

[0081] The final output of the context aggregation module is expressed as:

[0082]

[0083] Where MABL represents the feature map output by the context aggregation module, and f in represents the rain stripe feature in the input rainy image, F CPA (·) represents the output feature map after the pixel attention module and the channel attention module are connected in series, and MFEM(·) represents the output feature map of the multi-scale feature extraction module MFEM.

[0084] S4: Using a feature fusion module, the prior feature map and shallow features of rain streaks at different scales are fused to obtain a first comprehensive feature map.

[0085] Figure 5 Schematic diagram of the structure of the feature fusion module in an embodiment of the present invention. Figure 5 In the figure, Conv3×3 represents a 3×3 convolutional layer, SENet represents a SE network (Squeeze-and-Excitation Net), represents element-wise multiplication, Sigmoid(·) represents the Sigmoid activation function, and Conv1×1 represents a 1×1 convolutional layer.

[0086] The feature map obtained by the context feature aggregation module and the prior feature map output by the residual channel prior extraction module are respectively transformed by 3×3 convolution, and then the importance of each channel is adaptively adjusted through the SE (Squeeze-and-Excitation) module, which is specifically expressed as follows:

[0087] W r =Sigmoid(FC(GlobalPool(F r ))

[0088] W p =Sigmoid(FC(GlobalPool(F p ))

[0089] In the formula, F r represents the output of the rainy image after 3×3 convolution, F p represents the output of the prior feature after 3×3 convolution, W r represents the attention weight of the rainy image feature obtained after channel attention, W p Represents the attention weight of the prior features obtained after channel attention.

[0090] The shallow rain image features and prior features are fused by element-by-element multiplication, allowing the features to interact with each other to obtain the weighted feature F′ r , F′ p , which are specifically:

[0091] F′ r =F r ·W r ,F′ p =F p ·W p

[0092] In the formula, F′ r represents the weighted rainy image feature, F′ p represents the weighted prior features, represents the dot product,

[0093] The learnable scaling parameters α and β are used to control the weight of each feature in feature fusion.

[0094] F″ r =α·F′ r ,F″ p =β·F′ p

[0095] In the formula, F″ r Represents the scaled rain image feature, F″ p represents the scaled prior feature, α represents the first scaling parameter, β represents the second scaling parameter, and · represents the dot product.

[0096] After Sigmoid activation, a gating mechanism is used to perform further nonlinear control on the result of the element-by-element operation.

[0097] Finally, all fused features are combined into the final output features through a 1×1 convolutional layer.

[0098] F fusion =Conv 1×1 (γ(F″ r ×F″ p +F″ r +F″ p ))

[0099] In the formula, F fusion It represents the output feature of the feature fusion module, and γ represents the gating parameter. By adjusting γ, the weight is adjusted to enhance the flexibility of different feature combinations.

[0100] S5: Use L recursive Transformer extraction modules RSTM to process the first comprehensive feature map to obtain a rain-free image.

[0101] Figure 6Schematic diagram of the structure of the recursive Transformer extraction module in an embodiment of the present invention. Figure 6 In the paper, the Recursive Transformer (RSTM) consists of the first normalization layer (Layer Norm, LN), the windowed multi-head self attention mechanism module (window-based multi-head self attention, WMSA), the second normalization layer LN and the Multi-Scale Feature Refinement Feedforward Network (Multi-Scale Feature Refinement Feedforward Network, MFRFN).

[0102] In some preferred embodiments, reference Figure 6 As shown in the figure, the recursive Transformer extraction module RSTM includes the first normalization layer LN, the windowed multi-head attention mechanism module WMSA, the second normalization layer LN and the multi-scale feature refinement feedforward network MFRFN in sequence. Each RSTM uses recursive and residual connections to stack Swin Transformer blocks.

[0103] In the deep feature extraction stage, recursive Swin Transformer blocks (RSTM) and convolution blocks are used. Each RSTM uses recursive and residual connections to stack Swin Transformer blocks. The specific structure is shown in the figure. Figure 6 Each STM can be described as:

[0104] STM=WMSA(LN(X))+MFRFN(LN(WMSA(LN(X))))

[0105] Where STM represents the improved Swin Transformer structure, X represents, LN(·) represents layer normalization, WMSA(·) represents the windowed multi-head attention mechanism, and MFRFN(·) represents the output features after processing by the MFRFN module.

[0106] In the Swin Transformer block, the feature map is divided into multiple non-overlapping window areas, and self-attention operations are performed in each window to obtain local information in the local window. The shifted window is used to establish connections between local windows to obtain the long-range relationship between pixels, thereby quickly and comprehensively extracting the features of the input rainy day image.

[0107] The WMSA module can be expressed as:

[0108] MultiHead(Q,K,V)=Concat(h1...h n )WO

[0109] h i =Attention(QW i Q ,KW i K ,VW i V )

[0110]

[0111] Among them, MultiHead(Q,K,V) represents multi-head self-attention, Q represents the query in multi-head attention, K represents the key in multi-head attention, V represents the value in multi-head attention, and h n represents the self-attention of each head, W O represents a learned output transformation matrix, h i Represents the output of weighted calculation by query, key, and value, W i Q represents the weight matrix, W i K represents the weight matrix, W i V represents the weight matrix, K T represents the transpose of the key matrix, Softmax(·) represents the Softmax function, represents the dimension of k, where k represents the key vector.

[0112] Let the number of STMs in each residual connection be U, the number of recursive calls be L, and the input be X i ∈R HW / / P×D , then RSTM can be described by the following formula:

[0113]

[0114] Among them, X i+1 represents the output of L RSTMs, Indicates the call to control RSTM. Indicates the combination of controlling different STMs, U=2, L=3, STM i (·) represents the output of the i-th STM.

[0115] At the end of each RSTM, a convolutional layer is used to process information that may be ignored by the Swin Transformer based on the local window.

[0116] The regular feedforward network processes the information of each pixel position separately and improves the feature representation through the self-attention mechanism. When the Swin Transformer processes the image through a hierarchical local window attention mechanism, it reduces the computational overhead brought by global self-attention and effectively captures the features of the local area. However, similar or repeated information may still exist in each local window, and there is still redundancy in the channel. To overcome this problem, a multi-scale feature refinement feedforward network (MFRFN) module is designed.

[0117] Figure 7 Schematic diagram of the structure of the multi-scale feature refinement feed-forward network MFRFN module in an embodiment of the present invention. Figure 7 In the MFRFN module, the layer normalization layer LN, the partial convolution layer PConv, the first linear layer Linner, the feature splitting layer Split, the feature reshaping layer Reshape, two parallel branches (which are DWConv5×5 and Relu function in series, DWConv3×3 and Relu function in series), the flatten layer Flatten and the second linear layer Linner. Among them, Indicates splicing, Represents matrix multiplication.

[0118] In some preferred embodiments, reference Figure 7 As shown, the multi-scale feature refinement feedforward network MFRFN includes a layer normalization layer LN, a partial convolution layer PConv, a first linear layer Linner, a feature splitting layer Split, a feature reshaping layer Reshape, two parallel branches, a flattening layer Flatten and a second linear layer Linner in sequence, and the two parallel branches are respectively: a series branch of DWConv5×5 and Relu function and a series branch of DWConv3×3 and Relu function.

[0119] The MFRFN module enhances the information elements in the features by introducing PConv operations and introduces depthwise separable convolutions to reduce the processing burden of redundant information. The rich multi-scale representation has fully demonstrated its effectiveness in better rain removal.

[0120] The MFRFN module is specifically represented as follows:

[0121]

[0122]

[0123] In the formula, Represents an output after some convolutional layers and linear projection. represents the output of the second LN layer in the STM module, PConv(·) represents the partial convolution operation, GELU(·) represents the activation function, W1 represents the first linear projection, [,] represents the channel slicing operation, It means that after the split, the channel becomes half of the original output. It means that the output of the other part of the original segmentation is flattened after the deep convolution operation of the two branches and then spliced. The output obtained by element-wise multiplication is, Represents matrix multiplication, R(·) represents reshape operation, F(·) represents Flatten operation, R(·) and F(·) are crucial to introduce locality into the architecture, concact(·) represents concatenation operation, DWConv(·) represents depthwise convolution operation, represents the final output of the MFRFN module, and W2 represents the second linear projection.

[0124] MFRFN is able to enhance feature representation by extracting representative features from the information flow while simplifying redundant features. It also provides the model with the opportunity to clean up uninformative features along the channel dimension.

[0125] S6: Calculate the loss between the derained image and the real rain-free image corresponding to the input rainy image sample, and optimize the network parameters by backpropagating the loss until the loss function converges and then stop the optimization.

[0126] The loss function adopts mean square error, which is specifically:

[0127]

[0128] Among them, MSE Loss represents the mean square error loss, y k represents the predicted value of the kth sample, represents the predicted value of the kth sample, and N represents the total number of samples. For multiple predictions, the final error measurement is obtained by calculating the average of the squares of the differences between the predicted values ​​and the actual observed values. A smaller MSE Loss value indicates a better fit of the model.

[0129] The mean square error loss is used to reduce the deviation in pixel values ​​between the original image and the target image, making the image cleaner after rain removal.

[0130] A person skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructing the relevant hardware through a program, and the program may be stored in a computer-readable storage medium, which may include: ROk, RAM, disk or CD, etc.

[0131] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multi-scale Transformer image deraining method based on residual channel prior guidance, characterized in that: include: Get a rainy image and input it into the trained image deraining network to get a derained image. The training process of the image deraining processing network includes: Obtain rainy image samples and their corresponding true rain-free images; The residual channel prior extraction module is used to extract the prior feature map of the rainy image samples; The context feature extraction module is used to extract features from rainy image samples to obtain shallow features of rain streaks at different scales. Using a feature fusion module, the prior feature map and shallow features of rain streaks at different scales are fused to obtain a first comprehensive feature map; The first comprehensive feature map is processed using L recursive Transformer extraction modules RSTM to obtain a rain-free image; The loss between the derained image and the real rain-free image corresponding to the input rainy image sample is calculated, and the network parameters are optimized by backpropagation through the loss until the loss function converges and the optimization is stopped.

2. According to claim 1, a multi-scale Transformer image deraining method based on residual channel prior guidance is characterized in that: The residual channel prior extraction module sequentially includes a residual layer based on the L2 norm, a first convolutional layer Conv, three sequentially connected channel attention networks ECANet and a second convolutional layer Conv.

3. The multi-scale Transformer image deraining method based on residual channel prior guidance according to claim 1, characterized in that: The residual channel prior extraction module is used to extract the prior feature map of the rainy image sample. The specific process includes: Calculate the color image intensity of the input rainy image sample; Based on the color image intensity, normalizing the input rainy image; Based on the normalized rainy image, the residual channel prior P is introduced to obtain the background image; The background image is passed through the first convolutional layer Conv in sequence to obtain an initial feature map, which is then input into three layers of stacked channel attention network ECANet and the second convolutional layer Conv for processing to obtain a priori feature map of the rainy image sample.

4. The multi-scale Transformer image deraining method based on residual channel prior guidance according to claim 1, characterized in that: The context aggregation module includes a channel attention module, a pixel attention module and a multi-scale feature extraction module MFEM. The channel attention module and the pixel attention module are connected in series in a product manner and then connected in parallel with the multi-scale feature extraction module MFEM.

5. The multi-scale Transformer image deraining method based on residual channel prior guidance according to claim 4, characterized in that: The input rainy image is processed by the channel attention module and the pixel attention module to obtain the first feature map F CPA (·), the input rainy image is processed by the multi-scale feature extraction module MFEM to obtain the second feature map MFEM(·), the first feature map F CPA (·) is connected in parallel with the second feature map MFEM(·) by multiplication, and the result of the connection is connected in parallel with the rain strip features in the rainy image to obtain the final output of the context aggregation module.

6. The multi-scale Transformer image deraining method based on residual channel prior guidance according to claim 4, characterized in that: The multi-scale feature extraction module MFEM is used to process the rain strip features in the input rainy image. The specific process is as follows: For a given rain strip feature f in To downsample at a rate of 2, we get features R of different scales m ; The features R of different scales m Convolve with 4 convolutional layers activated by LeakyReLU to get a feature output of different scales Output the features of different scales After three times of upsampling by a factor of 2, the output features of the multi-scale feature extraction module are obtained.

7. The multi-scale Transformer image deraining method based on residual channel prior guidance according to claim 4, characterized in that: The channel attention module is used to extract features from the input rainy image and assign different weighted information. The specific process is as follows: The global adaptive average pooling function is used to convert the global spatial information of the channel in the input rainy image into the channel descriptor G C ; Based on the channel descriptor G C , the input rainy image passes through two convolutional layers, Sigmoid and ReLu activation functions, and the weights of different channels are obtained; The rain stripe feature f in the input rainy image in and the channel weight W CA Multiply the corresponding elements to get the output features of the entire channel attention.

8. The multi-scale Transformer image deraining method based on residual channel prior guidance according to claim 1, characterized in that: The recursive Transformer extraction module RSTM consists of a first normalization layer LN, a windowed multi-head attention mechanism module WMSA, a second normalization layer LN, and a multi-scale feature refinement feedforward network MFRFN in sequence. Each RSTM uses recursive and residual connections to stack Swin Transformer blocks.

9. The multi-scale Transformer image deraining method based on residual channel prior guidance according to claim 8, characterized in that: The multi-scale feature refinement feedforward network MFRFN includes a layer normalization layer LN, a partial convolution layer PConv, a first linear layer Linner, a feature splitting layer Split, a feature reshaping layer Reshape, two parallel branches, a flattening layer Flatten and a second linear layer Linner in sequence, and the two parallel branches are respectively: a series branch of DWConv5×5 and Relu function and a series branch of DWConv3×3 and Relu function.

Images