A multi-scale transformer image rain removal method based on residual channel prior guidance

Abstract

The application discloses a multi-scale Transform image rain removal method based on residual channel prior guidance, comprising: acquiring a rainy image, inputting the rainy image into a trained image rain removal processing network, and obtaining a rain-removed image. The training process of the image rain removal processing network comprises: acquiring a rainy image sample and a corresponding real rain-free image; extracting a prior feature map of the rainy image sample by using a residual channel prior extraction module; extracting features of the rainy image sample by using a context aggregation module, and obtaining shallow features of rain streaks in different scales; performing feature fusion on the shallow features to obtain a first comprehensive feature map; processing the first comprehensive feature map by using a recursive Transform extraction module RSTM to obtain a rain-removed image; calculating a loss between the rain-removed image and the real rain-free image corresponding to the input rainy image sample, and optimizing network parameters by backward propagation of the loss. The application can remove rain streaks from different directions and scales, so that the rain-removed image is more consistent with a real image.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a multi-scale Transformer image rain removal method based on residual channel prior guidance. BACKGROUND

[0002] Images captured outdoors in rainy weather are easily disturbed by rain streak density, shape and direction, affecting their visual quality and causing degradation of some advanced computer vision tasks, such as object detection, image classification and image segmentation. How to effectively remove rain streaks from rainy images while preserving the details of the image has always been a research hotspot in the field of rain removal.

[0003] Current single-image rain removal algorithms mainly fall into two categories: model-driven and data-driven. In model-driven algorithms, traditional filters, dictionary learning, sparse coding and Gaussian mixture models cannot adapt to the diversity of rain marks, resulting in a large number of rain marks remaining in the rain-removed images.

[0004] Most of the recent state-of-the-art methods are based on convolutional networks, which have achieved impressive results. However, in complex scenarios, they are difficult to remove all rain streaks and restore the structural information of the image. This is because: most CNN-based methods learn the mapping relationship between rain-free images and rainy images, and achieve and obtain rain-free images under the constraints of the loss function. However, due to the change in rain streak density, too much or too little rain streaks are removed, resulting in incomplete structural information of the reconstructed image; these methods focus on the learning of rain streak structure, but pay less attention to the learning of object structure, ignoring the importance of image prior. At the same time, convolution has a limited receptive field, and CNN-based methods have difficulty effectively utilizing the global information of the rain image and the long-distance dependence between pixels, resulting in the loss of some details and structural information in the rain-removed image, and cannot obtain satisfactory rain removal results. Especially, the density, shape, position and size of rain streaks are uncertain, making it difficult for the network to recover a clean image from various degradations.

[0005] The Transformer has shown great potential in artificial intelligence applications. Initially, researchers applied the Transformer to natural language processing (NLP), achieving outstanding results there. Inspired by this, researchers extended it to computer vision tasks, resulting in the Vision Transformer (ViT), which has achieved breakthroughs in vision research such as object detection and image deblurring. However, the computational complexity of the original Vision Transformer grows quadratically with increasing spatial resolution of the input image. Therefore, its efficiency is significantly reduced when applied to vision tasks such as image deraining, failing to meet practical industry requirements.

[0006] Standard Transformers typically aggregate features using all attention relationships based on query-key pairs. Since tokens from the key are not always correlated with tokens from the query, the natively dense computational model of self-attention amplifies relatively small similarity weights, making feature interactions and the aggregation process susceptible to implicit noise. This leads to the introduction of corresponding redundant or irrelevant representations when modeling global feature dependencies. Therefore, these findings have inspired researchers to continuously explore how to resolve irrelevant redundant information in order to fully utilize these features for better image deraining. Summary of the Invention

[0007] To solve the above technical problems, the present invention proposes a multi-scale Transformer image deraining method based on residual channel prior guidance, which includes: obtaining a rainy image, inputting it into a trained image deraining processing network, and obtaining a derained image;

[0008] The training process of the image deraining processing network includes:

[0009] S1: Obtain rainy image samples and their corresponding true rain-free images;

[0010] S2: Use the residual channel prior extraction module to extract the prior feature map of the rainy image sample;

[0011] S3: Use the context feature extraction module to extract features from rainy image samples and obtain shallow features of rain streaks at different scales;

[0012] S4: using a feature fusion module to fuse the prior feature map and shallow features of rain streaks at different scales to obtain a first comprehensive feature map;

[0013] S5: Exploitation A recursive Transformer extraction module RSTM processes the first comprehensive feature map to obtain the rain-free image;

[0014] S6: Calculate the loss between the rain-removed image and the true rain-free image corresponding to the input rain image sample, optimize the network parameters through the loss, and stop optimization until the loss function converges.

[0015] The present application has the following advantages: The residual channel prior extraction module is introduced in the present application, which introduces a residual based on L2 norm, focuses on the local difference between channels, enhances the blurred or low-contrast rain streaks, effectively removes these disturbances in the rain removal process, and guides the high-quality image reconstruction; at the same time, considering the difference in different scales and densities of rain lines, a context aggregation module is introduced to adapt the algorithm to the diversity of rain trace distribution. The SwinTransformer is introduced to process the image, which can capture global information and long-distance dependence between pixels, reduce the computational overhead brought by global self-attention, effectively capture the characteristics of local regions, but in each local window, there may still be similar or repeated information, and there is still redundancy in the channel. The PConv operation is introduced in the recursive Transformer extraction module to enhance the information elements in the features, and the depth separable convolution is introduced to reduce the processing burden of redundant information. The present application can remove rain streaks from different directions and different scales with less parameter amount and less computing resource, so that the rain-removed image is more consistent with the effect of the real image. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 The training step flow chart of the image rain removal processing network in the embodiment of the present application is shown in the figure.

[0017] Figure 2 The structure diagram of the image rain removal processing network in the embodiment of the present application is shown in the figure.

[0018] Figure 3 The structure diagram of the residual channel prior extraction module in the embodiment of the present application is shown in the figure.

[0019] Figure 4 The structure diagram of the context aggregation module in the embodiment of the present application is shown in the figure.

[0020] Figure 5 The structure diagram of the feature fusion module in the embodiment of the present application is shown in the figure.

[0021] Figure 6 The structure diagram of the recursive Transformer extraction module RSTM in the embodiment of the present application is shown in the figure.

[0022] Figure 7 The structure diagram of the multi-scale feature refinement feedforward network MFRFN in the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0023] The terms "first", "second", "third", and the like in the description and claims of the application and the above figure are used for distinguishing between similar objects and not necessarily for describing a specific sequential or chronological order. It is to be understood that the terms so used are interchangeable under appropriate circumstances and are merely employed in the description of embodiments of the application for descriptive purposes.

[0024] The technical solutions in the embodiments of the application will be apparently and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the application.

[0025] The embodiment of the application provides a multi-scale Transformer image rain removal method based on residual channel prior guidance, which comprises the following steps: acquiring a rainy image, inputting the rainy image into a trained image rain removal processing network, and obtaining a rain-removed image.

[0026] Figure 1 The training step flowchart of the image rain removal processing network in the embodiment of the application.

[0027] Figure 2 The structural schematic diagram of the image rain removal processing network in the embodiment of the application. Figure 2 In the embodiment, the input image is a rainy image sample, and the image rain removal processing network is used for processing the input image to output a rain-removed image. The image rain removal processing network comprises a parallel residual channel prior extraction module and a context feature extraction module, a feature fusion module, a Patch Embedding, a plurality of RSTM modules and a Patch Unembed. The prior feature map of the input rainy image is extracted through the residual channel prior extraction module; the features of the input rainy image are extracted through the context feature extraction module to obtain shallow features of rain streaks at different scales; the prior feature map and the shallow features of rain streaks at different scales are fused through the feature fusion module to obtain a first comprehensive feature map; the first comprehensive feature map is divided into a plurality of small patches through the Patch Embedding, and then each small patch is converted into a fixed-dimensional vector representation, that is, the image is mapped from the pixel space to an embedding space; the rainy image is processed through the plurality of RSTM modules (that is, the recursive Transformer extraction module); and the image is reconstructed through the Patch Unembed (which is used for converting the features processed through the plurality of RSTM modules back to the image) to obtain a rain-removed image.

[0028] Reference Figure 1As shown, the training process of the image rain removal processing network includes:

[0029] S1: Obtain a rain image sample and its corresponding real rain-free image.

[0030] The synthetic data set used in the present embodiment includes pairs of rain images and corresponding real rain-free images, which are one-to-one corresponding, such as 1input corresponding to 1target, 2input corresponding to 2target, 1input representing a rain image numbered 1, and 1target representing a real rain-free image corresponding to the rain image numbered 1. If the output image obtained by the image rain removal processing network is named 1output, then 1output and 1target are compared. The loss of the image rain removal processing network in the training process is calculated by calculating the loss between the output image (such as 1output) of the network and its corresponding real rain-free image (such as 1target), and the training is stopped by making the loss converge.

[0031] S2: Extract the prior feature map of the rain image sample using the residual channel prior extraction module.

[0032] Figure 3 The structure of the residual channel prior extraction module in the present embodiment is shown in the figure. Figure 3 In the figure, the residual channel prior extraction module sequentially includes a residual layer based on L2 norm, a first convolutional layer Conv, a channel attention network ECANet stacked in three layers, and a second convolutional layer Conv. Mean Channel represents the mean value in the channel dimension.

[0033] In some preferred embodiments, the prior feature map of the rain image sample is extracted using the residual channel prior extraction module, and the specific process includes:

[0034] S201: Calculate the color image intensity of the input rain image sample (including rain streaks), which is specifically represented as:

[0035] ,

[0036] wherein, represents the color vector of the color intensity in the input rain image, represents the pixel of the input image, represents the time of the raindrop passing through the pixel , represents the refraction coefficient, consisting of the refraction coefficient, the specular reflection coefficient and the internal reflection coefficient of the raindrop; represents the three-channel brightness vector transmitted by the raindrop, , represents the sum of the brightness vectors, for the exposure time of the camera, the luminance vector representing red, r represents red, the luminance vector representing green, g represents green, the luminance vector representing blue, b represents blue; R represents the three-channel luminance vector of the background reflected light, , the sum of the background reflection vectors, the background reflection vector representing red, the background reflection vector representing green, the background reflection vector representing blue; the normalized chrominance vector of B, , .

[0037] In the formula, the first term ( ) is the rain mark term, and the second term ( ) is the background term.

[0038] S202: Based on the color image intensity, the input rain image is normalized, and the specific normalization formula is:

[0039] ,

[0040] ,

[0041] wherein, denotes the normalized image, denotes the pixel of the input image, is estimated by using the existing color constant algorithm, denotes the rain mark term, , is a short form of , which represents the refraction coefficient, , denotes the background term, , and the vector division is the element division.

[0042] When the input rain image is normalized, the light chrominance is cancelled, and the color effect of the spectral sensitivity is also cancelled. Therefore, the residual channel prior P is introduced, which only contains a transformed background detail to achieve the effect of maintaining the structure of the rain-removed image.

[0043] S203: Based on the normalized rain image, the residual channel prior P is introduced to obtain the background image.

[0044] The residual channel prior P is defined as , and its expression is:

[0045] ,

[0046] ,

[0047] In the formula, represents the L2-based residual value of the input image in the channel dimension, that is, the mean value of all pixels on each channel (R, G, B) is calculated, R represents red, G represents green, and B represents blue, represents the pixel of the input image, represents the pixel value of in the channel dimension, represents the mean value in the channel dimension, H represents the height of the feature map, i represents the position index of the feature map height, W represents the width of the feature map, and j represents the position index of the feature map width.

[0048] S204: The background image is sequentially subjected to the first convolution layer Conv to obtain an initial feature map, and the initial feature map is input into the three-layer channel attention network ECANet and the second convolution layer Conv for processing to obtain the prior feature map of the rainy image sample.

[0049] Specifically, the residual prior is introduced into the model to guide the image rain removal. The rainy image obtains the background image through the residual prior, is sequentially subjected to convolution to obtain an initial feature map, and then reduces the noise in the initial feature and enriches the semantic information of the feature. The channel attention mechanism is used to extract the deep features of the prior.

[0050] S3: The context aggregation module is used to extract features of the rainy image sample to obtain shallow features of rain patterns at different scales.

[0051] Figure 4 It is a structural schematic diagram of the context aggregation module in the embodiment of the application. Figure 4 In the embodiment, the context aggregation module includes a channel attention module, a pixel attention module, and a multi-scale feature extraction module (MFEM). Pooling represents a pooling layer, Conv represents a convolution layer, Relu represents a Relu activation function, Sigmoid represents a Sigmoid activation function, and Leaky Relu represents an activation function.

[0052] In some preferred embodiments, reference is made to Figure 4 ​As shown, the context aggregation module includes a channel attention module, a pixel attention module, and a multi-scale feature extraction module (MFEM). The channel attention module and the pixel attention module are connected in series via a product method and then in parallel with the multi-scale feature extraction module. The context aggregation module introduces a pyramid structure to extract rain streak distribution information at different scales, addressing the problem of insufficient feature extraction at a single scale. Multi-scale learning is a key step in this process.

[0053] In some preferred embodiments, reference Figure 4 As shown in the figure, the multi-scale feature extraction module (MFEM) is used to process the rain stripe features in the input rain image. The specific process is as follows:

[0054] S301: For a given rain strip feature To downsample at a rate of 2, we get features of different scales, specifically:

[0055] ,

[0056] ,

[0057] ,

[0058] Where, It represents a feature obtained after the rain strip feature passes through convolution and activation function. Represents the given rain strip features, Perform a 3×3 convolution operation on the input rain strip features. represents the activation function, Indicates the use of residual blocks The features obtained by processing, represents the residual block operation, Represents different scales, Represents the maximum pooling, using right Downsampling, Represents the features of different scales obtained after downsampling, where scale m = 1, 2, 4, 8.

[0059] S302: Features of different scales ( , , , ) is convolved with 4 convolutional layers activated by LeakyReLU to obtain a feature output of different scales , which are specifically:

[0060] ,

[0061] wherein, denotes a feature output of different scales after a plurality of convolution and activation functions, m = 1, 2, 4, 8.

[0062] S303: the feature outputs of different scales are sequentially concatenated to obtain a feature output of the multi-scale feature extraction module. After sequentially undergoing 3 times of 2 times upsampling processing respectively, the output features of the multi-scale feature extraction module are obtained.

[0063] Specifically, the feature output of the first scale is obtained by upsampling the input image by 2 times and then performing convolution on the output. The feature output of the second scale is obtained by upsampling the feature output of the first scale by 2 times and then performing convolution on the output. The feature output of the third scale is obtained by upsampling the feature output of the second scale by 2 times and then performing convolution on the output. The feature output of the fourth scale is obtained by upsampling the feature output of the third scale by 2 times and then performing convolution on the output. The output of the multi-scale feature extraction module is obtained by upsampling the feature output of the fourth scale by 2 times and then performing convolution on the output. The output of the multi-scale feature extraction module is obtained by upsampling the feature output of the fourth scale by 2 times and then performing convolution on the output.

[0064] ,

[0065] ,

[0066] wherein, denotes gradually concatenating four different scale features (two adjacent scales, the small scale feature is first upsampled, and its output is concatenated with the latter until the original size is restored), is a linear interpolation operation, is a concatenation operation, denotes the final output obtained by the MFEM module, denotes the output obtained by the original multiple, denotes the output obtained by downsampling by 2 times, denotes the output obtained by downsampling by 4 times, denotes the output feature obtained by downsampling by 8 times.

[0067] The channel attention module mainly focuses on different channel features containing completely different weighted information, extracts important features of the rainy image, and assigns different weighted information.

[0068] In some preferred embodiments, as shown in FIG. Figure 4 , the channel attention module is used to extract features of the input rainy image and assign different weighted information, and the specific process is as follows:

[0069] S311: use a global adaptive average pooling function to convert the global spatial information of the channels in the input rainy image into channel descriptors:

[0070] ,

[0071] wherein, the output of the adaptive global average pooling, denotes an adaptive global average pooling function, denotes a given rain streak feature, H denotes the height of the feature map, and W denotes the width of the feature map, denotes the value of the Cth channel at the position, denotes the height position index of the image, denotes the width position index of the image.

[0072] S312: based on the channel descriptor After the input rainy image passes through two convolutional layers, Sigmoid and ReLu activation functions, the weight values of different channels are obtained, which are specifically as follows:

[0073] ,

[0074] In the formula, denotes the weight value of the channel, denotes a convolution operation, denotes a ReLu activation function, denotes an activation function.

[0075] S313: multiplying the rain streak feature in the input rainy image by the weight value of the channel to obtain the output feature of the whole channel attention, which is specifically as follows:

[0076] ,

[0077] In the formula, denotes the output feature of the whole channel attention, denotes element-wise multiplication.

[0078] Considering that the rain streaks are unevenly distributed on different image pixels, the pixel attention module is used to make the network pay more attention to the information features of the rain streaks and high-frequency regions in the rainy image, and to pay more attention to preserving the texture detail information of the objects in the input image.

[0079] The output feature of the whole channel attention is input into two convolutional layers, including ReLu and Sigmoid activation functions, and after processing, the output feature of the pixel attention module is obtained, which is specifically as follows:

[0080] ,

[0081] ​ ,

[0082] In the formula, represents the output feature of the pixel attention module, and the input and The element multiplication method is used, represents the output feature after the pixel attention and the channel attention module are connected in series.

[0083] The first feature map of the input rainy image after being processed by the channel attention module and the pixel attention module The second feature map of the input rainy image after being processed by the multi-scale feature extraction module MFEM The first feature map and the second feature map are connected in parallel by multiplication, and the parallel results are connected in parallel with the rain strip features in the rainy image to obtain the final output of the context aggregation module, that is, the feature map finally output by the module, which includes the shallow features of the rain stripes in different scales extracted by the module.

[0084] The final output of the context aggregation module is represented as:

[0085] ,

[0086] In the formula, represents the feature map output by the context aggregation module, represents the rain strip features in the input rainy image, represents the output feature map after the pixel attention and the channel attention module are connected in series, represents the output feature map of the multi-scale feature extraction module MFEM.

[0087] S4: using the feature fusion module to perform feature fusion on the prior feature map and the shallow features of the rain stripes in different scales to obtain a first comprehensive feature map.

[0088] Figure 5 It is a structural schematic diagram of the feature fusion module in the embodiment of the application. Figure 5 In the formula, Conv3x3 represents a 3x3 convolution layer, SENet represents an SE network (Squeeze-and-Excitation Net), represents element multiplication, represents an activation function, and Conv1x1 represents a 1x1 convolution layer.

[0089] The feature map obtained by the context feature aggregation module and the prior feature map output by the residual channel prior extraction module are respectively processed by After the convolution transformation, the importance of each channel is adaptively adjusted through the SE (Squeeze-and-Excitation) module, which is specifically represented as:

[0090] ;

[0091] In the formula, represents the output of the rain image after 3 convolutions, represents the output of the prior feature after 3 convolutions, represents the attention weight of the rain image feature obtained after the channel attention, represents the attention weight of the prior feature obtained after the channel attention.

[0092] The shallow rain image feature and the prior feature are fused through element-wise multiplication, allowing the features to interact with each other, obtaining the weighted feature , which is specifically:

[0093] ,

[0094] In the formula, represents the weighted rain image feature, represents the weighted prior feature, represents the dot product,

[0095] Through the learnable scaling parameters and , the weight of each feature in feature fusion is controlled.

[0096] ,

[0097] In the formula, represents the scaled rain image feature, represents the scaled prior feature, represents the first scaling parameter, represents the second scaling parameter, represents the dot product.

[0098] After Sigmoid activation, the result of the element-wise operation is further controlled using a gating mechanism.

[0099] Finally, all the fused features are combined into the final output feature through convolution layers.

[0100] ,

[0101] In the formula, output features of the feature fusion module, gate parameters, by regulating weight adjustment, enhancing the flexibility of different feature combinations.

[0102] S5: using a recursive Transformer extraction module RSTM to process the first comprehensive feature map to obtain a rain-removed image.

[0103] Figure 6 is a structural schematic diagram of the recursive Transformer extraction module in the embodiment of the application. Figure 6 In the recursive Transformer (RSTM), a first layer normalization layer (Layer Norm, LN), a window-based multi-head self attention mechanism module (WMSA), a second layer normalization layer LN and a multi-scale feature refinement feedforward network (MFRFN) are sequentially included.

[0104] In some preferred embodiments, with reference to Figure 6 , the recursive Transformer extraction module RSTM sequentially includes a first layer normalization layer LN, a window-based multi-head self attention mechanism module WMSA, a second layer normalization layer LN and a multi-scale feature refinement feedforward network MFRFN, and each RSTM uses recursion and residual connection to stack Swin Transformer blocks.

[0105] In the deep feature extraction stage, recursive Swin Transformer blocks (RSTMs) and convolutional blocks are used, each RSTM uses recursion and residual connection to stack Swin Transformer blocks, and the specific structure is as shown in the accompanying drawing Figure 6 . Each STM can be described as:

[0106] ,

[0107] In the formula, denotes an improved Swin Transformer structure, denotes, denotes layer normalization processing, denotes a window-based multi-head self attention mechanism, denotes an output feature after processing by the module.

[0108] In the Swin Transformer block, the feature map is divided into multiple disjoint window regions, and a self-attention operation is performed in each window to obtain local information within the local window. The shifted window is used to establish the relationship between each local window to obtain the long-range relationship between each pixel, and then the features of the input rainy image are quickly and comprehensively extracted.

[0109] The WMSA module can be represented as:

[0110] ,

[0111] ,

[0112] ,

[0113] wherein, represents multi-head self-attention, represents the query in multi-head attention, represents the key in multi-head attention, represents the value in multi-head attention, represents the self-attention of each head, represents a learned output transformation matrix, represents the output calculated by weighting the query, the key and the value, represents a weight matrix, represents a weight matrix, represents a weight matrix, represents the transpose of the key matrix, represents function, represents dimension of represents the key vector.

[0114] The number of STMs in each residual connection is represented as , and the number of recursive calls is represented as L, then the input is represented as , can be described by the following formula,

[0115] ,

[0116] wherein, represents the output of L , represents the call to control , represents the combination of different , =2, L=3, represents the first One The output.

[0117] At the end of each Convolutional layers are used to process information that may be ignored by the local window Swin Transformer.

[0118] The regular feedforward network processes the information of each pixel position alone, and improves the feature representation through the self-attention mechanism. When the Swin Transformer processes the image through the hierarchical local window attention mechanism, the computational overhead brought by the global self-attention is reduced, and the features of the local area are effectively captured, but in each local window, there may still be similar or repetitive information, and there is still redundancy in the channel. In order to overcome this problem, a multi-scale feature refinement feedforward network (MFRFN) module is designed.

[0119] Figure 7 The structure diagram of the multi-scale feature refinement feedforward network MFRFN module in the embodiment of the application. Figure 7 In the embodiment, the MFRFN module includes a layer normalization layer LN, a partial convolution layer PConv, a first linear layer Linner, a feature splitting layer Split, a feature reshaping layer Reshape, two parallel branches (which are a DWConv5x5 and a Relu function in series, and a DWConv3x3 and a Relu function in series, respectively), a flattening layer Flatten and a second linear layer Linner. Among them, represents splicing, represents matrix multiplication.

[0120] In some preferred embodiments, as shown in Figure 7 The multi-scale feature refinement feedforward network MFRFN sequentially includes a layer normalization layer LN, a partial convolution layer PConv, a first linear layer Linner, a feature splitting layer Split, a feature reshaping layer Reshape, two parallel branches, a flattening layer Flatten and a second linear layer Linner, and the two parallel branches are respectively: a series branch of DWConv5x5 and a Relu function, and a series branch of DWConv3x3 and a Relu function.

[0121] The MFRFN module enhances the information elements in the features by introducing the PConv operation, and reduces the processing burden of redundant information by introducing the depth separable convolution. The rich multi-scale representation has fully proved the effectiveness in better rain removal.

[0122] The MFRFN module is specifically represented as:

[0123] ,

[0124] ,

[0125] ,

[0126] ,

[0127] In the formula, represents an output obtained after a partial convolution layer and a linear projection, represents the output of the second LN layer in the STM module, represents a partial convolution operation, represents an activation function, represents a first linear projection, represents a channel slicing operation, , represents an output in which the channel is reduced to half after splitting, represents an output obtained by performing a deep convolution operation on two branches, then performing splicing, flattening, and then combining with the output of another part of the original split an output obtained by element multiplication, represents matrix multiplication, represents a reshape operation, represents a Flatten operation, and It is essential to introduce locality into the architecture, represents a splicing operation, represents a deep convolution operation, represents the final output of the MFRFN module, represents a second linear projection.

[0128] The MFRFN can enhance feature representation by extracting representative features from the information flow, while simplifying redundant features. It also provides the model with the opportunity to clear the non-information features along the channel dimension.

[0129] S6: Calculate the loss between the rain-removed image and the true rain-free image corresponding to the input rain image sample, optimize the network parameters through the loss, and stop optimization until the loss function converges.

[0130] The loss function adopts mean square error, which is specifically:

[0131] ,

[0132] wherein, represents the mean square error loss, represents the predicted value of the i th sample, represents the predicted value of the i th sample, represents the predicted value of the i th sample, N is the number of samples, and N is the total number of samples. For multiple predictions, the final error measure is obtained by calculating the average of the square of the difference between the predicted value and the actual observed value. Smaller values indicate a better fitting of the model.

[0133] The mean square error loss is used to reduce the deviation of the original image and the target image in the pixel value, so that the image after rain removal is cleaner.

[0134] A person of ordinary skill in the art can understand that all or part of the steps in the above-mentioned embodiments can be completed by a program instructing related hardware, and the program can be stored in a computer readable storage medium, which can include ROM, RAM, magnetic disk or optical disk, etc.

[0135] Although the embodiments of the present application have been shown and described, it should be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A multi-scale Transformer image deraining method based on residual channel prior guidance, characterized in that, The method comprises the following steps: An image with rain is obtained, and the image is input into a trained image rain removal processing network to obtain a rain-removed image. The training process of the image rain removal processing network comprises the following steps: An image sample with rain and a corresponding real image without rain are obtained. A prior feature map of the image sample with rain is extracted by using a residual channel prior extraction module, the residual channel prior extraction module comprises, in sequence, a residual layer based on an L2 norm, a first convolutional layer Conv, a three-layer channel attention network ECANet, and a second convolutional layer Conv, and the specific process of extracting the prior feature map comprises the following steps: calculating the intensity of a color image of the input image with rain; performing normalization processing on the input image with rain based on the intensity of the color image; introducing a residual channel prior P based on the normalized image with rain to obtain a background image; inputting the background image into the first convolutional layer Conv to obtain an initial feature mapping, and inputting the initial feature mapping into the three-layer channel attention network ECANet and the second convolutional layer Conv in sequence to obtain the prior feature map of the image sample with rain. A feature of the image sample with rain is extracted by using a context aggregation module to obtain shallow features of rain streaks at different scales. The prior feature map and the shallow features of rain streaks at different scales are fused by using a feature fusion module to obtain a first comprehensive feature map. Utilizing a recursive transformer extraction module RSTM to process the first comprehensive feature map to obtain a rain-removed image; A loss between the rain-removed image and the real image without rain corresponding to the input image sample with rain is calculated, and the network parameters are optimized through the loss in a backward propagation manner until the loss function converges, and then the optimization is stopped.

2. The multi-scale Transformer image deraining method based on residual channel prior guidance according to claim 1, characterized in that, The context aggregation module comprises a channel attention module, a pixel attention module, and a multi-scale feature extraction module MFEM, the channel attention module and the pixel attention module are connected in series in a multiplication manner, and the multi-scale feature extraction module MFEM is connected in parallel.

3. The multi-scale Transformer image deraining method based on residual channel prior guidance according to claim 2, characterized in that, The input rainy image is processed by the channel attention module and the pixel attention module to obtain a first feature map The input rainy image is processed by the multi-scale feature extraction module MFEM to obtain a second feature map The first feature map The second feature map The first feature map and the second feature map are connected in parallel in a multiplication manner, and the parallel connection result is connected in parallel with the rain strip feature in the rainy image to obtain the final output of the context aggregation module.

4. The multi-scale Transformer image deraining method based on residual channel prior guidance according to claim 2, characterized in that, The rain streak features in the input image with rain are processed by using the multi-scale feature extraction module MFEM, and the specific process comprises the following steps: For a given rain streak feature To downsample at a rate of 2, resulting in features of different scales ; Features of different scales Convolve with 4 convolutional layers activated by LeakyReLU to obtain a feature output of different scales , m∈{1,2,4,8}; outputting the features of different scales After 3 times of 2 times upsampling processing respectively, the output features of the multi-scale feature extraction module are obtained.

5. The multi-scale Transformer image deraining method based on residual channel prior guidance according to claim 2, characterized in that, The features of the input image with rain are extracted and different weighting information is assigned by using the channel attention module, and the specific process comprises the following steps: applying a global adaptive average pooling function to convert global spatial information of channels in the input rainy image into channel descriptors ; based on the channel descriptor After the input rain image passes through two convolutional layers, a Sigmoid function and a ReLU activation function, weight values of different channels are obtained. extracting rain streak features from the input rainy image weights of the channels performing multiplication on corresponding elements to obtain output features of the whole channel attention.

6. The multi-scale Transformer image deraining method based on residual channel prior guidance according to claim 1, characterized in that, The recursive Transformer extraction module RSTM comprises, in sequence, a first normalization layer LN, a windowed multi-head attention mechanism module WMSA, a second normalization layer LN, and a multi-scale feature refinement feedforward network MFRFN, and each RSTM uses recursion and residual connection to stack Swin Transformer blocks.

7. The multi-scale Transformer image deraining method based on residual channel prior guidance according to claim 6, characterized in that, The multi-scale feature refinement feedforward network MFRFN comprises, in sequence, a layer normalization layer LN, a partial convolutional layer PConv, a first linear layer Linner, a feature splitting layer Split, a feature reshaping layer Reshape, two parallel branches, a flattening layer Flatten, and a second linear layer Linner, and the two parallel branches are a series connection branch of a DWConv5x5 and a Relu function and a series connection branch of a DWConv3x3 and a Relu function.

Images