Image rain removal method and device

Abstract

The application discloses an image rain removal method and device, and aims to solve the technical problem that the existing rain removal technology is difficult to accurately balance rain line removal and background detail recovery, resulting in poor image rain removal effect. The method comprises the following steps: acquiring a rain image, and performing multi-scale downsampling on the rain image to output a multi-scale rain image; inputting the rain image and the multi-scale rain image into a preset multi-scale dual-domain collaborative rain removal network, wherein the preset multi-scale dual-domain collaborative rain removal network comprises an encoder, a multi-scale spatial interaction feature fusion module and a decoder; encoding the rain image and the multi-scale rain image by using the encoder to output a plurality of target rain coding features; performing feature fusion on the plurality of target rain coding features by using the multi-scale spatial interaction feature fusion module to output target fusion features; and inputting the rain image, the multi-scale rain image, the target fusion features and the target rain coding features into the decoder for decoding to generate a target rain removal image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to an image rain removal method and device. BACKGROUND

[0002] In the field of computer vision, low-quality images have a significant impact on the performance of various applications, and the restoration of rain images is particularly critical and challenging.

[0003] Rainwater, as a common natural phenomenon, forms a rain line pattern that is both unknown and difficult to predict from the distribution of the real background, resulting in problems such as reduced contrast, blurred images, and loss of detail information in images taken in the rain. This degradation not only affects the intuitive perception of the scene by the human eye, but also interferes with the normal operation of computer vision systems, making the accuracy of basic tasks such as object detection and target tracking greatly compromised. Therefore, the study of image rain removal in bad weather is an urgent need to improve the reliability of outdoor imaging systems.

[0004] Existing rain removal techniques mostly tend to model features in a single dimension (such as spatial dimension or channel dimension) or limited domain (such as frequency domain, spatial domain), relying on models such as Transformers for attention calculation and feature extraction. However, this approach often overlooks the supporting role of multi-scale features of the encoder in learning by the decoder; at the same time, when multi-domain joint modeling is involved, it is also prone to limitations such as difficulty in effectively integrating high and low frequency features and lack of innovation in feature correlation mining. These problems, compounded, make it difficult for existing techniques to accurately balance rain line removal and background detail restoration, resulting in poor image rain removal results. SUMMARY

[0005] The present application provides an image rain removal method and device to solve the technical problem that existing rain removal techniques cannot accurately balance rain line removal and background detail restoration, resulting in poor image rain removal results.

[0006] The first aspect of the present application provides an image rain removal method, comprising:

[0007] Obtaining a rain image and performing multi-scale down-sampling on the rain image to output a multi-scale rain image;

[0008] Inputting the rain image and the multi-scale rain image into a pre-set multi-scale dual-domain collaborative rain removal network, the pre-set multi-scale dual-domain collaborative rain removal network comprising an encoder, a multi-scale spatial interaction feature fusion module and a decoder;

[0009] Encoding the rain image and the multi-scale rain image using the encoder to output a plurality of target rain encoding features;

[0010] The multi-scale space interaction feature fusion module is used for performing feature fusion on the plurality of target rain encoding features, and outputs target fusion features;

[0011] The rain image, the multi-scale rain image, the target fusion features and the target rain encoding features are input into the decoder for decoding, and a target rain removal image is generated.

[0012] Optionally, the multi-scale rain image includes a first scale rain image and a second scale rain image; the encoder includes a cooperative feature enhancement module and a 3*3 deep hyper-parameterized convolution layer; the plurality of target rain encoding features includes a first rain encoding feature, a second rain encoding feature and a third rain encoding feature; and the encoding of the rain image and the multi-scale rain image by using the encoder to output the plurality of target rain encoding features includes:

[0013] The 3*3 deep hyper-parameterized convolution layer is used to perform deep hyper-parameterized convolution on the rain image, and output deep convolution features;

[0014] The cooperative feature enhancement module is used to perform feature enhancement on the deep convolution features, and output the first rain encoding feature;

[0015] Feature extraction is performed on the first scale rain image, and first scale rain image features are output;

[0016] Down-sampling is performed on the first rain encoding feature, and the first rain encoding feature after down-sampling is output;

[0017] The first scale rain image features and the first rain encoding feature after down-sampling are fused to generate first fusion features;

[0018] The cooperative feature enhancement module is used to perform feature enhancement on the first fusion features, and output the second rain encoding feature;

[0019] Feature extraction is performed on the second scale rain image, and second scale rain image features are output;

[0020] Down-sampling is performed on the second rain encoding feature, and the second rain encoding feature after down-sampling is output;

[0021] The second scale rain image features and the second rain encoding feature after down-sampling are fused to generate second fusion features;

[0022] The cooperative feature enhancement module is used to perform feature enhancement on the second fusion features, and output the third rain encoding feature.

[0023] Optionally, the collaborative feature enhancement module comprises a frequency focus enhancement convolution block and a double-flow sparse interaction block based on a self-attention mechanism; the feature processing step of the collaborative feature enhancement module is specifically:

[0024] The input feature input into the collaborative feature enhancement module is focused and enhanced by the frequency focus enhancement convolution block to generate a focused and enhanced feature;

[0025] The focused and enhanced feature is taken as the input of the double-flow sparse interaction block based on the self-attention mechanism to generate an output feature.

[0026] Optionally, the frequency focus enhancement convolution block comprises a 3x3 deep hyper-parameterized convolution layer, a ReLU activation function layer, a 1x1 convolution layer and a Sigmoid activation function layer; the input feature input into the collaborative feature enhancement module is focused and enhanced by the frequency focus enhancement convolution block to generate a focused and enhanced feature, comprising:

[0027] The input feature is deep hyper-parameterized convolved by the 3x3 deep hyper-parameterized convolution layer to output an intermediate deep convolution feature, and the intermediate deep convolution feature is taken as the input of the ReLU activation function layer to output a first nonlinear feature;

[0028] The first nonlinear feature is deep hyper-parameterized convolved by the 3x3 deep hyper-parameterized convolution layer to generate a local feature, and the local feature is subjected to Fourier transform to output a first real part feature and a first imaginary part feature;

[0029] The first real part feature and the first imaginary part feature are spliced to output a first spliced feature, and the first spliced feature is convolved by the 1x1 convolution layer to output a first convolution feature;

[0030] The first convolution feature is taken as the input of the ReLU activation function layer to output a second nonlinear feature, and the second nonlinear feature is convolved by the 1x1 convolution layer to generate a second convolution feature;

[0031] The second convolution feature is divided into a second real part feature and a second imaginary part feature along the channel, and the second real part feature and the second imaginary part feature are subjected to inverse Fourier transform to output a high-frequency feature;

[0032] The local feature is subjected to global average pooling to generate a pooled feature, and the pooled feature is convolved by the 1x1 convolution layer to output a third convolution feature;

[0033] The third convolution feature is taken as the input of the ReLU activation function layer to output a third nonlinear feature, and the third nonlinear feature is convolved by the 1x1 convolution layer to generate a fourth convolution feature;

[0034] mapping the fourth convolutional feature by using a Sigmoid activation function layer to output a mapping feature, and multiplying the mapping feature and the local feature element by element to output a multiplication feature;

[0035] performing a fast Fourier transform convolution on the multiplication feature to generate a low-frequency feature, and adding the low-frequency feature, the high-frequency feature and the input feature element by element to generate a focus-enhanced feature.

[0036] Optionally, the double-flow sparse interaction block based on a self-attention mechanism includes a 1x1 convolutional layer, a Softmax activation function layer, an adaptive interaction module, and a gated depth-wise convolution feedforward network; the focus-enhanced feature is input into the double-flow sparse interaction block based on the self-attention mechanism to generate an output feature, including:

[0037] performing layer normalization on the focus-enhanced feature to generate a first normalized feature;

[0038] generating an initial query feature, an initial key feature and an initial value feature from the first normalized feature by using a 1x1 convolutional layer, and performing group convolution on the initial query feature to generate a query tensor;

[0039] performing depth-wise convolution on the initial key feature and the initial value feature respectively to generate a key tensor and a value tensor, and reshaping the query tensor, the key tensor and the value tensor respectively to output a reshaped query tensor, a reshaped key tensor and a reshaped value tensor;

[0040] performing matrix multiplication on the reshaped query tensor and the reshaped key tensor to generate a multiplication feature, and inputting the multiplication feature into a Softmax activation function layer to output a sparse attention map;

[0041] performing matrix multiplication on the sparse attention map and the reshaped value tensor to generate an attention output feature, and reshaping the attention output feature to generate a reshaped attention output feature;

[0042] performing convolution operation on the value tensor by using a 1x1 convolutional layer to output a convolution value tensor, and performing depth-wise convolution on the convolution value tensor to generate a depth-wise convolution value tensor;

[0043] adding the depth-wise convolution value tensor and the convolution value tensor element by element to output a first addition vector, and inputting the first addition vector and the reshaped attention output feature into an adaptive interaction module for feature interaction to output a target interaction feature;

[0044] convolve the target interaction feature by using a 1x1 convolution layer to output a convolution interaction feature, and perform element-wise addition on the convolution interaction feature and the focus enhancement feature to output a second added feature;

[0045] perform layer normalization on the second added feature to output a second normalized feature, and generate a feedforward feature according to the second normalized feature by using a gated depth-wise convolution feedforward network;

[0046] perform element-wise addition on the feedforward feature and the second added feature to generate an output feature.

[0047] Optionally, the adaptive interaction module includes a 1x1 convolution layer, a GeLU activation function layer, and a Sigmoid activation function layer; and the inputting of the first added vector and the reshaped attention output feature into the adaptive interaction module for feature interaction to output a target interaction feature includes:

[0048] convolve the first added vector by using a 1x1 convolution layer to generate a convolution added feature, and input the convolution added feature into a GeLU activation function layer to output a first nonlinear local feature;

[0049] convolve the first nonlinear local feature by using a 1x1 convolution layer to generate a convolution nonlinear local feature, and input the convolution nonlinear local feature into a Sigmoid activation function layer to output a second nonlinear local feature;

[0050] perform element-wise multiplication on the second nonlinear local feature and the reshaped attention output feature to generate a first initial interaction feature, and perform global average pooling on the reshaped attention output feature to generate a pooled global feature;

[0051] convolve the pooled global feature by using a 1x1 convolution layer to generate a convolution global feature, and input the convolution global feature into a GeLU activation function layer to output a first nonlinear global feature;

[0052] convolve the first nonlinear global feature by using a 1x1 convolution layer to generate a convolution nonlinear global feature, and input the convolution nonlinear global feature into a Sigmoid activation function layer to output a second nonlinear global feature;

[0053] perform element-wise multiplication on the second nonlinear global feature and the first added vector to generate a second initial interaction feature, and perform element-wise addition on the second initial interaction feature and the first initial interaction feature to generate a target interaction feature.

[0054] Optionally, the feature fusion of the plurality of target rain encoding features by the multi-scale space interaction feature fusion module outputs a target fusion feature, comprising:

[0055] The first rain encoding feature, the second rain encoding feature, and the third rain encoding feature are respectively preprocessed to output a first rain initial feature, a second rain initial feature, and a third rain initial feature;

[0056] The first rain initial feature, the second rain initial feature, and the third rain initial feature are sequentially subjected to 1x1 convolution operation, nonlinear mapping, and 3x3 deep over-parameterized convolution to generate a first rain intermediate feature, a second rain intermediate feature, and a third rain intermediate feature;

[0057] The first rain intermediate feature, the second rain intermediate feature, and the third rain intermediate feature are spliced to output a rain spliced feature, and the rain spliced feature is sequentially subjected to dilated convolution and nonlinear mapping to generate a rain mapping feature;

[0058] The rain mapping feature is subjected to depth-wise convolution to generate a rain depth convolution feature, and the rain depth convolution feature and the rain depth convolution feature are element-wise added to output a rain addition feature;

[0059] The rain addition feature is sequentially subjected to nonlinear mapping and compression convolution to generate a rain compression convolution feature, and the rain compression convolution feature is subjected to channel attention calculation and spatial attention calculation, respectively, to output a channel attention feature and a spatial attention feature;

[0060] The channel attention feature and the spatial attention feature are element-wise added to generate a rain enhancement feature, and the rain enhancement feature is respectively spliced with the first rain intermediate feature, the second rain intermediate feature, and the third rain intermediate feature to generate a first initial fusion feature, a second initial fusion feature, and a third initial fusion feature;

[0061] The first initial fusion feature, the second initial fusion feature, and the third initial fusion feature are respectively subjected to 3x3 deep over-parameterized convolution to generate a first key spatial map, a second key spatial map, and a third key spatial map;

[0062] The first key spatial map, the rain enhancement feature, and the first rain intermediate feature are weighted summed to generate a first spatial perception feature;

[0063] The second key spatial map, the rain enhancement feature, and the second rain intermediate feature are weighted summed to generate a second spatial perception feature;

[0064] perform weighted summation on the third key spatial map, the rain enhancement feature, and the third rain intermediate feature to generate a third spatial perception feature;

[0065] perform summation on the first spatial perception feature, the second spatial perception feature, and the third spatial perception feature to output a summation feature, and sequentially perform 1x1 convolution operation and nonlinear mapping on the summation feature to generate an intermediate fusion feature;

[0066] perform element-wise addition on the intermediate fusion feature, the first rain initial feature, the second rain initial feature, and the third rain initial feature to output a target fusion feature.

[0067] Optionally, the target fusion feature includes a first fusion feature and a second fusion feature; the decoder includes a collaborative feature enhancement module, a 3x3 deep hyper-parameterized convolution layer, and a 1x1 convolution layer; the target rain-removed image includes a first rain-removed image, a second rain-removed image, and a third rain-removed image; and the inputting of the rain image, the multi-scale rain image, the target fusion feature, and the target rain-encoding feature to the decoder for decoding to generate the target rain-removed image includes:

[0068] perform feature enhancement on the third rain-encoding feature by using the collaborative feature enhancement module to output a first initial rain-removed feature, perform upsampling on the first initial rain-removed feature to output an upsampled first initial rain-removed feature, and concatenate the upsampled first initial rain-removed feature and the second fusion feature to output a first concatenated rain-removed feature;

[0069] perform convolution operation on the first concatenated rain-removed feature by using the 1x1 convolution layer to generate a first convolution rain-removed feature, and perform feature enhancement on the first convolution rain-removed feature by using the collaborative feature enhancement module to output a second initial rain-removed feature;

[0070] perform upsampling on the second initial rain-removed feature to output an upsampled second initial rain-removed feature, concatenate the upsampled second initial rain-removed feature and the first fusion feature to generate a second concatenated rain-removed feature, and perform convolution operation on the second concatenated rain-removed feature by using the 1x1 convolution layer to generate a second convolution rain-removed feature;

[0071] perform feature enhancement on the second convolution rain-removed feature by using the collaborative feature enhancement module to output a third initial rain-removed feature, and input the third initial rain-removed feature to the 3x3 deep hyper-parameterized convolution layer to perform deep hyper-parameterized convolution to output a third initial rain-removed convolution feature;

[0072] perform element-wise addition on the third initial rain-removed convolution feature and the rain image to generate a first rain-removed image;

[0073] performing deep hyper-parameterized convolution on the first initial rain removal feature and the second initial rain removal feature respectively by using 3*3 deep hyper-parameterized convolution layers, and outputting a first intermediate rain removal feature and the second intermediate rain removal feature;

[0074] performing element-wise addition on the first intermediate rain removal feature and the first scale rain-bearing image, and generating a second rain removal image;

[0075] performing element-wise addition on the second intermediate rain removal feature and the second scale rain-bearing image, and generating a third rain removal image.

[0076] Optionally, the model training process of the preset multi-scale dual-domain collaborative rain removal network is specifically as follows:

[0077] obtaining a rain-bearing image for model training, and performing multi-scale down-sampling on the rain-bearing image for model training, and outputting a multi-scale rain-bearing image for model training.

[0078] inputting the rain-bearing image for model training and the multi-scale rain-bearing image for model training into an initial multi-scale dual-domain collaborative rain removal network as inputs, and outputting a rain removal image for model training;

[0079] substituting the rain removal image for model training into a preset loss function and deriving, and outputting a model gradient;

[0080] updating model parameters of the initial multi-scale dual-domain collaborative rain removal network by using the model gradient, determining an intermediate multi-scale dual-domain collaborative rain removal network, and counting an updating number in real time;

[0081] determining whether the updating number reaches a preset training number;

[0082] if yes, taking the intermediate multi-scale dual-domain collaborative rain removal network as the preset multi-scale dual-domain collaborative rain removal network trained.

[0083] The second aspect of the present application provides an image rain removal device, which comprises:

[0084] an acquisition module, configured to acquire a rain-bearing image, and perform multi-scale down-sampling on the rain-bearing image, and output a multi-scale rain-bearing image;

[0085] an input module, configured to input the rain-bearing image and the multi-scale rain-bearing image into a preset multi-scale dual-domain collaborative rain removal network, wherein the preset multi-scale dual-domain collaborative rain removal network comprises an encoder, a multi-scale spatial interaction feature fusion module and a decoder;

[0086] An encoding module is configured to encode the rainy image and the multi-scale rainy image by using the encoder to output a plurality of target rainy encoding features;

[0087] A fusion module is configured to fuse the plurality of target rainy encoding features by using the multi-scale spatial interaction feature fusion module to output a target fusion feature;

[0088] A decoding module is configured to input the rainy image, the multi-scale rainy image, the target fusion feature and the target rainy encoding feature into the decoder for decoding to generate a target rain-removed image.

[0089] From the above technical solutions, the present application has the following advantages:

[0090] The above technical solutions of the present application provide an image rain-removal method, which acquires a rainy image, performs multi-scale down-sampling on the rainy image to output a multi-scale rainy image, inputs the rainy image and the multi-scale rainy image into a preset multi-scale dual-domain collaborative rain-removal network, and performs encoding, feature fusion and decoding on the rainy image and the multi-scale rainy image by using the preset multi-scale dual-domain collaborative rain-removal network to generate a target rain-removed image. The preset multi-scale dual-domain collaborative rain-removal network includes an encoder, a multi-scale spatial interaction feature fusion module and a decoder. The encoder is used to encode the rainy image and the multi-scale rainy image to output a plurality of target rainy encoding features. The multi-scale spatial interaction feature fusion module is used to fuse the plurality of target rainy encoding features to output a target fusion feature. The decoder is used to decode the rainy image, the multi-scale rainy image, the target fusion feature and the target rainy encoding feature to generate the target rain-removed image. Based on the above scheme, the multi-scale spatial interaction feature fusion module is used to represent the key region features at different scales, which can guide the decoder to better distinguish rain lines and background textures, thereby improving the image rain-removal effect. BRIEF DESCRIPTION OF DRAWINGS

[0091] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0092] Figure 1 A step flow chart of an image rain-removal method provided by the first embodiment of the present application;

[0093] Figure 2 A structure schematic diagram of a preset multi-scale dual-domain collaborative rain-removal network provided by the first embodiment of the present application;

[0094] Figure 3 A structure schematic diagram of a frequency focusing enhancement convolution block provided by the first embodiment of the present application;

[0095] Figure 4 A structural schematic diagram of a double-flow sparse interaction block based on a self-attention mechanism is provided for the first embodiment of the present application.

[0096] Figure 5 A structural schematic diagram of an adaptive interaction module is provided for the first embodiment of the present application.

[0097] Figure 6 A structural schematic diagram of a multi-scale spatial interaction feature fusion module is provided for the first embodiment of the present application.

[0098] Figure 7 A step flowchart of a training process of a preset multi-scale double-domain collaborative rain removal network is provided for the second embodiment of the present application.

[0099] Figure 8 A Rain200H data set visualization result is provided for the second embodiment of the present application.

[0100] Figure 9 A structural block diagram of an image rain removal device is provided for the third embodiment of the present application. DETAILED DESCRIPTION

[0101] The embodiments of the present application provide an image rain removal method and device, and aim to solve the technical problem that the existing rain removal technology is difficult to accurately balance rain line removal and background detail recovery, resulting in poor image rain removal effect.

[0102] In order to make the purposes, characteristics and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the drawings in the embodiments of the present application. Obviously, the following described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0103] Please refer to Figure 1 , Figure 1 A step flowchart of an image rain removal method is provided for the first embodiment of the present application.

[0104] The image rain removal method provided by the present application comprises:

[0105] In step 101, a rainy image is obtained, and the rainy image is down-sampled at multiple scales to output a multi-scale rainy image.

[0106] It should be noted that the input rainy image (rainy image) is down-sampled to half and one fourth of the resolution, respectively, and is recorded as a first scale rainy image I2 and a second scale rainy image I3.

[0107] Step 102, input the rainy image and the multi-scale rainy image into a preset multi-scale dual-domain collaborative deraining network, the preset multi-scale dual-domain collaborative deraining network comprising an encoder, a multi-scale spatial interaction feature fusion module and a decoder.

[0108] It should be noted that, please refer to Figure 2 The preset multi-scale dual-domain collaborative deraining network (MDCDNet) mainly comprises an encoder, a multi-scale spatial interaction feature fusion module and a decoder, which adopts a 3-level U-shaped structure marked as , which is specially designed for image deraining.

[0109] Further, for the input rainy image (the rainy image), the MDCDNet first extracts shallow features through a 3x3 depth over-parameterized convolution layer (DO-Conv), wherein C represents the number of channels, and the spatial resolution. These shallow features are then processed by a 3-level encoder to obtain deep features (i.e., multiple target rainy encoding features). Each encoder and decoder level is composed of a collaborative feature enhancement module (CFEM). As the features pass through the encoder, the spatial resolution gradually decreases, while the channel dimension increases. In order to effectively utilize multi-scale information, the MDCDNet adopts a multi-scale input-output strategy. Specifically, the input rainy image is down-sampled to half and quarter resolution, denoted as I2 and I3. These down-sampled images are processed using existing extraction modules for feature extraction and existing fusion modules for feature fusion. The processed features are re-fused into the main processing stream to enhance the multi-scale context-aware ability of rain removal. Then, the encoded deep features are sent to a three-level decoder to gradually reconstruct high-resolution features. In order to further guide the decoder, the present application introduces a multi-scale spatial interaction feature fusion module (MSIFFM), which fuses the features of the multi-level encoder to enrich the decoding process and improve the accuracy of recovery. The output features of the second and third level decoders are refined by 3x3 DO-Conv and added to the down-sampled images of the corresponding resolution to recover low-resolution images, which are further up-sampled to the original resolution.Figure 2 The simplicity of the encoder is not shown. Finally, the first-level decoder recovers the full-resolution features, which are further processed by a 3x3 DO-Conv and combined with the original input image to generate the final high-quality rain removal output, i.e., the target rain-removed image.

[0110] Step 103, encoding the rain-bearing image and the multi-scale rain-bearing image by using the encoder to output a plurality of target rain-bearing encoded features.

[0111] The plurality of target rain-bearing encoded features includes a first rain-bearing encoded feature, a second rain-bearing encoded feature, and a third rain-bearing encoded feature.

[0112] It should be noted that the encoder is mainly composed of three collaborative feature enhancement module stacks, a 3x3 deep hyper-parameterized convolution layer, two extraction modules, and two fusion modules. The extraction module is used for feature extraction of the input image, and the fusion module is used for feature fusion of the input features. The processing principles of the two processing processes can refer to the existing extraction module and fusion module, and the present application will not be described in more detail.

[0113] Further, step 103 can include the following sub-steps S31-S310:

[0114] Step S31, performing deep hyper-parameterized convolution on the rain-bearing image by using the 3x3 deep hyper-parameterized convolution layer to output a deep convolution feature;

[0115] Step S32, performing feature enhancement on the deep convolution feature by using the collaborative feature enhancement module to output a first rain-bearing encoded feature;

[0116] Step S33, performing feature extraction on the first scale rain-bearing image to output a first scale rain-bearing image feature;

[0117] Step S34, downsampling the first rain-bearing encoded feature to output a first rain-bearing encoded feature after downsampling;

[0118] Step S35, performing feature fusion on the first scale rain-bearing image feature and the first rain-bearing encoded feature after downsampling to generate a first fusion feature;

[0119] Step S36, performing feature enhancement on the first fusion feature by using the collaborative feature enhancement module to output a second rain-bearing encoded feature;

[0120] Step S37, performing feature extraction on the second scale rain-bearing image to output a second scale rain-bearing image feature;

[0121] Step S38, downsampling the second rain-bearing encoded feature to output a second rain-bearing encoded feature after downsampling;

[0122] Step S39, the second scale rain image feature and the second rain encoding feature after down-sampling are fused to generate a second fusion feature;

[0123] Step S310, the second fusion feature is enhanced by using a collaborative feature enhancement module to generate a third rain encoding feature.

[0124] It should be noted that the collaborative feature enhancement module is composed of a frequency-focused enhanced convolution block (FFECB) and a dual-stream sparse interactive block based on a self-attention mechanism (i.e., a dual-stream sparse interactive transformer block (DSITB)). The frequency-focused enhanced convolution block is mainly composed of a 3x3 deep hyper-parameterized convolution layer, a ReLU activation function layer, a 1x1 convolution layer, and a Sigmoid activation function layer. The dual-stream sparse interactive block based on the self-attention mechanism is mainly composed of a 1x1 convolution layer, a Softmax activation function layer, an adaptive interaction module, and a gated deep convolution feedforward network.

[0125] Optionally, the feature processing step of the collaborative feature enhancement module can include the following sub-steps S1-S2:

[0126] Step S1, the input feature input into the collaborative feature enhancement module is focused and enhanced by the frequency-focused enhanced convolution block to generate a focused enhanced feature.

[0127] It should be noted that the input feature is a feature map input into the collaborative feature enhancement module. It can be understood that the input feature can correspond to any feature map input into the collaborative feature enhancement module for image processing in the process of training or detecting the preset multi-scale dual-domain collaborative rain removal network.

[0128] Further, step S1 can include the following sub-steps:

[0129] The input feature is deep hyper-parameterized convolved by the 3x3 deep hyper-parameterized convolution layer to output an intermediate deep convolution feature, and the intermediate deep convolution feature is input into the ReLU activation function layer to output a first nonlinear feature.

[0130] The first nonlinear feature is deep hyper-parameterized convolved by the 3x3 deep hyper-parameterized convolution layer to generate a local feature, and the local feature is Fourier transformed to output a first real part feature and a first imaginary part feature.

[0131] The first real part feature and the first imaginary part feature are spliced to output a first spliced feature, and a 1x1 convolution layer is used to perform convolution operation on the first spliced feature to output a first convolution feature;

[0132] The first convolution feature is taken as an input of a ReLU activation function layer to output a second nonlinear feature, and a 1x1 convolution layer is used to perform convolution operation on the second nonlinear feature to generate a second convolution feature;

[0133] The second convolution feature is divided into a second real part feature and a second imaginary part feature along a channel, and inverse Fourier transform is performed on the second real part feature and the second imaginary part feature to output a high-frequency feature;

[0134] Global average pooling is performed on the local feature to generate a pooled feature, and a 1x1 convolution layer is used to perform convolution operation on the pooled feature to output a third convolution feature;

[0135] The third convolution feature is taken as an input of a ReLU activation function layer to output a third nonlinear feature, and a 1x1 convolution layer is used to perform convolution operation on the third nonlinear feature to generate a fourth convolution feature;

[0136] A Sigmoid activation function layer is used to perform nonlinear mapping on the fourth convolution feature to output a mapped feature, and the mapped feature and the local feature are multiplied element by element to output a multiplied feature;

[0137] Fast Fourier transform convolution is performed on the multiplied feature to generate a low-frequency feature, and the low-frequency feature, the high-frequency feature and the input feature are added element by element to generate a focus-enhanced feature.

[0138] It should be noted that, please refer to Figure 3 , first, the input feature is given, and the FFECB first extracts local features through two consecutive 3x3 deep hyperparameterized convolutions with ReLU activation functions as intermediate layers.

[0139] ;

[0140] wherein, represents a 3x3 deep hyperparameterized convolution (i.e., a 3x3 deep hyperparameterized convolution layer); represents a ReLU activation function layer. The local feature is then processed through two parallel frequency modeling branches. Fast Fourier transform convolution (FFT-Conv, Fast Fourier Transform Convolution) is used to model high-frequency and low-frequency features.

[0141] In the high-frequency branch, the present application uses FFT-Conv to explore frequency features. Specifically, the local feature Firstly, the real part F R and the imaginary part F I , i.e., the first real part feature and the first imaginary part feature, are converted by Fast Fourier Transform (FFT) to obtain the frequency domain. Then, the two parts are spliced along the channel dimension. Next, the features of specific frequencies (i.e., the second convolutional features) are extracted by two consecutive 1x1 convolutional layers with ReLU activation functions in between. Finally, the output features are re-separated into real and imaginary parts (i.e., the second real part features and the second imaginary part features ), and converted back to the spatial domain using Inverse Fast Fourier Transform (IFFT). The calculation process of the high-frequency features can be represented as:

[0142] ;

[0143] wherein, and represent Fast Fourier Transform (FFT) and Inverse Fast Fourier Transform (IFFT), respectively; represents a 1x1 convolutional layer.

[0144] Further, in the low-frequency branch, channel attention is first applied to suppress high-frequency components and retain low-frequency information, and then the features are further optimized by FFT-Conv.

[0145] Finally, the final output of the FFECB module, i.e., the focus-enhanced features is obtained by adding the high-frequency branch output and the low-frequency branch output (low-frequency features) , and performing residual connection with the original input features . The calculation process of the FFECB module is as follows:

[0146] ;

[0147] wherein, represents channel attention, represents Fast Fourier Transform convolution.

[0148] It is worth mentioning that each feature map generated in the process of processing the feature map by the frequency focus-enhanced convolutional block (i.e., the first nonlinear feature, the first real part feature, and the first imaginary part feature, etc.) is an intermediate map generated in the collaborative feature enhancement module.

[0149] Step S2, taking the focus-enhanced features as the input of the dual-flow sparse interaction block based on the self-attention mechanism to generate output features.

[0150] It should be noted that the output feature is the feature map output by the cooperative feature enhancement module, which can be understood as any output feature map after the cooperative feature enhancement module processes the image during the training or detection of the preset multi-scale dual-domain cooperative rain removal network.

[0151] Further, step S2 can include the following sub-steps:

[0152] Layer normalization is performed on the focus enhanced feature to generate a first normalized feature;

[0153] An initial query feature, an initial key feature, and an initial value feature are generated from the first normalized feature using a 1x1 convolution layer, and group convolution is performed on the initial query feature to generate a query tensor;

[0154] Depth-wise convolution is performed on the initial key feature and the initial value feature respectively to generate a key tensor and a value tensor, and the query tensor, the key tensor, and the value tensor are reshaped respectively to output a reshaped query tensor, a reshaped key tensor, and a reshaped value tensor;

[0155] Matrix multiplication is performed on the reshaped query tensor and the reshaped key tensor to generate a multiplication feature, and the multiplication feature is input into a Softmax activation function layer to output a sparse attention map;

[0156] Matrix multiplication is performed on the sparse attention map and the reshaped value tensor to generate an attention output feature, and the attention output feature is reshaped to generate a reshaped attention output feature;

[0157] A 1x1 convolution layer is used to perform convolution operation on the value tensor to output a convolution value tensor, and depth-wise convolution is performed on the convolution value tensor to generate a depth-wise convolution value tensor;

[0158] The depth-wise convolution value tensor and the convolution value tensor are element-wise added to output a first addition vector, and the first addition vector and the reshaped attention output feature are input into an adaptive interaction module for feature interaction to output a target interaction feature;

[0159] A 1x1 convolution layer is used to perform convolution operation on the target interaction feature to output a convolution interaction feature, and the convolution interaction feature and the focus enhanced feature are element-wise added to output a second addition feature;

[0160] Layer normalization is performed on the second addition feature to output a second normalized feature, and a gated depth-wise convolution feedforward network is used to generate a feedforward feature from the second normalized feature;

[0161] The feedforward feature and the second addition feature are element-wise added to generate an output feature.

[0162] It should be noted that please refer toFigure 4 The dual-stream sparse interaction block based on the self-attention mechanism effectively reduces channel redundancy by forcing sparsity in attention calculation, and realizes adaptive fusion of global attention features and local information compensation features. For the input in the (l-1) block, that is, the input of the dual-stream sparse interaction block based on the self-attention mechanism , which corresponds to the focus enhancement feature, the DSITB processing procedure is designed as:

[0163] ;

[0164] wherein, represents a dual-stream sparse interactive self-attention mechanism (DSISA); is a gated-Dconv feed-forward network (GDFN); denotes layer normalization; is the second added feature; is the output feature.

[0165] Further, the DSISA realizes a sparse attention mechanism by performing self-attention calculation through a small number of information channels. Specifically, for the layer-normalized output, that is, the first normalized feature , the present application constructs a query tensor , a key tensor , and a value tensor . Specifically, and branches reduce the channel dimension to , (0, 1] through 1×1 convolution, and then perform spatial encoding through 3×3 depthwise convolution. In contrast, the query branch uses 1×1 convolution and then performs 3×3 group convolution (GConv), while maintaining the complete channel resolution. The calculation process is as follows:

[0166] ;

[0167] wherein, denotes group convolution, and the number of groups is set to . denotes 3×3 depthwise convolution.

[0168] Next, and are rearranged (i.e., reshaped) so that their dot product interaction generates a sparse attention map The attention output feature is calculated as follows:

[0169] ;

[0170] wherein, , and the reshaped versions of query tensor key tensor and value tensor ; is a learnable scaling parameter to stabilize the Softmax operation. The final result is reshaped into the reshaped attention output feature

[0171] . Like the standard multi-head self-attention mechanism, the application divides the channels into multiple heads to learn the attention map in parallel.

[0172] ;

[0173] wherein, is a convolution value tensor; is a first addition vector.

[0174] Further, the obtained first addition vector and the reshaped attention output feature are input into an adaptive interaction module for feature interaction, and an output target interaction feature is output; wherein the adaptive interaction module mainly consists of a 1×1 convolution layer, a GeLU activation function layer, and a Sigmoid activation function layer, and the process of outputting the target interaction feature by the adaptive interaction module is as follows:

[0175] The 1×1 convolution layer is used to perform convolution operation on the first addition vector to generate a convolution addition feature, and the convolution addition feature is taken as the input of the GeLU activation function layer to output a first nonlinear local feature;

[0176] The 1×1 convolution layer is used to perform convolution operation on the first nonlinear local feature to generate a convolution nonlinear local feature, and the convolution nonlinear local feature is taken as the input of the Sigmoid activation function layer to output a second nonlinear local feature;

[0177] The second nonlinear local feature and the reshaped attention output feature are multiplied element by element to generate a first initial interaction feature, and the reshaped attention output feature is globally averaged to generate a pooled global feature;

[0178] The 1x1 convolution layer is adopted to perform convolution operation on the pooled global feature to generate a convolution global feature, and the convolution global feature is taken as an input of a GeLU activation function layer to output a first nonlinear global feature;

[0179] The 1x1 convolution layer is adopted to perform convolution operation on the first nonlinear global feature to generate a convolution nonlinear global feature, and the convolution nonlinear global feature is taken as an input of a Sigmoid activation function layer to output a second nonlinear global feature;

[0180] The second nonlinear global feature and the first addition vector are element-wise multiplied to generate a second initial interaction feature, and the second initial interaction feature and the first initial interaction feature are element-wise added to generate a target interaction feature.

[0181] It should be noted that, in order to integrate global and local features, the present application adopts an adaptive interaction module (AIM), as shown in Figure 5 The AIM enhances the representation ability by adaptively weighting and fusing global and local features, which is described as follows:

[0182] ;

[0183] wherein, and represent a channel attention map (channel attention feature) and a spatial attention map (spatial attention feature), respectively; represents a global average pooling; and represent a GeLU activation function and a Sigmoid activation function, respectively; represents an element-wise multiplication; is the second nonlinear global feature; is the second nonlinear local feature; is a reshaped attention output feature, corresponding to the global feature; is the first addition vector, corresponding to the local feature.

[0184] Further, the 1x1 convolution layer is adopted to perform convolution operation on the target interaction feature to output a convolution interaction feature, and the convolution interaction feature and the focus enhancement feature are element-wise added to output a second addition feature; the second addition feature is subjected to layer normalization to output a second normalized feature, and a gated depth-wise convolution feedforward network is adopted to generate a feedforward feature based on the second normalized feature; the feedforward feature and the second addition feature are element-wise added to generate an output feature. The process of outputting the feedforward feature based on the gated depth-wise convolution feedforward network can refer to the processing process of the existing gated depth-wise convolution feedforward network, and the present application will not be described in more detail.

[0185] It is worth mentioning that each feature map (i.e., the first normalized feature, the initial query feature, the initial key feature, the initial value feature, etc.) generated in the process of processing the feature map by the dual-flow sparse interaction block based on the self-attention mechanism is an intermediate map generated in the collaborative feature enhancement module.

[0186] In step 104, the multi-scale spatial interaction feature fusion module is used to perform feature fusion on the plurality of target rain encoding features, and target fusion features are output.

[0187] It should be noted that the feature fusion performed by the multi-scale spatial interaction feature fusion module can strengthen the association of the rain line pattern and the background details in each scale feature, retain the rain line edge details of the small-scale feature, integrate the background global information of the large-scale feature, and finally output target fusion features with multi-scale complementarity and spatial correlation, thereby laying a foundation for accurately distinguishing rain lines and backgrounds in the subsequent decoding stage.

[0188] Further, step 104 can include the following sub-steps S41-S412:

[0189] In step S41, the first rain encoding feature, the second rain encoding feature, and the third rain encoding feature are respectively preprocessed, and first rain initial features, second rain initial features, and third rain initial features are output.

[0190] In step S42, 1x1 convolution operation, non-linear mapping, and 3x3 deep over-parameterized convolution are sequentially performed on the first rain initial feature, the second rain initial feature, and the third rain initial feature, respectively, to generate first rain intermediate features, second rain intermediate features, and third rain intermediate features.

[0191] In step S43, the first rain intermediate feature, the second rain intermediate feature, and the third rain intermediate feature are spliced to output a rain splicing feature, and dilated convolution and non-linear mapping are sequentially performed on the rain splicing feature to generate a rain mapping feature.

[0192] In step S44, depth-wise convolution is performed on the rain mapping feature to generate a rain depth convolution feature, and the rain depth convolution feature is added element by element to output a rain addition feature.

[0193] In step S45, non-linear mapping and compression convolution are sequentially performed on the rain addition feature to generate a rain compression convolution feature, and channel attention calculation and spatial attention calculation are respectively performed on the rain compression convolution feature to output channel attention features and spatial attention features.

[0194] Step S46 element-wise adds the channel attention feature and the spatial attention feature to generate a rain enhancement feature, and splices the rain enhancement feature with the first rain intermediate feature, the second rain intermediate feature, and the third rain intermediate feature respectively to generate a first initial fusion feature, a second initial fusion feature, and a third initial fusion feature;

[0195] Step S47, 3x3 depth over-parameterized convolution is respectively performed on the first initial fusion feature, the second initial fusion feature, and the third initial fusion feature to generate a first key spatial map, a second key spatial map, and a third key spatial map;

[0196] Step S48, weighted summation is performed on the first key spatial map, the rain enhancement feature, and the first rain intermediate feature to generate a first spatial perception feature;

[0197] Step S49, weighted summation is performed on the second key spatial map, the rain enhancement feature, and the second rain intermediate feature to generate a second spatial perception feature;

[0198] Step S410, weighted summation is performed on the third key spatial map, the rain enhancement feature, and the third rain intermediate feature to generate a third spatial perception feature;

[0199] Step S411, summation is performed on the first spatial perception feature, the second spatial perception feature, and the third spatial perception feature to output a summation feature, and 1x1 convolution operation and nonlinear mapping are sequentially performed on the summation feature to generate an intermediate fusion feature;

[0200] Step S412, element-wise addition is performed on the intermediate fusion feature, the first rain initial feature, the second rain initial feature, and the third rain initial feature to output a target fusion feature.

[0201] The target fusion feature includes the first fusion feature and the second fusion feature.

[0202] It should be noted that, please refer to Figure 6 In order to make the spatial dimensions of the fusion consistent, the output first rain initial feature , the second rain initial feature , and the third rain initial feature The first, second and third rain-belt encoded features are pre-processed by upsampling or downsampling to match the resolution of the decoder, and first, second and third rain-belt initial features are output, wherein the first rain-belt initial feature comprises the original first rain-belt encoded feature and the first rain-belt encoded feature after one time of downsampling; the second rain-belt initial feature comprises the second rain-belt encoded feature after one time of upsampling and the original second rain-belt encoded feature; and the third rain-belt initial feature comprises the third rain-belt encoded feature after two times of upsampling and the third rain-belt encoded feature after one time of upsampling. The multi-scale spatial interaction feature fusion module has two groups of inputs, the first group of inputs being the original first rain-belt encoded feature, the second rain-belt encoded feature after one time of upsampling and the third rain-belt encoded feature after two times of upsampling, and the first fusion feature being output according to the above-described step process; and the second group of inputs being the first rain-belt encoded feature after one time of downsampling, the original second rain-belt encoded feature and the third rain-belt encoded feature after one time of upsampling, and the second fusion feature being output according to the above-described step process.

[0203] It is worth mentioning that when the input of the multi-scale spatial interaction feature fusion module is the first group of inputs, the first fusion feature is obtained by element-wise addition of the first group of inputs and the intermediate fusion feature corresponding to the first group of inputs after the intermediate fusion feature corresponding to the first group of inputs is obtained according to the above-described step process; and when the input of the multi-scale spatial interaction feature fusion module is the second group of inputs, the second fusion feature is obtained by element-wise addition of the second group of inputs and the intermediate fusion feature corresponding to the second group of inputs after the intermediate fusion feature corresponding to the second group of inputs is obtained according to the above-described step process.

[0204] Further, in the initial feature fusion and enhancement stage, each input is subjected to a series of convolution operations and nonlinear activation. Subsequently, the enhanced features of all scales (i.e. the first, second and third rain-belt intermediate features) are concatenated and fused through MobiVari2 variants. Finally, the features are enhanced through a simple channel attention mechanism and a spatial attention mechanism, respectively, so as to obtain the output of the first stage, i.e. the rain-belt enhanced features . The calculation process of stage 1 can be represented as:

[0205] ;

[0206] wherein, and represent 1×1 expansion group convolution and 3×3 depth hyper-parameterized depth-wise convolution with an expansion rate of 1.2, respectively; and These represent channel attention mechanism and spatial attention mechanism, respectively. These are intermediate features of the rainy zone, including the first intermediate feature of the rainy zone, the second intermediate feature of the rainy zone, and the third intermediate feature of the rainy zone. ; The target rainy coding features include the first rainy coding feature, the second rainy coding feature, and the third rainy coding feature; This is a rain-mapping feature; This is a characteristic of the first rain belt; This is a characteristic of the second rain belt; This is a characteristic of the middle of the third rain belt; This is a rain-compressed convolution feature; This is a feature enhanced by rain.

[0207] Furthermore, to further refine key spatial details at different scales, this invention performs spatial interaction operations. Specifically, and features at each scale The features are concatenated and input into a 3×3 DO-Conv operation, whereby the rain-enhanced features are concatenated with the first, second, and third rain-intermediate features to generate the first, second, and third initial fused features. Then, the first, second, and third initial fused features are subjected to 3×3 depthwise overparameterized convolutions to generate the first, second, and third key space maps.

[0208] Furthermore, the specific process of outputting spatial perception features (including the first spatial perception feature, the second spatial perception feature, and the third spatial perception feature) is as follows: the output key spatial map S (including the first key spatial map, the second key spatial map, and the third key spatial map) is divided into two spatial maps. and Spatial perception features are calculated by weighted summation. Finally, the output of all levels. After summation and refinement using 1×1 convolution and nonlinear functions, the target fusion features are obtained by connecting them to the residuals of the original input. This ensures the stability of the gradient flow. The calculation process in the second stage is as follows:

[0209] ;

[0210] Step 105: Input the rainy image, multi-scale rainy image, target fusion features and target rainy coding features into the decoder for decoding to generate the target de-rained image.

[0211] The decoder comprises a cooperative feature enhancement module, a 3x3 deep hyper-parameterized convolution layer, and a 1x1 convolution layer.

[0212] The target rain-removed image comprises a first rain-removed image, a second rain-removed image, and a third rain-removed image.

[0213] It should be noted that when the target fusion feature and the target rain-bearing encoding feature are input into the decoder for decoding, the decoder performs hierarchical fusion on local details (such as small-scale rain line edges and background textures) in the third rain-bearing encoding feature in the target rain-bearing encoding feature and global correlation information (such as large-scale rain curtain distribution and scene structure) in the target fusion feature through the skip connection; at the same time, the rain line feature weight is dynamically inhibited and the background feature expression is strengthened by means of the attention mechanism, the rain line interference is accurately distinguished and stripped, the occluded scene details are completely restored, and finally the target rain-removed image with complete rain line removal and faithful background information is generated.

[0214] Further, the step 105 can comprise the following sub-steps S51-S58:

[0215] In step S51, the third rain-bearing encoding feature is subjected to feature enhancement by using the cooperative feature enhancement module, to output a first initial rain-removed feature; the first initial rain-removed feature is subjected to up-sampling, to output the first initial rain-removed feature after up-sampling, and the first initial rain-removed feature after up-sampling and the second fusion feature are spliced to output a first spliced rain-removed feature;

[0216] In step S52, the first spliced rain-removed feature is subjected to convolution operation by using the 1x1 convolution layer, to generate a first convolution rain-removed feature, and the first convolution rain-removed feature is subjected to feature enhancement by using the cooperative feature enhancement module, to output a second initial rain-removed feature;

[0217] In step S53, the second initial rain-removed feature is subjected to up-sampling, to output the second initial rain-removed feature after up-sampling, the second initial rain-removed feature after up-sampling and the first fusion feature are spliced to generate a second spliced rain-removed feature, and the second spliced rain-removed feature is subjected to convolution operation by using the 1x1 convolution layer, to generate a second convolution rain-removed feature;

[0218] In step S54, the second convolution rain-removed feature is subjected to feature enhancement by using the cooperative feature enhancement module, to output a third initial rain-removed feature, and the third initial rain-removed feature is input into the 3x3 deep hyper-parameterized convolution layer for deep hyper-parameterized convolution, to output a third initial rain-removed convolution feature;

[0219] In step S55, the third initial rain-removed convolution feature and the rain-bearing image are subjected to element-by-element addition, to generate a first rain-removed image;

[0220] Step S56, a 3x3 deep hyper-parameterized convolution layer is used to perform deep hyper-parameterized convolution on the first initial rain removal feature and the second initial rain removal feature respectively, to output a first intermediate rain removal feature and a second intermediate rain removal feature;

[0221] Step S57, the first intermediate rain removal feature and the first scale rain-bearing image are added element by element to generate a second rain removal image;

[0222] Step S58, the second intermediate rain removal feature and the second scale rain-bearing image are added element by element to generate a third rain removal image.

[0223] It should be noted that the full resolution feature is recovered by the decoder, the obtained feature is further processed by 3x3 DO-Conv, and combined with the original input image (rain-bearing image) to generate the final high-quality rain removal output, i.e. the first rain removal image, which is taken as the final rain removal result.

[0224] As a comparison of technical effects, reference can be made in combination with the prior art. Low-quality images seriously affect the performance of many computer vision applications. In the task of low-quality image restoration, image rain removal is a particularly challenging basic vision task because the real background and rain line pattern distribution are both unknown and difficult to predict. Therefore, the research on image rain removal under the condition of heavy rain is imminent.

[0225] The background of image rain removal technology can be divided into prior-based methods and deep learning-based methods. Prior-based methods mainly rely on manual setting based on rain line and background scene statistical features. However, the effectiveness of these methods is limited, especially in complex degradation conditions such as heavy rain or rain fog mixing, resulting in poor robustness and generalization ability in practical applications. With the rise of deep learning, learning-based methods have made significant progress by designing various convolutional neural networks (CNNs). Due to the locality of convolution operation and its translational invariance, CNNs are usually difficult to capture global dependencies. This limitation seriously affects their modeling ability for complex rain scene patterns, such as wide rain bands or rain fog interlacing. Recently, due to the rapid development of deep learning technology and the improvement of hardware, Transformer-based rain removal methods have shown excellent performance in image rain removal by utilizing their global attention mechanism. However, single-dimensional Transformer rain removal methods are often subject to different limitations, so the rain removal performance cannot be optimized. Specifically, spatial dimension Transformer is limited by high computational cost or local receptive field, while channel dimension Transformer lacks excellent spatial mapping and the ability to emphasize useful information.

[0226] For the Restormer deraining scheme, it starts from the channel dimension and does not consider the weak spatial mapping ability and attention generalization problem of the Transformer modeling in the channel dimension. In addition, it does not consider the importance of the multi-scale features of the encoder to the learning of the decoder.

[0227] For the DRSformer deraining scheme, it considers the redundancy problem of self-attention calculation from the channel dimension, overcomes the attention generalization problem of the channel dimension Transformer, and fully utilizes the most useful features. However, this scheme still does not consider the weak spatial mapping ability of the channel dimension Transformer, which limits the deraining performance of the system. In addition, it also does not consider the importance of the multi-scale features of the encoder to the learning of the decoder.

[0228] For the MSDT deraining scheme, it overcomes the limitation of single-dimensional Transformer global feature integration to some extent by modeling in the frequency domain and the spatial domain. However, it does not realize that it is difficult to achieve effective integration of high and low frequency features when using the same branch to model high and low frequency features. In addition, the modeling of the Transformer in the spatial domain is similar to the DRSformer, lacking sufficient innovation.

[0229] Based on the above, the present application focuses on the limitations of single-dimensional Transformer deraining methods. Starting from the frequency domain and the spatial domain, a pre-set multi-scale dual-domain collaborative deraining network (Multi-scale dual-domain collaborative deraining network, MDCDNet) is proposed to further improve the image deraining performance, so as to meet the needs of various computer vision tasks and be applied in the field of computer vision under bad weather conditions. Specifically, first, the frequency domain operation is used to process high and low frequency features, providing efficient and global spatial representation for subsequent spatial domain modeling and enhancing the discriminability of the features. Subsequently, the Transformer modeling in the spatial domain generates a sparse attention map by a channel compression method to focus on important features, and a parallel local compensation branch is designed. Finally, the two branches are adaptively fused to extract high-quality channel dimension features. In addition, in order to effectively fuse the multi-scale encoder information to promote the effective learning of the decoder, a fusion module is designed. In general, the present application can remove rain streaks while preserving clear textures and details, generating high-quality derained images.

[0230] In summary, in view of the limitations of single-dimensional Transformer, the application fuses frequency domain and spatial domain modeling techniques. The core architecture is composed of the proposed collaborative feature enhancement module (CFEM), which integrates the frequency-focused enhanced convolution block (FFECB) and the dual-stream sparse interactive transformer block (DSITB). At the same time, FFECB and DSITB aim to take advantage of the field and work collaboratively. FFECB enhances feature discriminability and global spatial representation by processing high and low frequency features in the frequency domain, while DSITB constructs high-quality channel dimension features by using channel compression to sparsify the attention map, dual-branch compensation, and adaptive fusion technology in the spatial domain. In addition, for effective fusion of multi-scale encoder features, the application proposes a multi-scale spatial interaction feature fusion module (MSIFFM) to represent key region features at different scales and guide the decoder to better distinguish rain lines and background textures.

[0231] Compared with existing single-dimensional Transformer rain removal methods, the frequency-space domain dual-domain collaborative rain removal method proposed by the application is more effective in rain removal performance. Specifically, the frequency domain provides rich global spatial representation, and then the channel dimension Transformer method can effectively integrate features and enrich multi-dimensional feature representation. The proposed FFECB can improve feature discriminability and rich global spatial representation by processing high and low frequency features in the frequency domain. At the same time, the DSITB proposed by the application can force attention calculation to be sparse in the global branch by using a novel channel compression method in the spatial domain, effectively reducing channel redundancy, effectively integrating channel information, and realizing adaptive fusion of global attention features and local information compensation features. In addition, the MSIFFM proposed by the application can effectively enhance key region representation at different scales through progressive feature fusion enhancement and spatial interaction mechanism, effectively fuse deep semantic information and shallow texture information in the encoder, and guide the decoder to better distinguish rain lines and background textures.

[0232] In the embodiment of the present application, the present application provides a method for removing rain from an image, obtaining a rain-bearing image, and performing multi-scale down-sampling on the rain-bearing image to output a multi-scale rain-bearing image; inputting the rain-bearing image and the multi-scale rain-bearing image into a preset multi-scale dual-domain collaborative rain removal network, the preset multi-scale dual-domain collaborative rain removal network comprising an encoder, a multi-scale spatial interaction feature fusion module and a decoder; using the encoder to encode the rain-bearing image and the multi-scale rain-bearing image to output a plurality of target rain-bearing encoded features; using the multi-scale spatial interaction feature fusion module to fuse the plurality of target rain-bearing encoded features to output a target fused feature; and inputting the rain-bearing image, the multi-scale rain-bearing image, the target fused feature and the target rain-bearing encoded features into the decoder for decoding to generate a target rain-removed image.

[0233] For better illustration, refer to Figure 7 , a step flowchart of a training process of the preset multi-scale dual-domain collaborative rain removal network provided by the second embodiment of the present application is shown, which can include the following steps:

[0234] Step 701, obtaining a rain-bearing image for model training, and performing multi-scale down-sampling on the rain-bearing image for model training to output a multi-scale rain-bearing image for model training.

[0235] Step 702, inputting the rain-bearing image for model training and the multi-scale rain-bearing image for model training as inputs of an initial multi-scale dual-domain collaborative rain removal network to output a rain-removed image for model training.

[0236] Step 703, substituting the rain-removed image for model training into a preset loss function and deriving to output a model gradient.

[0237] Step 704, updating model parameters of the initial multi-scale dual-domain collaborative rain removal network using the model gradient to determine an intermediate multi-scale dual-domain collaborative rain removal network, and counting the number of updates in real time.

[0238] Step 705, determining whether the number of updates reaches a preset number of training.

[0239] Step 706, if yes, taking the intermediate multi-scale dual-domain collaborative rain removal network as the preset multi-scale dual-domain collaborative rain removal network trained.

[0240] It should be noted that the preset loss function is specifically:

[0241] ;

[0242] wherein, is a loss value of the preset loss function; , , are scalar weights, empirically set as 1, 0.05 and 0.01; are multi-scale Charbonnier losses, , are penalty coefficients, are set as 10 -3 , are reconstructed images at the s scale (i.e. rain-removed images used for model training), are real images (GT, Ground Truth) at the s scale; are multi-scale edge losses, , are Laplacian operators; are multi-scale frequency losses, .

[0243] It is worth mentioning that when the number of updates does not reach the preset number of training times, the intermediate multi-scale dual-domain collaborative rain removal network is taken as a new initial multi-scale dual-domain collaborative rain removal network, and step 702 is jumped to execute until the number of updates reaches the preset number of training times, and the intermediate multi-scale dual-domain collaborative rain removal network determined when the number of updates reaches the preset number of training times is taken as the trained preset multi-scale dual-domain collaborative rain removal network.

[0244] Further, in the experiment, the comparison results of the Rain200H dataset are as shown in Figure 8 . Compared with the rain-removed pictures generated by other methods, the image quality generated by the method of the present application is optimal. Each method has achieved certain results in generating rain-removed images, and can achieve rain removal and repair at the same time for the general content of the image, but lacks in details. Specifically, in the enlarged area of the tail of the helicopter, the method of the present application can generate a general outline, while other methods cannot achieve this.

[0245] In the embodiment of the present application, the present application proposes a preset multi-scale dual-domain collaborative rain removal network, the core of which is to integrate frequency domain and spatial domain modeling technology, which is composed of the proposed CFEM module, which integrates FFECB and DSITB, which aims to utilize the advantages of the field and work collaboratively. FFECB enhances feature discriminability and global spatial representation by processing high and low frequency features in the frequency domain, while DSITB generates a sparse attention map in the spatial domain using channel compression, a double-branch compensation and adaptive fusion technology to construct high-quality channel dimension features. In addition, the MSIFFM module is proposed to effectively fuse multi-scale encoder features, enhance key region feature representation at different scales, and guide the decoder to better distinguish rain lines and background textures.

[0246] Please refer to Figure 9 , Figure 9A structural block diagram of an image rain removing device is provided for Embodiment Three of the present application.

[0247] The present application provides an image rain removing device, comprising:

[0248] The acquisition module 901 is configured to acquire a rain-bearing image, and perform multi-scale down-sampling on the rain-bearing image to output a multi-scale rain-bearing image.

[0249] The input module 902 is configured to input the rain-bearing image and the multi-scale rain-bearing image into a preset multi-scale dual-domain collaborative rain removing network, wherein the preset multi-scale dual-domain collaborative rain removing network comprises an encoder, a multi-scale spatial interaction feature fusion module, and a decoder.

[0250] The encoding module 903 is configured to encode the rain-bearing image and the multi-scale rain-bearing image by using the encoder to output a plurality of target rain-bearing encoded features.

[0251] The fusion module 904 is configured to perform feature fusion on the plurality of target rain-bearing encoded features by using the multi-scale spatial interaction feature fusion module to output target fused features.

[0252] The decoding module 905 is configured to input the rain-bearing image, the multi-scale rain-bearing image, the target fused features, and the target rain-bearing encoded features into the decoder for decoding to generate a target rain-removed image.

[0253] Further, the multi-scale rain-bearing image comprises a first-scale rain-bearing image and a second-scale rain-bearing image; the encoder comprises a collaborative feature enhancement module and a 3x3 deep hyper-parameterized convolution layer; the plurality of target rain-bearing encoded features comprises a first rain-bearing encoded feature, a second rain-bearing encoded feature, and a third rain-bearing encoded feature; and the input module 902 is specifically configured to:

[0254] perform deep hyper-parameterized convolution on the rain-bearing image by using the 3x3 deep hyper-parameterized convolution layer to output deep convolution features;

[0255] perform feature enhancement on the deep convolution features by using the collaborative feature enhancement module to output the first rain-bearing encoded feature;

[0256] perform feature extraction on the first-scale rain-bearing image to output first-scale rain-bearing image features;

[0257] perform down-sampling on the first rain-bearing encoded feature to output a down-sampled first rain-bearing encoded feature;

[0258] perform feature fusion on the first-scale rain-bearing image features and the down-sampled first rain-bearing encoded feature to generate first fused features;

[0259] perform feature enhancement on the first fused features by using the collaborative feature enhancement module to output the second rain-bearing encoded feature;

[0260] perform feature extraction on the second scale rain-bearing image to output a second scale rain-bearing image feature;

[0261] perform down-sampling on the second rain-bearing coding feature to output a down-sampled second rain-bearing coding feature;

[0262] perform feature fusion on the second scale rain-bearing image feature and the down-sampled second rain-bearing coding feature to generate a second fusion feature;

[0263] perform feature enhancement on the second fusion feature by using a collaborative feature enhancement module to generate a third rain-bearing coding feature.

[0264] Optionally, the collaborative feature enhancement module comprises a frequency focus enhancement convolution block and a dual-flow sparse interaction block based on a self-attention mechanism; the feature processing steps of the collaborative feature enhancement module are specifically as follows:

[0265] focus enhance the input feature input into the collaborative feature enhancement module by using the frequency focus enhancement convolution block to generate a focus enhanced feature;

[0266] input the focus enhanced feature into the dual-flow sparse interaction block based on the self-attention mechanism as an input to generate an output feature.

[0267] Optionally, the frequency focus enhancement convolution block comprises a 3x3 deep hyper-parameterized convolution layer, a ReLU activation function layer, a 1x1 convolution layer and a Sigmoid activation function layer; the input feature input into the collaborative feature enhancement module is focus enhanced by using the frequency focus enhancement convolution block to generate the focus enhanced feature, which comprises:

[0268] perform deep hyper-parameterized convolution on the input feature by using the 3x3 deep hyper-parameterized convolution layer to output an intermediate deep convolution feature, and input the intermediate deep convolution feature into the ReLU activation function layer as an input to output a first nonlinear feature;

[0269] perform deep hyper-parameterized convolution on the first nonlinear feature by using the 3x3 deep hyper-parameterized convolution layer to generate a local feature, and perform Fourier transform on the local feature to output a first real part feature and a first imaginary part feature;

[0270] splice the first real part feature and the first imaginary part feature to output a first spliced feature, and perform convolution operation on the first spliced feature by using the 1x1 convolution layer to output a first convolution feature;

[0271] input the first convolution feature into the ReLU activation function layer as an input to output a second nonlinear feature, and perform convolution operation on the second nonlinear feature by using the 1x1 convolution layer to generate a second convolution feature;

[0272] The second convolutional feature is divided into a second real part feature and a second imaginary part feature along the channel, and inverse Fourier transform is performed on the second real part feature and the second imaginary part feature, and a high-frequency feature is output;

[0273] The local feature is globally averaged and pooled to generate a pooled feature, and a 1x1 convolutional layer is used to convolve the pooled feature to output a third convolutional feature;

[0274] The third convolutional feature is input into a ReLU activation function layer to output a third nonlinear feature, and a 1x1 convolutional layer is used to convolve the third nonlinear feature to generate a fourth convolutional feature;

[0275] A Sigmoid activation function layer is used to perform nonlinear mapping on the fourth convolutional feature to output a mapped feature, and the mapped feature and the local feature are multiplied element by element to output a multiplied feature;

[0276] Fast Fourier transform convolution is performed on the multiplied feature to generate a low-frequency feature, and the low-frequency feature, the high-frequency feature, and the input feature are added element by element to generate a focus-enhanced feature.

[0277] Optionally, the dual-flow sparse interaction block based on the self-attention mechanism includes a 1x1 convolutional layer, a Softmax activation function layer, an adaptive interaction module, and a gated depth-wise convolution feedforward network; the focus-enhanced feature is input into the dual-flow sparse interaction block based on the self-attention mechanism to generate an output feature, including:

[0278] Layer normalization is performed on the focus-enhanced feature to generate a first normalized feature;

[0279] A 1x1 convolutional layer is used to generate an initial query feature, an initial key feature, and an initial value feature based on the first normalized feature, and group convolution is performed on the initial query feature to generate a query tensor;

[0280] Depth-wise convolution is performed on the initial key feature and the initial value feature respectively to generate a key tensor and a value tensor, and the query tensor, the key tensor, and the value tensor are reshaped respectively to output a reshaped query tensor, a reshaped key tensor, and a reshaped value tensor;

[0281] Matrix multiplication is performed on the reshaped query tensor and the reshaped key tensor to generate a multiplication feature, and the multiplication feature is input into a Softmax activation function layer to output a sparse attention map;

[0282] Matrix multiplication is performed on the sparse attention map and the reshaped value tensor to generate an attention output feature, and the attention output feature is reshaped to generate a reshaped attention output feature;

[0283] The 1×1 convolution layer is adopted to perform convolution operation on the value tensor, and a convolution value tensor is output, and the convolution value tensor is subjected to depth-wise convolution to generate a depth-wise convolution value tensor;

[0284] The depth-wise convolution value tensor and the convolution value tensor are subjected to element-wise addition, and a first addition vector is output, and the first addition vector and the reshaped attention output feature are input into the adaptive interaction module for feature interaction, and a target interaction feature is output;

[0285] The 1×1 convolution layer is adopted to perform convolution operation on the target interaction feature, and a convolution interaction feature is output, and the convolution interaction feature and the focus enhancement feature are subjected to element-wise addition, and a second addition feature is output;

[0286] The second addition feature is subjected to layer normalization, and a second normalized feature is output, and the feedforward feature is generated according to the second normalized feature by using the gated depth-wise convolution feedforward network;

[0287] The feedforward feature and the second addition feature are subjected to element-wise addition to generate an output feature.

[0288] Optionally, the adaptive interaction module comprises a 1×1 convolution layer, a GeLU activation function layer and a Sigmoid activation function layer; the first addition vector and the reshaped attention output feature are input into the adaptive interaction module for feature interaction, and a target interaction feature is output, which comprises:

[0289] The 1×1 convolution layer is adopted to perform convolution operation on the first addition vector to generate a convolution addition feature, and the convolution addition feature is input into the GeLU activation function layer to output a first nonlinear local feature;

[0290] The 1×1 convolution layer is adopted to perform convolution operation on the first nonlinear local feature to generate a convolution nonlinear local feature, and the convolution nonlinear local feature is input into the Sigmoid activation function layer to output a second nonlinear local feature;

[0291] The second nonlinear local feature and the reshaped attention output feature are subjected to element-wise multiplication to generate a first initial interaction feature, and the reshaped attention output feature is subjected to global average pooling to generate a pooled global feature;

[0292] The 1×1 convolution layer is adopted to perform convolution operation on the pooled global feature to generate a convolution global feature, and the convolution global feature is input into the GeLU activation function layer to output a first nonlinear global feature;

[0293] The 1×1 convolution layer is adopted to perform convolution operation on the first nonlinear global feature to generate a convolution nonlinear global feature, and the convolution nonlinear global feature is input into the Sigmoid activation function layer to output a second nonlinear global feature;

[0294] Element-wise multiplication is performed on the second nonlinear global feature and the first addition vector to generate a second initial interaction feature, and element-wise addition is performed on the second initial interaction feature and the first initial interaction feature to generate a target interaction feature.

[0295] Further, the fusion module 904 is specifically configured to:

[0296] The first rain-bearing encoding feature, the second rain-bearing encoding feature, and the third rain-bearing encoding feature are respectively preprocessed to output a first rain-bearing initial feature, a second rain-bearing initial feature, and a third rain-bearing initial feature.

[0297] The first rain-bearing initial feature, the second rain-bearing initial feature, and the third rain-bearing initial feature are sequentially subjected to 1x1 convolution operation, nonlinear mapping, and 3x3 deep over-parameterized convolution to generate a first rain-bearing intermediate feature, a second rain-bearing intermediate feature, and a third rain-bearing intermediate feature.

[0298] The first rain-bearing intermediate feature, the second rain-bearing intermediate feature, and the third rain-bearing intermediate feature are spliced to output a rain-bearing spliced feature, and the rain-bearing spliced feature is sequentially subjected to dilated convolution and nonlinear mapping to generate a rain-bearing mapping feature.

[0299] The rain-bearing mapping feature is subjected to depth-wise convolution to generate a rain-bearing depth-wise convolution feature, and the rain-bearing depth-wise convolution feature is element-wise added to output a rain-bearing addition feature.

[0300] The rain-bearing addition feature is sequentially subjected to nonlinear mapping and compression convolution to generate a rain-bearing compression convolution feature, and the rain-bearing compression convolution feature is respectively subjected to channel attention calculation and spatial attention calculation to output a channel attention feature and a spatial attention feature.

[0301] The channel attention feature and the spatial attention feature are element-wise added to generate a rain-bearing enhancement feature, and the rain-bearing enhancement feature is respectively spliced with the first rain-bearing intermediate feature, the second rain-bearing intermediate feature, and the third rain-bearing intermediate feature to generate a first initial fusion feature, a second initial fusion feature, and a third initial fusion feature.

[0302] The first initial fusion feature, the second initial fusion feature, and the third initial fusion feature are respectively subjected to 3x3 deep over-parameterized convolution to generate a first key spatial map, a second key spatial map, and a third key spatial map.

[0303] The first key spatial map, the rain-bearing enhancement feature, and the first rain-bearing intermediate feature are weightedly summed to generate a first spatial perception feature.

[0304] The second key spatial map, the rain-bearing enhancement feature, and the second rain-bearing intermediate feature are weightedly summed to generate a second spatial perception feature.

[0305] perform weighted summation on the third key spatial map, the rain enhancement feature, and the third rain intermediate feature to generate a third spatial perception feature;

[0306] perform summation on the first spatial perception feature, the second spatial perception feature, and the third spatial perception feature to output a summation feature, and sequentially perform 1x1 convolution operation and nonlinear mapping on the summation feature to generate an intermediate fusion feature;

[0307] perform element-wise addition on the intermediate fusion feature, the first rain initial feature, the second rain initial feature, and the third rain initial feature to output a target fusion feature.

[0308] Further, the target fusion feature includes a first fusion feature and a second fusion feature; the decoder includes a collaborative feature enhancement module, a 3x3 deep hyper-parameterized convolution layer, and a 1x1 convolution layer; the target rain-removed image includes a first rain-removed image, a second rain-removed image, and a third rain-removed image; and the decoding module 905 is specifically configured to:

[0309] perform feature enhancement on the third rain-removed encoding feature by using the collaborative feature enhancement module to output a first initial rain-removed feature, perform upsampling on the first initial rain-removed feature to output an upsampled first initial rain-removed feature, and concatenate the upsampled first initial rain-removed feature and the second fusion feature to output a first concatenated rain-removed feature;

[0310] perform convolution operation on the first concatenated rain-removed feature by using the 1x1 convolution layer to generate a first convolution rain-removed feature, and perform feature enhancement on the first convolution rain-removed feature by using the collaborative feature enhancement module to output a second initial rain-removed feature;

[0311] perform upsampling on the second initial rain-removed feature to output an upsampled second initial rain-removed feature, concatenate the upsampled second initial rain-removed feature and the first fusion feature to generate a second concatenated rain-removed feature, and perform convolution operation on the second concatenated rain-removed feature by using the 1x1 convolution layer to generate a second convolution rain-removed feature;

[0312] perform feature enhancement on the second convolution rain-removed feature by using the collaborative feature enhancement module to output a third initial rain-removed feature, and input the third initial rain-removed feature into the 3x3 deep hyper-parameterized convolution layer to perform deep hyper-parameterized convolution to output a third initial rain-removed convolution feature;

[0313] perform element-wise addition on the third initial rain-removed convolution feature and the rain image to generate the first rain-removed image;

[0314] perform deep hyper-parameterized convolution on the first initial rain-removed feature and the second initial rain-removed feature by using the 3x3 deep hyper-parameterized convolution layer respectively to output a first intermediate rain-removed feature and a second intermediate rain-removed feature;

[0315] performing element-wise addition on the first intermediate rain removal feature and the first scale rainy image to generate a second rain removal image;

[0316] performing element-wise addition on the second intermediate rain removal feature and the second scale rainy image to generate a third rain removal image.

[0317] In an alternative apparatus embodiment, further comprising:

[0318] The first module is configured to obtain a rainy image for model training and perform multi-scale down-sampling on the rainy image for model training to output a multi-scale rainy image for model training.

[0319] The second module is configured to input the rainy image for model training and the multi-scale rainy image for model training as inputs of an initial multi-scale dual-domain collaborative rain removal network to output a rain removal image for model training.

[0320] The third module is configured to substitute the rain removal image for model training into a preset loss function and derive to output a model gradient.

[0321] The fourth module is configured to update a model parameter of the initial multi-scale dual-domain collaborative rain removal network using the model gradient to determine an intermediate multi-scale dual-domain collaborative rain removal network and count an update number in real time.

[0322] The fifth module is configured to determine whether the update number reaches a preset training number.

[0323] The sixth module is configured to, if yes, take the intermediate multi-scale dual-domain collaborative rain removal network as a preset multi-scale dual-domain collaborative rain removal network trained.

[0324] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described apparatus and modules can refer to the corresponding process in the foregoing method embodiments, which will not be described herein.

[0325] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other ways. For example, the apparatus embodiments described above are only schematic, for example, the division of units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed mutual units can be indirect coupling or communication connection through some interfaces, apparatuses or units, and can be electrical, mechanical or other forms.

[0326] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may also be distributed to multiple network units. Part or all of the units can be selected to achieve the purpose of the embodiment scheme according to actual needs.

[0327] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image deraining method, characterized in that, include: Acquire rain-bearing images and perform multi-scale downsampling on the rain-bearing images to output multi-scale rain-bearing images; The rainy image and the multi-scale rainy image are input into a preset multi-scale dual-domain collaborative rain removal network, which includes an encoder, a multi-scale spatial interaction feature fusion module, and a decoder. The encoder is used to encode the rainy image and the multi-scale rainy image to output multiple target rainy coding features; The multi-scale spatial interaction feature fusion module performs feature fusion on multiple target rain-encoded features and outputs target fused features. The rainy image, the multi-scale rainy image, the target fusion feature, and the target rainy encoding feature are input into the decoder for decoding to generate a de-rained target image. The multi-scale rainy image includes a first-scale rainy image and a second-scale rainy image; the encoder includes a collaborative feature enhancement module and a 3×3 deep hyperparameterized convolutional layer; the multiple target rainy coding features include a first rainy coding feature, a second rainy coding feature, and a third rainy coding feature; the encoder is used to encode the rainy image and the multi-scale rainy image to output multiple target rainy coding features, including: A 3×3 depth hyperparameterized convolutional layer is used to perform depth hyperparameterized convolution on the rainy image to output depth convolution features. The deep convolutional features are enhanced by a collaborative feature enhancement module, and the first rainy encoded feature is output. Feature extraction is performed on the rainy image at the first scale, and the features of the rainy image at the first scale are output. The first rainy coding feature is downsampled, and the downsampled first rainy coding feature is output. The first scale rainy image features and the downsampled first rainy encoded features are fused together to generate a first fused feature; The first fused feature is enhanced using a collaborative feature enhancement module, and a second rain-coded feature is output. Feature extraction is performed on the rainy image at the second scale, and the features of the rainy image at the second scale are output. The second rainy coding feature is downsampled, and the downsampled second rainy coding feature is output. The second-scale rainy image features and the downsampled second rainy encoded features are fused to generate a second fused feature. The second fused feature is enhanced using a collaborative feature enhancement module to generate a third rain-coded feature. The collaborative feature enhancement module includes a frequency-focused enhanced convolutional block and a two-stream sparse interaction block based on a self-attention mechanism; the feature processing steps of the collaborative feature enhancement module are as follows: The input features to the collaborative feature enhancement module are enhanced by a frequency-focusing enhancement convolutional block to generate focused enhancement features; The focused enhancement features are used as input to a two-stream sparse interaction block based on a self-attention mechanism to generate output features.

2. The image deraining method according to claim 1, characterized in that, The frequency-focusing enhancement convolutional block includes a 3×3 depth hyperparameterized convolutional layer, a ReLU activation function layer, a 1×1 convolutional layer, and a Sigmoid activation function layer; the input features input to the collaborative feature enhancement module are enhanced by the frequency-focusing enhancement convolutional block to generate focused enhanced features, including: The input features are subjected to depth-overparameterized convolution through a 3×3 depth hyperparameterized convolutional layer to output intermediate depth convolutional features. These intermediate depth convolutional features are then used as input to a ReLU activation function layer to output the first nonlinear feature. A 3×3 deep hyperparameterized convolutional layer is used to perform deep hyperparameterized convolution on the first nonlinear feature to generate local features, and Fourier transform is performed on the local features to output the first real part feature and the first imaginary part feature. The first real part feature and the first imaginary part feature are concatenated to output the first concatenated feature, and the first concatenated feature is convolved through a 1×1 convolutional layer to output the first convolutional feature; The first convolutional feature is used as the input to the ReLU activation function layer to output the second nonlinear feature, and a 1×1 convolutional layer is used to perform convolution operation on the second nonlinear feature to generate the second convolutional feature. The second convolutional feature is divided into a second real feature and a second imaginary feature along the channel, and an inverse Fourier transform is performed on the second real feature and the second imaginary feature to output high-frequency features; Global average pooling is performed on the local features to generate pooled features, and a 1×1 convolutional layer is used to perform convolution operations on the pooled features to output a third convolutional feature; The third convolutional feature is used as the input of the ReLU activation function layer to output the third nonlinear feature, and the third nonlinear feature is convolved through a 1×1 convolutional layer to generate the fourth convolutional feature. A Sigmoid activation function layer is used to perform non-linear mapping on the fourth convolutional feature, outputting the mapped feature. The mapped feature and the local feature are then multiplied element-wise to output the multiplied feature. The multiplied features are subjected to Fast Fourier Transform convolution to generate low-frequency features. The low-frequency features, the high-frequency features, and the input features are then added element-wise to generate focus enhancement features.

3. The image deraining method according to claim 1, characterized in that, The self-attention-based two-stream sparse interaction block includes a 1×1 convolutional layer, a Softmax activation function layer, an adaptive interaction module, and a gated depthwise convolutional feedforward network; the step of using the focused enhancement features as input to the self-attention-based two-stream sparse interaction block to generate output features includes: The focused enhancement features are layer normalized to generate the first normalized features; A 1×1 convolutional layer is used to generate initial query features, initial key features, and initial value features based on the first normalized features, and group convolution is performed on the initial query features to generate a query tensor; The initial key features and the initial value features are respectively convolved at depthwise to generate a key tensor and a value tensor. The query tensor, the key tensor and the value tensor are then reshaped to output a reshaped query tensor, a reshaped key tensor and a reshaped value tensor. Matrix multiplication is performed on the reshaped query tensor and the reshaped key tensor to generate multiplication features, and the multiplication features are used as input to the Softmax activation function layer to output a sparse attention map; Matrix multiplication is performed on the sparse attention map and the reshaped value tensor to generate attention output features, and the attention output features are reshaped to generate reshaped attention output features; A 1×1 convolutional layer is used to perform convolution operation on the value tensor to output a convolutional value tensor, and the convolutional value tensor is then subjected to depthwise convolution to generate a depthwise convolutional value tensor. The depthwise convolutional value tensor and the convolutional value tensor are added element-wise to output a first added vector. The first added vector and the reshaped attention output feature are then input into the adaptive interaction module for feature interaction to output the target interaction feature. A 1×1 convolutional layer is used to perform convolution operations on the target interaction features to output convolutional interaction features. The convolutional interaction features and the focus enhancement features are then added element by element to output a second added feature. The second additive feature is layer normalized to output the second normalized feature, and a gated depthwise convolutional feedforward network is used to generate feedforward features based on the second normalized feature; The feedforward feature and the second additive feature are added element by element to generate the output feature.

4. The image deraining method according to claim 3, characterized in that, The adaptive interaction module includes a 1×1 convolutional layer, a GeLU activation function layer, and a Sigmoid activation function layer; the step of inputting the first summed vector and the reshaped attention output features into the adaptive interaction module for feature interaction, and outputting target interaction features, includes: A 1×1 convolutional layer is used to perform convolution operation on the first additive vector to generate convolutional additive features, and the convolutional additive features are used as the input of the GeLU activation function layer to output the first nonlinear local features; A 1×1 convolutional layer is used to perform convolution operation on the first nonlinear local feature to generate a convolutional nonlinear local feature. The convolutional nonlinear local feature is then used as the input of the Sigmoid activation function layer to output a second nonlinear local feature. The second nonlinear local feature and the reshaped attention output feature are multiplied element-wise to generate the first initial interaction feature, and the reshaped attention output feature is subjected to global average pooling to generate the pooled global feature. A 1×1 convolutional layer is used to perform convolution operations on the pooled global features to generate convolutional global features. The convolutional global features are then used as the input to the GeLU activation function layer to output the first nonlinear global features. A 1×1 convolutional layer is used to perform convolution operation on the first nonlinear global feature to generate a convolutional nonlinear global feature. The convolutional nonlinear global feature is then used as the input of the Sigmoid activation function layer to output a second nonlinear global feature. The second nonlinear global feature and the first additive vector are multiplied element-wise to generate the second initial interaction feature, and the second initial interaction feature and the first initial interaction feature are added element-wise to generate the target interaction feature.

5. The image deraining method according to claim 1, characterized in that, The step of fusing multiple target rain-encoded features through the multi-scale spatial interaction feature fusion module to output target fused features includes: The first rainy coding feature, the second rainy coding feature, and the third rainy coding feature are preprocessed respectively to output the first rainy initial feature, the second rainy initial feature, and the third rainy initial feature; The first rainy initial feature, the second rainy initial feature, and the third rainy initial feature are respectively subjected to 1×1 convolution operation, non-linear mapping, and 3×3 depth-overparameterized convolution to generate the first rainy intermediate feature, the second rainy intermediate feature, and the third rainy intermediate feature. The first rainy intermediate feature, the second rainy intermediate feature, and the third rainy intermediate feature are concatenated to output a rainy concatenated feature. The rainy concatenated feature is then subjected to dilated convolution and nonlinear mapping in sequence to generate a rainy mapped feature. The rain-mapped features are subjected to depthwise convolution to generate rain-mapped depthwise convolution features. The rain-mapped depthwise convolution features are then added element-wise to the rain-mapped features to output rain-mapped additive features. The rain-laden additive features are sequentially subjected to nonlinear mapping and compressed convolution to generate rain-laden compressed convolution features. Channel attention and spatial attention are then calculated on the rain-laden compressed convolution features to output channel attention features and spatial attention features. The channel attention features and the spatial attention features are added element by element to generate rain-enhanced features. The rain-enhanced features are then concatenated with the first rain-intermediate feature, the second rain-intermediate feature, and the third rain-intermediate feature to generate the first initial fusion feature, the second initial fusion feature, and the third initial fusion feature. Perform 3×3 depthwise overparameterized convolutions on the first initial fusion feature, the second initial fusion feature, and the third initial fusion feature respectively to generate the first key space map, the second key space map, and the third key space map; The first key spatial map, the rain-enhanced feature, and the first rain-intermediate feature are weighted and summed to generate the first spatial perception feature; The second key spatial map, the rain-enhanced feature, and the second rain-intermediate feature are weighted and summed to generate the second spatial perception feature; The third key spatial map, the rain-enhanced feature, and the third rain-intermediate feature are weighted and summed to generate the third spatial perception feature; The first spatial perception feature, the second spatial perception feature, and the third spatial perception feature are summed to output the summed feature. Then, the summed feature is sequentially subjected to 1×1 convolution and nonlinear mapping to generate intermediate fusion features. The intermediate fusion feature, the first rainy initial feature, the second rainy initial feature, and the third rainy initial feature are added element by element to output the target fusion feature.

6. The image deraining method according to claim 1, characterized in that, The target fusion feature includes a first fusion feature and a second fusion feature; the decoder includes a collaborative feature enhancement module, a 3×3 deep hyperparameterized convolutional layer, and a 1×1 convolutional layer; the target derained image includes a first derained image, a second derained image, and a third derained image; the step of inputting the rainy image, the multi-scale rainy image, the target fusion feature, and the target rainy encoded feature into the decoder for decoding to generate the target derained image includes: The third rainy coding feature is enhanced by a collaborative feature enhancement module, and a first initial deraining feature is output. The first initial deraining feature is upsampled, and the upsampled first initial deraining feature is output. The upsampled first initial deraining feature and the second fusion feature are concatenated to output a first concatenated deraining feature. A 1×1 convolutional layer is used to perform convolution operation on the first concatenated rain removal feature to generate the first convolutional rain removal feature, and a collaborative feature enhancement module is used to enhance the first convolutional rain removal feature to output the second initial rain removal feature. The second initial deraining feature is upsampled to output the upsampled second initial deraining feature. The upsampled second initial deraining feature and the first fused feature are concatenated to generate the second concatenated deraining feature. A 1×1 convolutional layer is used to perform convolution operation on the second concatenated deraining feature to generate the second convolutional deraining feature. The second convolutional deraining feature is enhanced by a collaborative feature enhancement module, and a third initial deraining feature is output. The third initial deraining feature is then input into a 3×3 depth hyperparameterized convolutional layer for depth hyperparameterized convolution, and a third initial deraining convolutional feature is output. The third initial deraining convolutional feature and the rainy image are added element by element to generate the first deraining image; A 3×3 depth hyperparameterized convolutional layer is used to perform depth hyperparameterized convolution on the first initial rain removal feature and the second initial rain removal feature respectively, and the first intermediate rain removal feature and the second intermediate rain removal feature are output. The first intermediate de-raining feature and the first scale rainy image are added element by element to generate the second de-raining image; The second intermediate deraining feature and the second scale rainy image are added element by element to generate the third deraining image.

7. The image deraining method according to claim 1, characterized in that, The model training process of the pre-set multi-scale dual-domain collaborative rain removal network is as follows: Acquire rainy images for model training, and perform multi-scale downsampling on the rainy images for model training to output multi-scale rainy images for model training. The rainy image used for model training and the multi-scale rainy image used for model training are used as inputs to the initial multi-scale dual-domain collaborative deraining network, and the output is the derained image used for model training. Substitute the rain-removed image used for model training into the preset loss function and take the derivative to output the model gradient; The model gradient is used to update the model parameters of the initial multi-scale dual-domain collaborative rain removal network, determine the intermediate multi-scale dual-domain collaborative rain removal network, and count the number of updates in real time. Determine whether the number of updates has reached the preset number of training iterations; If this is achieved, the intermediate multi-scale dual-domain collaborative rain removal network will be used as the trained pre-set multi-scale dual-domain collaborative rain removal network.

8. An image deraining device, applied to the image deraining method of claim 1, characterized in that, include: The acquisition module is used to acquire rainy images, perform multi-scale downsampling on the rainy images, and output multi-scale rainy images; The input module is used to input the rainy image and the multi-scale rainy image into a preset multi-scale dual-domain collaborative rain removal network. The preset multi-scale dual-domain collaborative rain removal network includes an encoder, a multi-scale spatial interaction feature fusion module, and a decoder. The encoding module is used to encode the rainy image and the multi-scale rainy image using the encoder, and output multiple target rainy encoding features; The fusion module is used to perform feature fusion on multiple target rain-encoded features through the multi-scale spatial interaction feature fusion module, and output target fusion features; The decoding module is used to input the rainy image, the multi-scale rainy image, the target fusion feature, and the target rainy encoding feature into the decoder for decoding to generate a target de-rained image.

Images