An image rain removal method based on dual-channel fusion

Abstract

This invention discloses an image deraining method based on dual-channel fusion, comprising the following steps: establishing a dataset containing original rainy images and their corresponding clear, rain-free images; constructing a dual-channel fusion network for image deraining; using a hybrid loss function composed of loss, edge loss, and frequency loss to quantify the difference between the derained image and the clear, rain-free image; training the dual-channel fusion network using the dataset; and testing the trained dual-channel fusion network on the rainy image to be derained to obtain the clear image after deraining. This invention introduces modules such as dual-channel fusion enhanced attention and a hybrid scale-gated feedforward network into the model, improving the model's robustness and feature extraction capabilities, enabling it to better adapt to raindrop interference in different real-world scenarios.

Description

Technical Field

[0001] This invention relates to the field of computer image processing technology, and in particular to an image deraining method based on dual-channel fusion. Background Technology

[0002] Images captured outdoors are often susceptible to weather conditions, with rain being the most common precipitation phenomenon that severely degrades image quality. Rain streaks can obscure texture details and blur object outlines, thus affecting subsequent high-level visual tasks such as object detection, tracking, and recognition. Therefore, effectively removing rain streaks from rain-affected images and restoring a clear background has always been an important research problem in the field of image restoration.

[0003] Existing image deraining methods are mainly divided into three categories: prior-based, convolutional neural network (CNN)-based, and Transformer-based. Prior-based deraining methods typically rely on hand-designed image features and constraint models, such as sparse representation and low-rank decomposition; these methods are poorly adaptable to complex rain conditions and are prone to background information loss or incomplete deraining. With the development of deep learning, convolutional neural networks have been widely used in image deraining tasks. CNNs can automatically learn rain patterns and background features from large-scale data, but their receptive field is limited, making it difficult to effectively capture long-distance dependencies, and their recovery effect on large-scale rain patterns and global structures is not ideal. In recent years, the application of Transformers in visual tasks has made significant progress. Its self-attention mechanism can model global contextual information, and its application in image deraining can improve the recovery quality to a certain extent. However, when facing real-world deraining scenarios, due to the diversity of rain pattern morphology and the complexity of the background, a single architecture is often insufficient to comprehensively address the issue. To this end, existing methods combine Transformers and CNNs to achieve complementary advantages. However, existing models have a significant limitation: the lack of an effective interaction and fusion mechanism between the two architectures. Specifically, CNN focuses on extracting local features, while Transformer focuses on modeling global semantics. This difference in feature information makes it difficult for the features extracted by the two to work together effectively, which limits further improvement in model performance. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a simple and robust image deraining method based on dual-channel fusion.

[0005] The technical solution of this invention to solve the above-mentioned technical problems is: an image deraining method based on dual-channel fusion, comprising the following steps:

[0006] Step 1: Create a dataset A containing the original rainy images and their corresponding clear, rain-free images;

[0007] Step 2: Construct a dual-channel fusion network for image deraining, denoted as DF TransNet;

[0008] Step 3: Utilize Loss, edge loss, and frequency loss constitute a hybrid loss function. This is used to quantify the difference between rain-removed images and clear, rain-free images:

[0009] Step 4: Train DF TransNet using dataset A;

[0010] Step 5: Use the trained DF TransNet to test the rainy image to be de-rained and obtain a clear image after de-raining.

[0011] In the above-mentioned dual-channel fusion-based image deraining method, step two involves DF TransNet consisting of a main path and branch paths.

[0012] The main path adopts a U-shaped structure with skip connections, where the encoding path consists of 4 encoders and the decoding path consists of 4 decoders; the branch paths consist of a 3×3 convolution and two downsampling operations.

[0013] In this branch, the original input image is first passed through a 3×3 convolution, and then downsampled sequentially to obtain three feature maps of different scales, denoted as […]. , , This serves as the input to the decoder. In the main path, the first, second, and third encoders have the same structure, consisting of multiple dual-channel fusion blocks and a downsampling connection. The dual-channel fusion block is the DF Transformer block. The number of DF Transformer blocks in the first, second, and third encoders are respectively... , , The fourth encoder is... The image is composed of Transformer blocks in each region; the original rainy image is first passed through a 3×3 convolution, which is then used as the input of the first encoder, and the output of the previous encoder becomes the input of the next encoder.

[0014] The first and second decoders consist of an upsampling module, a 1×1 convolution, and multiple concatenated region Transformer blocks. The number of concatenated region Transformer blocks in the first and second decoders are respectively... The third decoder consists of an upsampled and The fourth decoder consists of cascaded Transformer blocks in several regions. Each region consists of a Transformer and a 3×3 convolution; due to the introduction of skip connections, the input of the first decoder is the output of the third and fourth encoders and the feature maps of the branches. The input to the second decoder is the output of the second encoder, the output of the first decoder, and the feature maps of the branches. The input to the third decoder is the output of the first encoder, the output of the second decoder, and the feature maps of the branches. The input of the fourth decoder is the output of the third decoder; finally, the output of the fourth decoder is added to the original rainy image, and the result is used as the output of DF TransNet.

[0015] The above-mentioned image deraining method based on dual-channel fusion, wherein the DF Transformer block specifically consists of two layers of normalization, a dual-channel fusion enhanced attention module, and a hybrid scale-gated feedforward network; the dual-channel fusion enhanced attention module is constructed by a self-attention module, a focus aggregation module, and a feature attention fusion module;

[0016] For input features First, perform layer normalization to obtain features. , Where H, W, and 2C represent features respectively. Height, width, number of channels, Represent the real number field; then uniformly divide it into two sub-feature maps along the channel dimension. and , , The data are then fed into two parallel branches; one branch uses a self-attention module, and the other uses a focus aggregation module, thereby obtaining the features respectively. and , , Then, through two splicing operations, and , and Connect the channels to obtain the concatenated features. and , , ,Will and The input is fed into the feature attention fusion module to obtain the output. , ;Will and Adding them together yields the features. , Next, layer normalization is performed to normalize the features. Input the mixed-scale gated feedforward network and combine the obtained features with... Add them together to get the final output. , .

[0017] The aforementioned image deraining method based on dual-channel fusion includes a self-attention module structure comprising a layer normalization layer and a cross-window multi-head attention mechanism; for the input... First, layer normalization is performed, and then the layers are divided into horizontal feature maps based on the average number of channels. and vertical feature map , , In the horizontal direction, Divide evenly into M segments of width The non-overlapping horizontal strip regions are then flattened into a region of length [missing information]. The sequence is multiplied by W, and then the sequence is input into the multi-head attention module to obtain the attention features in the horizontal direction. , The specific process is as follows:

[0018]

[0019]

[0020]

[0021] Where M is the number of horizontal strip regions. , The width of the horizontal strip area is adjusted by... A trade-off can be struck between computational efficiency and receptive field; for The first in One element, for The first in One element, ; This represents a flattening operation, which flattens the strip-shaped region into a sequence vector. It is a multi-head self-attention module that learns the dependencies between each feature in a sequence through matrix multiplication;

[0022] for The same method is used to calculate the attention weights in the vertical direction to obtain the attention features in the vertical direction. , Finally, through the connection and The output of the self-attention module is obtained. .

[0023] The aforementioned image deraining method based on dual-channel fusion, wherein the focus aggregation module specifically comprises one layer normalization layer and one focus aggregation layer; for the input... First, layer normalization is performed, and then context aggregation is performed through linear mapping. The specific process is as follows:

[0024]

[0025]

[0026]

[0027] in, Indicates to Using projection matrix Features obtained through projection. , , yes The l-th slice in; Indicates to Using matrix Features obtained through projection. , , For the features at layer l in the context aggregation operation, , Through the Perform depthwise convolution and GeLU activation function get, Indicates the number of aggregation layers. This represents element-wise multiplication; For contextual aggregation features obtained through hierarchical depthwise convolution, , As a gate function, it acquires contextual features from different aggregation layers through a broadcast mechanism; finally, it performs linear projection on the aggregated features and performs element-wise multiplication with the original input features to obtain... The specific steps are as follows:

[0028]

[0029]

[0030] in, This represents the features obtained by linearly projecting the aggregated features onto the context. It is a projection matrix. , Indicates the projection matrix The obtained linear projection, .

[0031] The above-mentioned image deraining method based on dual-channel fusion has a feature attention fusion module divided into channel branches and spatial branches. The specific structure includes 2 average pooling, 2 max pooling, 4 convolutions, 2 Fourier transforms, 2 filters, 2 inverse Fourier transforms, and 4 Softmax functions.

[0032] In the channel branch, to obtain the channel weight, and Channel pooling operations are performed separately, which include parallel global average pooling and global max pooling operations. Then, the features are concatenated and aggregated. The specific operations are as follows:

[0033]

[0034] in, This indicates a splicing operation. Indicates the aggregated features. and These represent global average pooling and global max pooling in channel pooling, respectively. The result is obtained through two one-dimensional convolutions and the Softmax function. , Represent and Each channel weight;

[0035] In the spatial branch, to obtain the spatial weights, and Spatial pooling operations are performed separately, which include parallel global average pooling and global max pooling operations. Then, aggregated features are obtained in the same way. , The result is obtained through two 2D convolutions and the Softmax function. and Each of their spatial weights Next, and The channel weights and spatial weights are added together to obtain the overall weight, which determines the important part between the two features. Finally, the weights are... and Perform frequency domain state denoising and then sum the values ​​with the weighted sums to obtain the output features. The specific process is as follows:

[0036]

[0037]

[0038] in, They represent and Features after denoising and These are the Fast Fourier Transform and its inverse operation, respectively. This represents a filter.

[0039] The above-mentioned image deraining method based on dual-channel fusion specifically includes a hybrid scale-gated feedforward network module with four convolutions and one Gelu function. For the input features, a 1×1 convolution is first used to expand the number of channels, and then the features are divided into two parts along the channel dimension. The features are extracted by 3×3 and 5×5 convolutions respectively, and residual connections are introduced to obtain an intermediate feature map. and Then on Executing Gelu functions, its output is the same as Multiply to obtain the feature map Finally, a 1×1 convolution is used to shrink the image. The number of channels is used to obtain the output of the hybrid scale gated feedforward network module.

[0040] The above-mentioned image deraining method based on dual-channel fusion includes a specific structure of a region Transformer cascade block comprising one rain pattern mask region Transformer, one background mask region Transformer, one region Transformer, and one 1×1 convolutional layer. First, the input features are fed into two masked region Transformers, and their outputs are concatenated with the input features of the main path in the channel dimension. Then, the channel dimension is reduced by a 1×1 convolution. Finally, the region Transformer is used for processing to obtain the output features.

[0041] The rain ripple mask region Transformer and the background mask region Transformer have the same structure, both containing one region mask attention module, one hybrid scale gated feedforward network, and two layer normalization layers. First, the input features are normalized layer by layer, then the normalized features are fed into the region mask attention module to obtain intermediate features. Finally, the intermediate features are added to the input features to obtain the final feature. , will feature Layer normalization is performed, and features are extracted using a hybrid scale-gated feedforward network. Finally, the extracted features are compared with... The final output features are obtained by adding them together;

[0042] The Region Transformer consists of one region attention module, one hybrid scale gated feedforward network, and two layer normalizations. Its computation process is the same as that of the Rainflake Mask Region Transformer.

[0043] In the region attention module, the input features are first expanded to have more channels through a 3×3 depthwise convolution, and then divided into three equal parts along the channel dimension, which are respectively for querying... ,key Sum Finally, the attention weights are calculated. The specific steps are as follows:

[0044]

[0045] in This represents the Softmax function. Indicates the input dimension, superscript This indicates transpose.

[0046] In the aforementioned image deraining method based on dual-channel fusion, the region mask attention module subtracts the input from the branch path from the input from the main path, sets a threshold, and compares it to obtain the rain ripple layer mask. At the same time, the feature map with a global feature value of 1 is combined with Subtracting them gives the background mask. ; In the Transformer of the rain ripple mask region, query ,key Separately and Multiply and reuse the value The attention weights are calculated using the softmax function; in the Transformer within the background mask region, they are calculated using... Multiply.

[0047] In the above-described image deraining method based on dual-channel fusion, step three involves a hybrid loss function. The calculation formula is:

[0048]

[0049]

[0050]

[0051]

[0052] in, for loss, For edge loss, For frequency loss, It is the sample size. It is the first The true value of each sample It is the first The original values ​​of each sample, The model is for the first The predicted value for each sample, This represents the decomposition of the Laplace pyramid. Represents a constant; This is a hyperparameter used to balance the various loss values.

[0053] The beneficial effects of this invention are as follows:

[0054] 1. This invention introduces a dual-channel fusion enhanced attention and a hybrid scale gated feedforward network module into the model, which improves the robustness and feature extraction capability of the model, enabling it to better adapt to raindrop interference in different real-world scenarios.

[0055] 2. The dual-channel fusion enhanced attention module of the present invention adopts a parallel design of self-attention branch and focus aggregation branch in structure, which are used to extract global context information and local detail features respectively. The feature attention fusion module realizes dynamic weighting and adaptive fusion of the two features. The dual-channel fusion enhanced attention module can fully explore the salient information of the image in the spatial domain and channel domain, thereby effectively preserving image texture and edge details while removing rain patterns, and significantly improving the model's adaptability under different rain intensities and different rain pattern morphologies.

[0056] 3. The hybrid scale gated feedforward network of the present invention adopts a multi-scale convolutional structure. It extracts multi-level feature information under different receptive fields by combining 3×3 convolution and 5×5 convolution, and uses a gating function to dynamically adjust the response intensity of features at different scales. This effectively enhances the expressive power and anti-interference ability of features, and can better distinguish rain patterns from background information in complex rainfall environments, thereby improving the clarity and visual consistency of the image after rain removal. Attached Figure Description

[0057] Figure 1 This is the overall flowchart of the present invention.

[0058] Figure 2 This is a schematic diagram of the DF TransNet network structure of the present invention.

[0059] Figure 3 This is a schematic diagram of the dual-channel fusion enhanced attention module structure of the present invention.

[0060] Figure 4 This is a schematic diagram of the hybrid scale gated feedforward network structure of the present invention.

[0061] Figure 5 This is a schematic diagram of the self-attention module structure of the present invention.

[0062] Figure 6 This is a schematic diagram of the focus aggregation module structure of the present invention.

[0063] Figure 7 for Figure 6 A schematic diagram of the focal aggregation layer structure.

[0064] Figure 8 This is a schematic diagram of the feature attention fusion module structure of the present invention.

[0065] Figure 9 The following are examples of rain removal results of the present invention, wherein (a) to (c) are the original rain maps, and (d) to (f) are the results of rain removal of (a) to (c) using the present invention. Detailed Implementation

[0066] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0067] like Figure 1 As shown, an image deraining method based on dual-channel fusion includes the following steps:

[0068] Step 1: Create a dataset A containing the original rainy images and their corresponding clear, rain-free images.

[0069] Step 2: Construct a dual-channel fusion network for image deraining, denoted as DF TransNet.

[0070] DF TransNet is divided into main paths and branch paths;

[0071] The main path adopts a U-shaped structure with skip connections, where the encoding path consists of 4 encoders and the decoding path consists of 4 decoders; the branch paths consist of a 3×3 convolution and two downsampling operations.

[0072] In this branch, the original input image is first passed through a 3×3 convolution, and then downsampled sequentially to obtain three feature maps of different scales, denoted as […]. , , This serves as the input to the decoder. In the main path, the first, second, and third encoders have the same structure, consisting of multiple dual-channel fusion blocks and a downsampling connection. The dual-channel fusion block is the DF Transformer block. The number of DF Transformer blocks in the first, second, and third encoders are respectively... , , The fourth encoder is... The image is composed of Transformer blocks in each region; the original rainy image is first passed through a 3×3 convolution, which is then used as the input of the first encoder, and the output of the previous encoder becomes the input of the next encoder.

[0073] The first and second decoders consist of an upsampling module, a 1×1 convolution, and multiple concatenated region Transformer blocks. The number of concatenated region Transformer blocks in the first and second decoders are respectively... The third decoder consists of an upsampled and The fourth decoder consists of cascaded Transformer blocks in several regions. Each region consists of a Transformer and a 3×3 convolution; due to the introduction of skip connections, the input of the first decoder is the output of the third and fourth encoders and the feature maps of the branches. The input to the second decoder is the output of the second encoder, the output of the first decoder, and the feature maps of the branches. The input to the third decoder is the output of the first encoder, the output of the second decoder, and the feature maps of the branches. The input of the fourth decoder is the output of the third decoder; finally, the output of the fourth decoder is added to the original rainy image, and the result is used as the output of DF TransNet.

[0074] like Figure 2 As shown, the specific structure of the DF Transformer block includes two layers of normalization, a dual-channel fusion-enhanced attention module, and a hybrid-scale gated feedforward network; as... Figure 3 As shown, the dual-channel fusion enhanced attention module consists of a self-attention module, a focus aggregation module, and a feature attention fusion module;

[0075] For input features First, perform layer normalization to obtain features. , Where H, W, and 2C represent features respectively. Height, width, number of channels, Represent the real number field; then uniformly divide it into two sub-feature maps along the channel dimension. and , , The data are then fed into two parallel branches; one branch uses a self-attention module, and the other uses a focus aggregation module, thereby obtaining the features respectively. and , , Then, through two splicing operations, and , and Connect the channels to obtain the concatenated features. and , , ,Will and The input is fed into the feature attention fusion module to obtain the output. , ;Will and Adding them together yields the features. , Next, layer normalization is performed to normalize the features. Input the mixed-scale gated feedforward network and combine the obtained features with... Add them together to get the final output. , .

[0076] like Figure 5 As shown, the self-attention module specifically includes a layer normalization and a cross-window multi-head attention mechanism; for the input... First, layer normalization is performed, and then the layers are divided into horizontal feature maps based on the average number of channels. and vertical feature map , , In the horizontal direction, Divide evenly into M segments of width The non-overlapping horizontal strip regions are then flattened into a region of length [missing information]. The sequence is multiplied by W, and then the sequence is input into the multi-head attention module to obtain the attention features in the horizontal direction. , The specific process is as follows:

[0077]

[0078]

[0079]

[0080] Where M is the number of horizontal strip regions. , The width of the horizontal strip area is adjusted by... A trade-off can be struck between computational efficiency and receptive field; for The first in One element, for The first in One element, ; This represents a flattening operation, which flattens the strip-shaped region into a sequence vector. It is a multi-head self-attention module that learns the dependencies between each feature in a sequence through matrix multiplication;

[0081] for The same method is used to calculate the attention weights in the vertical direction to obtain the attention features in the vertical direction. , Finally, through the connection and The output of the self-attention module is obtained. .

[0082] like Figure 6 , Figure 7 As shown, the specific structure of the focus aggregation module includes one layer normalization layer and one focus aggregation layer; for the input... First, layer normalization is performed, and then context aggregation is performed through linear mapping. The specific process is as follows:

[0083]

[0084]

[0085]

[0086] in, Indicates to Using projection matrix Features obtained through projection. , , yes The l-th slice in; Indicates to Using matrix Features obtained through projection. , , For the features at layer l in the context aggregation operation, , Through the Perform depthwise convolution and GeLU activation function get, Indicates the number of aggregation layers. This represents element-wise multiplication; For contextual aggregation features obtained through hierarchical depthwise convolution, , As a gate function, it acquires contextual features from different aggregation layers through a broadcast mechanism; finally, it performs linear projection on the aggregated features and performs element-wise multiplication with the original input features to obtain... The specific steps are as follows:

[0087]

[0088]

[0089] in, This represents the features obtained by linearly projecting the aggregated features onto the context. It is a projection matrix. , Indicates the projection matrix The obtained linear projection, .

[0090] like Figure 8 As shown, the feature attention fusion module is divided into channel branch and spatial branch, and its specific structure includes 2 average pooling, 2 max pooling, 4 convolutions, 2 Fourier transforms, 2 filters, 2 inverse Fourier transforms, and 4 Softmax functions;

[0091] In the channel branch, to obtain the channel weight, and Channel pooling operations are performed separately, which include parallel global average pooling and global max pooling operations. Then, the features are concatenated and aggregated. The specific operations are as follows:

[0092]

[0093] in, This indicates a splicing operation. Indicates the aggregated features. and These represent global average pooling and global max pooling in channel pooling, respectively. The result is obtained through two one-dimensional convolutions and the Softmax function. , Represent and Each channel weight;

[0094] In the spatial branch, to obtain the spatial weights, and Spatial pooling operations are performed separately, which include parallel global average pooling and global max pooling operations. Then, aggregated features are obtained in the same way. , The result is obtained through two 2D convolutions and the Softmax function. and Each of their spatial weights Next, and The channel weights and spatial weights are added together to obtain the overall weight, which determines the important part between the two features. Finally, the weights are... and Perform frequency domain state denoising and then sum the values ​​with the weighted sums to obtain the output features. The specific process is as follows:

[0095]

[0096]

[0097] in, They represent and Features after denoising and These are the Fast Fourier Transform and its inverse operation, respectively. This represents a filter.

[0098] like Figure 4 As shown, the hybrid scale-gated feedforward network module specifically includes four convolutions and one Gelu function. For the input features, a 1×1 convolution is first used to expand the number of channels, and then the features are divided into two parts along the channel dimension. The features are extracted by 3×3 convolution and 5×5 convolution respectively. At the same time, residual connections are introduced to obtain an intermediate feature map. and Then on Executing Gelu functions, its output is the same as Multiply to obtain the feature map Finally, a 1×1 convolution is used to shrink the image. The number of channels is used to obtain the output of the hybrid scale gated feedforward network module.

[0099] The specific structure of the region Transformer cascade block includes one rain ripple mask region Transformer, one background mask region Transformer, one region Transformer, and one 1×1 convolutional layer. First, the input features are fed into two masked region Transformers, and their outputs are concatenated with the input features of the main path in the channel dimension. Then, the channel dimension is reduced by a 1×1 convolution. Finally, the region Transformer is used for processing to obtain the output features.

[0100] The rain ripple mask region Transformer and the background mask region Transformer have the same structure, both containing one region mask attention module, one hybrid scale gated feedforward network, and two layer normalization layers. First, the input features are normalized layer by layer, then the normalized features are fed into the region mask attention module to obtain intermediate features. Finally, the intermediate features are added to the input features to obtain the final feature. , will feature Layer normalization is performed, and features are extracted using a hybrid scale-gated feedforward network. Finally, the extracted features are compared with... The final output features are obtained by adding them together.

[0101] The Region Transformer consists of one region attention module, one hybrid scale gated feedforward network, and two layer normalizations. Its computation process is the same as that of the Rainflake Mask Region Transformer.

[0102] In the region attention module, the input features are first expanded to have more channels through a 3×3 depthwise convolution, and then divided into three equal parts along the channel dimension, which are respectively for querying... ,key Sum Finally, the attention weights are calculated. The specific steps are as follows:

[0103]

[0104] in This represents the Softmax function. Indicates the input dimension, superscript This indicates transpose.

[0105] In the region mask attention module: the input from the branch is subtracted from the input from the main path, and a threshold is set and compared with it to obtain the rain ripple layer mask. At the same time, the feature map with a global feature value of 1 is combined with Subtracting them gives the background mask. ; In the Transformer of the rain ripple mask region, query ,key Separately and Multiply and reuse the value The attention weights are calculated using the softmax function; in the Transformer within the background mask region, they are calculated using... Multiply.

[0106] Step 3: Utilize Loss, edge loss, and frequency loss constitute a hybrid loss function. This is used to quantify the difference between rain-removed images and clear, rain-free images.

[0107] Hybrid loss function The calculation formula is:

[0108]

[0109]

[0110]

[0111]

[0112] in, for loss, For edge loss, For frequency loss, It is the sample size. It is the first The true value of each sample It is the first The original values ​​of each sample, The model is for the first The predicted value for each sample, This represents the decomposition of the Laplace pyramid. Represents a constant; This is a hyperparameter used to balance the various loss values.

[0113] Step 4: Train DF TransNet using dataset A; the specific training strategy includes: setting the batch size to 2, setting the initial learning rate to 0.003, using cosine annealing, and gradually reducing the final learning rate to 2e-6.

[0114] Step 5: Use the trained DF TransNet to test the rainy image to be de-rained and obtain a clear image after de-raining.

[0115] This invention was used to conduct experiments on the Rain200H public dataset. Rain200H contains 2000 synthetic rainy images and corresponding clear, rainless images; 1800 image pairs are used as the training set and 200 image pairs are used as the test set. Figure 9 The experimental results on the test set are shown, where (a) to (c) are the original rainy images, and (d) to (f) are the results of de-raining (a) to (c) using the present invention. It can be observed that the present invention can effectively remove the interference of rain streaks in the image and obtain a high-quality image with high restoration and clear details.

Claims

1. An image deraining method based on dual-channel fusion, characterized in that, Includes the following steps: Step 1: Create a dataset A containing the original rainy images and their corresponding clear, rain-free images; Step 2: Construct a dual-channel fusion network for image deraining, denoted as DF TransNet; In step two, DF TransNet is divided into main paths and branch paths; The main path adopts a U-shaped structure with skip connections, where the encoding path consists of 4 encoders and the decoding path consists of 4 decoders; the branch paths consist of a 3×3 convolution and two downsampling operations. In this branch, the original input image is first passed through a 3×3 convolution, and then downsampled sequentially to obtain three feature maps of different scales, denoted as […]. , , This serves as the input to the decoder. In the main path, the first, second, and third encoders have the same structure, consisting of multiple dual-channel fusion blocks and a downsampling connection. The dual-channel fusion block is the DF Transformer block. The number of DF Transformer blocks in the first, second, and third encoders are respectively... , , The fourth encoder is... The image is composed of Transformer blocks in each region; the original rainy image is first passed through a 3×3 convolution, which is then used as the input of the first encoder, and the output of the previous encoder becomes the input of the next encoder. The first and second decoders consist of an upsampling module, a 1×1 convolution, and multiple concatenated region Transformer blocks. The number of concatenated region Transformer blocks in the first and second decoders are respectively... The third decoder consists of an upsampled and The fourth decoder consists of cascaded Transformer blocks in several regions. Each region consists of a Transformer and a 3×3 convolution; due to the introduction of skip connections, the input of the first decoder is the output of the third and fourth encoders and the feature maps of the branches. The input to the second decoder is the output of the second encoder, the output of the first decoder, and the feature maps of the branches. The input to the third decoder is the output of the first encoder, the output of the second decoder, and the feature maps of the branches. The input of the fourth decoder is the output of the third decoder; finally, the output of the fourth decoder is added to the original rainy image, and the result is used as the output of DF TransNet. The specific structure of the DF Transformer block consists of two layers of normalization, a dual-channel fusion enhanced attention module, and a hybrid scale gated feedforward network; the dual-channel fusion enhanced attention module is constructed by a self-attention module, a focus aggregation module, and a feature attention fusion module. For input features First, perform layer normalization to obtain features. , Where H, W, and 2C represent features respectively. Height, width, number of channels, Represent the real number field; then uniformly divide it into two sub-feature maps along the channel dimension. and , , , and input them into two parallel branches respectively; One branch uses a self-attention module, and the other branch uses a focus aggregation module, thus obtaining features respectively. and , , Then, through two splicing operations, and , and Connect the channels to obtain the concatenated features. and , , ,Will and The input is fed into the feature attention fusion module to obtain the output. , ;Will and Adding them together yields the features. , Next, layer normalization is performed to normalize the features. Input the mixed-scale gated feedforward network and combine the obtained features with... Add them together to get the final output. , ; The specific structure of the region Transformer cascade block includes one rain ripple mask region Transformer, one background mask region Transformer, one region Transformer, and one 1×1 convolutional layer. First, the input features are fed into two masked region Transformers, and their outputs are concatenated with the input features of the main path in the channel dimension. Then, the channel dimension is reduced by a 1×1 convolution. Finally, the region Transformer is used for processing to obtain the output features. The rain ripple mask region Transformer and the background mask region Transformer have the same structure, both containing one region mask attention module, one hybrid scale gated feedforward network, and two layer normalizations; In the region mask attention module: the input from the branch is subtracted from the input from the main path, and a threshold is set and compared with it to obtain the rain ripple layer mask. At the same time, the feature map with a global feature value of 1 is combined with Subtracting them gives the background mask. ; In the Transformer region of the rain ripple mask, query ,key Separately and Multiply and reuse the value The attention weights are calculated using the softmax function; in the Transformer within the background mask region, they are calculated using... Multiply; Step 3: Utilize Loss, edge loss, and frequency loss constitute a hybrid loss function. This is used to quantify the difference between rain-removed images and clear, rain-free images: Step 4: Train DF TransNet using dataset A; Step 5: Use the trained DF TransNet to test the rainy image to be de-rained and obtain a clear image after de-raining.

2. The image deraining method based on dual-channel fusion according to claim 1, characterized in that, The self-attention module specifically includes a layer normalization and a cross-window multi-head attention mechanism; for the input... First, layer normalization is performed, and then the layers are divided into horizontal feature maps based on the average number of channels. and vertical feature map , , In the horizontal direction, Divide evenly into M segments of width The non-overlapping horizontal strip regions are then flattened into a region of length [missing information]. The sequence is multiplied by W, and then the sequence is input into the multi-head attention module to obtain the attention features in the horizontal direction. , The specific process is as follows: ； ； ； Where M is the number of horizontal strip regions. , The width of the horizontal strip area is adjusted by... A trade-off can be struck between computational efficiency and receptive field; for The first in One element, for The first in One element, ; This represents a flattening operation, which flattens the strip-shaped region into a sequence vector. It is a multi-head self-attention module that learns the dependencies between each feature in a sequence through matrix multiplication; for The same method is used to calculate the attention weights in the vertical direction to obtain the attention features in the vertical direction. , Finally, through the connection and The output of the self-attention module is obtained. .

3. The image deraining method based on dual-channel fusion according to claim 2, characterized in that, The specific structure of the focus aggregation module includes one layer normalization layer and one focus aggregation layer; for the input... First, layer normalization is performed, and then context aggregation is performed through linear mapping. The specific process is as follows: ； ； ； in, Indicates to Using projection matrix Features obtained through projection. , , yes The l-th slice in; Indicates to Using matrix Features obtained through projection. , , For the features at layer l in the context aggregation operation, , Through the Perform depthwise convolution and GeLU activation function get, Indicates the number of aggregation layers. This represents element-wise multiplication; For contextual aggregation features obtained through hierarchical depthwise convolution, , As a gate function, it acquires contextual features from different aggregation layers through a broadcast mechanism; finally, it performs linear projection on the aggregated features and performs element-wise multiplication with the original input features to obtain... The specific steps are as follows: ； ； in, This represents the features obtained by linearly projecting the aggregated features onto the context. It is a projection matrix. , Indicates the projection matrix The obtained linear projection, .

4. The image deraining method based on dual-channel fusion according to claim 3, characterized in that, The feature attention fusion module is divided into channel branch and spatial branch. The specific structure includes 2 average pooling, 2 max pooling, 4 convolutions, 2 Fourier transforms, 2 filters, 2 inverse Fourier transforms, and 4 Softmax functions. In the channel branch, to obtain the channel weight, and Channel pooling operations are performed separately, which include parallel global average pooling and global max pooling operations. Then, the features are concatenated and aggregated. The specific operations are as follows: ； in, This indicates a splicing operation. This represents the aggregated features. and These represent global average pooling and global max pooling in channel pooling, respectively. The result is obtained through two one-dimensional convolutions and the Softmax function. , Represent and Each channel weight; In the spatial branch, to obtain the spatial weights, and Spatial pooling operations are performed separately, which include parallel global average pooling and global max pooling operations. Then, aggregated features are obtained in the same way. , The result is obtained through two 2D convolutions and the Softmax function. and Each of their spatial weights Next, and The channel weights and spatial weights are added together to obtain the overall weight, which determines the important part between the two features. Finally, the weights are... and Perform frequency domain state denoising and then sum the values ​​with the weighted sums to obtain the output features. The specific process is as follows: ； ； in, They represent and Features after denoising and These are the Fast Fourier Transform and its inverse operation, respectively. This represents a filter.

5. The image deraining method based on dual-channel fusion according to claim 4, characterized in that, The hybrid-scale gated feedforward network module specifically includes four convolutions and one Gelu function. For the input features, a 1×1 convolution is first used to expand the number of channels, and then the features are divided into two parts along the channel dimension. The features are extracted by 3×3 convolution and 5×5 convolution respectively. Residual connections are introduced to obtain an intermediate feature map. and Then on Executing Gelu functions, its output is the same as Multiply to obtain the feature map Finally, a 1×1 convolution is used to shrink the image. The number of channels is used to obtain the output of the hybrid scale gated feedforward network module.

6. The image deraining method based on dual-channel fusion according to claim 5, characterized in that, The rain pattern mask region Transformer and the background mask region Transformer have the same structure. First, the input features are normalized by layers, and then the normalized features are fed into the region mask attention module to obtain intermediate features. The intermediate features are added to the input features to obtain the feature. , will feature Layer normalization is performed, and features are extracted using a hybrid scale-gated feedforward network. Finally, the extracted features are compared with... The final output features are obtained by adding them together; The Region Transformer consists of one region attention module, one hybrid scale gated feedforward network, and two layer normalizations. Its computation process is the same as that of the Rainflake Mask Region Transformer. In the region attention module, the input features are first expanded to have more channels through a 3×3 depthwise convolution, and then divided into three equal parts along the channel dimension, which are respectively for querying... ,key Sum Finally, the attention weights are calculated. The specific steps are as follows: ； in This represents the Softmax function. Indicates the input dimension, superscript This indicates transpose.

7. The image deraining method based on dual-channel fusion according to claim 6, characterized in that, In step three, the hybrid loss function The calculation formula is: ； ； ； ； in, for loss, For edge loss, For frequency loss, It is the sample size. It is the first The true value of each sample It is the first The original values ​​of each sample, The model is for the first The predicted value for each sample, This represents the decomposition of the Laplace pyramid. Represents a constant; This is a hyperparameter used to balance the various loss values.

Images