Image rain removal method based on multi-scale Transform
By employing a multi-scale Transformer architecture and a frequency domain detail preservation module, combined with edge detection and self-attention mechanisms, this method addresses the problem that existing image deraining methods struggle to effectively remove rain streaks in complex scenes, achieving efficient image deraining effects across multi-scale scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHONGBEI UNIV
- Filing Date
- 2026-01-16
- Publication Date
- 2026-04-14
AI Technical Summary
Existing image deraining methods are insufficient in handling complex rain patterns, multi-scale features, and common characteristics of rain pattern degradation, making it difficult to effectively remove rain patterns and preserve image details in complex scenes.
Employing a multi-scale Transformer architecture, combining a frequency domain detail preservation module, a multi-scale enhancement module, and an image reconstruction and loss optimization module, cross-scale feature fusion and optimization are achieved through multi-scale downsampling, frequency domain decomposition, edge detection, and self-attention mechanisms.
While maintaining computational efficiency, it effectively suppresses rain pattern interference, improves the edge fidelity and texture clarity of images, adapts to diverse and complex rain pattern scenes, and enhances the generalization and practicality of rain removal tasks.
Smart Images
Figure CN121860883A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision technology, specifically relating to an image deraining method based on multi-scale Transformer. Background Technology
[0002] Image deraining, a visual task in computer vision, aims to eliminate the interference of raindrop degradation on image quality in order to recover clear, detailed, rain-free images. This task is crucial for applications such as autonomous driving, intelligent visual inspection, and video surveillance. However, existing deraining methods still have significant shortcomings in handling complex rain patterns, multi-scale features, and common characteristics of rain pattern degradation, making it difficult to meet the needs of real-world scenarios.
[0003] Image deraining methods mainly include traditional methods and deep learning methods. Traditional methods simulate the rain formation process using mathematical models, employing image decomposition, low-rank representation, discriminative sparse coding, Gaussian mixture models, etc. These methods use a small number of parameters and do not require powerful GPUs for training or inference. However, they have poor generalization ability; their performance is poor in complex scenes such as heavy rain, downpours, rain streaks of varying directions, and mixed rain and fog. Furthermore, they cannot handle complex background textures resembling rain streaks and cannot distinguish between the background and rain streaks. With the development of deep learning technology, Convolutional Neural Networks (CNNs) were introduced. These methods directly learn the mapping relationship from rainy images to rain-free images by building end-to-end network models, avoiding the dependence on prior knowledge of traditional methods. However, their local receptive field is an inherent limitation. Subsequently, the Transformer architecture was introduced. It captures long-range dependencies in images through a self-attention mechanism, showing significant advantages in global information modeling and effectively solving the local receptive field problem of CNNs. However, most existing Transformer-based rain removal methods employ single-scale features, which cannot simultaneously capture information at different scales. This makes it difficult for Transformers to balance rain removal with detail preservation when dealing with complex rain patterns. Therefore, multi-scale information is crucial for image rain removal. Multi-scale strategies, by applying Transformers at lower resolutions, simplify computation and enable the model to efficiently model global contextual relationships.
[0004] Since the formation of rain patterns is affected by a variety of factors (such as raindrop size, density, falling speed, lens surface characteristics, and lighting conditions), these factors result in a high degree of diversity in the appearance of rain patterns in images. Therefore, the key to solving the problem of rain removal in images is to adapt to complex rain patterns and scene requirements, preserve texture and edge information, and adapt to the diversity of real-world scenes. Summary of the Invention
[0005] To address the limitations of existing image deraining methods in complex scenarios, this invention provides an image deraining method based on multi-scale Transformer. The aim is to provide a method that can restore rainy images to clear, rain-free images in various environments, whether it's a heavy rain or a light rain, and whether it's a synthetic or real dataset. This method introduces a multi-scale Transformer, a frequency domain detail-preserving module, and an efficient multi-scale enhancement module, thereby improving the model's representational power and computational efficiency while maintaining the advantages of multi-scale methods.
[0006] To achieve the above objectives, the present invention employs the following technical solutions:
[0007] A multi-scale Transformer-based image deraining method, comprising the following steps:
[0008] Step 1: Construct an image deraining model that includes a multi-scale downsampling module, a frequency domain detail preservation module, a Transformer encoding and decoding module, a multi-scale enhancement module, and an image reconstruction and loss optimization module. Input the rainy image into the model and use bilinear interpolation to perform multi-scale downsampling on the image to obtain a multi-scale image representation.
[0009] The multi-scale downsampling module in step 1: inputs the original image into the multi-scale downsampling module and uses bilinear interpolation to downsample the image to obtain the sampled image;
[0010] Frequency domain detail preservation module: The feature map is decomposed into frequency domain components by Fourier transform to generate amplitude spectrum and phase spectrum. The frequency components are calculated by full-image integration, as shown in formula (1). Low-pass filtering is used to separate low-frequency and high-frequency components. The Sobel convolution operator is used to extract image edge information to generate a single-channel edge map. The edge map is then transformed into a multi-channel mask in the range of 0-1 through mapping and Sigmoid function. Finally, the mask is used to filter high-frequency details related to the edge and the low-frequency background is fused with the enhanced high-frequency details in the convolutional layer.
[0011] (1)
[0012] in, Representing the spatial domain image in coordinates Pixel value at that location, The frequency coordinates in the frequency domain are represented as The complex value of, Indicates image resolution, exponential term Represent the basis functions of the complex sine wave;
[0013] The Transformer encoding / decoding module is used to model rain streaks of different sizes. Since the pyramid-style downsampling structure constructs multi-level feature representations, the Transformer encoding / decoding module focuses on the overall structure at low resolution and on detailed information at high resolution when extracting features. It combines features from different scales to integrate information and enhance the understanding of complex scenes. The global attention mechanism handles large-scale dependencies, while local attention handles details. The Self-Attention mechanism in the Transformer encoding / decoding module can search for information at different scales and then complement it with cross-scale connections to generate better rain removal results.
[0014] Multi-scale enhancement module ECAfu: Introduces a cross-attention mechanism at the same scale, selectively retaining high-frequency information through the mask generated by edge detection, thereby enhancing the contours and details related to image structure while suppressing interference components related to rain patterns. Unlike traditional simple feature splicing or weighting, the multi-scale enhancement module ECAfu introduces an attention mechanism and edge guidance during feature fusion, combined with the edge mask generated by the Sobel operator, making the model more efficient and accurate in fusing global information provided by the coarse scale and local information provided by the fine scale. Finally, the features output by the multi-scale enhancement module ECAfu have both global consistency and local fineness, thereby improving edge fidelity and texture clarity in the rain removal task, ensuring the naturalness and credibility of the generated image. The formula for the attention mechanism is Equation (2), and the formula for the edge mask is Equation (3).
[0015] (2)
[0016] (3)
[0017] In the formula, It is a query vector generated from a certain scale feature. It is a key vector generated from features at another scale. It is a value vector generated from coarse-scale features. It is the dimension scaling factor of the key vector (to prevent the dot product from becoming too large). It is a 0 / 1 binary mask generated by the Sobel operator. It is Hadamaji.
[0018] Image reconstruction and loss optimization module: used to map the enhanced deep features back to the image space to generate rain-removed images, and guide the optimization of model parameters through multi-dimensional constraints;
[0019] The feature fusion and reconstruction unit in the image reconstruction and loss optimization module fuses cross-scale enhanced features and projects high-dimensional features into multi-scale RGB image outputs. The multi-dimensional loss constraint unit in the image reconstruction and loss optimization module is composed of Charbonnier loss, frequency pyramid loss, edge loss and total variation smoothing loss. It jointly constrains the training process of the model from multiple dimensions such as pixel accuracy, frequency domain consistency, edge texture and spatial smoothness, to ensure that the final generated image maintains structural integrity and detail clarity while removing rain patterns.
[0020] The input is a raw rain-affected image, 256×256 pixels. In this module, bilinear interpolation is first used to downsample the image, resulting in two versions at different scales: 128×128 and 64×64. During downsampling, the image resolution gradually decreases, compressing high-frequency details and leading to greater visual blur. Although some detail is lost, this scale reduction helps the model acquire complementary information at different levels: the coarser scale focuses on global structure and background information, while the finer scale retains more local texture and detail features. Through parallel extraction and fusion of multi-scale features, the model can better balance overall structure and local details while removing rain streaks. The final output images at each scale maintain an RGB three-channel structure for further processing and fusion in subsequent modules.
[0021] Step 2: Input the multi-scale image into the frequency domain detail preservation module to extract and enhance high-frequency detail features in the frequency domain;
[0022] The specific operation of step 2 is as follows:
[0023] Step 2.1: Input the multi-scale image into the frequency domain detail preservation module, transform the image from the spatial domain to the frequency domain through Fourier transform to obtain the amplitude spectrum and phase spectrum, and use low-pass filtering to separate the low-frequency and high-frequency components, while preserving the overall structure of the low-frequency part;
[0024] Step 2.2: By using the edge-aware gating in the frequency domain detail preservation module, the edge information of the image is extracted using Sobel convolution, the single-channel edge map is mapped back to the multi-channel, and a 0~1 mask is obtained by performing Sigmoid to filter out the high-frequency details related to the edge.
[0025] Step 2.3: Fuse the low-frequency components with the edge-enhanced high-frequency components in the convolutional layer to output a high-channel feature representation, providing richer-detailed input to the Transformer encoding and decoding module.
[0026] Step 3: Model the local and global relationships of multi-scale features through convolutional feature extraction and Transformer encoding / decoding modules;
[0027] The specific operation of step 3 is as follows:
[0028] When performing convolution to extract local features, the input is the image features enhanced by the frequency domain detail preservation module. These features are then concatenated channel by channel with the image features from the previous scale that have been enlarged to the current scale by bilinear interpolation. Subsequently, the number of channels is adjusted back to the target number of channels through a convolutional network, thus completing the initial feature fusion.
[0029] The preliminarily fused features are then input into the encoder-decoder module of the Transformer encoder-decoder module. The encoder implemented in this method adopts a hierarchical Transformer structure, with corresponding encoders and decoders configured for different scales. There is one encoder and decoder for small scales, two encoders and decoders for medium scales, and three encoders and decoders for large scales. The encoder extracts global context information through step-by-step downsampling, and the decoder recovers spatial structure and detail information through step-by-step upsampling. The encoder consists of a first encoding unit, a first downsampling unit, a second encoding unit, and a second downsampling unit. After the second encoding unit, the code enters the bottleneck layer, which aggregates the deep features extracted by the downsampling unit. Correspondingly, the decoder consists of a first upsampling unit, a first decoding unit, a second upsampling unit, and a second decoding unit.
[0030] Specifically, the feature map is used as input to the Encoder. First, a first encoding unit consisting of several Transformer blocks is executed, maintaining the spatial resolution and number of channels of the input, thereby modeling the correlation between features within the local receptive field. Then, a first downsampling unit downsamples the feature map, halving the resolution and increasing the number of channels to expand the receptive field and improve feature representation capacity. Next, a second encoding unit stacks several Transformer blocks again at the downsampled scale to extract scale-specific contextual information. Finally, a second downsampling unit further reduces the resolution and increases the number of channels before the features are fed into the bottleneck layer.
[0031] The bottleneck layer consists of several Transformer blocks. While maintaining the current scale, the Transformer blocks in each layer of the encoder use multi-head self-attention and feedforward networks to jointly model long- and short-range dependencies. This effectively suppresses rain ripple interference and preserves scene structure information while progressively reducing spatial resolution and gradually improving semantic abstraction.
[0032] Next, the decoder structure is introduced, which consists of alternating levels of decoding units and upsampling units to achieve progressive resolution restoration. The low-resolution, high-semantic features output from the bottleneck layer are first upsampled (by the first upsampling unit), doubling the spatial resolution and reducing the number of channels. At this scale, the decoder performs feature concatenation with the feature maps in the same layer as the encoder, fusing the local details preserved at the encoder with the progressively restored global semantics at the decoder. Then, convolution reduces the number of channels before sending the results to the first decoding unit. This unit, composed of several Transformer blocks, further refines the global feature representation while maintaining the current scale. Based on this, the output features are further amplified in resolution by the second upsampling unit, restoring them to a scale close to the original input. Channel concatenation and convolution adjust the number of channels again before being sent to the second decoding unit, where Transformer blocks perform feature decoding once more to enhance semantic consistency and repair local textures at higher resolution. Finally, the features enter the convolutional reconstruction module, where convolution yields the final output.
[0033] Step 4: Introduce the ECAfu multi-scale enhancement module into the bottleneck layer, combining edge perception and channel attention to enhance rain removal features;
[0034] Multi-scale enhancement module: Its main function is to realize the interaction and fusion of features between scales, and to enhance useful high-frequency details and suppress rain ripple noise by guiding edge information. For multi-bottleneck layer features at the same resolution scale, an edge-guided cross-attention fusion unit (ECAfu) is set up.
[0035] Specifically, on the same scale branch, the first bottleneck layer features and the second bottleneck layer features with the same spatial resolution are obtained respectively. These two features are input into the multi-scale enhancement module ECAfu. One of the features is used as a query vector to guide the attention mapping of the other feature. The high-frequency details are filtered by combining edge masks to obtain the fused enhanced features. This process maintains the number of channels and spatial resolution unchanged while enhancing the features.
[0036] The enhanced features after fusion are subjected to bilinear interpolation to reduce their spatial resolution, enabling them to be element-wise added to the bottleneck layer features at coarse scales (such as the smallest scale). Through this strategy of "intra-scale enhancement and cross-scale transfer," after suppressing rain streaks and strengthening the global structure, the optimized feature information can be supplemented to the coarse-scale branch, which helps to recover more realistic texture details and thus significantly improves the overall rain removal effect.
[0037] In this structure, the input consists of two bottleneck layers of the same scale, with attention computation, edge guidance, feature fusion, and residuals in between. The output is an enhanced feature map of the same scale without changing its resolution or size.
[0038] Step 5: Use the image reconstruction and loss optimization module to perform convolutional reconstruction on the enhanced rain-removing features, output multi-scale rain-removing image results, and use multiple loss functions for optimization to obtain the rain-removed image.
[0039] The input is the feature maps output from different scale enhancement modules, which are added element-wise to the upsampled result of the previous scale to achieve cross-scale information fusion. Finally, the image is mapped to a 3-channel image space through a 3×3 convolution to obtain the reconstruction results at each scale, with the output of the original scale serving as the final derained image.
[0040] During the optimization training process, after the final output of the 3-channel image, multiple loss constraints are introduced, including Charbonnier loss, frequency pyramid loss, edge loss, and total variation smoothing loss. The total loss adopts a four-loss joint optimization mechanism, which optimizes both the global structure and local details.
[0041] The Charbonnier loss formula is as follows:
[0042] (4)
[0043] Among them, L recon For Charbonnier loss, I pred For the rain-removed image output by the model, I gt This is a true image without rain. It is a smoothing constant, usually taken as... , Indicates the total number of pixels in the image;
[0044] The frequency pyramid loss formula is as follows:
[0045] (5)
[0046] Among them, L freq It is the frequency domain pyramid loss, F(·) is the Fast Fourier Transform (FFT), ||·||1 is the L1 norm of the frequency domain amplitude spectrum, and I pred For the rain-removed image output by the model, I gt This is a true image without rain;
[0047] The edge loss formula is as follows:
[0048] (6)
[0049] Among them, Ledge It is the edge loss, ||·||1 is the L1 norm of the frequency domain amplitude spectrum, It is a gradient operator;
[0050] The formula for the total variation smoothing loss is as follows:
[0051] (7)
[0052] Among them, L tv It is the total variation smoothing loss. For the horizontal and vertical directions, I is used to calculate the intensity difference between the current pixel and the pixels to its right and below;
[0053] The total loss is optimized using a four-fold loss joint optimization mechanism, as shown in the following formula:
[0054] (8).
[0055] Compared with the prior art, the present invention has the following advantages:
[0056] This invention introduces a Frequency Domain Detail Preservation (FDM) module and an Edge-Guided Cross-Scale Attention Fusion (ECAfu) unit within a multi-scale Transformer framework, achieving effective suppression of rain streaks and high-quality restoration of image details in complex scenes. Specific beneficial effects are as follows:
[0057] (1) Frequency domain decomposition and edge guidance enhancement: This invention uses Fourier transform to perform frequency domain decomposition on the input image, distinguishes low-frequency background from high-frequency details through low-pass filtering, and combines the mask generated by edge detection to retain only the high-frequency components related to the structure, thereby effectively suppressing high-frequency interference of rain patterns and improving the restoration of edges and textures.
[0058] (2) Multi-scale Transformer modeling capability: The layered Transformer codec structure is adopted to encode and decode features at different scales. The coarse scale is responsible for global structure modeling, while the fine scale focuses on local details, thus achieving a balance between global consistency and local refinement.
[0059] (3) Cross-scale information interaction and fusion: The ECAfu module is used to enhance features within the same scale and transfer the enhanced features to adjacent scales, realizing a dual optimization mechanism of "same-scale enhancement - cross-scale transfer", which effectively reduces rain streak residue and loss of details.
[0060] (4) Joint constraint of multiple loss functions: During the training process, this invention combines Charbonnier loss, frequency domain pyramid loss, edge loss and total variation smoothing loss to comprehensively optimize the model performance at the pixel level, structural level and perceptual level, significantly improving the robustness and visual naturalness of the rain removal results.
[0061] In summary, this invention maintains computational efficiency while taking into account the modeling capabilities of both global and local features. Compared with existing methods, it can better adapt to diverse and complex rain pattern scenarios, thus improving the generalization and practicality of image deraining tasks. Attached Figure Description
[0062] Figure 1 Here is a flowchart of an image deraining method based on multi-scale Transformer;
[0063] Figure 2 This is an overall framework diagram of the present invention;
[0064] Figure 3 This is a structural diagram of the ECAfu module. Detailed Implementation
[0065] To gain a deeper understanding of this invention, we will provide a comprehensive and detailed description. However, this invention has various implementations and is not limited to the specific examples listed herein. These examples are presented to enhance a full understanding of the disclosure of this invention.
[0066] A multi-scale Transformer-based image deraining method, comprising the following steps:
[0067] Step 1: Construct an image deraining model that includes a multi-scale downsampling module, a frequency domain detail preservation module, a Transformer encoding and decoding module, a multi-scale enhancement module, and an image reconstruction and loss optimization module. Input the rainy image into the model and use bilinear interpolation to perform multi-scale downsampling on the image to obtain a multi-scale image representation.
[0068] The multi-scale downsampling module in step 1: inputs the original image into the multi-scale downsampling module and uses bilinear interpolation to downsample the image to obtain the sampled image;
[0069] Frequency domain detail preservation module: The feature map is decomposed into frequency domain components by Fourier transform to generate amplitude spectrum and phase spectrum. The frequency components are calculated by full-image integration, as shown in formula (1). Low-pass filtering is used to separate low-frequency and high-frequency components. The Sobel convolution operator is used to extract image edge information to generate a single-channel edge map. The edge map is then transformed into a multi-channel mask in the range of 0-1 through mapping and Sigmoid function. Finally, the mask is used to filter high-frequency details related to the edge and the low-frequency background is fused with the enhanced high-frequency details in the convolutional layer.
[0070] (1)
[0071] in, Representing the spatial domain image in coordinates Pixel value at that location, The frequency coordinates in the frequency domain are represented as The complex value of, Indicates image resolution, exponential term Represent the basis functions of the complex sine wave;
[0072] The Transformer encoding / decoding module is used to model rain streaks of different sizes. Since the pyramid-style downsampling structure constructs multi-level feature representations, the Transformer encoding / decoding module focuses on the overall structure at low resolution and on detailed information at high resolution when extracting features. It combines features from different scales to integrate information and enhance the understanding of complex scenes. The global attention mechanism handles large-scale dependencies, while local attention handles details. The Self-Attention mechanism in the Transformer encoding / decoding module can search for information at different scales and then complement it with cross-scale connections to generate better rain removal results.
[0073] Multi-scale enhancement module ECAfu: Introduces a cross-attention mechanism at the same scale, selectively retaining high-frequency information through the mask generated by edge detection, thereby enhancing the contours and details related to image structure while suppressing interference components related to rain patterns. Unlike traditional simple feature splicing or weighting, the multi-scale enhancement module ECAfu introduces an attention mechanism and edge guidance during feature fusion, combined with the edge mask generated by the Sobel operator, making the model more efficient and accurate in fusing global information provided by the coarse scale and local information provided by the fine scale. Finally, the features output by the multi-scale enhancement module ECAfu have both global consistency and local fineness, thereby improving edge fidelity and texture clarity in the rain removal task, ensuring the naturalness and credibility of the generated image. The formula for the attention mechanism is Equation (2), and the formula for the edge mask is Equation (3).
[0074] (2)
[0075] (3)
[0076] In the formula, It is a query vector generated from a certain scale feature. It is a key vector generated from features at another scale. It is a value vector generated from coarse-scale features. It is the dimension scaling factor of the key vector (to prevent the dot product from becoming too large). It is a 0 / 1 binary mask generated by the Sobel operator. It is Hadamaji.
[0077] Image reconstruction and loss optimization module: used to map the enhanced deep features back to the image space to generate rain-removed images, and guide the optimization of model parameters through multi-dimensional constraints;
[0078] The feature fusion and reconstruction unit in the image reconstruction and loss optimization module fuses cross-scale enhanced features and projects high-dimensional features into multi-scale RGB image outputs. The multi-dimensional loss constraint unit in the image reconstruction and loss optimization module is composed of Charbonnier loss, frequency pyramid loss, edge loss and total variation smoothing loss. It jointly constrains the training process of the model from multiple dimensions such as pixel accuracy, frequency domain consistency, edge texture and spatial smoothness, to ensure that the final generated image maintains structural integrity and detail clarity while removing rain patterns.
[0079] The input is a raw rain-affected image, 256×256 pixels. In this module, bilinear interpolation is first used to downsample the image, resulting in two versions at different scales: 128×128 and 64×64. During downsampling, the image resolution gradually decreases, compressing high-frequency details and leading to greater visual blur. Although some detail is lost, this scale reduction helps the model acquire complementary information at different levels: the coarser scale focuses on global structure and background information, while the finer scale retains more local texture and detail features. Through parallel extraction and fusion of multi-scale features, the model can better balance overall structure and local details while removing rain streaks. The final output images at each scale maintain an RGB three-channel structure for further processing and fusion in subsequent modules.
[0080] Step 2: Input the multi-scale image into the frequency domain detail preservation module to extract and enhance high-frequency detail features in the frequency domain;
[0081] The specific operation of step 2 is as follows:
[0082] Step 2.1: Input the multi-scale image into the frequency domain detail preservation module. Transform the image from the spatial domain to the frequency domain through Fourier transform to obtain the amplitude spectrum and phase spectrum. Use low-pass filtering to separate the low-frequency and high-frequency components. The low-frequency part retains the overall structure, mainly including the background, contour and smooth area, while the high-frequency part includes rain patterns and detail information.
[0083] Step 2.2: By using edge-aware gating in the frequency domain detail preservation module, the edge information of the image is extracted using Sobel convolution, the single-channel edge map is mapped back to the multi-channel, and a 0~1 mask is obtained by performing Sigmoid, thereby filtering out high-frequency details related to the edge and suppressing rain pattern interference.
[0084] Step 2.3: The low-frequency components are fused with the edge-enhanced high-frequency components in a convolutional layer. After edge-aware gating, the high-frequency components remove rain pattern interference, retaining only texture information related to image details. The output is a 48-channel feature representation. The outputs at all three scales are 48-channel, providing richer-detailed input for the subsequent Transformer encoder.
[0085] Step 3: Model the local and global relationships of multi-scale features through convolutional feature extraction and Transformer encoding / decoding modules;
[0086] The specific operation of step 3 is as follows:
[0087] When performing convolution to extract local features, the input is the image features enhanced by the frequency domain detail preservation module. These features are then concatenated channel by channel with the image features from the previous scale that have been enlarged to the current scale by bilinear interpolation. Subsequently, the number of channels is adjusted back to the target number of channels through a convolutional network, thus completing the initial feature fusion.
[0088] The preliminarily fused features are then input into the encoder-decoder module of the Transformer encoder-decoder module. The encoder implemented in this method adopts a hierarchical Transformer structure, with corresponding encoders and decoders configured for different scales. There is one encoder and decoder for small scales, two encoders and decoders for medium scales, and three encoders and decoders for large scales. The encoder extracts global context information through step-by-step downsampling, and the decoder recovers spatial structure and detail information through step-by-step upsampling. The encoder consists of a first encoding unit, a first downsampling unit, a second encoding unit, and a second downsampling unit. After the second encoding unit, the code enters the bottleneck layer, which aggregates the deep features extracted by the downsampling unit. Correspondingly, the decoder consists of a first upsampling unit, a first decoding unit, a second upsampling unit, and a second decoding unit.
[0089] Taking the intermediate scale feature map of this method as an example, the shape is (1, 48, 128, 128). Its features are stitched together channel by channel with the image that has been enlarged to the size of the previous scale by bilinear interpolation. The shape becomes (1, 96, 128, 128). The number of channels is adjusted back to the target number of channels through a convolutional network, and the shape is restored to (1, 48, 128, 128).
[0090] The data is then input into the encoding / decoding module. This method employs a hierarchical Transformer structure for the encoder: one encoder and decoder for small-scale operations, two encoders and decoders for medium-scale operations, and three encoders and decoders for large-scale operations. The encoder extracts global context information through progressive downsampling, while the decoder recovers spatial structure and detail information through progressive upsampling. The encoder consists of a first encoding unit, a first downsampling unit, a second encoding unit, and a second downsampling unit. After the second encoding unit, the data enters the bottleneck layer, as this layer contains the deepest features extracted after the second downsampling unit. Similarly, the decoder consists of a first upsampling unit, a first decoding unit, a second upsampling unit, and a second decoding unit. Data enters the bottleneck layer after passing through one encoder; if it passes through a second encoder, it enters the second bottleneck layer.
[0091] Specifically, taking the intermediate-scale feature map of this method as an example, with a shape of (1, 48, 128, 128), it is used as the input of the Encoder. First, feature encoding consisting of several Transformer blocks is performed, namely the first encoding unit. The spatial resolution and number of channels of the input remain unchanged, thereby modeling the correlation between features within the local receptive field.
[0092] Subsequently, the feature map is downsampled by the first downsampling unit to halve the resolution and increase the number of channels, with a shape of (1, 96, 64, 64) to expand the receptive field and improve the feature representation capacity.
[0093] The second encoding unit stacks several Transformer blocks again at the downsampled scale to extract scale context information. Then, the resolution is further reduced and the number of channels is increased by the second downsampling unit, with a shape of (1, 192, 32, 32), and the features are then fed into the bottleneck layer.
[0094] The bottleneck layer consists of several Transformer blocks; the bottleneck layer information is used for subsequent processing. While maintaining the current scale, the Transformer blocks of each layer of the encoder use multi-head self-attention and feedforward networks to jointly model long- and short-range dependencies. This effectively suppresses rain ripple interference and preserves scene structure information while progressively reducing spatial resolution and gradually improving semantic abstraction.
[0095] Next, we proceed to the decoder structure, which consists of alternating multi-level decoding units and upsampling units to achieve progressive resolution recovery. Taking the intermediate-scale feature map of this method as an example, the low-resolution, high-semantic features output by the bottleneck layer are first upsampled, i.e., the first upsampling unit, which doubles the spatial resolution and correspondingly reduces the number of channels, resulting in a shape of (1, 96, 64, 64). The decoder performs feature concatenation with the feature map at the same level as the encoder at this scale, resulting in a shape of (1, 192, 64, 64). Through the concatenation operation, the local details retained at the encoder end are fused with the global semantics gradually recovered at the decoder end. Then, a 1×1 convolution is used to reduce the number of channels without changing its size, resulting in (1, 96, 64, 64), which is then sent to the first decoding unit. This decoding unit consists of several Transformer blocks, used to further refine the global feature representation while maintaining the current scale. Building upon this, the output features are further upsampled by a second upsampling unit to restore their resolution to near the scale of the original input, with a shape of (1, 48, 128, 128). Similarly, channel concatenation is performed to achieve a shape of (1, 96, 128, 128), followed by a 1×1 convolution to achieve the same shape. These features are then fed into a second decoding unit, where they are again decoded using a Transformer block, resulting in a shape of (1, 48, 128, 128). This process enhances semantic consistency and repairs local textures at a higher resolution. Finally, the features enter a convolutional reconstruction module, where a 3×3 convolution yields an output without affecting the number of channels or the resolution.
[0096] Step 4: Introduce the ECAfu multi-scale enhancement module into the bottleneck layer, combining edge perception and channel attention to enhance rain removal features;
[0097] Multi-scale enhancement modules: such as Figure 3 As shown, its main function is to realize the interaction and fusion of features across scales, and to enhance useful high-frequency details and suppress rain ripple noise by guiding edge information. In the embodiments of the present invention, an edge-guided cross-attention fusion unit (ECAfu) is set for the multi-path bottleneck layer features at the same resolution scale.
[0098] Specifically, in the intermediate scale branch, first and second bottleneck layer features are obtained, both with the same spatial resolution and a shape of (1, 192, 32, 32). These two bottleneck layer features are input into the ECAfu module, using this scale feature as a query vector to guide the attention mapping of another feature. Combined with edge masks, high-frequency details are filtered to obtain the fused intermediate scale enhanced feature, with an output shape of (1, 192, 32, 32), without changing its channel number or spatial resolution. Further, this feature is subjected to bilinear interpolation to reduce its spatial resolution, allowing it to be added to the smallest scale bottleneck layer, resulting in a shape of (1, 192, 16, 16). This "intra-scale enhancement—cross-scale transfer" strategy allows for the suppression of rain patterns and enhancement of the global structure, followed by the supplementation of optimized feature information to the coarse scale. This helps the fine-scale branch recover more realistic texture details, significantly improving the overall rain removal effect.
[0099] In this architecture, the input consists of two bottleneck layers of the same scale, with attention calculation, edge guidance, feature fusion, and residuals in between. The output is an enhanced feature map of the same scale without changing its resolution or size.
[0100] Step 5: Use the image reconstruction and loss optimization module to perform convolutional reconstruction on the enhanced rain-removing features, output multi-scale rain-removing image results, and use multiple loss functions for optimization to obtain the rain-removed image.
[0101] The input consists of feature maps output from different scale enhancement modules. Taking the intermediate scale feature map of this method as an example, the shape is (1, 48, 128, 128), and it is added element-wise with the upsampled result of the previous scale to achieve cross-scale information fusion. Finally, it is mapped to the 3-channel image space through 3×3 convolution to obtain the reconstruction results of each scale, where the output of the original scale is used as the final rain-removed image.
[0102] During the optimization training process, after the final output of the 3-channel image, multiple loss constraints were introduced, including Charbonnier loss to enhance pixel-level robustness, frequency pyramid loss, edge loss to suppress high-frequency rain streak interference, and total variation smoothing loss to improve visual perception quality. The total loss adopts a four-loss joint optimization mechanism, which optimizes both the global structure and local details to ensure the clarity and naturalness of the rain-removed image.
[0103] The Charbonnier loss formula is as follows:
[0104] (4)
[0105] Among them, L reconFor Charbonnier loss, I pred For the rain-removed image output by the model, I gt This is a true image without rain. It is a smoothing constant, usually taken as... , This represents the total number of pixels in the image; as a robust variant of L1 loss, it smooths large errors through the square root function, effectively suppressing outlier interference. Its role is to reduce oversmoothing compared to MSE loss in pixel reconstruction of rainstorm areas, preserving more detail and texture.
[0106] The frequency pyramid loss formula is as follows:
[0107] (5)
[0108] Among them, L freq It is the frequency domain pyramid loss, F(·) is the Fast Fourier Transform (FFT), ||·||1 is the L1 norm of the frequency domain amplitude spectrum, and I pred For the rain-removed image output by the model, I gt As a real, rainless image, its function is to capture the frequency domain differences of rain streaks / structure / texture in layers, and to measure the difference between the predicted image and the real image in the frequency domain space at multiple scales.
[0109] The edge loss formula is as follows:
[0110] (6)
[0111] Among them, L edge It is the edge loss, ||·||1 is the L1 norm of the frequency domain amplitude spectrum, It is the gradient operator, I pred For the rain-removed image output by the model, I gt The image is a real, rain-free image, used to measure the similarity between the predicted image and the real image in terms of edge structure.
[0112] The formula for the total variation smoothing loss is as follows:
[0113] (7)
[0114] Among them, L tv It is the total variation smoothing loss. For the horizontal and vertical directions, I is used to calculate the intensity difference between the current pixel and the pixels to the right and below; its function is to eliminate stripe artifacts and enhance visual smoothness, which is equivalent to an implicit low-pass filter regularization term.
[0115] The total loss adopts a four-fold loss joint optimization mechanism, as shown in the following formula: it can supervise and constrain the model simultaneously from multiple levels and multiple perspectives, thereby significantly improving the visual quality, sharpness, detail preservation ability and realism of the rain-removed image.
[0116] (8)
[0117] As shown in Table 1, this invention selected the Rain200L dataset for testing and compared it with excellent algorithms in recent years, namely: GMM, DDN, RESCAN, PReNet, MSPFN, RCDNet, and MPRNet. The data in the table shows that, compared with other methods, this invention improves PSNR by 0.11 dB and SSIM by 0.0031 dB. The comparison results demonstrate that this method not only removes rain streaks but also makes images clearer and preserves more details.
[0118] Table 1. Quantitative results of PSNR and SSIM metrics on synthetic datasets.
[0119] Method PSNR SSIM GMM 28.66 0.8652 DDN 34.68 0.9671 RESCAN 36.09 0.9697 PReNet 37.80 0.9814 MSPFN 38.58 0.9827 MPRNet 39.47 0.9825 Ours 39.58 0.9856
[0120] Contents not described in detail in this specification are prior art known to those skilled in the art. Although illustrative specific embodiments of the invention have been described above to facilitate understanding by those skilled in the art, it should be understood that the invention is not limited to the scope of the specific embodiments. Various modifications are readily apparent to those skilled in the art as long as they fall within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of this invention are protected.
Claims
1. An image deraining method based on multi-scale Transformer, characterized in that, The method includes the following steps: Step 1: Construct an image deraining model that includes a multi-scale downsampling module, a frequency domain detail preservation module, a Transformer encoding and decoding module, a multi-scale enhancement module, and an image reconstruction and loss optimization module. Input the rainy image into the model and use bilinear interpolation to perform multi-scale downsampling on the image to obtain a multi-scale image representation. Step 2: Input the multi-scale image into the frequency domain detail preservation module to extract and enhance high-frequency detail features in the frequency domain; Step 3: Model the local and global relationships of multi-scale features through convolutional feature extraction and Transformer encoding / decoding modules; Step 4: Introduce the ECAfu multi-scale enhancement module into the bottleneck layer, combining edge perception and channel attention to enhance rain removal features; Step 5: Use the image reconstruction and loss optimization module to perform convolutional reconstruction on the enhanced rain-removing features, output multi-scale rain-removing image results, and use multiple loss functions for optimization to obtain the rain-removed image.
2. The image deraining method based on multi-scale Transformer according to claim 1, characterized in that, The multi-scale downsampling module in step 1: inputs the original image into the multi-scale downsampling module and uses bilinear interpolation to downsample the image to obtain the sampled image; Frequency domain detail preservation module: The feature map is decomposed into frequency domain components by Fourier transform to generate amplitude spectrum and phase spectrum. The frequency components are calculated by full-image integration, as shown in formula (1). Low-pass filtering is used to separate low-frequency and high-frequency components. The Sobel convolution operator is used to extract image edge information to generate a single-channel edge map. The edge map is then transformed into a multi-channel mask in the range of 0-1 through mapping and Sigmoid function. Finally, the mask is used to filter high-frequency details related to the edge and the low-frequency background is fused with the enhanced high-frequency details in the convolutional layer. （1） in, Represents the spatial domain image in coordinates Pixel value at that location, The frequency coordinates in the frequency domain are represented as The complex value of, Indicates image resolution, exponential term Represent the basis functions of the complex sine wave; Transformer encoding / decoding module: used to model rain patterns of different sizes. When extracting features, the Transformer encoding / decoding module focuses on the overall structure at low resolution and on detailed information at high resolution. It combines features at different scales to integrate information and enhance the understanding of complex scenes. The global attention mechanism handles large-scale dependencies, while the local attention mechanism handles details. Multi-scale enhancement module ECAfu: In the feature fusion process, attention mechanism and edge guidance are introduced. Combined with the edge mask generated by Sobel operator, the features output by multi-scale enhancement module ECAfu have both global consistency and local fineness. The formula for attention mechanism is Equation (2), and the formula for edge mask is Equation (3). （2） （3） In the formula, It is a query vector generated from a certain scale feature. It is a key vector generated from features at another scale. It is a value vector generated from coarse-scale features. It is the scaling factor for the dimension of the key vector. It is a 0 / 1 binary mask generated by the Sobel operator. It is a Hadama pile; Image reconstruction and loss optimization module: used to map the enhanced deep features back to the image space to generate rain-removed images, and guide the optimization of model parameters through multi-dimensional constraints; The feature fusion and reconstruction unit in the image reconstruction and loss optimization module fuses cross-scale enhanced features and projects high-dimensional features into multi-scale RGB image outputs. The multi-dimensional loss constraint unit in the image reconstruction and loss optimization module is composed of Charbonnier loss, frequency pyramid loss, edge loss and total variation smoothing loss, which jointly constrain the training process of the model from multiple dimensions.
3. The image deraining method based on multi-scale Transformer according to claim 2, characterized in that, The specific operation of step 2 is as follows: Step 2.1: Input the multi-scale image into the frequency domain detail preservation module, transform the image from the spatial domain to the frequency domain through Fourier transform to obtain the amplitude spectrum and phase spectrum, and use low-pass filtering to separate the low-frequency and high-frequency components, while preserving the overall structure of the low-frequency part; Step 2.2: By using the edge-aware gating in the frequency domain detail preservation module, the edge information of the image is extracted using Sobel convolution, the single-channel edge map is mapped back to the multi-channel, and a 0~1 mask is obtained by performing Sigmoid to filter out the high-frequency details related to the edge. Step 2.3: Fuse the low-frequency components with the edge-enhanced high-frequency components in the convolutional layer, and output a high-channel feature representation to provide input for the Transformer encoding and decoding module.
4. The image deraining method based on multi-scale Transformer according to claim 3, characterized in that, The specific operation of step 3 is as follows: Step 3: Model the local and global relationships of multi-scale features through convolutional feature extraction and Transformer encoding / decoding modules; When performing convolution to extract local features, the input is the image features enhanced by the frequency domain detail preservation module. These features are then concatenated channel by channel with the image features from the previous scale that have been enlarged to the current scale by bilinear interpolation. Subsequently, the number of channels is adjusted back to the target number of channels through a convolutional network, thus completing the initial feature fusion. The initially fused features are then input into the encoder-decoder module in the Transformer encoder-decoder module. The encoder extracts global context information through step-by-step downsampling, and the decoder recovers spatial structure and detail information through step-by-step upsampling. The encoder consists of a first encoding unit, a first downsampling unit, a second encoding unit, and a second downsampling unit. After the second encoding unit, the bottleneck layer is entered, which gathers the deep features extracted by the downsampling unit. The decoder consists of a first upsampling unit, a first decoding unit, a second upsampling unit, and a second decoding unit; The specific operation is as follows: the feature map is used as the input of the encoder. First, the first encoding unit, which consists of several Transformer blocks, is executed. The spatial resolution and number of channels of the input remain unchanged. The correlation between features is modeled in the local receptive field. Subsequently, the feature map is downsampled by the first downsampling unit, and the second encoding unit stacks several Transformer blocks again on the downsampled scale to extract the scale context information. The resolution is reduced and the number of channels is increased by the second downsampling unit, and then the features are sent to the bottleneck layer. The bottleneck layer consists of several Transformer blocks. The Transformer blocks of each layer of the encoder, while maintaining the current scale, use multi-head self-attention and feedforward network to complete the joint modeling of long and short-range dependencies. Next, we proceed to the decoder structure, which consists of alternating multi-level decoding units and upsampling units. The low-resolution, high-semantic features output from the bottleneck layer are first upsampled. The decoder performs feature concatenation with the feature maps at the same level as the encoder at this scale. Through concatenation, the local details preserved at the encoder end are fused with the global semantics gradually recovered at the decoder end. Then, the number of channels is reduced by convolution and sent to the first decoding unit, which consists of several Transformer blocks. Based on this, the output features are further amplified in resolution by the second upsampling unit to restore them to a scale close to the original input. Channel concatenation and convolution are also performed to adjust the number of channels before being sent to the second decoding unit. The features are decoded again by Transformer blocks to enhance semantic consistency and repair local textures. Finally, the features enter the convolutional reconstruction module, where convolution yields the final output result.
5. The image deraining method based on multi-scale Transformer according to claim 4, characterized in that, The specific operation of step 4 is as follows: Step 4: Introduce the ECAfu multi-scale enhancement module into the bottleneck layer, combining edge perception and channel attention to enhance rain removal features; Multi-scale enhancement module: Enables interaction and fusion of features between scales, and enhances useful high-frequency details and suppresses rain ripple noise by guiding edge information. For multi-bottleneck layer features at the same resolution scale, an edge-guided cross-attention fusion unit ECAfu is set up. On the same scale branch, the first bottleneck layer features and the second bottleneck layer features with the same spatial resolution are obtained respectively. These two features are input into the multi-scale enhancement module ECAfu. One of the features is used as a query vector to guide the attention mapping of the other feature. The high-frequency details are filtered by combining edge masks to obtain the fused enhanced features. The enhanced features after fusion are subjected to bilinear interpolation to reduce their spatial resolution, enabling them to be added element-wise with the coarse-scale bottleneck layer features.
6. The image deraining method based on multi-scale Transformer according to claim 5, characterized in that, The specific operation of step 5 is as follows: The input is the feature maps output from different scale enhancement modules, which are added element-wise to the upsampled result of the previous scale to achieve cross-scale information fusion. Finally, the image is mapped to a 3-channel image space through a 3×3 convolution to obtain the reconstruction results at each scale, with the output of the original scale serving as the final derained image. During the optimization training process, after the final output of the 3-channel image, multiple loss constraints are introduced, including Charbonnier loss, frequency pyramid loss, edge loss, and total variation smoothing loss. The total loss adopts a four-loss joint optimization mechanism, which optimizes both the global structure and local details. The Charbonnier loss formula is as follows: （4） Among them, L recon For Charbonnier loss, I pred For the rain-removed image output by the model, I gt This is a true image without rain. It is a smoothing constant, usually taken as... , Indicates the total number of pixels in the image; The frequency pyramid loss formula is as follows: （5） Among them, L freq It is the frequency domain pyramid loss, F(·) is the Fast Fourier Transform (FFT), ||·||1 is the L1 norm of the frequency domain amplitude spectrum, and I pred For the rain-removed image output by the model, I gt This is a true image without rain; The edge loss formula is as follows: （6） Among them, L edge It is the edge loss, ||·||1 is the L1 norm of the frequency domain amplitude spectrum, It is a gradient operator; The formula for the total variation smoothing loss is as follows: （7） Among them, L tv It is the total variation smoothing loss. For the horizontal and vertical directions, I is used to calculate the intensity difference between the current pixel and the pixels to its right and below; The total loss is optimized using a four-fold loss joint optimization mechanism, as shown in the following formula: （8）。