Image snow removal method based on transformer modeling and multi-scale dynamic filtering

Abstract

The application discloses an image snow removal method based on a Transformer modeling and multi-scale dynamic filtering, and comprises the following steps: a data set containing a snow scene image and a corresponding snow-free image is established; an image snow removal network fusing global Transformer modeling and multi-scale dynamic filtering is constructed, and the image snow removal network is denoted as TranFusionNet; the TranFusionNet is trained by using the data set until a preset loss function converges; and a to-be-removed snow image is processed by using the trained image snow removal network to obtain a snow removal result. The application realizes deep fusion of global dependence modeling and local detail enhancement, achieves a good balance between structure consistency maintenance and detail recovery, can effectively remove complex snow particle interference, and reconstructs a clear and natural visual image, thereby providing an efficient solution for the image recovery field.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to an image desnowing method based on Transformer modeling and multi-scale dynamic filtering. Background Technology

[0002] Image restoration is an important research direction in computer vision, with wide practical value in applications such as intelligent surveillance, autonomous driving, remote sensing mapping, and computational photography. Snow in the natural environment significantly impacts image quality, producing randomly distributed snow particles, snow fog, and brightness interference, leading to problems such as structural degradation, texture blurring, and color deviation, severely affecting subsequent recognition, detection, and analysis tasks. The goal of image desnowing is to recover a clear, snow-free, realistic scene from images contaminated with snow particles. However, the randomness, multi-scale nature, and semi-transparent overlay characteristics of snow particle distribution make image desnowing a challenging task.

[0003] In recent years, with the development of deep learning technology, image snow removal methods based on Convolutional Neural Networks (CNNs) have made significant progress. Typical structures such as U-Net and ResNet achieve multi-scale feature extraction and image reconstruction through multiple layers of convolution and residual connections. Although these methods can effectively restore local textures, they are limited by the local receptive field of fixed convolutional kernels, making it difficult to fully capture global dependencies and resulting in insufficient restoration capabilities for large-scale snow fog or uneven brightness areas. In addition, traditional CNNs lack adaptability when dealing with multi-scale and non-uniform snow particle distributions, making it difficult to simultaneously maintain high-frequency details and low-frequency structural consistency, easily leading to detail loss or over-smoothing.

[0004] To overcome the limitations of CNNs, the Transformer architecture has been introduced into image desnow removal tasks. Its global self-attention mechanism can capture long-range dependencies and improve structural consistency. However, the standard Transformer has high computational complexity and insufficient ability to model local texture details, often resulting in texture blurring or artifacts in areas with dense snow particles and high-frequency details. Meanwhile, while some existing CNN-Transformer fusion methods attempt to consider both local and global features, they lack adaptive modeling capabilities for snow particle distributions at different scales, and their feature fusion methods are simplistic, leading to insufficient utilization of multi-scale contextual information. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides an image desnowing method based on Transformer modeling and multi-scale dynamic filtering, which features a simple algorithm and good desnowing effect.

[0006] The technical solution of this invention to solve the above-mentioned technical problems is: an image desnowing method based on Transformer modeling and multi-scale dynamic filtering, comprising the following steps:

[0007] Step 1: Create a dataset A containing snow scene images and corresponding snowless images;

[0008] Step 2: Construct an image desnowing network that integrates global Transformer modeling and multi-scale dynamic filtering. The image desnowing network is denoted as TranFusionNet.

[0009] TranFusionNet uses an encoder-decoder structure with skip connections as its overall framework. The encoder consists of three layers connected sequentially: the first layer has four residual blocks, the second layer has two coding blocks with star modules, and the third layer is an enhancement coding block. The enhancement coding block incorporates the residual structure of a lightweight Transformer. The Transformer in the enhancement coding block consists of a Local Window Attention (LSA) mechanism and a gated feedforward network (GFFN).

[0010] Step 3: Train TranFusionNet using dataset A until the pre-set loss function converges;

[0011] Step 4: Use the trained image desnowing network to process the image to be desnowed to obtain the desnowing result.

[0012] In the above-mentioned image desnowing method based on Transformer modeling and multi-scale dynamic filtering, in step two, the decoder and encoder are symmetrically constructed, and the decoder structure is the same as the encoder. The second and third layers of the decoder pass the multi-scale features of the first and second layers of the encoder to the corresponding decoding layer for feature fusion through skip connections, so as to obtain the high-dimensional features inside the decoder. The third layer of the decoder is a multi-scale bottleneck module between the encoder and the decoder. Finally, a 3×3 convolution is used to map the fused high-dimensional features back to the image space and output the desnowed image.

[0013] In the above-mentioned image desnowing method based on Transformer modeling and multi-scale dynamic filtering, in step two, in the star module, firstly, layer normalization (LN) is performed on the input feature F1, then spatial local features are extracted through a 3×3 depthwise separable convolution, and then channel recombination is completed through two 1×1 convolutions. Next, group normalization is performed on one of the convolution results to stabilize the feature distribution, and nonlinear enhancement of expressive power is introduced through the GELU activation function. Then, after multiplying with the other convolution result, a 1×1 convolution is performed for channel fusion and mapping, thereby obtaining the enhanced feature that integrates local structural information. Finally, the enhanced feature is added element-wise with the input feature F1 to form a residual connection to obtain feature F2, thereby realizing feature reuse and gradient stable propagation.

[0014] The image desnowing method based on Transformer modeling and multi-scale dynamic filtering described above, in step two, the operations in the enhancement coding block are as follows: First, the layer normalization performed on feature F2 is standardized, and query vector Q, key vector K, and value vector V are generated. Self-attention mapping within each window is calculated through multi-head attention. Then, the standardized features are input into the window translation module windows-shitf. In the window translation module windows-shitf, the standardized features are first divided according to the specified window, and then attention output is obtained through linear projection. Subsequently, the attention output is added to feature F2 to form a residual connection to obtain feature F3. Feature F3 is input into a gated feedforward network. In the gated feedforward network, feature F3 is first enlarged by 1×1 convolution, and then local features are extracted by 3×3 depthwise separable convolution. The convolution output is split into two parts along the channel dimension. One part is activated by GELU as a nonlinear transformation, and the other part is used as a gate signal and multiplied element-wise with the former. Finally, the dimensionality is reduced by 1×1 convolution to obtain output feature F4. Output feature F4 is added to the residual of feature 3.

[0015] In the above-mentioned image desnowing method based on Transformer modeling and multi-scale dynamic filtering, in step two, the multi-scale bottleneck module adopts a parallel design of multiple scale branches, and extracts feature representations under different receptive fields through pooling operations at different scales; after upsampling and alignment of the outputs of all scale branches, channel fusion is performed by 1×1 convolution, and finally a feature representation that integrates multi-scale context and global dependency is obtained, which is then fed into the decoder.

[0016] In the above-mentioned image desnowing method based on Transformer modeling and multi-scale dynamic filtering, in step two, the multi-scale bottleneck module adopts three scale branches, which perform 8×8, 4×4 and 2×2 average pooling on the input features to obtain downsampled features at different scales; the features of each scale branch are independently processed by an adaptive dynamic filtering block (ADFB) to extract rich local and channel information.

[0017] In the above-mentioned image desnowing method based on Transformer modeling and multi-scale dynamic filtering, step two involves ADFB, a dual-branch feature enhancement unit comprising a first branch and a second branch. The first branch utilizes a Dynamic Filter to extract locally variable convolutional features. By changing the value of the dynamic convolution kernel in the Dynamic Filter, it captures long-range dependent features across channels and space, obtaining vertical and horizontal features. Finally, the vertical and horizontal features are added to the input residual of the first branch and integrated through a 1×1 convolution. The second branch changes the value of the dynamic convolution kernel and simultaneously models the weights of the horizontal and vertical directions to achieve square attention extraction. Finally, the outputs of the first and second branches are added together.

[0018] The image desnowing method based on Transformer modeling and multi-scale dynamic filtering described above, in step two, the first branch extracts local variable convolutional features using a Dynamic Filter. The specific process is as follows: First, global context information of the input feature F5 is extracted using global average pooling. Then, a 1×1 convolution is used to generate dynamic convolutional kernel weights for each channel, allowing each convolutional kernel to dynamically adjust its parameters based on the global information of the input feature, thus effectively responding to the local and global structures of different inputs. Next, reflection padding and feature block unfolding operations are used to extract local feature blocks. The dynamic convolutional kernel is multiplied point-by-point with the local feature blocks and summed to obtain low-frequency features. Combine input feature F5 with low-frequency features The difference is used as a high-frequency feature Introducing channel-level learnable parameters for low-frequency features and high frequency characteristics Adaptive weighted fusion is performed separately; finally, the low-frequency and high-frequency responses are adjusted by the channel scaling parameter to achieve adaptive dynamic filtering of the input feature F5 in both spatial and channel dimensions.

[0019] In the above-described image desnowing method based on Transformer modeling and multi-scale dynamic filtering, step three uses the overall loss, composed of both spatial domain loss and frequency domain loss, as the objective function for model training. The spatial domain loss... for:

[0020] ;

[0021] in, Represents the predicted image. Represents a real image. This represents the calculation of the L1 norm between the predicted image and the ground truth image, and the result is the sum of the absolute differences between the predicted image and the ground truth image in the spatial domain;

[0022] Frequency domain loss for:

[0023] ;

[0024] in, This represents the frequency domain information obtained by Fourier transforming the predicted image. This represents the frequency domain information obtained by Fourier transforming a real image.

[0025] Total loss for:

[0026] ;

[0027] in, Indicates the number of pixels. These are the frequency domain loss weighting coefficients, used to adjust the frequency domain loss. The proportion of contribution to the overall loss.

[0028] The beneficial effects of this invention are as follows:

[0029] 1. This invention introduces a lightweight Transformer structure. Through the synergistic effect of the Local Window Attention (LSA) mechanism and the gated feedforward network (GFFN), it not only captures long-range dependencies across regions, but also maintains rich texture details in the local area. This effectively improves structural consistency and global information expression capabilities, while reducing computational complexity and memory usage.

[0030] 2. The multi-scale dynamic filtering module designed in this invention utilizes the adaptive adjustment capability of dynamic convolution kernels to model the multi-scale, non-uniform, and directional random distribution of snow particles, thereby realizing adaptive feature fusion and direction perception enhancement at different spatial scales.

[0031] 3. This invention achieves a deep fusion of global dependency modeling and local detail enhancement, achieving a good balance between maintaining structural consistency and restoring details. It can effectively remove complex snow particle interference and reconstruct clear and natural visual images, providing an efficient solution for the field of image restoration. Attached Figure Description

[0032] Figure 1 This is the overall flowchart of the present invention.

[0033] Figure 2 This is a structural diagram of the TranFusionNet image desnowing network of the present invention.

[0034] Figure 3 This is a schematic diagram of the star module structure.

[0035] Figure 4 This is a schematic diagram of the gated feedforward network of the present invention.

[0036] Figure 5 This is a schematic diagram of the local window attention mechanism of the present invention.

[0037] Figure 6 This is a schematic diagram of the adaptive dynamic filter block of the present invention.

[0038] Figure 7 This is a schematic diagram of the first image with snow.

[0039] Figure 8 This is a schematic diagram showing the snow removal process applied to the first snowy image using the method of the present invention.

[0040] Figure 9 This is a schematic diagram of the second image with snow.

[0041] Figure 10 This is a schematic diagram showing the snow removal process applied to a second snowy image using the method of the present invention. Detailed Implementation

[0042] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0043] like Figure 1 As shown, an image desnowing method based on Transformer modeling and multi-scale dynamic filtering includes the following steps:

[0044] Step 1: Create a dataset A containing snow scene images and corresponding snowless images.

[0045] Step 2: Construct an image desnowing network that integrates global Transformer modeling and multi-scale dynamic filtering. The image desnowing network is denoted as TranFusionNet.

[0046] like Figure 2As shown, TranFusionNet uses an encoder-decoder structure with skip connections as its overall framework. The encoder consists of three layers connected sequentially: the first layer has four residual blocks, the second layer has two coding blocks with a star module, and the third layer has an enhancement coding block that incorporates the residual structure of a lightweight Transformer. The decoder is constructed symmetrically to the encoder, and its structure is identical. The second and third layers of the decoder pass the multi-scale features of the first and second layers of the encoder to the corresponding decoding layers for feature fusion through skip connections, resulting in high-dimensional features within the decoder. The third layer of the decoder is connected to the encoder via a multi-scale bottleneck module. Finally, a 3×3 convolution is used to map the fused high-dimensional features back to the image space, outputting the desnowed image.

[0047] like Figure 3 As shown, in the star module, the input feature F1 is first subjected to Layer Normalization (LN), and then spatial local features are extracted through a 3×3 depthwise separable convolution (3×3DW-Conv). Channel reorganization is then performed through two 1×1 convolutions. Next, one of the convolution results is subjected to group normalization to stabilize the feature distribution, and nonlinear enhancement is introduced through the GELU activation function. After multiplying with the other convolution result, a 1×1 convolution is performed for channel fusion and mapping, thereby obtaining the enhanced feature that integrates local structural information. Finally, the enhanced feature is added element-wise with the input feature F1 to form a residual connection, resulting in feature F2, thus realizing feature reuse and gradient stable propagation.

[0048] The Transformer in the enhanced coding block consists of a Local Window Attention (LSA) mechanism and a gated feedforward network (GFFN), such as... Figure 4 , Figure 5As shown, the operations in the enhanced coding block are as follows: First, the layer normalization performed on feature F2 is standardized, and then a query vector Q, a key vector K, and a value vector V are generated by passing them through a 1×1 convolution and a 3×3 depthwise convolution (3×3DConv). The self-attention mapping within each window is calculated through multi-head attention, and each attention head generates an attention response matrix R, which is used to represent the relevance of the head at different positions within the window. Then, the standardized features are input into the window translation module windows-shitf. Only Q and K are windowed and translated, while V remains in place. In windows-shitf, the standardized features are first divided into specified windows (local translation shift can be selected). After concatenating or weighting the R values ​​of all attention heads along the channel dimension, the multi-head outputs are remapped along the channel dimension through a 1×1 convolution. Subsequently, the attention output is added to feature F2 to form a residual connection, resulting in feature F3. Feature F3 is input into a gated feedforward network. In the gated feedforward network, feature F3 is up-dimensioned through a 1×1 convolution, and then local features are extracted using a 3×3 depthwise separable convolution. The convolution output is split into two parts along the channel dimension. One part is activated by GELU as a non-linear transformation, and the other part is used as a gate signal and multiplied element-wise with the former. Finally, the dimensionality is reduced by a 1×1 convolution to obtain the output feature F4, which is then added to the residual of feature 3.

[0049] The multi-scale bottleneck module employs a parallel design with three scale branches: 8×8, 4×4, and 2×2 average pooling (AP) on the input features to obtain downsampled features at different scales, extracting feature representations under different receptive fields. The features of each scale branch are independently processed by an adaptive dynamic filtering block (ADFB) to extract rich local and channel information. The outputs of all scale branches are upsampled and aligned, then fused by 1×1 convolution to obtain a feature representation that integrates multi-scale context and global dependencies. This feature representation is then fed into the decoder.

[0050] like Figure 6 As shown, ADFB is a dual-branch feature enhancement unit, consisting of a first branch and a second branch. The first branch uses a Dynamic Filter to extract locally variable convolutional features. By changing the value of the dynamic convolution kernel in the Dynamic Filter, it captures long-range dependent features across channels and space, obtaining vertical and horizontal features. Finally, the vertical and horizontal features are added to the input residual of the first branch and integrated through a 1×1 convolution. The second branch changes the value of the dynamic convolution kernel and simultaneously models the weights of the horizontal and vertical directions to achieve square attention extraction. Finally, the outputs of the first and second branches are added together.

[0051] The first branch utilizes a Dynamic Filter to extract locally variable convolutional features. The process is as follows: First, global context information of the input feature F5 is extracted using Global Average Pooling (GAP). Then, a 1×1 convolution Conv generates dynamic convolutional kernel weights for each channel. Next, the Tanh hyperbolic tangent activation function is used for feature modulation, enabling each convolutional kernel to dynamically adjust its parameters based on the global information of the input features, thus effectively responding to different local and global input structures. Finally, reflection padding and feature block unrolling operations are used to extract local feature blocks. The dynamic convolutional kernel is then multiplied point-by-point with the local feature blocks and summed to obtain low-frequency features. Combine input feature F5 with low-frequency features The difference is used as a high-frequency feature Introducing channel-level learnable parameters for low-frequency features and high frequency characteristics Adaptive weighted fusion is performed separately; finally, the low-frequency and high-frequency responses are adjusted by the channel scaling parameter to achieve adaptive dynamic filtering of the input feature F5 in both spatial and channel dimensions.

[0052] Step 3: Train TranFusionNet using dataset A until the pre-set loss function converges. Use the overall loss, which is composed of spatial domain loss and frequency domain loss, as the objective function for model training.

[0053] Spatial domain loss for:

[0054] ;

[0055] in, Represents the predicted image. Represents a real image. This represents the calculation of the L1 norm between the predicted image and the ground truth image, and the result is the sum of the absolute differences between the predicted image and the ground truth image in the spatial domain;

[0056] Frequency domain loss for:

[0057] ;

[0058] in, This represents the frequency domain information obtained by Fourier transforming the predicted image. This represents the frequency domain information obtained by Fourier transforming a real image.

[0059] Total loss for:

[0060] ;

[0061] in, Indicates the number of pixels. These are the frequency domain loss weighting coefficients, used to adjust the frequency domain loss. The proportion of contribution to the overall loss.

[0062] Step 4: Use the trained image desnowing network to process the image to be desnowed to obtain the desnowing result.

[0063] The method of this invention was used to conduct experiments on the composite snow dataset CSD, which consists of 10,000 synthetic snow scene images. Figures 7-10 The experimental results are shown, in which Figure 7 and Figure 9 These are two original images with snow. Figure 8 and Figure 10 For comparison using the present invention Figure 7 and Figure 9 The results of the snow removal process show that the present invention can effectively remove snow streaks and snow spots from snow maps.

Claims

1. An image desnowing method based on Transformer modeling and multi-scale dynamic filtering, characterized in that, Includes the following steps: Step 1: Create a dataset A containing snow scene images and corresponding snowless images; Step 2: Construct an image desnowing network that integrates global Transformer modeling and multi-scale dynamic filtering. The image desnowing network is denoted as TranFusionNet. TranFusionNet uses an encoder-decoder structure with skip connections as its overall framework. The encoder consists of three layers connected sequentially: the first layer has four residual blocks, the second layer has two coding blocks with star modules, and the third layer is an enhancement coding block. The enhancement coding block incorporates the residual structure of a lightweight Transformer. The Transformer in the enhancement coding block consists of a Local Window Attention (LSA) mechanism and a gated feedforward network (GFFN). In the star module, the input feature F1 is first subjected to a layer normalization (LN) process. Then, a 3×3 depthwise separable convolution is used to extract local spatial features. Next, two 1×1 convolutions are used to reorganize the channels. Then, one of the convolution results is subjected to group normalization to stabilize the feature distribution. The GELU activation function is used to introduce nonlinear enhancement of expressive power. After multiplying with the other convolution result, a 1×1 convolution is performed to perform channel fusion and mapping, thereby obtaining an enhanced feature that integrates local structural information. Finally, this enhanced feature is added element-wise to the input feature F1 to form a residual connection, thus obtaining feature F2, thereby realizing feature reuse and gradient stable propagation. Step 3: Train TranFusionNet using dataset A until the pre-set loss function converges; Step 4: Use the trained image desnowing network to process the image to be desnowed to obtain the desnowing result.

2. The image desnowing method based on Transformer modeling and multi-scale dynamic filtering according to claim 1, characterized in that, In step two, the decoder and encoder are constructed symmetrically, and the decoder structure is the same as the encoder. The second and third layers of the decoder pass the multi-scale features of the first and second layers of the encoder to the corresponding decoding layer for feature fusion through skip connections, so as to obtain the high-dimensional features inside the decoder. The third layer of the decoder is a multi-scale bottleneck module between the encoder and the decoder. Finally, a 3×3 convolution is used to map the fused high-dimensional features back to the image space and output the desnowed image.

3. The image desnowing method based on Transformer modeling and multi-scale dynamic filtering according to claim 2, characterized in that, In step two, the operations in the enhanced coding block are as follows: First, the layer normalization performed on feature F2 is standardized, and query vector Q, key vector K, and value vector V are generated. Self-attention mapping within each window is calculated through multi-head attention. Then, the standardized features are input into the window translation module windows-shitf. In the window translation module windows-shitf, the standardized features are first divided according to the specified window, and then attention output is obtained through linear projection. Subsequently, the attention output is added to feature F2 to form a residual connection to obtain feature F3. Feature F3 is input into a gated feedforward network. In the gated feedforward network, feature F3 is first enlarged by 1×1 convolution, and then local features are extracted by 3×3 depthwise separable convolution. The convolution output is split into two parts along the channel dimension. One part is activated by GELU as a nonlinear transformation, and the other part is used as a gate signal and multiplied element-wise with the former. Finally, the output feature F4 is obtained by 1×1 convolution for dimensionality reduction. The output feature F4 is added to the residual of feature 3.

4. The image desnowing method based on Transformer modeling and multi-scale dynamic filtering according to claim 3, characterized in that, In step two, the multi-scale bottleneck module adopts a parallel design of multiple scale branches and extracts feature representations under different receptive fields through pooling operations at different scales. The outputs of all scale branches are upsampled and aligned, then channel-fused via 1×1 convolution to obtain a feature representation that integrates multi-scale context and global dependencies. This feature representation is then fed into the decoder.

5. The image desnowing method based on Transformer modeling and multi-scale dynamic filtering according to claim 4, characterized in that, In step two, the multi-scale bottleneck module uses three scale branches to perform 8×8, 4×4, and 2×2 average pooling on the input features to obtain downsampled features at different scales. The features of each scale branch are independently processed by an adaptive dynamic filter block (ADFB) to extract rich local and channel information.

6. The image desnowing method based on Transformer modeling and multi-scale dynamic filtering according to claim 5, characterized in that, In step two, ADFB is a dual-branch feature enhancement unit, comprising a first branch and a second branch. The first branch uses a Dynamic Filter to extract locally variable convolutional features. By changing the value of the dynamic convolution kernel in the Dynamic Filter, it captures long-range dependent features across channels and space, obtaining vertical and horizontal features. Finally, the vertical and horizontal features are added to the input residual of the first branch and integrated into the output through a 1×1 convolution. The second branch changes the value of the dynamic convolution kernel and simultaneously models the weights of the horizontal and vertical directions to achieve square attention extraction. Finally, the outputs of the first and second branches are added together and fused.

7. The image desnowing method based on Transformer modeling and multi-scale dynamic filtering according to claim 6, characterized in that, In step two, the specific process of extracting local variable convolutional features using the Dynamic Filter in the first branch is as follows: First, global context information of the input feature F5 is extracted through global average pooling, and dynamic convolutional kernel weights corresponding to each channel are generated through a 1×1 convolution, so that each convolutional kernel can dynamically adjust its own parameters according to the global information of the input feature, thereby effectively responding to the local and global structures of different inputs; then, local feature blocks are extracted using reflection padding and feature block unfolding operations, and the dynamic convolutional kernel is multiplied point by point with the local feature blocks and summed to obtain low-frequency features. Combine input feature F5 with low-frequency features The difference is used as a high-frequency feature Introducing channel-level learnable parameters for low-frequency features and high frequency characteristics Adaptive weighted fusion is performed separately; finally, the low-frequency and high-frequency responses are adjusted by the channel scaling parameter to achieve adaptive dynamic filtering of the input feature F5 in both spatial and channel dimensions.

8. The image desnowing method based on Transformer modeling and multi-scale dynamic filtering according to claim 1, characterized in that, In step three, the overall loss, composed of both spatial domain loss and frequency domain loss, is used as the objective function for model training. The spatial domain loss... for: ； in, Represents the predicted image. Represents a real image. This represents the calculation of the L1 norm between the predicted image and the real image, and the result is the sum of the absolute differences between the predicted image and the real image in the spatial domain; Frequency domain loss for: ； in, This represents the frequency domain information obtained by Fourier transforming the predicted image. This represents the frequency domain information obtained by Fourier transforming a real image. Total loss for: ； in, Indicates the number of pixels. These are the frequency domain loss weighting coefficients, used to adjust the frequency domain loss. The proportion of contribution to the overall loss.

Images