A snow day image restoration method, device and storage medium

Abstract

The application discloses a snow day image recovery method and device and a storage medium, and the method comprises the steps of constructing a multi-scale snow day image recovery network; the multi-scale snow day image recovery network comprises an encoder module based on a fast Fourier transform pyramid model, a feature fusion module and a decoder module based on a global visual transformer (GlobalViT); an input of the multi-scale snow day image recovery network is obtained by using a snow day imaging physical model, and the snow day image recovery network is trained to obtain a trained network; and the trained network is used for snow day image recovery. The application has the beneficial effects that the color deviation, the masking effect, the artifact and other problems in the snow removal process are successfully solved.

Description

Technical Field

[0001] This invention relates to the field of snow image processing, and more particularly to a method for restoring snow images. Background Technology

[0002] Snowy image restoration is one of the main types of image restoration in computer vision. Based on different snowflake sizes, it can be divided into small snowflakes, medium snowflakes, and large snowflakes.

[0003] Existing snow removal methods can be broadly categorized into three types. The first type is traditional methods, such as those using filtering and color space transformation, which are easy to implement. The second type employs an end-to-end deep learning snow removal framework. The third type is a framework for recovering snow in all weather conditions.

[0004] Existing algorithms cannot completely remove snow from different types of snowy images, especially large snowflakes. Current methods often produce blurring and artifacts, making recovery extremely difficult. Specifically, in the field of autonomous driving, falling snowflakes can obstruct the autonomous driving system's analysis of the surrounding environment, posing a significant safety hazard and easily leading to traffic accidents. Furthermore, in object detection, snowy images can hinder advanced tasks such as pedestrian detection, pedestrian falls, and pedestrian boundary crossings. Due to the irregular characteristics of snowflakes, such as their variable size and shape, the adsorption of different microparticles during snow crystal formation causing transparency issues, and the unpredictable trajectory of falling snowflakes influenced by atmospheric conditions, traditional snow removal methods rely on prior knowledge, making human factors highly influential on snow removal effectiveness. Traditional methods typically locate snowflakes individually, but due to the complex and variable nature of snowy weather, many types of snowflakes may go undetected, and the resulting clear images may suffer from distortion or color misalignment. This further complicates snowy image recovery, making it an even more challenging problem. Summary of the Invention

[0005] To address the technical problems of color deviation, occlusion effect, and artifacts in existing snow removal methods, this invention proposes a snow image restoration method, device, and storage medium. The method employs an end-to-end fusion framework FTNet that integrates Fast Fourier Transform and Global Vision Transformer (GlobalViT) under snowy weather conditions. This is an effective and novel Transformer-based image snow removal architecture, consisting of an encoder composed of convolutional neural network blocks and a decoder composed of GlobalViT.

[0006] Specifically, the method includes the following steps:

[0007] S1. Construct a multi-scale snow image restoration network; the multi-scale snow image restoration network includes: an encoder module based on the Fast Fourier Transform pyramid model, a feature fusion module, and a decoder module based on GlobalViT;

[0008] S2. Train the snowy image recovery network to obtain the trained network;

[0009] S3. Use the trained network to restore snowy images.

[0010] A storage medium storing instructions and data for implementing the snowy image restoration method.

[0011] A snow-covered image restoration device includes a processor and a storage medium; the processor loads and executes instructions and data in the storage medium to implement the snow-covered image restoration method.

[0012] The beneficial effects provided by this invention are: it designs an end-to-end framework FTNet that fuses Fast Fourier Transform and Global Vision Transformer (GlobalViT) under snowy weather conditions. This is an effective and novel Transformer-based image desnowing architecture, which consists of an encoder of convolutional neural network blocks and a decoder of GlobalViT. It successfully solves problems such as color deviation, occlusion effect and artifacts in the snow removal process. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0014] Figure 2 This is a schematic diagram of the structure of the multi-scale snow image restoration network of the present invention;

[0015] Figure 3 This is another flowchart illustrating the structure of the multi-scale snow image restoration network of this invention;

[0016] Figure 4 This is a schematic diagram of the encoder structure;

[0017] Figure 5 This is a schematic diagram of the internal structure of the decoder module;

[0018] Figure 6 This is a schematic diagram of a multi-axis gating (MAG) unit structure;

[0019] Figure 7 These are experimental results of the present invention;

[0020] Figure 8 This is a schematic diagram of the hardware device of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0022] Please refer to Figure 1 , Figure 1 This is a schematic diagram of the method flow of the present invention;

[0023] This invention provides a method for restoring snowy images, comprising:

[0024] S1. Construct a multi-scale snow image restoration network; the multi-scale snow image restoration network includes: an encoder module based on the Fast Fourier Transform pyramid model, a feature fusion module, and a decoder module based on GlobalViT;

[0025] It should be noted that the encoder module based on the fast Fourier transform pyramid model includes: a Fourier transform unit, a convolution unit, an inverse Fourier transform unit, a PPM unit, and a ReLU unit.

[0026] It should be noted that the GlobalViT-based decoder module includes: LinearQ, K, V units and a multi-axis gating (MAG) unit.

[0027] It should be noted that the LinearQ unit is replaced by a snow query module, which uses a hidden layer.

[0028] S2. Train the snowy image recovery network to obtain the trained network;

[0029] Please refer to Figure 2 and Figure 3 , Figure 2 This is a schematic diagram of the structure of the multi-scale snow image restoration network of the present invention. Figure 3 This is another flowchart illustrating the structure of the multi-scale snow image restoration network of this invention;

[0030] Figure 2 In this context, Encoderblock represents the encoder module, Decoderblock represents the decoder module, and Latentblock represents the hidden layer; Featurefusion represents feature fusion, Convolution represents convolution, and Element-wisesummation represents addition.

[0031] The process of image processing by the multi-scale snow image restoration network in step S2 is as follows:

[0032] S21. Input multi-scale snowy images to the encoder module based on the fast Fourier transform pyramid model to obtain image features at different scales;

[0033] It should be noted that the top layer of the input image is the original snowy image. After downsampling the original snowy image twice, two downsampled images at different scales are obtained. The downsampled images and the original snowy image together are called multi-scale snowy images.

[0034] S22. Input image features of different scales into the feature fusion module to obtain fused features;

[0035] S23. Input the fused features into the GlobalViT-based decoder module to obtain the output features;

[0036] S24. Add the output features to the original multi-scale image to obtain the snow-free image.

[0037] It should be noted that this is different from the previous top-down or parallel transmission of single information in the network.

[0038] The multi-scale snow image recovery network structure in this invention samples degraded snow images into snow images of various scales. This enables the accurate capture of all hidden data existing in low-quality images at different resolutions during the encoding stage.

[0039] During the downsampling process, the extracted snow feature maps are fused with these sampled multi-scale snowy images and used as input to the encoder module.

[0040] By utilizing supplementary data from degraded multi-scale images and extracted feature maps, the encoder module can effectively extract hidden, clear information from images.

[0041] In addition, the decoder also uses a multi-scale output method to recover snow-free images by combining feature information from each scale and extracting global information from feature maps generated by the network at different scales.

[0042] refer to Figure 3 ,by Figure 2 Based on the network structure, Figure 3 The process is as follows:

[0043] Based on the single-image snow removal model, three different scales of snowy and snowless images are constructed as inputs;

[0044] By leveraging the automatic learning capabilities of the Fast Fourier Pyramid module and the Global Globa iViT network, clear background information is extracted for each stage of snowy weather.

[0045] The snow features at each stage are fused using a fusion module to ultimately output clear, snow-free images at three scales.

[0046] By using frequency reconstruction loss, edge loss, and Charbonnier loss functions, the model can restore snow-free images at three scales that more closely approximate real snow-free images.

[0047] Please refer to Figure 4 , Figure 4 This is a schematic diagram of the encoder structure.

[0048] It should be noted that this invention designs a novel encoder—the Fast Fourier Transform Pyramid Model (FPM). It integrates frequency domain information from the Fast Fourier Transform (FFT) and global detail texture information learned from the Pyramid Module (PPM). The encoder structure is as follows: Figure 4 As shown.

[0049] Fourier transform maps snow information with similar amplitude and frequency to the same point in the frequency domain, thus effectively utilizing the encoded Fourier information to extract clear information. It overcomes the limitation of convolutional methods in not incorporating global contextual information in the early stages. In the frequency domain, an image displays its global information. Different states of snowflakes in a snowy image have their fixed frequencies, while clear background images have their unique frequencies. Clearly, the frequency of any information carries global information. This method is beneficial for presenting global information in the initial stage and enhancing the model to uncover hidden information. PPM is a special pooling model that can fuse features at different scales, thereby improving the model's ability to acquire global information, especially when repairing large snowflakes, effectively repairing local artifacts.

[0050] The internal processing procedure of the encoder module based on the Fast Fourier Transform pyramid model is as follows:

[0051] The input multi-scale image is divided into two branches before entering the encoder module;

[0052] The first branch undergoes the following processing steps in sequence: Fast Fourier Transform, 1×1 Convolution, ReLU Unit, 1×1 Convolution, and Inverse Fourier Transform, to obtain the output features of the first branch.

[0053] The second branch undergoes the following processing steps in sequence: 3×3 convolution, 3×3 convolution, PPM unit, 3×3 convolution, and 3×3 convolution to obtain the output features of the second branch.

[0054] The output features of the first branch and the output features of the second branch are added together to obtain image features at different scales.

[0055] Please refer to Figure 5 , Figure 5 This is a schematic diagram of the internal structure of the decoder module;

[0056] It should be noted that incorporating global information during snow removal can enhance the generation of clear images. To address this issue, this invention developed the GlobalViT module, such as... Figure 5 As shown.

[0057] This module employs an efficient patchwise cross-attention mechanism, combining global and local information. This integration ensures a harmonious balance between complexity and performance. Furthermore, a global snow query, Qs, is designed, which may contain valuable, overlooked information from features at each scale. Qs is a dynamic query generated from multiple snowscape images at different scales. Using it as LinearQ in GlobalViT self-attention can generate more realistic snow-free images. In this embodiment, LinearQ is represented by a hidden layer. Its attention formula is as follows:

[0058]

[0059] Where K i and V i Q is a feature from the i-th layer of the decoder. s It is a dynamic query generated from multiple snow scene images of different scales, where D is the scaling factor and p is the relative position.

[0060] The internal processing procedure of the GlobalViT-based decoder module is as follows:

[0061] The first result is obtained by passing multi-scale snowy images through a hidden layer;

[0062] The fused features are processed by a LinearK unit to obtain the second result;

[0063] The attention matrix is ​​obtained by taking the dot product of the transposes of the first and second results.

[0064] The fused features are processed by a LinearV unit to obtain a third result;

[0065] The attention matrix and the third result are multiplied by a dot product to obtain the fourth result;

[0066] The fourth result is input into the MAG unit to obtain the fifth result;

[0067] Multi-axis gated MAG units replace the linear layers in the GlobalViT structure.

[0068] The fused features are subjected to a Fast Fourier Transform to obtain the sixth result;

[0069] The fifth and sixth results are concatenated and convolved to obtain the seventh result;

[0070] The seventh result, after being added to the fusion features, yields the final snow-free image.

[0071] For GlobalViT's MLP layer, this invention creates an easily expandable multi-axis gating module. For example... Figure 6 , Figure 6 This is a schematic diagram of the MAG unit structure;

[0072] The MAG unit, a readily expandable module, is beneficial for image reconstruction, as it can adapt to test feature maps of arbitrary input size. Essentially, it performs fixed-step subsampling on the attention region using different block-slicing methods for local and global attention. Given a feature map of size (H, W, C), the feature map is sliced ​​into (b, b) non-overlapping feature blocks, each block being (H...). b H w The overall shape is represented as (b*b, H). b *H w (C) First, the feature map is divided into two branches along the channel dimension. One branch is a local branch, which divides the feature map using a fixed window and performs local feature processing using a first-axis gating unit. The other branch is a global branch, which divides the feature map using a fixed grid and performs global feature processing using a second-axis gating unit. The parallel application of multi-axis operations allows the first axis of the local branch and the second axis of the global branch to share along other spatial dimensions, effectively fusing multi-axis information. The MAG block proposed in this invention represents both global and local receptive fields at arbitrary input resolutions. Compared to processing fixed-size image patches, this method avoids boundary artifacts and improves the model's reconstruction performance.

[0073] In this embodiment of the invention, the MAG unit replaces the linear layer in the GlobalViT structure and is implemented using a multi-axis gating unit.

[0074] The internal processing procedure based on the MAG unit is as follows:

[0075] The feature map processed by the attention mechanism is segmented using different multi-axis local and global segmentation methods, and the feature map is sliced ​​to obtain the first result.

[0076] Local and global branches are processed by gating units to obtain the second result;

[0077] The gating unit processing procedure is as follows: First, the feature map is processed through a linear layer, splitting it into two parts by channel, and then each part is processed by a linear layer. Next, the results of the two linear layers are concatenated along the channel dimension, and then the concatenated result is processed by a linear layer. Finally, the result is added to the input feature map to obtain the final result of the gating unit.

[0078] The local and global branches are added to the input feature map to obtain the third result of the MAG module.

[0079] S3. Use the trained network to obtain clear images at three scales and perform snowy image restoration.

[0080] Step S3 is as follows:

[0081] S31. Construct a loss function to make the generated image approximate the real image. The loss function is defined as follows:

[0082] 1) Frequency reconstruction loss in the frequency domain, where F is the Fast Fourier Transform (FFT) of the image signal transmitted to the frequency domain, Y is the generated snow-free image, and X is the real image:

[0083]

[0084] 2) Constructing an edge loss function can effectively focus on high-frequency texture structures. E represents the obtained edge features, and |YX| represents the absolute error between the snow-free image and the real image. However, we are more concerned with edge information, so we use E to weight the edges, increasing the proportion of edge components and better optimizing edge information.

[0085]

[0086] 3) The Charbonnier loss function with enhanced robustness, where Y is the generated snow-free image and X is the real image, and ∈ represents the constant 10. -3 :

[0087]

[0088] 4) Finally, this invention primarily uses frequency reconstruction loss, supplemented by other loss functions, to obtain the total loss function as follows:

[0089] L = L char +α·L edges +β·L fr

[0090] According to the experiment, the optimal balance parameter α is 0.01 and β is 0.1.

[0091] As an example, in order to evaluate the effectiveness of the technology of the present invention, the CSD Snow-100K dataset was selected.

[0092] It should be noted that CSD is a typical example of synthetic data, while Snow-100K is a typical example of real data.

[0093] This invention uses structural similarity (SSIM) and peak signal-to-noise ratio (PSNR) to evaluate the performance of each snow removal method on synthetic datasets. FTNet is compared with 11 state-of-the-art snow removal techniques and general methods on both datasets. The results are shown in Table 1. In the case of real snow images, no definitively clear images are available for comparison. Therefore, this invention employs two no-reference (NR) metrics for qualitative evaluation, where higher image quality corresponds to a smaller IL-NIQE and a larger NIMA value.

[0094] Table 1 Comparison of results between the method of the present invention and other methods.

[0095]

[0096]

[0097] Experimental results show that the performance of this invention is superior to other methods. As shown in Table 1, the method of this invention has the best performance metrics, followed by Restormer, an efficient transformer for high-resolution image restoration. Clearly, Restormer is currently the most effective and efficient method. Compared to leading methods, the method of this invention achieves a PSNR gain of 1.755 dB and an SSIM gain of 0.006 on the CSD dataset. Similarly, on the Snow100K dataset, it improves PSNR by 1.15 dB and SSIM by 0.008.

[0098] Please refer to Figure 7 , Figure 7 This is an experimental result diagram of the present invention. To intuitively understand the results on the synthetic dataset, three types of snowy weather were selected to illustrate the results. Qualitative results are shown in Table 1, and the effects are as follows: Figure 7 As shown, DesnowNet produces significant blurring when attempting to recover snow images, resulting in the worst possible outcome. JSTASR recovers a clear image with a distinct snow line and severe chromatic aberration. In contrast, All-in-one fails to eliminate large areas and small snowflakes, especially large snowflakes obscuring a clear background, and cannot recover underlying background information. MPRNet and gUnet produce white masking artifacts during image boundary recovery, hindering the overall recovery effect. Conversely, TransWeather, Uformer, and NAFNet successfully recover large snowflakes, but remain susceptible to high-frequency snowflake boundary interference, leading to high-frequency noise and artifacts in the final image. Previous methods have encountered some challenges in retrieving large snowflakes. In summary, the network of this invention successfully integrates local and global information, recovering areas covered by large snowflakes while accurately locating boundaries and removing snow lines.

[0099] Please see Figure 8, Figure 8 This is a schematic diagram of the hardware device in operation according to an embodiment of the present invention. The hardware device specifically includes: a snowy image restoration device 401, a processor 402, and a storage medium 403.

[0100] A snowy day image restoration device 401: The snowy day image restoration device 401 implements the snowy day image restoration method.

[0101] Processor 402: The processor 402 loads and executes the instructions and data in the storage medium 403 to implement the snowy image restoration method.

[0102] Storage medium 403: The storage medium 403 stores instructions and data; the storage medium 403 is used to implement the snowy image restoration method.

[0103] As one embodiment, the processor includes two parts for training and inference. The training part can use a processing chip that meets the corresponding performance requirements, while the inference part, as one embodiment, can use the Atlas200DK.

[0104] The Atlas200DK is an AI acceleration module developed by Huawei. Integrating the Ascend processor, the Atlas200DK enables image recognition, image classification, and image restoration on the device side, and is widely used in edge AI scenarios such as smart cameras, robots, and drones.

[0105] The processor 402 can invoke logical instructions in the storage medium 403 to execute a snowy image restoration method.

[0106] Furthermore, the logical instructions in the aforementioned storage medium 403 can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0107] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated.

[0108] The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium or a semiconductor medium, etc. Unless otherwise specified, the model numbers of the devices in this embodiment of the invention are not limited; any device capable of performing the above functions is acceptable.

[0109] The beneficial effects of this invention are: it designs an end-to-end framework FTNet that fuses Fast Fourier Transform and Global Vision Transformer (GlobalViT) under snowy weather conditions. This is an effective and novel Transformer-based image desnowing architecture, which consists of an encoder of convolutional neural network blocks and a decoder of GlobalViT. It successfully solves problems such as color deviation, occlusion effect and artifacts in the desnowing process.

[0110] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for restoring images from snowy weather, characterized in that: Includes the following steps: S1. Construct a multi-scale snow image restoration network; the multi-scale snow image restoration network includes: an encoder module based on the Fast Fourier Transform pyramid model, a feature fusion module, and a decoder module based on GlobalViT; S2. Train the snowy day image recovery network to obtain the trained network; S3. Use the trained network to restore snowy images; The encoder module based on the fast Fourier transform pyramid model includes: a Fourier transform unit, a convolution unit, an inverse Fourier transform unit, a pyramid module (PPM) unit, and a ReLU unit. The GlobalViT-based decoder module includes: LinearQ, K, V units and a multi-axis gated MAG unit; The LinearQ unit is replaced by a snow query module, which uses a hidden layer.

2. The snow scene image restoration method as described in claim 1, characterized in that: The process of image processing by the multi-scale snow image restoration network in step S2 is as follows: S21. Input multi-scale snowy images to the encoder module based on the fast Fourier transform pyramid model to obtain image features at different scales; S22. Input image features of different scales into the feature fusion module to obtain fused features; S23. Input the fused features into the GlobalViT-based decoder module to obtain the output features; S24. Add the output features to the original multi-scale image to obtain the snow-free image.

3. The snow scene image restoration method as described in claim 1, characterized in that: The internal processing procedure of the encoder module based on the Fast Fourier Transform pyramid model is as follows: The input multi-scale image is divided into two branches before entering the encoder module; The first branch undergoes the following processing steps in sequence: Fast Fourier Transform, 1×1 Convolution, ReLU Unit, 1×1 Convolution, and Inverse Fourier Transform, to obtain the output features of the first branch. The second branch undergoes the following processing steps in sequence: 3×3 convolution, 3×3 convolution, PPM unit, 3×3 convolution, and 3×3 convolution to obtain the output features of the second branch. The output features of the first branch and the output features of the second branch are added together to obtain image features at different scales.

4. The snow scene image restoration method as described in claim 1, characterized in that: The internal processing procedure of the GlobalViT-based decoder module is as follows: The first result is obtained by passing multi-scale snowy images through a hidden layer; The fused features are processed by a LinearK unit to obtain the second result; The attention matrix is ​​obtained by taking the dot product of the transposes of the first and second results. The fused features are processed by a LinearV unit to obtain a third result; The attention matrix and the third result are multiplied by a dot product to obtain the fourth result; The fourth result is input into the MAG unit to obtain the fifth result; The fused features are subjected to a Fast Fourier Transform to obtain the sixth result; The fifth and sixth results are concatenated and convolved to obtain the seventh result; The seventh result, after being added to the fusion features, yields the final snow-free image.

5. The snow scene image restoration method as described in claim 1, characterized in that: The MAG unit replaces the linear layer in the GlobalViT structure.

6. A storage medium, characterized in that: The storage medium stores instructions and data to implement the snowy image restoration method according to any one of claims 1 to 5.

7. A snow-themed image restoration device, characterized in that: include: A processor and a storage medium; the processor loads and executes instructions and data in the storage medium to implement a snowy image restoration method according to any one of claims 1 to 5.

Images