A method for removing snow based on transfer regularization image physics framework (TRIP)

Abstract

The application relates to a snow removing method based on a transfer regularization image physics framework TRIP, and belongs to the technical field of image snow removing.The method comprises the following steps: a snow removing system based on the transfer regularization image physics framework TRIP is constructed, the snow removing system is provided with an image acquisition module and a snow removing physics model which are sequentially connected; the image acquisition module acquires a snow scene image I and transmits the snow scene image I to the snow removing physics model; an input layer in the snow removing physics model acquires the snow scene image I and transmits the snow scene image I to a snow mask generation module; the snow mask generation module generates a corresponding snow mask M according to the snow scene image I and transmits the snow mask M to an initialization module; the initialization module pre-fills a background image region covered by the snow mask M to generate an initialization image and transmits the initialization image to an ADMM-DIP physics restorer; the ADMM-DIP physics restorer performs image restoration on the initialization image, and then outputs a clear image after restoration through an output layer. Effects: the snow removing performance and precision of a snow removing model can be improved.

Description

Technical Field

[0001] This invention relates to the field of image desnowing technology, and in particular to a desnowing method based on the TRIP image physical framework. Background Technology

[0002] In outdoor vision systems, severe weather conditions such as rain, snow, and fog often lead to significant image quality degradation, impacting the performance of downstream tasks such as object detection and semantic segmentation. Among these degradation types, snowflakes, due to their diversity in shape, size, transparency, and trajectory, are considered one of the most difficult to handle. Single-image desnow removal aims to recover a clear background image from a tainted snow scene image I, which has significant application value in autonomous driving and video surveillance.

[0003] Early snow removal methods relied on hand-designed prior features, such as frequency separation or color space properties, but these often failed in complex real-world scenarios. In recent years, deep neural networks have been widely applied in this field. Among them, Liu et al. proposed DesnowNet, a deep network with environmental awareness suitable for snow removal operations, employing a multi-stage processing mechanism; Chen et al. proposed JSTASR, a snow removal algorithm that balances snow volume and transparency based on improved partial convolution and occlusion removal techniques, taking into account both snowflake size and transparency; and Valanarasu et al. proposed TransWeather, an image inpainting technique for severe weather conditions based on Transformer, which introduces Transformer to handle various weather degradations.

[0004] Despite the significant progress made by the above approaches, existing end-to-end deep learning methods still face two key limitations:

[0005] 1. Insufficient Domain Differences and Generalization Ability: Existing snow removal models heavily rely on synthetic datasets, making it difficult to generalize to real-world scenes with complex and varied snowflake shapes. Because synthetic snowflakes typically follow simplified physical models, these models often overfit to the features of the synthetic data. When faced with complex snowflakes in the real world, such as light spots in front of the lens or dense snow curtains in the distance, the performance of these models drops significantly.

[0006] 2. Loss of image details: Standard encoder-decoder architectures are prone to irreversibly losing high-frequency background information during downsampling and feature extraction. This often results in oversmoothing of the desnowed image, erasing subtle textures in the background while removing snowflakes.

[0007] Disadvantages of existing technologies: Existing image desnowing models suffer from insufficient domain differences and generalization ability, are prone to losing image details, and have poor desnowing performance and low accuracy. Summary of the Invention

[0008] This invention provides a snow removal method based on the TRIP image physical framework, which can improve the snow removal performance and accuracy of the snow removal model.

[0009] To achieve the above objectives, this invention provides a snow removal method based on the TRIP (Transfer Regularization Image Physical Framework), which, as a key feature, includes the following steps:

[0010] Step 1: Construct a snow removal system based on the TRIP image physical framework, which consists of an image acquisition module and a snow removal physical model connected in sequence.

[0011] The snow removal physical model is configured with an input layer, a snow mask generation module, an initialization module, an ADMM-DIP physical repairer, and an output layer connected in sequence.

[0012] Step 2: The image acquisition module acquires the snow scene image I and passes it to the snow removal physical model;

[0013] Step 3: The input layer in the snow removal physical model acquires the snow scene image I and passes it to the snow mask generation module;

[0014] Step 4: The snow mask generation module generates a corresponding snow mask M based on the snow scene image I and passes it to the initialization module;

[0015] Step 5: The initialization module pre-fills the background image area covered by the snow mask M to generate an initialization image. And pass it to the ADMM-DIP physical repairer;

[0016] Step 6: The ADMM-DIP physical repair tool applies the initialization image. The image is repaired, and then the repaired, clear image is output through the output layer. .

[0017] Through the above design, this invention proposes a novel detection-inpainting general framework that combines advanced transfer learning techniques with physical constraint optimization. First, instead of training the detection network from scratch, this invention utilizes transfer learning to pre-train visual foundational models on large-scale datasets, including ResNet101, SegFormer, and SAM, to obtain snowflake masks with extremely strong generalization capabilities. Second, this invention abandons purely data-driven inpainting networks and designs a physical optimization solver based on the Alternating Direction Multiplier Method (ADMM), introducing untrained depth image priors (DIPs) as regularization terms to form the ADMM-DIP physical inpainter. This ADMM-DIP physical inpainter can achieve physically consistent texture reconstruction using the statistical properties of the image itself without requiring large-scale paired real data. This invention not only improves the generalization ability of the desnowing model but also effectively preserves the detailed features of the background image, effectively improving the desnowing performance and accuracy of the desnowing model.

[0018] Preferably, the model expression for the snow mask M generated by the snow mask generation module is:

[0019] ;

[0020] in, This is a transfer learning snowflake mask generation model built into the snow mask generation module. Image of a snow scene; This is an image processing function used to divide the pixel values ​​in an image into two parts: one part is below a set threshold, and the other part is above a set threshold.

[0021] Preferably, the snowflake mask generation model is trained using a transfer learning strategy, and a hybrid loss is used as the total loss function of the model during training. The expression is:

[0022] ;

[0023] in, express Loss weight value; This represents the weighted binary cross-entropy loss, used to address the imbalance problem where snowflake pixels account for a very small proportion of the entire image. express Loss weight value; express Coefficient loss is used to further constrain the structural consistency of the snowflake shape;

[0024] Weighted binary cross-entropy loss The expression is:

[0025] ;

[0026] coefficient loss The expression is:

[0027] ;

[0028] in, Indicates the total number of pixels. Represents the j-th pixel. This represents the weighting coefficient, used to adjust the proportion of contribution of positive and negative samples (snowflake pixels and background pixels) to the loss; This represents the true label (truth mask) of the j-th pixel, typically 1 for the snowflake region and 0 for the background region; This represents the probability value that the model predicts the j-th pixel to be a snowflake; This represents a very small constant (smoothing term), which prevents the denominator from being zero, thereby enhancing the stability of numerical calculations.

[0029] To guide the pre-trained model to quickly converge from general features to snowflake mask generation, this invention designs a hybrid loss function, employing a weighted binary cross-entropy loss. To address the imbalance issue where snowflake pixels constitute a very small proportion of the overall image, and to introduce... coefficient loss This further constrains the structural consistency of the snowflake shape.

[0030] Preferably, the transfer learning snowflake mask generation model is either a transfer learning snowflake mask generation model based on semantic segmentation SegFormer, or a transfer learning snowflake mask generation model based on residual network ResNet-101, or a transfer learning snowflake mask generation model based on segmentation of all SAM.

[0031] Through the above design, this invention utilizes three fundamental visual models—ResNet101, SegFormer, and SAM—to learn powerful semantic representation capabilities on large-scale general datasets. By adjusting these models, their powerful feature extraction capabilities are transferred to the snowflake detection task, thereby generating high-precision snowflake masks even in real-world scenarios. This effectively solves the generalization bottleneck of snow removal models and improves their generalization ability.

[0032] Preferably, the initialization module uses the Fast Moving Model (FMM) inpainting algorithm to pre-fill the snow scene image area covered by the snow mask M to generate an initialization image. The expression is:

[0033] ;

[0034] in, To cover the snow, Image of a snow scene; This is an image inpainting function based on the fast traversal algorithm and the hydrodynamic inpainting algorithm. It analyzes pixel information in the image and diffuses pixel values ​​from the surrounding area to the area to be inpainted, thereby achieving image restoration.

[0035] As a preferred embodiment, the ADMM-DIP physics repair tool is used to model the snow removal problem as an optimization problem under physical constraints, the optimization problem being defined as follows:

[0036] ;

[0037] ;

[0038] in, To initialize the image, To cover the snow; The background image recovered from snow scene image I; These are parameters for the DIP texture generator. As an auxiliary variable, For the total variation TV smoothness constraint, Calculate the square of the L2 norm. Calculate the L1 norm. For element-wise multiplication, For spatial gradient operators, For parameters DIP texture generator;

[0039] The augmented Lagrangian function corresponding to the optimization problem is:

[0040] ;

[0041] in, For penalty parameters, ; To scale the dual variable.

[0042] Preferably, the ADMM-DIP physics repairer is provided with an interconnected DIP texture generator and an ADMM physics solver.

[0043] The DIP texture generator is used to perform prior prediction on the initialized image to obtain the prior structure. The expression is: ;

[0044] The ADMM physical solver is used to decompose the optimization problem into a background image using the Alternating Direction Multiplier Method (ADMM). Update, auxiliary variables Updating and scaling dual variables The three subproblems are updated, and the preprocessed conjugate gradient method (PCG) is used to solve the subproblems.

[0045] The beneficial effects of this invention are:

[0046] (1) A two-stage general framework for snow removal combining transfer learning and physical optimization is proposed, which not only improves the generalization ability of the snow removal model, but also effectively preserves the detailed features of the background image, thus effectively improving the snow removal performance and accuracy of the snow removal model.

[0047] (2) The powerful generalization ability of three different basic models, ResNet, SegFormer and SAM, is used to learn the snowflake mask, which greatly reduces the training time without losing accuracy.

[0048] (3) An ADMM-DIP physical inpainting tool was designed, and a closed-form update step was derived, enabling self-supervised image inpainting without the need for external large-scale paired data. Experimental data on synthetic and real datasets demonstrate that the invention is effective in preserving background details and removing complex snowflakes. Attached Figure Description

[0049] Figure 1 This is a flowchart illustrating the overall framework of the TRIP algorithm in this embodiment.

[0050] Figure 2 The images show the snowflake mask results from the training of the three models in this embodiment.

[0051] Figure 3 The diagram shows the results of the two stages in the embodiment;

[0052] Figure 4(a) is a comparison of the snow removal effect on the Snow100K-synthetic dataset in the example;

[0053] Figure 4(b) is a comparison of the snow removal effect on the real scene dataset Snow100K-Realistic in the embodiment;

[0054] Figure 5(a) shows the total variational regularization coefficients in the embodiment. Sensitivity analysis line graph;

[0055] Figure 5(b) shows the ADMM penalty term coefficients in the example. Sensitivity analysis line graph;

[0056] Figure 5(c) is a line graph showing the sensitivity analysis of the PCG_TOL tolerance for solving the linear subproblem in the embodiment. Detailed Implementation

[0057] The present invention will be further described in detail below with reference to the accompanying drawings and specific examples. The following embodiments or drawings are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0058] A snow removal method based on the TRIP image physical framework, comprising the following steps:

[0059] Step 1: Construct a snow removal system based on the TRIP image physical framework, which consists of an image acquisition module and a snow removal physical model connected in sequence.

[0060] The snow removal physical model is configured with an input layer, a snow mask generation module, an initialization module, an ADMM-DIP physical repairer, and an output layer connected in sequence, such as... Figure 1 As shown;

[0061] Step 2: The image acquisition module acquires the snow scene image I and passes it to the snow removal physical model;

[0062] Step 3: The input layer in the snow removal physical model acquires the snow scene image I and passes it to the snow mask generation module;

[0063] Step 4: The snow mask generation module generates a corresponding snow mask M based on the snow scene image I and passes it to the initialization module;

[0064] Step 5: The initialization module pre-fills the background image area covered by the snow mask M to generate an initialization image. And pass it to the ADMM-DIP physical repairer;

[0065] Step 6: The ADMM-DIP physical repair tool applies the initialization image. The image is repaired, and then the repaired, clear image is output through the output layer. .

[0066] The TRIP architecture consists of two cascaded stages, such as... Figure 1 As shown in (a), the two stages are snowflake mask generation based on transfer learning and ADMM-DIP physical repairer, respectively.

[0067] This invention models a snow-covered image I as a nonlinear superposition of a potential snow-free background u and a snowflake component S. This physical process is modulated by a binary mask M ∈ {0,1}, expressed as:

[0068] ;

[0069] Here, ⊙ represents element-wise multiplication. Non-zero elements in mask M indicate snowflake regions, and zero elements indicate background regions. The objective of this invention is to recover a clear background image from observed image I. Therefore, this invention first generates a mask M and an initial desnowed image. Then, the ADMM-DIP joint optimization algorithm further refines the recovery results.

[0070] The model expression for the snow mask M generated by the snow mask generation module is as follows:

[0071] ;

[0072] in, This is a transfer learning snowflake mask generation model built into the snow mask generation module. Image of a snow scene; This is an image processing function used to divide the pixel values ​​in an image into two parts: one part is below a set threshold, and the other part is above a set threshold.

[0073] The snowflake mask generation model is trained using a transfer learning strategy, and a hybrid loss is used as the overall loss function of the model during training. The expression is:

[0074] ;

[0075] in, express Loss weight value; This represents the weighted binary cross-entropy loss, used to address the imbalance problem where snowflake pixels account for a very small proportion of the entire image. express Loss weight value; express Coefficient loss is used to further constrain the structural consistency of the snowflake shape;

[0076] Weighted binary cross-entropy loss The expression is:

[0077] ;

[0078] coefficient loss The expression is:

[0079] ;

[0080] in, Indicates the total number of pixels. Represents the j-th pixel. This represents the weighting coefficient, used to adjust the proportion of contribution of positive and negative samples (snowflake pixels and background pixels) to the loss; This represents the true label (truth mask) of the j-th pixel, typically 1 for the snowflake region and 0 for the background region; This represents the probability value that the model predicts the j-th pixel to be a snowflake; This represents a very small constant (smoothing term), which prevents the denominator from being zero, thereby enhancing the stability of numerical calculations.

[0081] like Figure 1 As shown in (b), the transfer learning snowflake mask generation model is either a transfer learning snowflake mask generation model based on semantic segmentation SegFormer, or a transfer learning snowflake mask generation model based on residual network ResNet-101, or a transfer learning snowflake mask generation model based on segmentation of all SAM.

[0082] (1) Snowflake mask generation model based on semantic segmentation SegFormer transfer learning

[0083] The advantage of the snowflake mask generation model based on semantic segmentation SegFormer lies in capturing global semantics through a hierarchical MixTransformer encoder. Unlike general segmentation tasks, this invention focuses on transfer learning adaptation to the physical characteristics of snowflakes.

[0084] 1. Geometric Continuity Preservation Overlapping Patch Merging Strategy: To effectively address the local spatial information fragmentation caused by non-overlapping patch embedding in the traditional Vision Transformer (ViT) image processing deep learning model based on the Transformer architecture, this invention utilizes the Overlapping Patch Merging mechanism of SegFormer to enhance the perception of snowflake edges. By maintaining the convolutional downsampling settings during transfer training, the model can achieve cross-domain transfer from general semantics to desnow semantics and robustly maintain the geometric continuity of snowflake particles, effectively avoiding information loss of small-scale snowflake features during downsampling.

[0085] 2. Knowledge Transfer Logic Adapted for Snowflake Recognition: The core challenge of snowflake detection lies in distinguishing between white background objects and snowflakes. This invention uses a MiT-B5 encoder pre-trained on the ADE20K large-scale semantic dataset as the knowledge source. Transfer learning plays a crucial role in enhancing discriminative judgment.

[0086] Semantic reuse of efficient self-attention ESA: Utilizing existing sequence reduction strategies in pre-trained ESA modules, end-to-end fine-tuning is possible on high-resolution snow scene images. Its computational complexity is reduced from... Reduce to , Indicates the time complexity of the algorithm. This represents the total number of pixels in the image. This indicates the sequence reduction ratio, ensuring that the snowflake distribution across the entire image can be modeled even with limited computing power.

[0087] Mix-FFN positional encoding generalization method: Since the snow removal task needs to process real-world images of different resolutions, this invention utilizes the implicit positional bias provided by the 3×3 convolution in the SegFormer feedforward network to avoid the significant performance degradation of traditional ViT when migrating to new resolutions.

[0088] This is achieved through a task-specific objective function, namely the total loss function. The pre-trained SegFormer decoder can accurately process multi-scale features. The mapping is a high-fidelity binary mask, where, Represents a linear function. This represents vector concatenation. and This represents the eigenvector.

[0089] (2) Snowflake mask generation model based on ResNet-101 residual network

[0090] The snowflake mask generation model based on the ResNet-101 residual network adopts the classic U-Net architecture, which includes an encoder, bridging layers, and a decoder. The encoder part utilizes ResNet-101 pre-trained on ImageNet. This transfer learning strategy aims to leverage the powerful general visual feature extraction capabilities of ResNet-101, especially in low-level texture and structure perception.

[0091] Architecture and Transfer Strategies: The U-Net encoder transforms the input snow scene image I into a multi-scale feature map set. Then, an end-to-end full-parameter fine-tuning strategy is adopted. To best adapt to the distribution of snowflake features, this invention performs joint training on the entire ResNet-101 based on the pre-trained ImageNet weights. This strategy allows the low-level feature extractor to be fine-tuned according to the specific needs of the snow removal task, thereby more accurately capturing the edge and texture details of snowflakes.

[0092] Feature Enhancement and Fusion: To enhance the model's sensitivity to complex snowflake patterns, this invention focuses on optimizing feature fusion in skip connections between the encoder and decoder. This embodiment introduces an SCSE feature fusion module in the skip connection at encoder / decoder layer i∈{1, 2, 3, 4}. The SCSE feature fusion module combines spatial compression and channel excitation mechanisms to adaptively recalibrate feature weights.

[0093] The SCSE feature fusion module comprises a globally averaged magnetization layer, a first fully connected layer, and a second fully connected layer connected in sequence. The output of the first fully connected layer incorporates a ReLU activation function, and the output of the second fully connected layer incorporates a Sigmoid activation function. The specific implementation of the SCSE feature fusion module is as follows:

[0094] 1. Space compression operation Channel descriptors are obtained by performing global average pooling (GAP) along the spatial dimension. :

[0095] ;

[0096] ;

[0097] in, The total length of the input features. For the input feature length index, The total width of the input features. For the input feature width index, This represents the total number of input feature channels. Indicates the input feature channel index. For channel Descriptors, This is the output feature map of the current encoder layer;

[0098] 2. Channel excitation operation The activation vector s is obtained by learning the nonlinear dependency between channels through two fully connected layers.

[0099] ;

[0100] in, It is the ReLU activation function. For the Sigmoid activation function, This is for the first fully connected layer operation. This is for the operation of the second fully connected layer.

[0101] Finally, the feature map enhanced by SCSE It will be used in the decoder for pixel prediction. Its training process and optimizer are consistent with SegFormer, both employing hybrid loss. Optimize.

[0102] (3) Transfer learning snowflake mask generation model based on segmentation of all SAM

[0103] The Segmentation All-Simple (SAM) model, as a general segmentation foundation model, demonstrates remarkable zero-shot segmentation capabilities. However, SAM is inherently class-agnostic, making it difficult to directly distinguish snowflakes from background white dots. To transfer the powerful segmentation prior knowledge of SAM to a specific snow removal task, while avoiding the enormous computational overhead of full parameter fine-tuning, this embodiment employs an efficient parameter fine-tuning strategy for the mask decoder.

[0104] 1. Frozen Encoder and Feature Extraction: SAM consists of three main components: an image encoder, a cue encoder, and a mask decoder. Given that the ViT-B-based image encoder has a moderate number of parameters and contains rich general visual features, this embodiment completely freezes it during training. For a given input snow scene image I, its image embedding is pre-calculated:

[0105] ;

[0106] in, Indicates image embedding, Indicates image encoder operation, This indicates that the parameters are frozen and the gradient is not calculated.

[0107] 2. Global Cue Construction: SAM segmentation is highly dependent on cues. Snowflakes are typically scattered throughout the entire image space, making sparse point cues extremely inefficient. Therefore, this invention constructs a bounding box cue that covers the entire image. As a geometric prior, it informs the model to focus on targets globally. The cue encoder maps it to a cue embedding:

[0108] ;

[0109] in, This indicates that the prompt is embedded. This indicates a prompt for encoder operation. This indicates a prompt box;

[0110] 3. Decoder Fine-tuning and Mask Generation: In this embodiment, the frozen image is embedded... And prompts to embed into a lightweight mask decoder. All parameters of the decoder. Fine-tune it to learn to map general edge features to a specific snowflake distribution:

[0111] ;

[0112] in, This represents the snow mask generated by a snow mask generation model based on the transfer learning of the SAM (Segmentation of All Objects) model. This is for mask decoder operation.

[0113] This strategy allows the model to quickly adapt to snowflake detection tasks while retaining SAM's powerful edge segmentation capabilities, and the number of training parameters is less than 5% of the total parameters. A hybrid loss mechanism is used. Optimize.

[0114] After obtaining the snow mask M, in order to provide a good texture starting point for the optimization process and accelerate convergence, the initialization module uses the Fast Moving Model (FMM) inpainting algorithm to pre-fill the snow scene image area covered by the snow mask M, generating an initialization image. The expression is:

[0115] ;

[0116] in, To cover the snow, Image of a snow scene; This is an image inpainting function based on the fast traversal algorithm and the hydrodynamic inpainting algorithm. It analyzes pixel information in the image and diffuses pixel values ​​from the surrounding area to the area to be inpainted, thereby achieving image restoration.

[0117] like Figure 1 As shown in (c), the ADMM-DIP physics repair tool is used to model the snow removal problem as an optimization problem under physical constraints, and the optimization problem is defined as follows:

[0118] ;

[0119] ;

[0120] in, To initialize the image, To cover the snow, Ensure that data fidelity measures are only effective in non-snow areas; The background image recovered from snow scene image I; These are parameters for the DIP texture generator. As an auxiliary variable, For the total variation TV smoothness constraint, Calculate the square of the L2 norm. Calculate the L1 norm. For element-wise multiplication, For spatial gradient operators, For parameters DIP texture generator;

[0121] The augmented Lagrangian function corresponding to the optimization problem is:

[0122] ;

[0123] in, For penalty parameters, ; To scale the dual variable.

[0124] The optimization objective function includes a data fidelity term for non-snow regions, a total variational (TV) smoothness constraint, and image priors parameterized by the deep network. To facilitate numerical solution, this invention employs a scaling form of ADMM, introducing an auxiliary variable z and a scaling dual variable d. Figure 1 As shown in (e), ADMM decomposes the problem into subproblems and solves them iteratively in an alternating manner.

[0125] The ADMM-DIP physics repairer is equipped with interconnected DIP texture generators and ADMM physics solvers.

[0126] like Figure 1 As shown in (d), this invention designs a U-Net architecture DIP network based on MobileNetV2 as... To strictly adhere to the self-supervised design of DIP, all parameters of this network are randomly initialized without loading any pre-trained weights. The DIP network leverages the lightweight feature extraction capabilities of MobileNetV2 combined with the skip connections of U-Net to effectively preserve high-frequency details and multi-scale contextual information of the image. During optimization, a warm-start strategy is employed to process the initial images generated in the first stage. As the input to the network, i.e. This significantly accelerates the convergence of DIP and improves reconstruction quality. By minimizing the data fidelity term and gradient constraint term, the DIP network gradually adapts to the texture features of snow-free areas during the iteration process, and uses its structural priors to naturally extend these features to the snowflake-covered areas, thereby achieving physically consistent texture restoration.

[0127] The DIP texture generator is used to perform prior prediction on the initialized image to obtain the prior structure. The expression is: ;

[0128] The ADMM physical solver is used to decompose the optimization problem into a background image using the Alternating Direction Multiplier Method (ADMM). Update, auxiliary variables Updating and scaling dual variables The three subproblems are updated, and the preprocessed conjugate gradient method (PCG) is used to solve the subproblems.

[0129] The ADMM-DIP physical repair tool initializes the image. The specific steps for image restoration are as follows:

[0130] Step S1: Initialize variables: Set the initial background image Initial auxiliary variables Initial scaling dual variable ;

[0131] Step S2: Initialize the DIP texture generator The initial weight parameters are generated using a random number generator;

[0132] Step S3: Enter the external ADMM iteration, number of external ADMM iterations ;

[0133] Step S4: The DIP texture generator is parameterized. The optimization iterations are performed, with a total of L iterations within the DIP layer. The parameters of the DIP texture generator are... The update expression is:

[0134] ;

[0135] ;

[0136] ;

[0137] in, To approach in non-snow areas Data loss, The surrogate loss is used to approximate the auxiliary variable objective (zd) with the gradient field;

[0138] When the total number of iterations within the DIP reaches L, the DIP texture generator performs prior prediction on the initialized image based on the parameters obtained in the Lth iteration, thus obtaining the prior structure. The expression is:

[0139] ;

[0140] Step S5: Image Reconstruction: For the background image The updated subproblem is that the ADMM physics solver will update the prior structure predicted by the DIP texture generator. As background image The initial values ​​of the updated subproblem are used to perform update calculations to obtain the updated background image. The expression is:

[0141] ;

[0142] Regarding the above Update expression about By taking the derivative and setting it to zero, we can obtain the corresponding system of linear equations:

[0143] ;

[0144] in, This indicates extracting the diagonal elements of a matrix or creating a diagonal matrix from a vector; For penalty parameters, It is a Laplace operator with convolutional properties;

[0145] Because the left-hand linear operator contains a mask of spatial variation. Laplace operator with convolution properties Since it cannot be directly inverted, the preprocessed conjugate gradient method (PCG) is used for solving it. To improve efficiency, a linear operator based on FFT is constructed. Specifically, for the operator... ;in Operations can be efficiently computed in the frequency domain using the optical transfer function (OTF):

[0146] ;

[0147] in, Represents the Fast Fourier Transform. Represents the difference operator in the horizontal direction. Represents the difference operator in the vertical direction. Represents the square of the absolute value;

[0148] This invention runs the PCG solver in parallel and independently on the three RGB channels, and displays the output of the DIP. As an initial value for a hot start, this greatly accelerates convergence.

[0149] Step S6: Auxiliary variable update: For auxiliary variables The updated subproblem, the ADMM physics solver first calculates the current gradient. Then, by using soft thresholding for denoising, the updated auxiliary variables are obtained. The expression is:

[0150] ;

[0151] ;

[0152] in, For symbolic functions, It is a function with maximum value. This is the spatial gradient operator; this step removes noise in the gradient domain, achieving total variational denoising.

[0153] Step S7: Scaling Dual Variable Update: For scaling dual variables... The updated subproblem, calculated by the ADMM physics solver, is obtained through the following update expression:

[0154] ;

[0155] This step measures the residual of the gradient constraint and feeds it back to the next iteration.

[0156] Step S8: Repeat steps S4-S7 until the external ADMM iteration count reaches... At this point, the background image obtained from the last iteration will be... As a clear image after restoration Output.

[0157] In each outer iteration of ADMM, the DIP texture generator parameters θ are updated L times using the Adam optimizer. This enables the DIP network to learn an image manifold that conforms to both the observed data and the current gradient smoothness constraints.

[0158] This invention decomposes the snow removal task into two sub-problems: snowflake detection and background restoration, and addresses each sub-problem specifically. To overcome the generalization bottleneck, in the first stage, three basic visual models—ResNet101, SegFormer, and SAM—learn powerful semantic representation capabilities on large-scale general datasets. By adjusting these models, their powerful feature extraction capabilities are transferred to the snowflake detection task, thereby generating high-precision snowflake masks even in real-world scenes. Figure 2 The snowflake masking results for three models are presented, and the accuracy results are also shown in the experimental section. To overcome the problem of image detail loss, in the second stage, a DIP network based on the MobileNetV2 architecture is used as a regularization term. Its inductive bias on the statistical characteristics of natural images avoids overfitting to synthetic data. The ADMM algorithm is then used for solving the problem. The results of the two stages are as follows: Figure 3 As shown, after the snowflake position is accurately identified by the mask generation model in stage 1, preliminary results are obtained. In stage 2, the image details are preserved to the greatest extent by jointly optimizing ADMM and DIP.

[0159] The following experiments demonstrate the details of the training and testing process of the snow removal physics model proposed in this invention, and compare the estimation results of the three models on a public dataset.

[0160] (1) Data set and reference indicators

[0161] Datasets: To enhance the model's adaptability to different snowflake patterns, this embodiment uses the Snow Synthetic dataset (CSD) and the Snow100K training set as training data. This hybrid training strategy helps the model learn richer snowflake features. During the testing phase, the synthetic Snow100K dataset is used as the test set, while the real Snow100K-Realistic and RealSnow10K datasets are used for real-world snow scene testing.

[0162] Reference Metrics: To comprehensively evaluate image restoration quality, this invention employs two types of metrics: full-reference (FR) and no-reference (NR). The full-reference FR metric is used on the ground-valued Snow100K-synthetic test set, including Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM), with higher values ​​indicating better restoration quality; and the color difference metric CIEDE2000, with lower values ​​indicating less color deviation. The no-reference NR metric is used on the ground-valued Snow100K-Realistic and RealSnow10K real-world scene datasets. Four widely used no-reference image quality assessment metrics, NR-IQA, are then employed: BRISQUE, NIQE, PIQE, and PAQ2PIQ. NR-IQA metrics better reflect the human eye's perception of image naturalness, distortion, and overall quality.

[0163] Among them, BRISQUE is the spatial quality assessment index for blind / no-reference images, NIQE is the naturalness image quality assessment index, and PIQE is the perception-based image quality assessment index. The lower the scores of BRISQUE, NIQE, and PIQE, the better and more natural the perceptual quality of the image. For PAQ2PIQ, the higher the score, the better.

[0164] (2) Training details

[0165] Phase 1: Mask Generation Network Training: Three mask generation networks with different architectures were trained on the Snow100K training set. All models were implemented using the PyTorch deep learning framework and trained on a single NVIDIA 3090 GPU. The optimizer used was Adam. Loss function weights... and All were set to 0.5.

[0166] 1. ResNet-101: Employs the U-Net architecture, using a ResNet-101 pre-trained image on the ImageNet image database as the encoder, and introducing an SCSE attention module in the decoder. To strike a balance between preserving texture details and computational efficiency, the input image is resized to 640×480. The batch size is set to 32. The AdamW optimizer is used, with the learning rate initialized to 1×1000. The learning rate was adjusted using a cosine annealing restart strategy. Training lasted for 100 epochs.

[0167] 2. SegFormer: Uses the MixTransformer-B5 (MiT-B5) backbone network, loaded with ImageNet pre-trained weights. Input resolution is also set to 640×480. Training epochs are 200, batch size is 8, and gradient accumulation is used. The optimizer is AdamW, with a learning rate of 1× The learning rate is also adjusted using a cosine annealing restart strategy. To simulate harsh real-world environments and improve generalization, data augmentation strategies including RandomRain, MotionBlur, ShiftScaleRotate, and Gaussian noise are introduced.

[0168] 3. SAM: The ViT-B version of the segmentation model SegmentAnythingModel was selected. The PEFT strategy, an efficient parameter fine-tuning approach, was employed, freezing the image encoder parameters and fine-tuning only the mask decoder. The input image maintained a high resolution of 1024×1024. The training epochs were 100, and the batch size was 12. The learning rate was set to 1× And use the learning rate scheduler ReduceLROnPlateau.

[0169] Phase Two: ADMM-DIP Optimization Parameters: During the inference phase, physical constraint optimization is performed based on the ADMM algorithm. The external ADMM iteration count K = 10, and the internal DIP iteration count L = 10. Regularization parameters... =0.002, penalty parameter =0.1. The DIP texture generator uses the standard U-Net architecture with 3 input and 3 output channels and a learning rate of 0.05.

[0170] (3) Model Analysis

[0171] To verify the performance of the mask generation module in the first stage, this invention quantitatively evaluated the binary mask prediction accuracy of three different backbone networks—ResNet-101, SegFormer, and SAM—on the Snow100K-synthetic test set. As shown in Table 1, this implementation was tested on three subsets, Snow100K-S, M, and L, representing different snowflake sizes and densities.

[0172] Experimental results show that, thanks to the transfer learning strategy, the accuracy of all models generally exceeds 96%. The ResNet-101 variant achieved the highest average accuracy, reaching 0.9927, 0.9901, and 0.9782 on the S, M, and L subsets, respectively. This indicates that the classic CNN architecture combined with the SCSE attention mechanism still has a strong advantage in capturing local high-frequency features such as snowflakes. The SegFormer variant followed closely behind, with its Transformer-based encoder having an advantage in global context modeling. Although it was slightly weaker than ResNet in pure pixel-level accuracy, it still provided a very high-quality mask. The SAM variant, despite employing a frozen encoder parameter fine-tuning strategy, still achieved satisfactory accuracy, with a minimum of 96.85%.

[0173] In a longitudinal comparison, the accuracy of each model showed a slight downward trend as the snowflake coverage area increased, indicating that large-area occlusion posed a challenge to feature extraction. However, even on the most challenging Snow100K-L, the lowest accuracy remained above 96.8%. This provided a highly reliable prior mask for the second-stage ADMM-DIP physical inpainting, ensuring that subsequent inpainting processes only targeted the snow-covered areas, thereby preserving background details to the greatest extent possible.

[0174] Table 1: Accuracy test results of mask generation network on the Snow100K-synthetic test dataset

[0175]

[0176] (4) Quantitative analysis

[0177] 1. On the Snow100K-synthetic dataset, the TRIP framework proposed in this invention was compared with existing state-of-the-art snow removal methods, including HDCWNet (a single-image snow removal algorithm based on hierarchical dual-tree composite wavelet representation and contradictory channel loss), MPRNet (a multi-stage progressive image inpainting algorithm), InvDN (an image snow removal algorithm through depth reversible separation), and NAFNet (SIDD / REDS) (a real noise removal algorithm). Evaluation metrics included Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), Color Difference Formula (CIEDE2000), and four no-reference metrics: BRISQUE, NIQE, PIQE, and PAQ2PIQ. Experiments covered subsets of snowflake sizes S, M, and L.

[0178] The results are shown in Table 2. At the M-scale, TRIP-ResNet achieved a PSNR of 28.660 dB and an SSIM of 0.898, significantly outperforming the suboptimal NAFNet-REDS and MPRNet. This fully demonstrates the efficiency of the proposed transfer learning-based mask generation network in capturing features of snowflakes of common sizes, and the superiority of ADMM-DIP in restoring textures of occluded regions. Furthermore, both variants maintained their leading position in perceptual metrics such as BRISQUE and NIQE, indicating that the restored images not only have small pixel errors but also exhibit natural visual effects. At the L-scale test set, although HDCWNet slightly outperformed in PSNR, this was mainly because PSNR is highly sensitive to pixel-level errors, while HDCWNet tends to produce overly smoothed results. In contrast, TRIP-ResNet achieved a significant first place in the SSIM metric, which better reflects structural information. This means that when dealing with large-area snowflake occlusion, our method can better preserve the original structure and edge information of the image, avoiding severe structural distortion. It is noteworthy that the CIEDE2000 color difference index values ​​of the method of this invention are generally below 3.0 across all three scales, while the comparison methods typically exceed 4.0. This demonstrates that the method of this invention can maintain global color consistency of the image very well, avoiding the color shift problem common in deep learning models. In terms of the PIQE index, TRIP-ResNet ranks among the top three on both the S and L scales, and is on par with NAFNet-REDS on the PAQ2PIQ index, which comprehensively reflects image quality, especially approaching the state-of-the-art (SOTA) level on the S scale. This indicates that our method achieves excellent results in terms of traditional fidelity and naturalness of visual perception.

[0179] Table 2: Comparison of quantitative metrics for each method on the Snow100K-synthetic dataset, including S / M / L scales. The best result is indicated in red, the second best in blue, and the third best in green.

[0180]

[0181] 2. The results on the Snow100K-Realistic dataset are shown in Table 3. It can be seen that the three variants of the proposed TRIP framework all achieved the top three scores on both the BRISQUE and NIQE metrics. This indicates that the proposed method, while removing snowflakes, generates images that closely resemble real images in texture and structure. Specifically, TRIP-ResNet achieved the best score of 19.3434 on BRISQUE, and TRIP-SAM achieved the best score of 3.0672 on NIQE. On the PIQE metric, TRIP-SegFormer and TRIP-ResNet of this invention followed HDCWNet, ranking second and third respectively. On the PAQ2PIQ metric, although NAFNet-REDS achieved the highest score, the TRIP-SegFormer and TRIP-SAM variants of this invention ranked second and third with a slight difference, indicating that their overall perceptual quality remained good. In summary, the TRIP-SegFormer variant consistently ranked in the top two or three across all four metrics, demonstrating balanced and robust overall performance. This is thanks to SegFormer's powerful global context modeling capabilities and ADMM-DIP's effective constraints on image details.

[0182] Table 3: Comparison of quantitative metrics for each method on the Snow100K-Realistic dataset. It includes four evaluation metrics, with the best results indicated in red, the second best in blue, and the third best in green.

[0183]

[0184] 3. The comparison results on the RealSnow10K dataset are shown in Table 4. As can be seen from the data in Table 4, the three variants proposed in this invention occupy the top three positions in the NIQE metric, slightly worse than HDCWNet in the PIQE metric, and slightly worse than NAFNet-SIDD in the PAQ2PIQ metric. TRIP-SegFormer's results in the PAQ2PIQ metric are right behind NAFNet-REDS. Although the NAFNet variants score highly on some perceptual metrics, their NIQE scores show significant instability. In contrast, the method proposed in this invention performs more evenly across all metrics. This demonstrates that the ADMM-DIP physics optimization strategy introduced in this invention can effectively avoid artifacts common in deep learning models, and the generated images best match the statistical characteristics of natural scenes.

[0185] Table 4: Comparison of quantitative metrics for each method on the RealSnow10K dataset. It includes four evaluation metrics, with the best result represented in red, the second best in blue, and the third best in green.

[0186]

[0187] (5) Qualitative analysis

[0188] To visually demonstrate the snow removal effect of the method of this invention in different scenarios, a comparative visualization was performed on the Snow100K-synthetic and Snow100KRealistic datasets. The comparison results on the Snow100K-synthetic dataset are as follows: Figure 4a As shown, while HDCWNet removes snowflake noise, it distorts the image colors. MPRNet and NAFNet-SIDD results still retain a large amount of snowflakes. NAFNet-REDS results show many colored noise points, and InvDSNet results over-smooth the image to remove snowflakes, resulting in blurry images. In contrast, the three variant models proposed in this invention not only remove snowflakes of various sizes but also excellently preserve the high-frequency texture of the background, resulting in significantly better image clarity than the comparison methods.

[0189] The comparison results of the Snow100K-Realistic dataset are as follows: Figure 4b As shown, while HDCWNet effectively removes snowflake noise, it loses a significant amount of background texture. MPRNet, NAFNet-SIDD, NAFNet-REDS, and InvDSNet remove only a small amount of snowflakes. In contrast, the three variant models proposed in this invention perform better, especially TRIPResNet, while TRIP-SegFormer performs slightly worse.

[0190] (6) Parameter study

[0191] To evaluate the impact of key hyperparameters in the ADMM-DIP framework on snow removal performance, this embodiment conducts a systematic parameter study experiment on the Snow100K-synthetic dataset using TRIP-ResNet as an example. The experiment focuses on the total variational regularization coefficient. ADMM penalty term coefficient And the PCG_TOL expansion for solving linear subproblems. The results are shown in Figure 5.

[0192] Total variational regularization :like Figure 5a As shown, the performance indicators exhibit a trend of first increasing and then decreasing. When At that point, the model reached its optimal balance, i.e., PSNR ≈ 28.96 dB. Too small a value would result in a PSNR of 28.96 dB. It cannot effectively suppress the noise generated during the snow removal process, and when After exceeding 0.004, both PSNR and SSIM show a significant decline, indicating that excessively strong total variation constraints can lead to over-smoothing of image background details.

[0193] ADMM penalty parameters Experiments such as Figure 5b show, The constraint strength between the reconstructed image J and the auxiliary variable Z was controlled, and the smaller A value such as 0.002 can result in a higher PSNR value. With... With the increase of [something], although SSIM remained relatively stable within a certain range, PSNR decreased significantly. This indicates that at high [something]... Under this setting, ADMM iterations prematurely fall into local optima, limiting the space for DIP priors to recover background textures.

[0194] Solving the tolerance PCG_TOL: This embodiment analyzes the convergence threshold of the PCG solver in logarithmic coordinates. For example... Figure 5c As shown, when PCG_TOL≤ At that time, the recovery quality remained at an extremely high level. However, once the tolerance exceeded [a certain threshold], [the quality declined]. The reconstruction performance deteriorated sharply, with the PSNR dropping below 28.45 dB. This verifies the importance of high-precision linear inverse operations in the physical consistency solution process, and also indicates... It is the ideal threshold that balances computational efficiency and accuracy.

[0195] (7) Ablation test

[0196] To verify the effectiveness of each module, this embodiment conducted ablation experiments on the Snow100K-synthetic dataset.

[0197] Table 5 shows the performance improvement of StepTwo, which incorporates the ADMM-DIP physical optimization stage, compared to StepOne, which only uses the mask generation stage. The results show that all variants exhibit significant improvements in PSNR and SSIM after introducing the second optimization stage. In particular, the ResNet-101 variant shows a 4.58% improvement in PSNR and a 1.39% improvement in SSIM. This fully demonstrates the effectiveness of the physical prior in correcting mask errors and restoring image details. In contrast, the SegFormer variant shows a relatively smaller improvement because the mask quality generated in its first stage is already high, leaving relatively limited room for subsequent optimization.

[0198] Table 5 Ablation Experiment Results

[0199]

[0200] Conclusion: This invention proposes a novel general framework for image physics TRIP using transfer regularization, effectively addressing the two major challenges of weak generalization ability and background texture loss in single-image snow removal tasks, thus improving the snow removal performance and accuracy of the snow removal model. By innovatively transferring general semantic knowledge from the visual foundation model to snowflake detection, robust mask estimation for complex realistic snow scenes is achieved, significantly reducing the difference between the synthetic and real domains. Simultaneously, the ADMM-DIP physical inpainting algorithm designed in this invention restores the fine texture of occluded areas, avoiding the over-smoothing problem of traditional end-to-end models. Extensive experimental results demonstrate that TRIP outperforms existing state-of-the-art methods in both qualitative and quantitative metrics. Future work will focus on further improving the inference speed of the iterative solver and exploring the extension of this framework to other severe weather recovery tasks such as rain removal and defogging.

[0201] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A snow removal method based on the TRIP (Transfer Regularization Image Physical Framework), characterized in that, Includes the following steps: Step 1: Construct a snow removal system based on the TRIP image physical framework, which consists of an image acquisition module and a snow removal physical model connected in sequence. The snow removal physical model is configured with an input layer, a snow mask generation module, an initialization module, an ADMM-DIP physical repairer, and an output layer connected in sequence. Step 2: The image acquisition module acquires the snow scene image I and passes it to the snow removal physical model; Step 3: The input layer in the snow removal physical model acquires the snow scene image I and passes it to the snow mask generation module; Step 4: The snow mask generation module generates a corresponding snow mask M based on the snow scene image I and passes it to the initialization module; Step 5: The initialization module pre-fills the background image area covered by the snow mask M to generate an initialization image. And pass it to the ADMM-DIP physical repairer; Step 6: The ADMM-DIP physical repair tool applies the initialization image. The image is repaired, and then the repaired, clear image is output through the output layer. .

2. The snow removal method based on the TRIP image physical framework according to claim 1, characterized in that: The model expression for the snow mask M generated by the snow mask generation module is as follows: ； in, This is a transfer learning snowflake mask generation model built into the snow mask generation module. Image of a snow scene; This is an image processing function.

3. The snow removal method based on the TRIP image physical framework according to claim 2, characterized in that: The snowflake mask generation model is trained using a transfer learning strategy, and a hybrid loss is used as the overall loss function of the model during training. The expression is: ； in, express Loss weight value; This represents the weighted binary cross-entropy loss; express Loss weight value; express Coefficient loss; Weighted binary cross-entropy loss The expression is: ； coefficient loss The expression is: ； in, Indicates the total number of pixels. Represents the j-th pixel. Indicates the weighting coefficient. This represents the true label of the j-th pixel; This represents the probability value that the model predicts the j-th pixel to be a snowflake; It is a constant.

4. A snow removal method based on the TRIP image physical framework according to claim 2 or 3, characterized in that: The transfer learning snowflake mask generation model is either a transfer learning snowflake mask generation model based on semantic segmentation SegFormer, a transfer learning snowflake mask generation model based on residual network ResNet-101, or a transfer learning snowflake mask generation model based on segmentation of all SAMs.

5. The snow removal method based on the TRIP image physical framework according to claim 1, characterized in that: The initialization module uses the Fast Moving Model (FMM) inpainting algorithm to pre-fill the snow scene image area covered by the snow mask M, generating an initialization image. The expression is: ； in, To cover the snow, Image of a snow scene; This is an image restoration function.

6. The snow removal method based on the TRIP image physical framework according to claim 1, characterized in that: The ADMM-DIP physics repair tool is used to model the snow removal problem as an optimization problem under physical constraints, and the optimization problem is defined as follows: ； ； in, To initialize the image, To cover the snow; The background image recovered from snow scene image I; These are parameters for the DIP texture generator. As an auxiliary variable, For the total variation TV smoothness constraint, Calculate the square of the L2 norm. Calculate the L1 norm. For element-wise multiplication, For spatial gradient operators, For parameters DIP texture generator; The augmented Lagrangian function corresponding to the optimization problem is: ； in, For penalty parameters, ; To scale the dual variable.

7. A snow removal method based on the TRIP image physical framework according to claim 6, characterized in that: The ADMM-DIP physics repairer is equipped with an interconnected DIP texture generator and an ADMM physics solver. The DIP texture generator is used to perform prior prediction on the initialized image to obtain the prior structure. The expression is: ; The ADMM physical solver is used to decompose the optimization problem into a background image using the Alternating Direction Multiplier Method (ADMM). Update, auxiliary variables Updating and scaling dual variables The three subproblems are updated, and the preprocessed conjugate gradient method (PCG) is used to solve the subproblems.

8. The snow removal method based on the TRIP image physical framework according to claim 7, characterized in that: The ADMM-DIP physical repair tool initializes the image. The specific steps for image restoration are as follows: Step S1: Initialize variables: Set the initial background image Initial auxiliary variables Initial scaling dual variable ; Step S2: Initialize the DIP texture generator The initial weight parameters are generated using a random number generator; Step S3: Enter the external ADMM iteration, number of external ADMM iterations ; Step S4: The DIP texture generator is parameterized. The optimization iterations are performed, with a total of L iterations within the DIP layer. The parameters of the DIP texture generator are... The update expression is: ； ； ； in, To approach in non-snow areas Data loss, The surrogate loss is used to approximate the auxiliary variable objective (zd) with the gradient field; When the total number of iterations within the DIP reaches L, the DIP texture generator initializes the image based on the parameters obtained in the Lth iteration. Perform prior prediction to obtain the prior structure The expression is: ； Step S5: Image Reconstruction: For the background image The updated subproblem is that the ADMM physics solver will update the prior structure predicted by the DIP texture generator. As background image The initial values ​​of the updated subproblem are used to perform update calculations to obtain the updated background image. The expression is: ； Step S6: Auxiliary variable update: For auxiliary variables The updated subproblem, the ADMM physics solver first calculates the current gradient. Then, by using soft thresholding for denoising, the updated auxiliary variables are obtained. The expression is: ； ； in, For symbolic functions, It is a function with maximum value. For spatial gradient operators; Step S7: Scaling Dual Variable Update: For scaling dual variables... The updated subproblem, calculated by the ADMM physics solver, is obtained through the following update expression: ； Step S8: Repeat steps S4-S7 until the external ADMM iteration count reaches... At this point, the background image obtained from the last iteration will be... As a clear image after restoration Output.

Images