Complex degraded image restoration method and system based on space-time decoupling double-path diffusion model

Abstract

The invention relates to the technical field of image restoration, in particular to a complex degraded image restoration method and system based on a space-time decoupling double-path diffusion model. Aiming at the problems of coupling and superposition of various degradation factors such as low illumination, raindrop shielding, motion blur and noise in a real scene and the defects that a traditional diffusion model is large in sampling step number, poor in spatial adaptability and incapable of processing mixed degradation in an end-to-end mode, a space-time decoupling mechanism is provided; the weights of the structure repair path and the detail generation path are dynamically distributed at the pixel level; three-stage progressive scheduling driven by a frequency domain signal-to-noise ratio is constructed in a time dimension, a diffusion process is strictly divided into a structure repair stage, a transition fusion stage and a detail generation stage, and the number of sampling steps is reduced by 35%. The method realizes end-to-end hybrid degradation restoration, does not need degradation prior, has the advantages of high quality, high efficiency, strong generalization, explainability and the like, and is suitable for monitoring, vehicle-mounted, satellite and other real-time visual systems.

Description

Technical Field

[0001] This invention relates to the field of image restoration technology, specifically to a method and system for restoring complex degraded images based on a spatiotemporally decoupled dual-path diffusion model. Background Technology

[0002] With the rapid development of computer vision and mobile imaging technologies, image restoration has become a core component of various visual perception systems, such as intelligent surveillance, vehicle vision, and satellite remote sensing, and its performance directly determines the decision-making accuracy of downstream tasks. However, image degradation in real-world scenarios often exhibits complex characteristics due to the coupling and superposition of multiple factors. For example, nighttime road monitoring often faces combined interference from low light, raindrop occlusion, motion blur, and sensor noise, while outdoor vehicle imaging may encounter the synergistic effects of fog attenuation, backlight glare, and vibration blur. These degradation factors interact and amplify each other, forming a nonlinear degradation process that poses a severe challenge to image restoration technology.

[0003] Traditional algorithms targeting a single degradation type, such as Wiener filtering for Gaussian noise and blind deconvolution for motion blur, are essentially mathematical models and targeted optimizations of specific degradation mechanisms. These methods lack the ability to globally adapt to the coupling relationships of multiple factors, and can only process them step by step in a fixed order. This easily introduces accumulated errors, and the distortions in the preceding processing are amplified by the subsequent steps, ultimately leading to structural distortions, loss of details, or artifact residues in the restored image. They are also difficult to maintain stable robustness under complex degradation conditions.

[0004] In recent years, end-to-end methods based on convolutional neural networks (CNNs) have achieved significant results in single degradation restoration tasks, but they still have obvious limitations. The parameters of these models are fitted to features of specific degradation types in the training set, and their performance is highly dependent on the degradation patterns of the training data. When faced with unseen combinations of degradations, the models lack corresponding feature learning and struggle to extract the core features of mixed degradations, leading to a sharp drop in performance. Simultaneously, CNN models have limited receptive fields and insufficient ability to balance global semantics with local details. When handling large-scale mixed degradations, they are prone to problems where local restoration is inconsistent with the global scene, limiting their generalization ability; as seen in Chinese patent applications with publication numbers CN117541508A and CN107730469A.

[0005] Diffusion probability models (DPMs) offer a new path for image restoration due to their stable training and high-quality generation. However, existing methods based on diffusion models still have many shortcomings: low computational efficiency (traditional DDPM / DDIM require 500-1000 iterative sampling steps, resulting in excessively long inference times and failing to meet the needs of real-time applications); poor spatial adaptability (globally uniform denoising strategies cannot adapt to local degradation differences, leading to insufficient restoration of severely degraded areas or excessive smoothing of non-degraded areas); limited ability to handle mixed degradation, relying on explicit task identifiers or degradation priors, making end-to-end processing difficult; and insufficient interpretability (lacking a staged control mechanism, failing to intuitively reflect the "structure first, details later" restoration logic, thus restricting model optimization and applications in high-reliability fields). Summary of the Invention

[0006] The purpose of this invention is to propose a method and system for restoring complex degraded images based on a spatiotemporally decoupled dual-path diffusion model, which achieves efficient inference, pixel-level adaptive repair, end-to-end processing of multi-factor coupled degradation, and staged interpretable restoration, while balancing model performance and deployment cost to meet the needs of real-time applications.

[0007] According to a first aspect of the present disclosure, a method for restoring complex degraded images based on a spatiotemporally decoupled dual-path diffusion model is provided, comprising the following steps: The degraded image to be restored is acquired, and the degraded image is preprocessed to obtain a preprocessed image; A spatiotemporally decoupled dual-path diffusion model is constructed. The model includes a residual diffusion path and a noise diffusion path set in parallel, as well as a gated residual attention network and a progressive three-stage scheduler. The residual diffusion path is used for structural degradation repair, and the noise diffusion path is used for detailed texture generation. The preprocessed image is input into a gated residual attention network to extract the spatial and temporal embedding features of the image and generate a pixel-level gated weight map, which is used to dynamically fuse the output of the residual diffusion path and the noise diffusion path. The frequency domain signal-to-noise ratio (FSNR) is obtained through a progressive three-stage scheduler. Based on the rate of change of FSNR(t), the diffusion process is divided into a structural repair stage, a transition fusion stage, and a detail generation stage. A continuous diffusion coefficient scheduling function is designed for each stage to dynamically adjust the diffusion coefficients of the two paths. The multi-task combined loss is obtained based on the difference between the output image after adjusting the diffusion coefficient and the real clear image. The model parameters are then optimized through backpropagation training. During the inference phase, based on the trained model, an adaptive importance sampling strategy is used to adjust the sampling density of different regions, and an early termination mechanism is triggered by the FSNR(t) growth threshold to output the final restored image.

[0008] In one embodiment, the degraded image to be restored is acquired, and the degraded image is preprocessed, specifically as follows: The original image is acquired by an imaging device, which contains at least two degradation factors among low light, raindrops, motion blur and noise. The original image is cropped into a sub-image of a preset size using a sliding window method. The edge deficiency is processed by mirror filling to ensure the integrity of the target information. The cropped RGB sub-images are white-balanced and then normalized to a preset range using a linear scale to generate a preprocessed tensor, which is used as input for the subsequent spatiotemporal decoupling dual-path diffusion model.

[0009] In one embodiment, the forward process of the residual diffusion path is as follows: ,in, Indicates the residual path at time step The hidden state, The residual diffusion coefficient is... The unit covariance matrix; The forward process of the noise propagation path is as follows: ;in, Indicates the noise path at time step The hidden state, The noise diffusion coefficient; The two paths mentioned above share the first three layers of the encoder and the last two layers of the decoder.

[0010] In one embodiment, the gated residual attention network includes an input projection layer, a depthwise separable convolutional layer, a channel attention module, a spatial attention module, a temporal embedding fusion layer, and an output layer. The specific processing procedure is as follows: Receive U-Net encoder output feature map After Conv1×1 projection, the dimension reduction is as follows: ; Receive the sine-cosine encoded vector of the current time step t Mapped to via a fully connected layer (FC) Then, it is expanded to space broadcasting. ; right Performing the sequence of operations 3×3DWConv→1×1PWConv→BN→ReLU→Res yields the enhanced feature map. ; Enhanced feature maps Perform global average pooling → FC↓ (64 → 128) → ReLU → FC↑ (128 → 64) → Sigmoid to generate channel weights. ,and The feature map is obtained by multiplying each channel. ; For feature maps Perform Conv3×3→Sigmoid to generate spatial weights. ,and Multiplying by spatial position yields ; After broadcast and Element-by-element addition: spliced ​​along the channel dimension as After Conv1×1 (128→1) and Sigmoid activation, a pixel-level gated weight map is output. .

[0011] In one embodiment, the frequency domain signal-to-noise ratio-based The diffusion process stage division and scheduling function design include: The frequency domain signal-to-noise ratio (FSNR)(t) is: in For stability parameters, Low-frequency energy, For high-frequency energy; the rate of change is defined using forward differential. To depict the temporal evolution trend of FSNR; The end point of the structural repair phase is: ; End point of the transition and integration phase: , The growth rate threshold; Upper limit of detail generation stage: Set growth rate threshold With the continuous verification window length K, satisfying ,by Premature termination of sampling; Residual coefficients in three stages With noise figure satisfy And maintain C at stage boundaries T1 and T2 1 continuous; Structural repair phase: Transition and Integration Phase: Detail generation stage: , This is the attenuation rate adjustment coefficient.

[0012] In one embodiment, the total number of sampling steps N is allocated to the structure repair stage, transition fusion stage, and detail generation stage in a ratio of N1:N2:N3=1:2:1, where N1+N2+N3=N and N≪T3; the step sizes for each stage are as follows: .

[0013] In one embodiment, the multi-task combination loss function is: in, These are the weighting coefficients for each loss component. To rebuild the losses, In order to perceive loss, To combat the losses, This is the loss due to regularization.

[0014] According to a second aspect of the present disclosure, a complex degraded image restoration system based on a spatiotemporally decoupled dual-path diffusion model is provided, comprising: The image preprocessing module acquires the degraded image to be restored and preprocesses the degraded image to obtain a preprocessed image. The dual-path model construction module constructs a spatiotemporally decoupled dual-path diffusion model. The model includes a residual diffusion path and a noise diffusion path set in parallel, as well as a gated residual attention network and a progressive three-stage scheduler. The residual diffusion path is used for structural degradation repair, and the noise diffusion path is used for detailed texture generation. The gated weight generation module inputs the preprocessed image into the gated residual attention network, extracts the spatial and temporal embedding features of the image, and generates a pixel-level gated weight map for dynamically fusing the output of the residual diffusion path and the noise diffusion path. The diffusion stage scheduling module obtains the frequency domain signal-to-noise ratio FSNR(t) through a progressive three-stage scheduler, and divides the diffusion process into a structural repair stage, a transition fusion stage, and a detail generation stage based on the rate of change of FSNR(t). A continuous diffusion coefficient scheduling function is designed for each stage to dynamically adjust the diffusion coefficients of the two paths. The model training and optimization module obtains a multi-task combined loss based on the difference between the output image after adjusting the diffusion coefficient and the real clear image, and optimizes the model parameters through backpropagation training. In the inference output module, during the inference phase, based on the trained model, an adaptive importance sampling strategy is used to adjust the sampling density of different regions, and an early termination mechanism is triggered by the FSNR(t) growth threshold to output the final restored image.

[0015] According to a third aspect of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and running on the memory. When the processor executes the program, it implements the aforementioned method for restoring complex degraded images based on a spatiotemporally decoupled dual-path diffusion model.

[0016] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the aforementioned method for restoring complex degraded images based on a spatiotemporally decoupled dual-path diffusion model.

[0017] The advantages of the above technical solutions adopted in this invention compared with the prior art are as follows: 1. Addressing the core challenge of multiple degradation factors coupled and superimposed in real-world scenarios, such as low light, raindrop occlusion, motion blur, and noise, this method achieves end-to-end restoration through a spatiotemporally decoupled dual-path architecture without relying on explicit degradation priors or task identifiers. Compared to traditional single-task-specific models, it can flexibly adapt to unseen degradation combinations, maintain stable performance in complex scenarios, and demonstrates significantly better generalization capabilities than existing methods.

[0018] 2. Spatial dimension: A lightweight gated residual attention network is used to dynamically allocate weights for structural restoration and detail generation at the pixel level, accurately matching local degradation differences in the image. The residual path focuses on capturing core information such as geometric structure and edge contours, while the noise path is responsible for generating texture details and high-frequency features. The synergy between the two ensures that the restoration result maintains structural integrity while possessing realistic and natural texture details.

[0019] 3. In the time dimension, a three-stage progressive scheduling mechanism is constructed based on the frequency domain signal-to-noise ratio. The diffusion process is divided into structural repair, transition fusion and detail generation stages. Combined with non-uniform step size allocation and early termination strategy, the number of sampling steps is greatly reduced.

[0020] 4. By observing the three-stage evolution of the frequency domain signal-to-noise ratio (FSNR), the temporal relationship between structural repair and detail generation is intuitively presented. The stage boundaries are automatically determined by the FSNR change rate, and the diffusion coefficient scheduling function satisfies C at the stage transition. 1 Continuous operation avoids numerical oscillations. Compared to the "black box" iteration of traditional diffusion models, the phased control mechanism of this invention makes the recovery process interpretable and traceable, facilitating subsequent optimization and debugging.

[0021] 5. By adopting a dual-path parameter sharing strategy, combined with a lightweight gated residual attention network and a fast feature extraction module, the model size is significantly reduced while ensuring performance. It can be flexibly deployed in different scenarios such as surveillance cameras, vehicle terminals, and satellite image processing systems, making it highly practical. Attached Figure Description

[0022] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an undue limitation of this application.

[0023] Figure 1This is a diagram illustrating the overall framework of the spatiotemporal decoupling progressive dual-path diffusion model (ST-DDPM) of this invention. Figure 2 This is a structural diagram of the gated residual attention network (GRAN) of the present invention; Figure 3 This is a diagram of the U-shaped backbone network and dual-path cooperative structure of the present invention; Figure 4 This is a detailed structural diagram of the fast feature extraction module FBblock of the present invention; Figure 5 The images show a comparison of the restoration effects of this invention in a mixed degradation scene (low light + raindrops + blur); Figure 6 This is an example diagram illustrating the three-stage evolution process of FSNR on a typical hybrid degradation sample according to the present invention. Detailed Implementation

[0024] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0025] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and systems according to various embodiments of this disclosure. It should be noted that each block in a flowchart or block diagram may represent a module, segment, or portion of code, which may include one or more executable instructions for implementing the logical functions specified in the various embodiments. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutively represented blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, may be implemented using a dedicated hardware-based system that performs the specified functions or operations, or using a combination of dedicated hardware and computer instructions.

[0026] Example 1: This embodiment provides a method for restoring complex degraded images based on a spatiotemporally decoupled dual-path diffusion model, including the following steps: S1. Obtain the degraded image to be restored, and preprocess the degraded image to obtain a preprocessed image; S1.1 Complex Degraded Image Acquisition and Preprocessing: An original image containing at least two of the following degradation factors—low light, raindrops, motion blur, and noise—is acquired using an imaging device. This original image is then cropped into 512×512 pixel sub-images using a sliding window method, with a sliding window step size of 384 pixels (i.e., 25% overlap). Insufficient edges are filled using mirroring to bring the image to 512×512. The filling formula is as follows: Ensure the integrity of the target information; S1.2 Normalization and Tensor Generation: The cropped RGB sub-image is first subjected to white balance correction, and then normalized to [-1,1] using a linear scale to obtain the preprocessed tensor. This is used as input for subsequent dual-path diffusion models.

[0027] This embodiment uses vehicle-mounted surveillance cameras and industrial cameras to acquire raw images in real and complex environments, including nighttime roads (low light + motion blur), rainy windshield (raindrop obstruction + noise), and underground parking lots (low light + compression artifacts). A total of 200 raw video sequences were acquired, with 100 keyframes extracted from each sequence, resulting in a total of 20,000 raw images at a uniform resolution of 1920×1080. To expand data diversity, a physical simulation chamber was constructed in a darkroom environment, where the following degradation factors were controllably superimposed: low light, raindrop obstruction, motion blur, and Gaussian noise. For each clear image, 2-3 degradation factors were randomly combined to generate paired data, ultimately constructing the Mixed-Degrade-5k dataset: 5,000 pairs for training, 500 pairs for validation, and 500 pairs for testing. The 1920×1080 image was cropped using a 512×512 window with a 25% overlap sliding window and a stride of 384 pixels. Insufficient edges were filled using mirroring. Nine sub-images were generated from each large image. The final training set was expanded to 45,000 sub-image-parameter pairs. After cropping, the RGB sub-images were first white-balanced and then linearly normalized to [-1,1], resulting in a preprocessed tensor I0∈[-1,1]^(3×512×512), which was then input into the model.

[0028] S2. Construct a spatiotemporally decoupled dual-path diffusion model, which includes a residual diffusion path and a noise diffusion path set in parallel, as well as a gated residual attention network and a progressive three-stage scheduler. The residual diffusion path is used for structural degradation repair, and the noise diffusion path is used for detailed texture generation. S2.1 The residual diffusion path and the noise diffusion path process the input image in parallel. The outputs of the two paths are dynamically fused through a gated residual attention network. The fusion weights are automatically adjusted by a progressive three-stage scheduler based on the frequency domain signal-to-noise ratio (FSNR)(t) at the current time step t. The residual diffusion path is responsible for structural degradation repair, and the noise diffusion path is responsible for detail texture generation. The two paths share the first 3 layers of the encoder and the last 2 layers of the decoder. The independent prediction head contains only 1×1 convolutions, and the amount of additional parameters is <30%.

[0029] The residual diffusion path described in S2.2 is used to model structural degradation, and its forward process is represented as follows: in, Indicates the residual path at time step The hidden state, The residual diffusion coefficient is... The residual diffusion path, which is a unit covariance matrix, captures the geometric structure, edge contours, and large-scale semantic information of the image through directional diffusion.

[0030] S2.3 Forward Modeling of Noise Diffusion Path. The noise diffusion path is used to model stochastic details, and its forward process is represented as follows: in, Indicates the noise path at time step The hidden state, The noise diffusion coefficient is used to generate texture details, high-frequency information, and random perturbations through standard Gaussian diffusion, ensuring the realism and diversity of the restoration results.

[0031] The residual path and noise path described in S2.4 share weights in the first 3 layers of the encoder and the last 2 layers of the decoder, with a sharing ratio of 70%. They only use independent 1×1 convolutions in their respective prediction heads. The independent prediction heads are used to map the fused features to the image space. This design compresses the overall parameter size to <30 M while maintaining the independent expressive power of the two paths, meeting the deployment requirements of mobile terminals and edge devices.

[0032] The gated residual attention network described in S2.5 includes a depthwise separable convolutional layer, a channel attention module, a spatial attention module, and a sigmoid output layer. The depthwise separable convolutional layer reduces computational cost, the channel attention module recalibrates feature channel weights, the spatial attention module captures pixel-level importance, and the sigmoid output layer generates a normalized gated weight map. .

[0033] The progressive three-stage scheduler described in S2.6 divides the diffusion process into a structural repair stage, a transition fusion stage, and a detail generation stage based on the frequency domain signal-to-noise ratio (FSNR)(t). The FSNR(t) is obtained by calculating energy in both the low-frequency and high-frequency ranges using 2D-DCT. The three-stage boundaries T1, T2, and T3 are determined by the FSNR change rate. Automatically determined and dynamically adjusted diffusion coefficient based on the current stage. , .

[0034] S2.7 Preprocessed Image Simultaneously input the residual path and the noise path, and the outputs of the two paths at time step t. , Weight graph generated by gated residual attention network ,according to After merging, the data is sent to the next stage. The progressive three-stage scheduler updates FSNR(t) at each time step and determines the current stage, and then updates... , It can be used for dual-path forward control to form a closed-loop control.

[0035] S3. Input the preprocessed image into a gated residual attention network, extract the spatial and temporal embedding features of the image, and generate a pixel-level gated weight map for dynamically fusing the output of the residual diffusion path and the noise diffusion path. The network includes an input projection layer, a depthwise separable convolutional layer, a channel attention module, a spatial attention module, a temporal embedding fusion layer, and an output layer, with a total number of parameters <0.3 M, which is only 1% of the backbone network.

[0036] Receive U-Net encoder output feature map After 1×1 projection by Conv, the dimension reduction is... The computational load is reduced by 87.5%; the sine-cosine encoded vector of the current time step t is received. Mapped through a fully connected layer FC (128→64) to Then, it is expanded to space broadcasting. ; opposing Performing a sequence of operations—3×3 DW Conv → 1×1 PW Conv → BN → ReLU → Res—results in enhanced features. .

[0037] Perform global average pooling on F1 → FC↓(64→128) → ReLU → FC↑(128→64) → Sigmoid to generate channel weights. Multiply by F1 channel by channel to get .

[0038] Perform Conv 3×3→Sigmoid on F2 to generate spatial weights. Multiplying by F2 spatially position yields .

[0039] After broadcast and Element-by-element addition: spliced ​​along the channel dimension as After Conv 1×1 (128→1) and Sigmoid activation, a pixel-level gated weight map is output. .

[0040] See Figure 1 The ST-DDPM model was constructed within the PyTorch framework, with the residual path and noise path running in parallel. The two paths share the last two layers of the U-Net encoder and decoder (70% sharing rate), and each independent prediction head contains a 1×1 convolution, with a total parameter count of 28.3M.

[0041] See Figure 3 The encoder uses 4-layer downsampling: Layer1: Conv(3→64) + ×2 + MaxPool2×2 Layer2: Conv(64→128) + ×2 + MaxPool2×2 Layer3: Conv(128→256) + ×2 + MaxPool2×2 Layer4: Conv(256→512) + ×2 The decoder uses symmetrical 4 layers of talent, and the horizontal jump connection is fused after channel attention weighting.

[0042] See Figure 4 Each include: Branch 1: PConv3×3 → Conv1×1 → Conv1×1 Branch 2: PConv1×1 Branch 3: Skip Connection After the three branches are concatenated and then dimensionality is reduced by 1×1 convolution, the number of parameters is only 40% of that of a standard C2f block.

[0043] See Figure 2 GRAN structure: Input projection: Conv1×1 (512→64) dimensionality reduction Depthwise separable convolution: 3×3 DW Conv → 1×1 PW Conv → BN → ReLU → Res CA module: GlobalAvgPool → FC (64→128) → ReLU → FC (128→64) → Sigmoid SA module: Conv3×3 → Sigmoid Time merging: FC (128→64) broadcast followed by summation Output layer: Conv1×1(128→1) → Sigmoid, generating G_t∈[0,1]^(32×32) (upsampled to 512×512 for fusion).

[0044] S4. The frequency domain signal-to-noise ratio FSNR(t) is obtained through a progressive three-stage scheduler, and the diffusion process is divided into a structural repair stage, a transition fusion stage, and a detail generation stage based on the rate of change of FSNR(t). A continuous diffusion coefficient scheduling function is designed for each stage to dynamically adjust the diffusion coefficients of the two paths. S4.1 Let For time step The noisy image, its two-dimensional discrete cosine transform (2D-DCT) is denoted as: in, Indicates the vertical frequency index. Representing the horizontal frequency index, it is normalized as follows: Based on the statistical law that "low frequencies represent structure and high frequencies represent details" in natural images, frequency domain energy is decomposed into low-frequency energy and high-frequency energy.

[0045] (1) Low-frequency energy : in These are DCT coefficients accessed using normalized coordinates. The central region of the corresponding DCT spectrum is used to measure the energy of the main structure.

[0046] (2) High-frequency energy : in The ring-shaped high-frequency region corresponding to the DCT spectrum, excluding the low-frequency region, includes high-frequency components such as edges, textures, details, and noise.

[0047] (3) Frequency domain signal-to-noise ratio (FSNR): in Used for stability. FSNR measures "structural energy percentage," thus reflecting the dominance of structure and details at the current time step.

[0048] S4.2 To automatically determine the boundaries between the "structural repair stage" and the "transition fusion stage" during back-diffusion, this invention constructs a discrete quantization criterion based on the temporal variation characteristics of FSNR. Through statistical analysis of the ImageNet-1k validation set (50k images), and to achieve robust detection in the discrete time domain, referencing the inflection point detection theory in signal processing and the non-uniform parameterization concept of EDM, the following provides a rigorous definition of the FSNR change rate and expressions for the boundaries of each stage.

[0049] To characterize the growth, saturation, and decline trends of FSNR over time, forward differencing is used to define its rate of change: When structural information is quickly recovered Larger; when the structure tends to stabilize or enters the detail generation stage, It will gradually approach zero. Back diffusion can be divided into three stages: "structural repair - transition fusion - detail generation". The changing trend of FSNR provides a natural basis for the stage division.

[0050] End of structural repair phase Before sampling Within a given timeframe, FSNR typically exhibits a rapid upward trend, reaching a growth rate threshold of [missing information]. At that point, the FSNR growth rate subsequently declined. The timing of the basic structural repair can be determined by the inflection point of the FSNR growth rate. Definition: This definition corresponds to the point in time when the growth rate of FSNR first shows a significant decline.

[0051] End of transition and integration phase In the mid-term, FSNR tends to saturate, at which point detailed generation becomes dominant. The first moment when the FSNR growth rate drops to this threshold is defined as: This moment marks the transition of FSNR from growth to stability, and the system enters the detail generation phase. A relatively small growth threshold (determined by the validation set, approximately 0.02–0.05).

[0052] Detail generation phase cap and early stopping mechanism After entering the detail generation stage, the change in FSNR becomes extremely slow, and high-frequency details are gradually added. To avoid unnecessary iterations in this stage, this invention sets an upper limit for the detail generation stage. An early cessation strategy is introduced based on the FSNR growth rate. A growth rate threshold is set. (Typically 0.005–0.01), and a continuous validation window length K (e.g., 20 or 30). If the FSNR changes below the threshold for K consecutive time steps, it is determined that the detail supplementation has saturated, and sampling is terminated early. This mechanism can significantly reduce invalid computations in the detail stage while maintaining the final image quality.

[0053] The empirical threshold is determined by grid search optimization on the validation set as follows: FSNR(t) evolves in three stages over time, as follows: Figure 2 As shown.

[0054] S4.3 After defining the boundaries of the three stages T1, T2, and T3, this invention constructs noise scheduling functions with different characteristics for each stage, ensuring that backdiffusion achieves the most suitable noise attenuation rate in all three stages: structure restoration, information fusion, and detail generation. To this end, the entire sampling interval is divided into stages I, II, and III. Continuous, monotonic, and physically meaningful residual coefficients are designed for each of the three intervals. With noise figure .

[0055] Phase I (Structural Repair): , Phase II (Transitional Integration): , Phase III (Detail Generation): , exist , At, α t β t All satisfy C 1 Continuity is maintained to avoid numerical oscillations. Here, λ is a coefficient that adjusts the decay rate. Through validation set search, setting λ=3.0 achieves a good balance between convergence speed and generation quality. To avoid numerical jumps at stage transitions, the above piecewise function... and Maintaining first-order continuity: S4.4 Non-uniform step size allocation and early termination.

[0056] Under the FSNR-based three-stage scheduling strategy, the time structure of backpropagation has a clear stage division, which allows for further optimization of step allocation and termination mechanisms to reduce computational costs and improve convergence efficiency. This invention accelerates the sampling process from two aspects: non-uniform step size allocation and adaptive early stopping mechanism.

[0057] Non-uniform step size allocation Because the convergence speed and information type differ across the three stages, distributing the total number of steps non-uniformly among them can significantly improve sampling efficiency. This is based on the variation pattern of FSNR: Structural repair phase The structure recovers relatively quickly; simply reduce the number of steps. Transition and Integration Phase The structure and details are adjusted simultaneously, with a moderate rate of change. Detail generation stage The process of generating details is smooth and does not require a large number of steps.

[0058] Therefore, the total number of sampling steps Proportional allocation in three stages ,in and The corresponding stage step size is: This strategy concentrates sampling steps in the intermediate stages, thereby reducing the total number of steps to approximately [a fraction] of the original number of steps while maintaining generation quality. about Adaptive Termination Criterion During the detail generation stage, if the FSNR has remained at an extremely low level for an extended period, it indicates that the image details have largely converged, and further iterations will yield very low marginal returns in detail changes. Therefore, adaptive early stopping can be achieved by monitoring the rate of increase in FSNR. Corresponding to Equation 9, this mechanism can effectively reduce redundant iterations in the detail stage without affecting visual quality, reducing computation by an average of 25%–35%.

[0059] Overall acceleration effect By combining non-uniform step size and early stopping strategy, the overall complexity of the entire backdiffusion process can be approximately reduced to that of the original process. The reconstruction quality is 60%–70%, while the reconstruction quality remains stable or even slightly improved in indicators such as FID and PSN (thanks to more reasonable stage division and scheduling function).

[0060] S5. Based on the difference between the output image after adjusting the diffusion coefficient and the real clear image, the multi-task combined loss is obtained, and the model parameters are optimized through backpropagation training. The multi-task combination loss function is: The reconstruction loss (Lrecon) employs a combination of L1 and SSIM losses. L1 loss ensures pixel-level accuracy, while SSIM loss maintains structural similarity. The SSIM metric better reflects the human visual system's perception of image quality and is therefore widely used for evaluation and optimization in image restoration tasks. The perceptual loss (Lperceptual) is based on the feature representation of a pre-trained VGG network, optimizing perceptual quality by comparing the differences in high-level semantic features between the generated and target images. This invention selects the conv34 and conv44 layers of the VGG-19 network for perceptual loss calculation; this choice is based on the optimal configuration validated through extensive experiments. The adversarial loss (Ladversarial) adopts an improved form of WGAN-GP, enhancing the realism of the generated image by introducing a discriminator network. WGAN-GP addresses the instability problem of the original GAN ​​training, providing a stable optimization foundation for adversarial training. The discriminator network uses a lightweight design, containing only 5 convolutional layers to avoid mode collapse during training. The regularization loss (Lreg) includes a weight decay term and a gated weight regularization term; the former prevents overfitting, while the latter ensures the stability of the gating mechanism. The weight parameters were set to λ1=1.0, λ2=0.1, λ3=0.01, and λ4=0.001. These parameters were optimized on the validation set through grid search.

[0061] S5.2 Three-stage course learning.

[0062] Phase I: Single degenerate pre-training for 50 epochs, lr=2×10-4→5×10-5, Lrecon only.

[0063] Phase II: Hybrid degeneration + gated activation for 100 epochs, lr=1×10-4, with the addition of Lperceptual and Lgate. Phase III: Adversarial fine-tuning for 80 epochs, lr=5×10⁻⁵, enabling full loss and EMA parameters. S5.3 optimized configuration. AdamW optimizer, gradient clipping threshold 1.0, mixed precision training, 4×A100 distributed, batch size=64, total training time approximately 72 hours.

[0064] The above three-stage course involves 230 epochs of training; each epoch takes approximately 18 minutes, for a total training time of 72 hours.

[0065] Hardware and data loading: 4×NVIDIA A100 GPUs (40GB VRAM), hybrid training of data parallelism and model parallelism, gradient Top-K sparsity (K=10%), and a 90% reduction in communication overhead. Data loading adopts the Prefetch mechanism, with CPU preprocessing overlapping with GPU training.

[0066] Convergence monitoring: PSNR / FID / inference time are evaluated on the validation set every 5 epochs, and the optimal model is saved. Training curves show that: in stage I, PSNR rises rapidly to 22 dB; in stage II, PSNR increases to 23.5 dB after gating activation; in stage III, after adversarial fine-tuning, PSNR finally stabilizes at 24.33 dB, and FID decreases to 21.7.

[0067] S6. In the inference phase, based on the trained model, an adaptive importance sampling strategy is adopted to adjust the sampling density of different regions, and combined with the FSNR(t) growth threshold to trigger an early termination mechanism, the final restored image is output.

[0068] S6.1 EMA parameter inference. The exponential moving average parameter (β=0.999) is enabled during the inference phase, improving model stability and increasing PSNR by 0.5 dB.

[0069] S6.2 Adaptive Sampling Optimization. The sampling step size is dynamically adjusted according to the gradient norm and the rate of energy change. Fine sampling is performed in high-importance regions, while skipping in low-importance regions. The average number of steps is ≤650 (35% less than DDIM).

[0070] S6.3 Mixed Precision and Early Termination. FP16 forward pass + FP32 accumulation reduces memory usage by 50%; early termination occurs when ‖FSNR(t)-FSNR(t-Δt)‖<0.01 for three consecutive steps, saving an average of 30% of steps.

[0071] S6.4 edge deployment. TensorRT INT8 quantization achieves 1.3 FPS on a 512×512 image on a Jetson Orin Nano with a power consumption of 7.8 W, meeting the requirements of real-time applications.

[0072] TensorRT INT8 quantization: Model size compressed to 7.2 MB, inference speed improved by 2.3x. Power consumption: 3.2 W idle, 7.8 W during inference, meeting the power consumption constraints of automotive / monitoring edge devices. Frame rate: 1.3 FPS for 512×512 images, 1.8 FPS for 640×480 images Additional verification was performed on the RK3588 (NPU) and HiSilicon Hi3559A (DV300): RK3588: FP16 mode, 0.9 FPS for 512×512 images, power consumption 5.2 W. Hi3559A: INT8 mode, 0.6 FPS for 512×512 graphics, 3.5 W power consumption. Experimental Results and Comparative Analysis: Comparison with PromptIR, Uformer, Restormer, and StableDiffusion-Inpainting on the Mixed-Degrade-5k test set.

[0073] Table 1 Comparison with PromptIR, Uformer, Restormer, and StableDiffusion-Inpainting The results are compared with state-of-the-art methods on three single tasks and one mixed task, as follows: Table 2 compares the results with state-of-the-art methods on three single tasks and one hybrid task. See Figure 5 Comparison of restoration effects in a mixed degradation scene (low light + raindrops + blur) and Figure 6 An example of the three-stage evolution of FSNR on a typical hybrid degradation sample. On the hybrid degradation sample, in stage I (0-200 steps), the FSNR rapidly increases from 0.1 to 0.35, repairing large-scale dark areas; in stage II (200-450 steps), the FSNR slowly increases to 0.38, balancing edge transitions; in stage III (450-650 steps), the FSNR stabilizes at 0.38-0.39, optimizing texture details.

[0074] See Table 3 for the data after adding each key component. Each module brings a performance improvement, and the final complete model reduces inference time by approximately 35% while maintaining high quality.

[0075] Table 3 Ablation Experiment Data Example 2: This embodiment provides a complex degraded image restoration system based on a spatiotemporal decoupled dual-path diffusion model, including: The image preprocessing module acquires the degraded image to be restored and preprocesses the degraded image to obtain a preprocessed image. The dual-path model construction module constructs a spatiotemporally decoupled dual-path diffusion model. The model includes a residual diffusion path and a noise diffusion path set in parallel, as well as a gated residual attention network and a progressive three-stage scheduler. The residual diffusion path is used for structural degradation repair, and the noise diffusion path is used for detailed texture generation. The gated weight generation module inputs the preprocessed image into the gated residual attention network, extracts the spatial and temporal embedding features of the image, and generates a pixel-level gated weight map for dynamically fusing the output of the residual diffusion path and the noise diffusion path. The diffusion stage scheduling module obtains the frequency domain signal-to-noise ratio FSNR(t) through a progressive three-stage scheduler, and divides the diffusion process into a structural repair stage, a transition fusion stage, and a detail generation stage based on the rate of change of FSNR(t). A continuous diffusion coefficient scheduling function is designed for each stage to dynamically adjust the diffusion coefficients of the two paths. The model training and optimization module obtains a multi-task combined loss based on the difference between the output image after adjusting the diffusion coefficient and the real clear image, and optimizes the model parameters through backpropagation training. In the inference output module, during the inference phase, based on the trained model, an adaptive importance sampling strategy is used to adjust the sampling density of different regions, and an early termination mechanism is triggered by the FSNR(t) growth threshold to output the final restored image.

[0076] The above modules can be deployed on the same device or distributed devices; the division of modules is only a functional logic description and does not limit the specific physical boundaries or implementation order.

[0077] Example 3: An electronic device is provided for running the aforementioned "a method for restoring complex degraded images based on a spatiotemporally decoupled dual-path diffusion model". The electronic device includes a processor, a memory, and optional communication interfaces / display devices / input devices, etc.; the memory stores a computer program that can run on the processor, and when the processor executes the program, it implements steps S1 to S6 of the method described in Embodiment 1, specifically including but not limited to: S1. Obtain the degraded image to be restored, and preprocess the degraded image to obtain a preprocessed image; S2. Construct a spatiotemporally decoupled dual-path diffusion model, which includes a residual diffusion path and a noise diffusion path set in parallel, as well as a gated residual attention network and a progressive three-stage scheduler. The residual diffusion path is used for structural degradation repair, and the noise diffusion path is used for detailed texture generation. S3. Input the preprocessed image into a gated residual attention network, extract the spatial and temporal embedding features of the image, and generate a pixel-level gated weight map for dynamically fusing the output of the residual diffusion path and the noise diffusion path. S4. The frequency domain signal-to-noise ratio FSNR(t) is obtained through a progressive three-stage scheduler, and the diffusion process is divided into a structural repair stage, a transition fusion stage, and a detail generation stage based on the rate of change of FSNR(t). A continuous diffusion coefficient scheduling function is designed for each stage to dynamically adjust the diffusion coefficients of the two paths. S5. Based on the difference between the output image after adjusting the diffusion coefficient and the real clear image, the multi-task combined loss is obtained, and the model parameters are optimized through backpropagation training. S6. In the inference phase, based on the trained model, an adaptive importance sampling strategy is adopted to adjust the sampling density of different regions, and combined with the FSNR(t) growth threshold to trigger an early termination mechanism, the final restored image is output.

[0078] The electronic device hardware can be one of a server, personal computer, workstation, industrial controller, edge computing device, or mobile terminal; the processor can be a general-purpose CPU, GPU, NPU, FPGA, or a combination thereof; the memory can be RAM, ROM, flash memory, or disk array. The device can interact with local / remote data storage (acquiring observation data and outputting inversion results) through a communication interface. The above hardware configuration does not constitute a limitation of the present invention.

[0079] Example 4: A computer-readable storage medium storing a computer program, which, when run on a processor of an electronic device, causes the program to execute the method steps S1 to S6 described in Embodiment 1; the storage medium may be a disk, optical disk, flash memory, solid-state drive, read-only memory, random access memory, or any combination of the above media.

[0080] Those skilled in the art will understand that the modules or steps described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, which can then be stored in a storage device for execution by a computer device. Alternatively, they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. This disclosure is not limited to any particular combination of hardware and software.

[0081] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0082] While the specific embodiments of this disclosure have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of this disclosure. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this disclosure are still within the scope of protection of this disclosure.

Claims

1. A method for restoring complex degraded images based on a spatiotemporally decoupled dual-path diffusion model, characterized in that, Includes the following steps: The degraded image to be restored is acquired, and the degraded image is preprocessed to obtain a preprocessed image; A spatiotemporally decoupled dual-path diffusion model is constructed. The model includes a residual diffusion path and a noise diffusion path set in parallel, as well as a gated residual attention network and a progressive three-stage scheduler. The residual diffusion path is used for structural degradation repair, and the noise diffusion path is used for detailed texture generation. The preprocessed image is input into a gated residual attention network to extract the spatial and temporal embedding features of the image and generate a pixel-level gated weight map, which is used to dynamically fuse the output of the residual diffusion path and the noise diffusion path. The frequency domain signal-to-noise ratio (FSNR) is obtained through a progressive three-stage scheduler. Based on the rate of change of FSNR(t), the diffusion process is divided into a structural repair stage, a transition fusion stage, and a detail generation stage. A continuous diffusion coefficient scheduling function is designed for each stage to dynamically adjust the diffusion coefficients of the two paths. The multi-task combined loss is obtained based on the difference between the output image after adjusting the diffusion coefficient and the real clear image. The model parameters are then optimized through backpropagation training. During the inference phase, based on the trained model, an adaptive importance sampling strategy is used to adjust the sampling density of different regions, and an early termination mechanism is triggered by the FSNR(t) growth threshold to output the final restored image.

2. The method for restoring complex degraded images based on a spatiotemporally decoupled dual-path diffusion model according to claim 1, characterized in that, The degraded image to be restored is acquired, and the degraded image is preprocessed, specifically as follows: The original image is acquired by an imaging device, which contains at least two degradation factors among low light, raindrops, motion blur and noise. The original image is cropped into a sub-image of a preset size using a sliding window method. The edge deficiency is processed by mirror filling to ensure the integrity of the target information. The cropped RGB sub-images are white-balanced and then normalized to a preset range using a linear scale to generate a preprocessed tensor, which is used as input for the subsequent spatiotemporal decoupling dual-path diffusion model.

3. The method for restoring complex degraded images based on a spatiotemporally decoupled dual-path diffusion model according to claim 1, characterized in that, The forward process of the residual diffusion path is as follows: ,in, Indicates the residual path at time step The hidden state, The residual diffusion coefficient is... The unit covariance matrix; The forward process of the noise propagation path is as follows: ;in, Indicates the noise path at time step The hidden state, The noise diffusion coefficient; The two paths mentioned above share the first three layers of the encoder and the last two layers of the decoder.

4. The method for restoring complex degraded images based on a spatiotemporally decoupled dual-path diffusion model according to claim 1, characterized in that, The gated residual attention network includes an input projection layer, a depthwise separable convolutional layer, a channel attention module, a spatial attention module, a temporal embedding fusion layer, and an output layer. The specific processing procedure is as follows: Receive U-Net encoder output feature map After Conv1×1 projection, the dimension reduction is as follows: ; Receive the sine-cosine encoded vector of the current time step t Mapped to via a fully connected layer (FC) Then, it is expanded to space broadcasting. ; right Performing the sequence of operations 3×3DWConv→1×1PWConv→BN→ReLU→Res yields the enhanced feature map. ; Enhanced feature maps Perform global average pooling → FC↓ (64 → 128) → ReLU → FC↑ (128 → 64) → Sigmoid to generate channel weights. ,and The feature map is obtained by multiplying each channel. ; For feature maps Perform Conv3×3→Sigmoid to generate spatial weights. ,and Multiplying by spatial position yields ; After broadcast and Element-by-element addition: spliced ​​along the channel dimension as After Conv1×1 (128→1) and Sigmoid activation, a pixel-level gated weight map is output. .

5. The method for restoring complex degraded images based on a spatiotemporally decoupled dual-path diffusion model according to claim 1, characterized in that, The frequency domain signal-to-noise ratio The diffusion process stage division and scheduling function design include: The frequency domain signal-to-noise ratio (FSNR)(t) is: in For stability parameters, Low-frequency energy, For high-frequency energy; the rate of change is defined using forward differential. To depict the temporal evolution trend of FSNR; The end point of the structural repair phase is: ; End point of the transition and integration phase: , The growth rate threshold; Upper limit of detail generation stage: Set growth rate threshold With the continuous verification window length K, satisfying ,by Premature termination of sampling; Residual coefficients in three stages With noise figure satisfy And maintain C at stage boundaries T1 and T2 1 continuous; Structural repair phase: Transition and Integration Phase: Detail generation stage: , This is the attenuation rate adjustment coefficient.

6. The method for restoring complex degraded images based on a spatiotemporally decoupled dual-path diffusion model according to claim 5, characterized in that, The total number of sampling steps N is allocated to the structural repair stage, transition fusion stage, and detail generation stage in the ratio N1:N2:N3=1:2:1, where N1+N2+N3=N and N≪T3; the step sizes for each stage are as follows: 。 7. The method for restoring complex degraded images based on a spatiotemporally decoupled dual-path diffusion model according to claim 1, characterized in that, The multi-task combination loss function is: in, These are the weighting coefficients for each loss component. To rebuild the losses, In order to perceive loss, To combat the losses, This is the loss due to regularization.

8. A complex degraded image restoration system based on a spatiotemporal decoupled dual-path diffusion model, characterized in that, include: The image preprocessing module acquires the degraded image to be restored and preprocesses the degraded image to obtain a preprocessed image. The dual-path model construction module constructs a spatiotemporally decoupled dual-path diffusion model. The model includes a residual diffusion path and a noise diffusion path set in parallel, as well as a gated residual attention network and a progressive three-stage scheduler. The residual diffusion path is used for structural degradation repair, and the noise diffusion path is used for detailed texture generation. The gated weight generation module inputs the preprocessed image into the gated residual attention network, extracts the spatial and temporal embedding features of the image, and generates a pixel-level gated weight map for dynamically fusing the output of the residual diffusion path and the noise diffusion path. The diffusion stage scheduling module obtains the frequency domain signal-to-noise ratio FSNR(t) through a progressive three-stage scheduler, and divides the diffusion process into a structural repair stage, a transition fusion stage, and a detail generation stage based on the rate of change of FSNR(t). A continuous diffusion coefficient scheduling function is designed for each stage to dynamically adjust the diffusion coefficients of the two paths. The model training and optimization module obtains a multi-task combined loss based on the difference between the output image after adjusting the diffusion coefficient and the real clear image, and optimizes the model parameters through backpropagation training. In the inference output module, during the inference phase, based on the trained model, an adaptive importance sampling strategy is used to adjust the sampling density of different regions, and an early termination mechanism is triggered by the FSNR(t) growth threshold to output the final restored image.

9. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and running thereon, characterized in that, When the processor executes the program, it implements the complex degraded image restoration method based on the spatiotemporal decoupled dual-path diffusion model as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for restoring complex degraded images based on a spatiotemporally decoupled dual-path diffusion model as described in any one of claims 1-7.

Images