Degradation-aware prior propagation model for image super-resolution

Abstract

This invention relates to the field of image processing technology, and more particularly to a degradation-aware prior diffusion model image super-resolution system and method. The image degradation-aware module initially reconstructs a base image from a low-resolution image, downsamples to construct a pseudo-low-resolution image, and generates a degradation uncertainty map through local differences in the feature space. The reconstruction noise prediction module fuses features and degradation priors to generate a spatiotemporal mask, performs feature mapping from both pixel fidelity and semantic detail dimensions, and completes self-attention feature modeling based on the mask and mapping matrix, outputting prediction noise at each time step in multiple levels. The residual purification diffusion module iteratively purifies residual latent features using prediction noise, obtaining pure latent features through temporal loops. The decoding and reconstruction module fuses the pure residual latent features with the base image latent features, decoding and outputting a high-quality reconstructed image. This invention achieves high-quality image super-resolution reconstruction with absolutely stable structure, no artifacts, and realistic high-frequency details.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to a diffusion model image super-resolution system and method with degradation-aware prior. Background Technology

[0002] Image super-resolution is a fundamental and core technology in computer vision, aiming to reconstruct high-resolution images with clear textures and rich details from low-resolution images. Due to the irreversibility of the degradation process and the spatially uneven distribution of its severity, achieving high-quality, artifact-free image reconstruction in highly challenging real-world scenarios has always been a technical bottleneck that academia and industry urgently need to overcome.

[0003] Early deep learning image super-resolution primarily relied on two types of models: CNN (Convolutional Neural Networks) and GAN (Generative Adversarial Networks). CNN methods optimize at the pixel level, resulting in high image fidelity, but the reconstructed image is overly smoothed, losing high-frequency texture details. GAN methods optimize visual perception and enhance realism through adversarial learning, but they are prone to generating artifacts and false textures when dealing with severely degraded regions outside the training distribution, and they also struggle to replicate complex real image structures. In recent years, pre-trained diffusion models have demonstrated outstanding performance in super-resolution tasks due to their powerful distribution modeling and generative prior capabilities, generating high-perceptual-quality images. However, their unconstrained generative nature can easily lead to over-creation rather than faithful reconstruction, deviating from the original image's structure and semantics, causing semantic illusions and feature shifts, thus plunging into the dilemma of balancing fidelity and realism.

[0004] Existing technologies alleviate the contradiction through two types of solutions: one is to introduce degenerate uncertainty gradient guidance to suppress artifacts in diffusion backsampling, but each inference step requires complex backpropagation, which is time-consuming and difficult to implement; the other is to adopt a dual-branch LoRA architecture (dual-branch low-rank adaptive), which splits the fidelity branch and detail branch and fuses features, which significantly improves inference efficiency, but has obvious drawbacks.

[0005] First, it lacks explicit physical degradation priors, and feature fusion relies on static black-box weights. It is prone to misjudgment in non-uniform and severely degraded scenes. Vulnerable areas are prone to semantic illusions and artifacts, while normal areas are over-constrained, resulting in smooth and distorted images. Secondly, the use of static fusion weights deviates from the temporal evolution of the diffusion model and ignores the temporal characteristics of reconstructing low-frequency structures in the early stage of denoising and generating high-frequency details in the later stage, resulting in inaccurate early structure anchoring and limited late-stage detail generation. Summary of the Invention

[0006] In view of this, the present invention aims to provide a degradation-aware prior diffusion model image super-resolution system and method. In the image degradation-aware module, a degradation uncertainty assessment mechanism is constructed to accurately quantify the severity of local image degradation, providing the network with physically meaningful regional guidance and fundamentally suppressing artifact bursts in vulnerable areas. The innovative design of the spatiotemporal dynamic control submodule and the dual LoRA mapping submodule deeply couples the degradation spatial prior with the diffusion time step, enabling the system to follow the physical law of "structure first, details later". In the early stage of denoising, the "fidelity branch" is globally strengthened to firmly anchor the underlying skeleton of the image, while in the late stage of denoising, the "detail branch" is dynamically released strictly according to the spatial prior to safely inject high-frequency textures.

[0007] To achieve the above objectives, the technical solution created by this invention is implemented as follows: A degradation-aware prior diffusion model image super-resolution system includes: an image degradation-aware module, which performs preliminary reconstruction of the input low-resolution image to obtain a base image, and then downsamples the base image to obtain a pseudo-low-resolution image; calculates the local difference between the low-resolution image and the pseudo-low-resolution image in the feature space to obtain a degradation uncertainty map; a reconstruction noise prediction module, whose input includes the degradation uncertainty map, the current time step, and the super-resolution residual latent features at the current time step; the reconstruction noise prediction module adopts a pre-trained U-Net model, and adds a spatiotemporal dynamic control submodule, a dual LoRA mapping submodule, and a fusion Transformer submodule to each layer of the U-Net model. At each time step; in the spatiotemporal dynamic control submodule, the input features of the current layer and the degradation uncertainty map are fused spatiotemporally according to the time step to obtain the spatiotemporal fusion mask of the current layer at the current time step; in the dual LoRA mapping submodule, pixel-level fidelity LoRA and semantic-level detail LoRA are used to perform spatiotemporal dual information fusion. oRA maps the input features of the current layer to obtain a mapping matrix. In the fusion Transformer submodule, using the corresponding spatiotemporal fusion mask as a mask, a Transformer self-attention operation is performed on the input features of the current layer based on the mapping matrix to obtain the output features of the current layer. The output features of the current layer are used as the input features of the next layer. After continuous multi-layer processing, the predicted noise affecting the super-resolution reconstruction at the current time step is output. The residual purification and diffusion module uses the predicted noise obtained at the current time step to purify the super-resolution residual latent features at the current time step to obtain the super-resolution residual latent features at the next time step. The super-resolution residual latent features at the next time step are then input into the reconstruction noise prediction module to complete the output of the predicted noise at the next time step, until the clean super-resolution residual latent features output at the final time step are obtained. The decoding and reconstruction module decodes the latent features obtained by combining the clean super-resolution residual latent features with the latent features of the base image to obtain the reconstructed image.

[0008] Furthermore, in the image degradation perception module: a pre-trained super-resolution reconstruction network is used to perform preliminary reconstruction of the low-resolution image to obtain a base image; a fixed VAE encoder is used to map the base image to the latent space to obtain basic latent features; bicubic downsampling is performed on the base image to obtain a pseudo-low-resolution image; a pre-trained VGG16 network is used to extract deep features from the pseudo-low-resolution image and the low-resolution image respectively, and convolution operation is performed on the difference between the two deep features; Gaussian smoothing filtering is performed on the convolution result to obtain a degradation uncertainty map.

[0009] Furthermore, the process of obtaining the spatiotemporal fusion mask in the spatiotemporal dynamic control submodule includes: performing position encoding and multilayer perceptron mapping on the current time step to obtain a temporal feature vector; downsampling the degenerate uncertainty map to the same spatial resolution as the input features of the current layer to obtain a downsampled degenerate map, and concatenating and fusing the downsampled degenerate map with the input features of the current layer to obtain a spatial feature map; adding the spatial feature map and the temporal feature vector with dynamic weights, and then performing sigmoid activation processing on the result to obtain the spatiotemporal fusion mask.

[0010] Furthermore, the process of fusing the output features of the current layer from the Transformer submodule includes obtaining the dynamic fused query features after fusing pixel-level fidelity LoRA and semantic-level detail LoRA using the following formula: ; Where Q represents the dynamically fused query features, X represents the input features of the current layer, and W... q W represents the query matrix in the Transformer self-attention operation. pix,q W represents the low-rank mapping matrix of the pixel-fidelity LoRA branch on the query projection. sem,q This represents the low-rank mapping matrix of the semantic-level detail LoRA branch on the query projection. M represents the Hadama product. ST Indicates a spatiotemporal fusion mask; The dynamic fused value feature, obtained by fusing pixel-level fidelity LoRA and semantic-level detail LoRA, is obtained using the following formula: ; Where V represents the dynamic fusion value feature, W v W represents the value matrix in the Transformer self-attention operation. pix,v W represents the low-rank mapping matrix of the pixel-fidelity LoRA branch on the value projection. sem,v The low-rank mapping matrix representing the semantic-level detail LoRA branch on the value projection; Self-attention characteristics are obtained by performing self-attention operations using the following formula: ; Where Att represents the self-attention weight, K represents the key matrix in the Transformer self-attention operation, and d represents the feature scaling dimension; The output features of the current layer are obtained using the following formula: X out =FFN(W o Att+X)+(W o Att+X); Among them, X out W represents the output features of the current layer, FFN represents the feedforward mapping operation in the Transformer's self-attention operation, and W represents the output features of the current layer. o This represents the self-attention output projection matrix.

[0011] Furthermore, the super-resolution residual latent features of the reconstructed noise prediction module at the initial time step satisfy a standard normal distribution.

[0012] Furthermore, in the residual purification diffusion module, the inverse diffusion process is used to gradually denoise the data, obtaining the super-resolution residual latent features from the previous time step: z t-1 =DDIM(z t ,ε t ,t); Among them, z t Let ε represent the super-resolution residual latent feature at time step t. t This represents the predicted noise output by the residual purification diffusion module at the current time step. DDIM represents the deterministic or semi-deterministic inverse diffusion update operator performed on the current latent variable based on preset diffusion scheduling parameters.

[0013] Furthermore, in the decoding and reconstruction module: the pure super-resolution residual latent features are added element-wise to the latent features of the base image to obtain the super-resolution latent features; the super-resolution latent features are then decoded using VAE to obtain the reconstructed image.

[0014] A degradation-aware prior diffusion model image super-resolution method includes: S1: Obtain a high-resolution image dataset, perform a degradation operation on the high-resolution image dataset to obtain the corresponding low-resolution image dataset, and the high-resolution image dataset and the low-resolution image dataset form a training set. S2: Construct a diffusion model image super-resolution system with degradation perception prior as provided in this invention; S3: Use the training set obtained in step S1 to train the image super-resolution system constructed in step S2 to obtain the image super-resolution model; S4: Input the low-resolution image to be reconstructed into the image super-resolution model obtained in step S3 to obtain the predicted reconstructed image.

[0015] Furthermore, step S3 includes a two-stage training process: In the first stage of training: the weights of the reconstruction noise prediction module, except for the pixel-level fidelity LoRA, are frozen, and the spatiotemporal fusion mask is set to 0. Only the weights of the pixel-level fidelity LoRA are trained and optimized, and the first stage of training is performed using the first loss function. ; Among them, L pixel Let z represent the first loss function. HR Represents the latent features of a true high-resolution image. z represents the clean super-resolution residual latent features output during the first stage of training. base Represents the latent features of the underlying image; In the second training phase: In the model obtained from the first training phase, only the pixel-level fidelity LoRA weights are frozen, and the second training phase is performed using the second loss function. L total =λ1L lpips (I SR ,I HR )+λ2L csd ; Among them, L total Let L represent the second loss function. lpips I represents perceived loss. SR I represents the predicted reconstructed image. HR L represents the corresponding true high-resolution image. csd Let λ1 and λ2 represent the CSD loss based on fractional distillation.

[0016] Compared with the prior art, the present invention can achieve the following beneficial effects: The present invention creates a diffusion model image super-resolution system and method with degradation perception prior, which introduces an explicit degradation uncertainty assessment mechanism in the image degradation perception module. This mechanism can accurately quantify and locate extremely fragile severely degraded regions and safe complex texture regions, thereby fundamentally suppressing the severe artifacts and semantic illusions caused by blind feature generation in the spatial dimension. This solves the shortcomings of existing solutions that rely only on static coarse-grained statistical information and lack local degradation physical modeling. To address the fatal weaknesses of existing technologies that completely abandon the diffusion time dynamics in pursuit of speed, or rigidly adopt fixed feature fusion weights throughout the entire reverse denoising cycle, this invention innovatively proposes a spatiotemporal dynamic control submodule and a dual LoRA mapping submodule. This deeply couples the spatial degradation prior with the time step of the diffusion model, perfectly matching the physical evolution law of the diffusion generation process of "first reconstructing the low-frequency structure and then generating high-frequency details". In the early stage of denoising, the "fidelity branch" is globally strengthened to firmly anchor the underlying skeleton of the image, while in the late stage of denoising, the "detail branch" is dynamically released strictly according to the spatial prior to safely inject high-frequency textures. Ultimately, when dealing with extremely complex and unevenly distributed degradation scenarios in the real world, this invention achieves high-quality image super-resolution reconstruction with extremely high robustness, resulting in absolutely stable structure, no artifacts, and extremely realistic high-frequency details. Attached Figure Description

[0017] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments and descriptions of the invention are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 A schematic diagram of the diffusion model image super-resolution system with degradation perception prior as described in an embodiment of the present invention; Figure 2 This is a schematic flowchart of the degradation-aware prior diffusion model image super-resolution method described in an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not constitute a limitation thereof. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0019] In the description of this invention, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0020] The invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0021] like Figure 1As shown in the embodiment of the present invention, the degradation-aware prior diffusion model image super-resolution system includes an image degradation-aware module, a reconstruction noise prediction module, a residual purification diffusion module, and a decoding reconstruction module.

[0022] The image degradation perception module performs preliminary reconstruction of the input low-resolution image to obtain a base image, and then downsamples the base image to obtain a pseudo-low-resolution image; it calculates the local differences between the low-resolution image and the pseudo-low-resolution image in the feature space to obtain a degradation uncertainty map.

[0023] In some embodiments, the feature processing of the image degradation perception module includes: performing preliminary reconstruction of the low-resolution image using a pre-trained super-resolution reconstruction network to obtain a base image; and mapping the base image to the latent space using a fixed VAE encoder to obtain basic latent features. When dealing with complex real-world degradation, to accurately quantify the severity of local unknown degradation and reconstruction uncertainty in each region of the image's internal space, this invention introduces a degradation closed-loop consistency check mechanism. Specifically, the base image is bicubic downsampled to obtain a pseudo-low-resolution image; deep features of the pseudo-low-resolution image and the low-resolution image are extracted using a pre-trained VGG16 network, and the difference between the two deep features is convolved; the convolution result is then subjected to Gaussian smoothing filtering to obtain a degradation uncertainty map. The process of obtaining the degradation uncertainty map is as follows: U = Gaussian(Conv) 3×3 (| (I LQ ')- (I LQ )|)); Where U represents the degenerate uncertainty graph, Gaussian represents the Gaussian smoothing filter operation, and Conv... 3×3 This represents a convolution operation with a 3×3 kernel, a stride of 1, and 1 output channel. I represents the feature extraction operation of the VGG16 network. LQ I represents a low-resolution image. LQ ' represents a simulated low-resolution image. In this embodiment of the invention, the degradation uncertainty map U is also normalized. In the processed degradation uncertainty map: the larger the pixel value, the more severe the degradation in that region, and the easier it is to generate semantic illusions during diffusion generation; the closer the pixel value is to 1, the more severe the degradation in that region, and the more likely it is to generate semantic illusions and artifacts in subsequent diffusion generation. In addition, in this embodiment of the invention, the existing Real-ESRGAN model is used to perform preliminary reconstruction of the low-resolution image to obtain the base image. The process of obtaining the basic latent features of the base image using a fixed VAE encoder is as follows: z base =VAE(Ibase ); Among them, z base Represents the basic latent features, I base The basic image represents the basic latent features, which mainly carry stable information such as overall structure, subject outline, color distribution, brightness levels, smooth areas, coarse edges, and geometric layout.

[0024] The input to the noise prediction module includes a degradation uncertainty map, the current time step, and the super-resolution residual latent features at the current time step. The noise prediction module employs a pre-trained U-Net model, and adds a spatiotemporal dynamic control submodule, a dual LoRA mapping submodule, and a fusion Transformer submodule to each layer of the U-Net model. In some embodiments, the super-resolution residual latent features at the initial time step of the noise prediction module satisfy a standard normal distribution.

[0025] At each time step, in the spatiotemporal dynamic control submodule, the input features of the current layer and the degradation uncertainty map are fused spatiotemporally according to the time step to obtain the spatiotemporal fusion mask of the current layer at the current time step, thereby realizing spatiotemporal coordinated regulation. In the dual LoRA mapping submodule, the input features of the current layer are mapped using pixel-level fidelity LoRA and semantic-level detail LoRA to obtain the mapping matrix. In the fusion Transformer submodule, the corresponding spatiotemporal fusion mask is used as a mask, and Transformer self-attention operation is performed on the input features of the current layer based on the mapping matrix to obtain the output features of the current layer. The output features of the current layer are used as the input features of the next layer. After continuous multi-layer processing, the prediction noise affecting super-resolution reconstruction at the current time step is output. Each LoRA consists of a dimensionality reduction matrix A and an increased dimensionality matrix B, i.e., W=BA, where W represents pixel-level fidelity LoRA or semantic-level detail LoRA.

[0026] Among them, pixel-level fidelity LoRA and semantic-level detail LoRA come from the paper "Pixel-level and Semantic-level Adjustable Super-resolution: A Dual-LoRA Approach" published at CVPR 2025. The paper proposes to learn two LoRA modules on pre-trained Stable Diffusion, one constrained by L2 loss for pixel-level regression, and the other constrained by LPIPS and CSD loss for semantic-level augmentation.

[0027] Understandably, the input features of the first layer in the noise prediction module are the super-resolution residual latent features at the current time step. In the first layer of the noise prediction module, within the spatiotemporal dynamic control submodule, the super-resolution residual latent features at the current time step are fused with the degradation uncertainty map according to the time step, resulting in a spatiotemporal fusion mask for the current layer at the current time step. In the dual LoRA mapping submodule, pixel-level fidelity LoRA and semantic-level detail LoRA are used to map the super-resolution residual latent features at the current time step, resulting in a mapping matrix. In the fusion Transformer submodule, the corresponding spatiotemporal fusion mask is used as a mask, and a Transformer self-attention operation is performed on the super-resolution residual latent features at the current time step based on the mapping matrix, resulting in the output features of the first layer. The output features of the first layer are then used as the input features of the second layer.

[0028] In some embodiments, the process of obtaining the spatiotemporal fusion mask in the spatiotemporal dynamic control submodule includes: performing position encoding and multilayer perceptron mapping on the current time step to obtain a temporal feature vector; downsampling the degradation uncertainty map to the same spatial resolution as the input features of the current layer to obtain a downsampled degradation map, and concatenating and fusing the downsampled degradation map with the input features of the current layer to obtain a spatial feature map; adding the spatial feature map and the temporal feature vector with dynamic weights, and then performing sigmoid activation processing on the result to obtain the spatiotemporal fusion mask.

[0029] In this embodiment of the invention, the processing procedure of the spatiotemporal dynamic control submodule specifically includes: The time step t is mapped to a time feature vector t using positional encoding (PE) of the sine-cosine time step and a multilayer perceptron (MLP). emb As shown in the following formula: t emb =MLP(PE(t)); The degraded uncertainty map U is downsampled to the same spatial resolution as the input feature X of the current layer to obtain the downsampled degraded map U'. The downsampled degraded map U' is then concatenated with the input feature X along the channel dimension, and spatial information is exchanged through a convolutional layer to obtain the spatial feature map F. s As shown in the following formula: F s =Conv(Concat(U',X)); Where Concat represents the channel-level concatenation operation, and Conv represents the convolution operation; Introducing spatiotemporal joint calibration parameters to dynamically fuse temporal and spatial features, the spatiotemporal fusion mask M, whose size matches the input features, is calculated using the following formula. ST : M ST =sigmoid(α×Expand(t emb )+β×F s ); Here, sigmoid represents the sigmoid activation function, Expand represents the dimensionality expansion operation (i.e., expanding a one-dimensional time vector to a two-dimensional spatial dimension), and α and β represent learnable scaling parameters.

[0030] The temporal feature component is physically designed such that, in the early stages of diffusion denoising (when t is relatively large), the output has a significant negative bias, making the spatiotemporal fusion mask M... ST The overall value approaches 0; in the late diffusion stage (when t is small), the output is positively biased, at which point the spatiotemporal fusion mask M... ST The distribution is determined by the spatial feature map F s leading.

[0031] In some embodiments, the process of fusing the output features of the current layer obtained from the Transformer submodule includes: obtaining the dynamic fused query features after fusing pixel-level fidelity LoRA and semantic-level detail LoRA using the following formula: ; Where Q represents the dynamically fused query features, X represents the input features of the current layer, and W... q W represents the query matrix in the Transformer self-attention operation. pix,q W represents the low-rank mapping matrix of the pixel-fidelity LoRA branch on the query projection. sem,q This represents the low-rank mapping matrix of the semantic-level detail LoRA branch on the query projection. M represents the Hadama product. ST This represents a spatiotemporal fusion mask.

[0032] The dynamic fused value feature, obtained by fusing pixel-level fidelity LoRA and semantic-level detail LoRA, is obtained using the following formula: ; Where V represents the dynamic fusion value feature, W v W represents the value matrix in the Transformer self-attention operation. pix,v W represents the low-rank mapping matrix of the pixel-fidelity LoRA branch on the value projection. sem,v This represents the low-rank mapping matrix of the semantic-level detail LoRA branch on the value projection.

[0033] Self-attention characteristics are obtained by performing self-attention operations using the following formula: ; Where Att represents the self-attention weight, K represents the key matrix in the Transformer self-attention operation, and d represents the feature scaling dimension; the key matrix is ​​not obtained through pixel-level fidelity LoRA and semantic-level detail LoRA in the above formula, but directly uses the key matrix K in the Transformer self-attention operation to stabilize the attention similarity calculation.

[0034] The physical meaning of the above formula is as follows: First, the similarity between the dynamic fusion query feature Q and the key feature K is calculated using the softmax function to obtain the attention weight distribution. Then, this weight distribution is used to perform weighted aggregation on the dynamic fusion value feature V, thereby enabling the system to selectively enhance information at different spatial locations within the current layer. Since the dynamic fusion query feature Q and the dynamic fusion value feature V simultaneously contain information from the U-Net model itself, the pixel-level fidelity LoRA branch, and the semantic-level detail LoRA branch, and the fusion ratio of the two types of LoRA branches is controlled by the spatiotemporal fusion mask, this self-attention process essentially achieves a spatiotemporal adaptive collaborative allocation of "structural fidelity information" and "detail generation information." It should be noted that the above self-attention feature Att only represents the enhanced feature output by the current Transformer self-attention sublayer, not the final reconstructed image.

[0035] To obtain a denoised prediction result that can be used for updating the inverse diffusion process, the attention feature Att needs to continue passing through the output mapping layer, residual connection, and feedforward network layer of the current Transformer block. Specifically, the output feature of the current layer is obtained by the following formula: X out =FFN(W o Att+X)+(W o Att+X); Among them, X out W represents the output features of the current layer, FFN represents the feedforward mapping operation in the Transformer's self-attention operation, and W represents the output features of the current layer. o This represents the self-attention output projection matrix.

[0036] It should be noted that the query matrix, value matrix, and key matrix of the Transformer self-attention operation mentioned above are consistent with existing methods for obtaining the query matrix, value matrix, and key matrix involved in the Transformer self-attention operation; that is, the existing methods for obtaining the query matrix, value matrix, and key matrix are directly adopted. As the features propagate layer by layer throughout the U-Net model, the final output is the prediction noise at the current time step t.

[0037] The residual purification and diffusion module uses the prediction noise obtained at the current time step to purify the super-resolution residual latent features at the current time step, obtains the super-resolution residual latent features at the next time step, and inputs the super-resolution residual latent features at the next time step into the reconstruction noise prediction module to complete the output of the prediction noise at the next time step, until the clean super-resolution residual latent features output at the final time step are obtained.

[0038] In some embodiments, the residual purification diffusion module performs a reverse diffusion process to gradually denoise the data, obtaining the super-resolution residual latent features from the previous time step: z t-1 =DDIM(z t ,ε t ,t); Among them, z t Let ε represent the super-resolution residual latent feature at time step t. t This represents the predicted noise output by the residual purification diffusion module at the current time step. DDIM represents the deterministic or semi-deterministic inverse diffusion update operator performed on the current latent variable based on preset diffusion scheduling parameters. Based on the output predicted noise, the currently noisy latent variable is pushed towards a cleaner, more closely resembling target high-resolution latent representation, thus gradually completing the inverse diffusion recovery from a high-noise state to a state with both high-quality structure and detail over time. DDIM is an existing diffusion model sampling algorithm, derived from (Jiaming Song, Chenlin Meng, Stefano Ermon, “Denoising Diffusion Implicit Models”, arXiv:2010.02502, 2020 / ICLR 2021). This algorithm constructs a non-Markovian inverse diffusion sampling process based on DDPM, enabling the recovery from noisy latent variables to clean latent variables with fewer sampling steps.

[0039] A further explanation is that the aforementioned fusion Transformer submodule and residual cleanup diffusion module are not executed in a single run, but rather repeatedly over all diffusion time steps. Specifically, in the early stages of denoising, the time step t is relatively large, resulting in a spatiotemporal fusion mask M. ST The overall size is relatively small, therefore, the pixel-level fidelity LoRA branch is globally enhanced to dominate the attention calculation, thus prioritizing the recovery of low-frequency skeleton information such as the main outline, spatial layout, brightness levels, and coarse-grained edges to anchor the underlying structure and avoid prematurely introducing unstable semantic details during high-noise stages. As the reverse diffusion process progresses, the time step t decreases, and the spatiotemporal fusion mask M... ST The spatial distribution of the degradation uncertainty map U initially dominates: in the safe texture region with lower degradation risk, the spatiotemporal fusion mask M... STIncreasing the value of the LoRA branch at the semantic level enhances its contribution to the dynamic fusion query features and dynamic fusion value features, thereby releasing stronger high-frequency texture representation capabilities during attention aggregation to generate more realistic local details; while in vulnerable regions with a high risk of degradation, the spatiotemporal fusion mask M... ST By maintaining relatively small values, the pixel-level fidelity LoRA branch continues to dominate, thereby suppressing artifacts, semantic illusions, and structural drift. Thus, this invention follows a "structure first, detail later" diffusion evolution pattern in the temporal dimension, and achieves differentiated control in the spatial dimension through "conservative restoration of high-risk areas and active enhancement of safe areas." After the residual purification diffusion module completes all time-step inverse diffusion iterations, the final clean super-resolution residual latent features are obtained. These clean super-resolution residual latent features represent the high-frequency realistic details and controlled semantic enhancement components that need to be added on top of the basic latent features.

[0040] The decoding and reconstruction module decodes the latent features obtained by combining the clean super-resolution residual latent features with the latent features of the base image to obtain the reconstructed image. In some embodiments, the corresponding elements of the clean super-resolution residual latent features and the latent features of the base image are added together to obtain the super-resolution latent features; the super-resolution latent features are then decoded using VAE to obtain the reconstructed image.

[0041] This invention also provides a diffusion model image super-resolution method based on degradation-aware priors, combined with... Figure 1 and Figure 2 ,include: S1: Obtain a high-resolution image dataset, perform a degradation operation on the high-resolution image dataset to obtain the corresponding low-resolution image dataset, and the high-resolution image dataset and the low-resolution image dataset together form the training set.

[0042] In this embodiment of the invention, the high-resolution image datasets are the publicly available high-quality image datasets DIV2K and Flickr2K. A complex high-order degradation model is constructed (simulating a mixed degradation process in the real world, including unknown blurring, Gaussian random noise, and JPEG compression artifacts) to generate corresponding low-resolution images. DIV2K and Flickr2K together contain 3450 paired images; 3000 pairs are selected as the training sample set, and 100 pairs as the validation sample set. Simultaneously, 100 complex natural scene images are selected from the publicly available real-world super-resolution benchmark dataset RealSR as the test sample set to verify the model's generalization ability in real-world unknown degradation environments.

[0043] S2: Construct a diffusion model image super-resolution system with degradation-aware prior as provided in this invention.

[0044] S3: Use the training set obtained in step S1 to train the image super-resolution system constructed in step S2 to obtain the image super-resolution model.

[0045] In some embodiments, step S3 includes a two-stage training process: In the first stage of training: the weights of all parts of the reconstruction noise prediction module except for pixel-level fidelity LoRA are frozen, and the spatiotemporal fusion mask is set to 0 (i.e., the detail branch is completely masked). Only the weights of pixel-level fidelity LoRA are trained and optimized, and the first stage of training is performed using the first loss function. ; Among them, L pixel Let z represent the first loss function. HR Represents the latent features of a true high-resolution image. z represents the clean super-resolution residual latent features output during the first stage of training. base This stage represents the latent features of the base image, aiming to teach the fidelity branch accurate low-frequency structure mappings. In this embodiment, a fixed VAE encoder is used to map the real high-resolution image to the latent space, obtaining the latent features z. HR This latent feature mainly carries microstructures such as fine textures, sharp edges, local contrast transitions, and hair / fabric / texture / skin texture.

[0046] In the second training phase: In the model obtained from the first training phase, only the pixel-level fidelity LoRA weights are frozen, and the second training phase is performed using the second loss function. L total =λ1L lpips (I SR ,I HR )+λ2L csd ; Among them, L total Let L represent the second loss function. lpips I represents perceived loss. SR I represents the predicted reconstructed image. HR L represents the corresponding true high-resolution image. csd Let λ1 and λ2 represent the CSD loss based on fractional distillation, and let λ1 and λ2 represent the loss weights. CSD loss also comes from the paper "Pixel-level and Semantic-level Adjustable Super-resolution: A Dual-LoRA Approach" published at CVPR 2025. CSD loss does not require complex two-layer optimization, which greatly reduces memory consumption and accelerates the convergence of realistic textures.

[0047] In addition, during the training process, the validation sample set is periodically input into the current model for forward inference to calculate the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM). If the indicators do not improve for several consecutive cycles, the early stopping mechanism is triggered to terminate the training in advance, and the weight parameters of the finally converged image super-resolution model are saved.

[0048] S4: Input the low-resolution image to be reconstructed into the image super-resolution model obtained in step S3 to obtain the predicted reconstructed image.

[0049] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0050] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A degradation-aware prior's diffusion model image super-resolution system, characterized in that, include: The image degradation perception module performs preliminary reconstruction on the input low-resolution image to obtain a base image, and then downsamples the base image to obtain a pseudo-low-resolution image. Calculate the local differences between the low-resolution image and the pseudo-low-resolution image in the feature space to obtain the degradation uncertainty map; The noise prediction module takes as input a degradation uncertainty map, the current time step, and the super-resolution residual latent features at the current time step. The noise prediction module uses a pre-trained U-Net model and adds a spatiotemporal dynamic control submodule, a dual LoRA mapping submodule, and a fusion Transformer submodule to each layer of the U-Net model at each time step. In the spatiotemporal dynamic control submodule, the input features of the current layer and the degradation uncertainty map are fused according to time steps to obtain the spatiotemporal fusion mask of the current layer at the current time step; In the dual LoRA mapping submodule, the input features of the current layer are mapped using pixel-level fidelity LoRA and semantic-level detail LoRA to obtain a mapping matrix; In the fusion Transformer submodule, the corresponding spatiotemporal fusion mask is used as a mask, and the Transformer self-attention operation is performed on the input features of the current layer based on the mapping matrix to obtain the output features of the current layer. The output features of the current layer are used as the input features of the next layer. After continuous multi-layer processing, the predicted noise affecting the super-resolution reconstruction at the current time step is output. The residual purification and diffusion module purifies the super-resolution residual latent features at the current time step using the prediction noise obtained at the current time step, obtains the super-resolution residual latent features at the next time step, and inputs the super-resolution residual latent features at the next time step into the reconstruction noise prediction module to complete the output of the prediction noise at the next time step, until the clean super-resolution residual latent features output at the final time step are obtained. The decoding and reconstruction module decodes the latent features obtained by combining the latent features of the clean super-resolution residual with the latent features of the base image to obtain the reconstructed image.

2. The degrading perceptual prior diffusion model image super-resolution system of claim 1, wherein, In the image degradation perception module: The low-resolution image is initially reconstructed using a pre-trained super-resolution reconstruction network to obtain the base image. A fixed VAE encoder is used to map the base image to the latent space to obtain the base latent features; A pseudo-low-resolution image is obtained by performing bicubic downsampling on the base image. The pre-trained VGG16 network is used to extract deep features from the simulated low-resolution image and the low-resolution image respectively. The difference between the two deep features is then subjected to a convolution operation. The result after convolution is then subjected to a Gaussian smoothing filter to obtain the degradation uncertainty map.

3. The degrading perceptual prior diffusion model image super-resolution system of claim 1, wherein, The process of obtaining the spatiotemporal fusion mask in the spatiotemporal dynamic control submodule includes: The current time step is encoded using position encoding and mapped using a multilayer perceptron to obtain a time feature vector. The degradation uncertainty map is downsampled to the same spatial resolution as the input features of the current layer to obtain the downsampled degradation map. The downsampled degradation map is then concatenated and fused with the input features of the current layer to obtain the spatial feature map. After dynamically weighting the spatial feature map and the temporal feature vector, the result is subjected to sigmoid activation to obtain the spatiotemporal fusion mask.

4. The degrading perceptual prior diffusion model image super-resolution system of claim 1, wherein, The process of fusing the output features of the current layer from the Transformer submodule includes: The dynamic fused query features, which combine pixel-level fidelity LoRA and semantic-level detail LoRA, are obtained using the following formula: ； Where Q represents the dynamically fused query features, X represents the input features of the current layer, and W... q W represents the query matrix in the Transformer self-attention operation. pix,q W represents the low-rank mapping matrix of the pixel-fidelity LoRA branch on the query projection. sem,q This represents the low-rank mapping matrix of the semantic-level detail LoRA branch on the query projection. M represents the Hadama product. ST Indicates a spatiotemporal fusion mask; The dynamic fused value feature, obtained by fusing pixel-level fidelity LoRA and semantic-level detail LoRA, is obtained using the following formula: ； Where V represents the dynamic fusion value feature, W v W represents the value matrix in the Transformer self-attention operation. pix,v W represents the low-rank mapping matrix of the pixel-fidelity LoRA branch on the value projection. sem,v The low-rank mapping matrix representing the semantic-level detail LoRA branch on the value projection; Self-attention characteristics are obtained by performing self-attention operations using the following formula: ； Where Att represents the self-attention weight, K represents the key matrix in the Transformer self-attention operation, and d represents the feature scaling dimension; The output features of the current layer are obtained using the following formula: X out = FFN(W o Att+X) + (W o Att+X); where X out represents the output features of the current layer, FFN represents a feed-forward mapping operation in the Transformer self-attention operation, W o represents a self-attention output projection matrix.

5. The degradation-aware prior diffusion model image super-resolution system according to claim 1, characterized in that, The super-resolution residual latent features at the initial time step of the reconstructed noise prediction module satisfy a standard normal distribution.

6. The degrading perceptual prior diffusion model image super-resolution system of claim 1, wherein, In the residual purification diffusion module, the inverse diffusion process is used to gradually remove noise, obtaining the super-resolution residual latent features of the previous time step: z t-1 = DDIM(z t , ε t , t); Among them, z t Let ε represent the super-resolution residual latent feature at time step t. t This represents the predicted noise output by the residual purification diffusion module at the current time step. DDIM represents the deterministic or semi-deterministic inverse diffusion update operator performed on the current latent variable based on preset diffusion scheduling parameters.

7. The degrading perceptual prior diffusion model image super-resolution system of claim 1, wherein, In the decoding and reconstruction module: The corresponding elements of the clean super-resolution residual latent features and the latent features of the base image are added together to obtain the super-resolution latent features; The super-resolution latent features are decoded using VAE to obtain the reconstructed image.

8. A diffusion model image super-resolution method of degradation-aware prior, characterized in that, include: S1: Obtain a high-resolution image dataset, perform a degradation operation on the high-resolution image dataset to obtain the corresponding low-resolution image dataset, and the high-resolution image dataset and the low-resolution image dataset form a training set. S2: Construct a diffusion model image super-resolution system with degradation perception prior as described in any one of claims 1 to 7; S3: Use the training set obtained in step S1 to train the image super-resolution system constructed in step S2 to obtain the image super-resolution model; S4: Input the low-resolution image to be reconstructed into the image super-resolution model obtained in step S3 to obtain the predicted reconstructed image.

9. The degrading perceptual prior diffusion model image super-resolution method of claim 8, wherein, Step S3 includes a two-stage training process: In the first stage of training: the weights of all parts of the reconstruction noise prediction module except for pixel-level fidelity LoRA are frozen, and the spatiotemporal fusion mask is set to 0. Only the weights of pixel-level fidelity LoRA are trained and optimized, and the first stage of training is performed using the first loss function. ； Among them, L pixel Let z represent the first loss function. HR Represents the latent features of a true high-resolution image. z represents the clean super-resolution residual latent features output during the first stage of training. base Represents the latent features of the underlying image; In the second training phase: In the model obtained from the first training phase, only the pixel-level fidelity LoRA weights are frozen, and the second training phase is performed using the second loss function. L total =λ1L lpips (I SR ,I HR )+λ2L csd ; Among them, L total Let L represent the second loss function. lpips I represents perceived loss. SR I represents the predicted reconstructed image. HR L represents the corresponding true high-resolution image. csd Let λ1 and λ2 represent the CSD loss based on fractional distillation.

Images