A video super-resolution reconstruction method and system based on a single-step diffusion model

Abstract

The application provides a video super-resolution reconstruction method and system based on a single-step diffusion model. The method comprises inputting a low-resolution video sequence into an encoder of a frozen-parameter three-dimensional variational autoencoder to obtain a low-resolution latent representation; inputting the low-resolution latent representation into a trained single-step diffusion Transformer model to perform single denoising prediction under the condition of a fixed diffusion time step, and generate a high-resolution latent representation; wherein the single-step diffusion Transformer model is constructed based on a pre-trained diffusion model, and a two-stage distillation strategy is adopted in the training process; inputting the high-resolution latent representation into a decoder of the frozen-parameter three-dimensional variational autoencoder, restoring the high-resolution latent representation to a pixel space by the decoder, and reconstructing a high-resolution video sequence. The application can realize high-quality and efficient video super-resolution reconstruction under the condition of limited computing resources.

Description

Technical Field

[0001] This application relates to the field of video super-resolution technology, specifically to a video super-resolution reconstruction method and system based on a single-step diffusion model. Background Technology

[0002] Video super-resolution (VSR) aims to recover high-resolution, high-quality video content from low-resolution video sequences and is a significant research area in computer vision. Compared to single-frame image super-resolution, video super-resolution not only needs to recover spatial details but also needs to simultaneously model the temporal correlations between frames to ensure the continuity and consistency of the video in the temporal dimension. Therefore, VSR tasks typically face the dual challenges of spatial detail reconstruction and temporal consistency modeling, making them technically challenging in practical applications.

[0003] Traditional video super-resolution methods are mainly based on convolutional neural networks (CNNs) or recurrent neural networks (RNNs), achieving information fusion by explicitly modeling the motion relationships between adjacent frames. For example, some methods use optical flow estimation for inter-frame alignment, followed by reconstruction using a feature fusion network; others employ temporal recursive structures or attention-based mechanisms to aggregate information from multiple frames. While these methods can improve reconstruction results to some extent, they still have significant limitations. On the one hand, explicit motion estimation methods are less robust to complex scenes (such as occlusion, rapid motion, and non-rigid deformation), easily leading to misalignment problems and resulting in blurry or artifact-prone reconstruction results. On the other hand, recursive or attention-based structures have high computational costs when modeling long sequences, making it difficult to meet the requirements of high-efficiency inference while maintaining performance.

[0004] In recent years, with the development of generative models, diffusion models have been gradually introduced into image and video super-resolution tasks. Diffusion models generate high-quality samples from random noise through progressive denoising, exhibiting strong distribution modeling and detail recovery capabilities, and have achieved excellent performance in visual generation tasks. Video super-resolution methods based on diffusion models can generate more realistic and detailed textures, and outperform traditional discriminative models in perceptual quality. However, existing diffusion-based video super-resolution methods typically rely on multi-step iterative denoising processes, requiring dozens or even hundreds of forward inference iterations for each frame or video segment to complete generation. This results in extremely high computational costs and slow inference speeds, making it difficult to meet the real-time or high-throughput requirements of practical applications.

[0005] Furthermore, to ensure temporal consistency in videos, existing diffusion-based VSR methods often introduce additional temporal alignment modules, motion modeling modules, or cross-frame attention mechanisms to enhance inter-frame correlation. However, while these designs improve temporal consistency, they also further increase model complexity and the number of parameters, resulting in high resource consumption for the overall system during both training and inference phases. On the other hand, although some methods attempt to reduce the number of diffusion steps through distillation or accelerated sampling strategies, under extreme single-step generation settings, the model often struggles to simultaneously maintain structural accuracy and detail realism, easily leading to texture loss or semantic shift issues. This is mainly due to the lack of sufficient progressive refinement during single-step generation, which limits the model's ability to model high-frequency details and complex motion patterns.

[0006] A literature search of existing technologies revealed Chinese patent CN120689209A, which proposes a video super-resolution method, system, terminal, and medium based on single-step diffusion acceleration. By using a more powerful base model and leveraging model priors, this method significantly improves the inference efficiency of video super-resolution and achieves higher-quality video super-resolution restoration, thus greatly enhancing its applicability in practical applications. However, this method does not employ distillation or accelerated sampling strategies to reduce the number of diffusion steps.

[0007] Therefore, how to effectively reduce the inference complexity of diffusion models while ensuring the quality of video super-resolution reconstruction, and simultaneously improve temporal consistency and perceptual realism, remains a key problem that urgently needs to be solved in the field of video super-resolution. Against this backdrop, it is necessary to design a new video super-resolution method that combines the prior knowledge of a pre-trained generative model with an efficient single-step inference mechanism to achieve high-quality, stable, and temporally consistent high-definition video reconstruction while reducing computational overhead. Summary of the Invention

[0008] To address the shortcomings of existing technologies, the purpose of this application is to provide a video super-resolution reconstruction method and system based on a single-step diffusion model.

[0009] According to the first aspect of this application, a video super-resolution reconstruction method based on a single-step diffusion model is provided, comprising: A low-resolution video sequence is input into the encoder in a 3D variational autoencoder with frozen parameters, and the encoder maps the low-resolution video sequence to the latent space to obtain a low-resolution latent representation. The low-resolution latent representation is input into the trained single-step diffusion Transformer model, and separate denoising prediction is performed under the condition of fixed diffusion time steps to generate the corresponding high-resolution latent representation; wherein, the single-step diffusion Transformer model is built based on the pre-trained diffusion model, and a two-stage distillation strategy including latent space consistency distillation and classifier fractional distillation is adopted during the training process. The high-resolution latent representation is input into the decoder in the three-dimensional variational autoencoder with frozen parameters. The decoder restores the high-resolution latent representation to the pixel space and reconstructs the final high-resolution video sequence.

[0010] Optionally, the separate denoising prediction includes: The low-resolution latent representation is used as an intermediate state of the diffusion process at a specific noise level. The low-resolution latent representation is input into a trained single-step diffusion Transformer model, and the denoising direction corresponding to the intermediate state is predicted by the single-step diffusion Transformer model in a velocity prediction mode. The corresponding high-resolution latent representation is reconstructed according to the preset diffusion process parameters and the denoising direction.

[0011] Optionally, the latent space consistency distillation is used in the first stage of training. In the first stage of training, pairs of low-resolution video samples and corresponding high-resolution video samples are acquired, and the parameters of the encoder and decoder in the 3D variational autoencoder are frozen. The encoder with frozen parameters maps the low-resolution video samples and corresponding high-resolution video samples to a unified latent space, respectively, to obtain low-resolution latent representations and true high-resolution latent representations. Using the low-resolution latent representations as input, the corresponding high-resolution latent representations are predicted and generated by the single-step diffusion Transformer model. The feature difference between the predicted high-resolution latent representations and the true high-resolution latent representations is used as the latent space consistency loss. The latent space consistency loss is used as the main loss for the first stage of training. The single-step diffusion Transformer model is trained in the first stage until convergence, and a student model is obtained after training.

[0012] Optionally, the classifier fractional distillation is used in the second-stage training process. In the second-stage training process, a pre-trained diffusion model is introduced as a teacher model, and different diffusion noises are added to the high-resolution latent representation output by the student model to obtain intermediate states under different diffusion noise levels. The intermediate states are then input into the teacher model to perform inference under conditional input and unconditional input respectively, and the noise prediction results under conditional input and unconditional input are obtained accordingly. The feature difference between the noise prediction results under conditional input and the noise prediction results under unconditional input is used as the classifier fractional distillation loss. The classifier fractional distillation loss is used as the main loss for the second stage of training. Backpropagation is used to update the network parameters of the student model, and the final training of the single-step diffusion Transformer model is completed.

[0013] Optionally, in the second stage of training, in addition to the main loss of the second stage training, an auxiliary loss is also introduced. The main loss and the auxiliary loss of the second stage training are weighted and summed as the overall optimization objective of the second stage training. The auxiliary loss includes at least one of perceptual loss and temporal consistency loss.

[0014] Optionally, the timing consistency loss is obtained in the following way: The inter-frame variation of each adjacent frame pair in the real high-resolution video sample and the reconstructed high-resolution video sequence is calculated respectively. The Euclidean distance between the inter-frame variation of the corresponding adjacent frame pairs in the real high-resolution video sample and the reconstructed high-resolution video sequence is extracted. The average value of the Euclidean distances corresponding to all adjacent frame pairs is taken to obtain the temporal consistency loss.

[0015] Optionally, the perceptual loss includes: Within the deep feature space of the single-step diffusion Transformer model, the structural and texture differences between real high-resolution video samples and reconstructed high-resolution video sequences are calculated, and the arithmetic mean of the structural and texture differences is used to obtain the perceptual loss.

[0016] According to a second aspect of this application, a video super-resolution reconstruction system based on a single-step diffusion model is provided, comprising: The encoding module is used to input a low-resolution video sequence into the encoder in the three-dimensional variational autoencoder of the frozen parameters, and the encoder maps the low-resolution video sequence to the latent space to obtain a low-resolution latent representation. The diffusion module is used to input the low-resolution latent representation into the trained single-step diffusion Transformer model, and perform separate denoising prediction under the condition of fixed diffusion time steps to generate the corresponding high-resolution latent representation; wherein, the single-step diffusion Transformer model is built based on the pre-trained diffusion model, and a two-stage distillation strategy including latent space consistency distillation and classifier fractional distillation is adopted during the training process. The reconstruction module is used to input the high-resolution latent representation into the decoder in the three-dimensional variational autoencoder with frozen parameters, and the decoder restores the high-resolution latent representation to the pixel space to reconstruct the final high-resolution video sequence.

[0017] According to a third aspect of this application, a non-transitory computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of a video super-resolution reconstruction method based on a single-step diffusion model as provided in the first aspect of this application.

[0018] According to a fourth aspect of this application, an electronic device is provided, comprising: At least one memory for storing program instructions; At least one processor is configured to invoke program instructions stored in the memory and execute the steps of a video super-resolution reconstruction method based on a single-step diffusion model as provided in the first aspect of this application, according to the obtained program instructions.

[0019] This application provides a video super-resolution reconstruction method based on a single-step diffusion model. It replaces the traditional multi-step diffusion sampling process with a single-step diffusion Transformer model, significantly improving inference efficiency and reducing computational resource consumption while maintaining the powerful generative and detail recovery capabilities of the diffusion model. A two-stage distillation strategy, including latent space consistency distillation and classifier fractional distillation, is employed to fully utilize the structural and semantic priors inherent in the pre-trained diffusion model. This allows the single-step diffusion Transformer model to learn high-quality video distribution features, achieving excellent reconstruction quality despite reducing sampling steps. This application enables high-quality and high-efficiency video super-resolution reconstruction under limited computational resources.

[0020] Other technical effects resulting from the additional features will be further illustrated in the corresponding embodiments. Attached Figure Description

[0021] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart of a video super-resolution reconstruction method in one embodiment of this application; Figure 2 This is a schematic diagram of a video super-resolution method based on a single-step diffusion model implemented through two-stage distillation in one embodiment of this application. Figure 3 This is a schematic diagram of the inference process of a video super-resolution method based on a single-step diffusion model implemented through two-stage distillation in one embodiment of this application; wherein, (a) is a schematic diagram of the first-stage distillation process, and (b) is a schematic diagram of the second-stage distillation process. Figure 4This is a subjective quality comparison chart of the video super-resolution method and other deep learning video super-resolution models in one embodiment of this application on the synthetic dataset (Sequence002) YouHQ40, where (a) is the second low-resolution image in the YouHQ40 dataset, (b) is a high-resolution cropped image generated by the BasicVSR++ method, (c) is a high-resolution cropped image generated by the RealBasicVSR method, (d) is a high-resolution cropped image generated by the STAR method, (e) is a high-resolution cropped image generated by the MGLD-VSR method, and (f) is a high-resolution cropped image generated by the MGLD-VSR method in one embodiment of this application (Our). (s) is the high-resolution cropped image generated, (g) is the second high-resolution cropped image in the YouHQ40 dataset, (h) is the tenth low-resolution image in the YouHQ40 dataset, (i) is the high-resolution cropped image generated by the BasicVSR++ method, (j) is the high-resolution cropped image generated by the RealBasicVSR method, (k) is the high-resolution cropped image generated by the STAR method, (l) is the high-resolution cropped image generated by the MGLD-VSR method, (m) is the high-resolution cropped image generated by the embodiment of this application, and (n) is the tenth high-resolution cropped image in the YouHQ40 dataset; Figure 5 This is a subjective quality comparison chart of a video super-resolution method according to an embodiment of this application and other deep learning video super-resolution models on the real dataset (Sequence016) VideoLQ, wherein (a) is the sixteenth low-resolution image in the VideoLQ dataset, (b) is a high-resolution cropped image generated by the DBVSR method, (c) is a high-resolution cropped image generated by the BasicVSR++ method, (d) is a high-resolution cropped image generated by the RealBasicVSR method, (e) is a high-resolution cropped image generated by the PiSA-SR method, (f) is a high-resolution cropped image generated by the STAR method, (g) is a high-resolution cropped image generated by the Upscale-A-Video method, (h) is a high-resolution cropped image generated by the MGLD-VSR method, and (i) is a high-resolution cropped image generated by an embodiment of this application. Figure 6 This is a schematic diagram of a video super-resolution reconstruction system in one embodiment of this application. Detailed Implementation

[0022] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all fall within the protection scope of the present application. Parts not described in detail in the following embodiments can be implemented using existing technology.

[0023] It should be noted that all information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with relevant regulations.

[0024] Video super-resolution technology is one of the important research directions in the field of computer vision. With the development of generative models, diffusion models have been gradually introduced into image and video super-resolution tasks. However, existing diffusion-based video super-resolution methods usually rely on multi-step iterative denoising processes, which leads to extremely high computational costs and slow inference speeds, making it difficult to meet the real-time or high-throughput requirements of practical applications; moreover, they are prone to texture loss or semantic shift problems. Based on the above problems, this application provides a video super-resolution reconstruction method based on a single-step diffusion model to solve the aforementioned problems.

[0025] Reference Figure 1 As shown, this application provides a video super-resolution reconstruction method based on a single-step diffusion model, including: M1. Input the low-resolution video sequence into the encoder in the three-dimensional variational autoencoder with frozen parameters. The encoder maps the low-resolution video sequence to the latent space to obtain the low-resolution latent representation. M2. Input the low-resolution latent representation into the trained single-step diffusion Transformer model, and perform separate denoising prediction under the condition of fixed diffusion time steps to generate the corresponding high-resolution latent representation. The single-step diffusion Transformer model is built based on the pre-trained diffusion model, and a two-stage distillation strategy including latent space consistency distillation and classifier fractional distillation is adopted during the training process. M3 inputs the high-resolution latent representation into the decoder in the three-dimensional variational autoencoder with frozen parameters. The decoder restores the high-resolution latent representation to the pixel space and reconstructs the final high-resolution video sequence.

[0026] The embodiments described above in this application employ a single-step diffusion Transformer model to replace the traditional multi-step diffusion sampling process. While maintaining the powerful generative and detail recovery capabilities of the diffusion model, this significantly improves inference efficiency and reduces computational resource consumption. A two-stage distillation strategy, including latent space consistency distillation and classifier fractional distillation, is employed to fully utilize the structural and semantic priors inherent in the pre-trained diffusion model. This allows the single-step diffusion Transformer model to learn high-quality video distribution features, achieving excellent reconstruction quality despite reducing sampling steps. This application enables high-quality, high-efficiency video super-resolution reconstruction under limited computational resources.

[0027] For example, for an input low-resolution video sequence, this application first inputs the low-resolution video sequence into the encoder of a three-dimensional variational autoencoder (3D-VAE) with frozen parameters, compressing and mapping the low-resolution video sequence to the latent space to obtain a low-resolution latent representation: in, This refers to the encoder of a 3D-VAE. This represents the input low-resolution video sequence. This indicates that the encoding yields a low-resolution latent representation of the low-resolution video sequence; The low-resolution latent representation is then input into a single-step diffusion Transformer model (DiT) for high-quality latent feature recovery. A single denoising prediction is performed under a fixed diffusion time step to directly generate a high-resolution latent representation. The single-step diffusion Transformer model is initialized by a pre-trained text-to-video diffusion model and optimized through a two-stage distillation strategy, thereby significantly reducing the inference overhead of the diffusion model while maintaining high-quality reconstruction capabilities.

[0028] Finally, the high-resolution latent representation is restored to pixel space using the decoder of the frozen 3D variational autoencoder, resulting in the final high-resolution video sequence. : in, This refers to the decoder for 3D-VAE.

[0029] Among them, the single-step diffusion Transformer model replaces the traditional multi-step iterative denoising process with a single forward inference, realizing a direct mapping from low-quality latent representation to high-quality latent representation.

[0030] In some specific embodiments of this application, the denoising prediction alone may further include: The low-resolution latent representation is used as the intermediate state of the diffusion process under a specific noise level. The low-resolution latent representation is input into the trained single-step diffusion Transformer model, and the denoising direction corresponding to the intermediate state is predicted by the single-step diffusion Transformer model in the velocity prediction mode. The corresponding high-resolution latent representation is reconstructed according to the preset diffusion process parameters and denoising direction.

[0031] For example, the low-resolution latent representation is regarded as the state under a specific noise level during the diffusion process. The corresponding denoising direction is directly predicted by the velocity prediction network in the single-step diffusion Transformer model, and the high-resolution latent representation is reconstructed based on the preset diffusion process parameters, thereby realizing the generation of high-quality video latent representation in a single step.

[0032] In some specific embodiments of this application, latent space consistency distillation is used in the first stage of training. In the first stage of training, pairs of low-resolution video samples and corresponding high-resolution video samples are acquired, and the parameters of the encoder and decoder in the 3D variational autoencoder are frozen. The encoder with frozen parameters maps the low-resolution video samples and corresponding high-resolution video samples to a unified latent space, respectively, to obtain low-resolution latent representations and true high-resolution latent representations. Using the low-resolution latent representation as input, the corresponding high-resolution latent representation is predicted and generated by the single-step diffusion Transformer model. The feature difference between the predicted high-resolution latent representation and the true high-resolution latent representation is used as the latent space consistency loss. The latent space consistency loss is used as the main loss for the first stage of training. The single-step diffusion Transformer model is trained in the first stage until convergence, and the student model is obtained after training.

[0033] For example, the first stage is latent space consistency distillation. By freezing the encoder, low-resolution video samples and their corresponding high-resolution video samples are mapped to the latent space, respectively. By constraining the consistency of these samples within the latent space, the single-step diffusion Transformer model learns the basic structural mapping relationship from low-resolution to high-resolution. Specifically, during the first stage of training, the encoder and decoder of the 3D variational autoencoder are frozen, and only the single-step diffusion Transformer model is trained. First, the low-resolution video samples and their corresponding high-resolution video samples are encoded into a unified latent space, obtaining low-resolution latent representations and high-resolution latent representations, respectively. Then, the single-step diffusion Transformer model is used to perform a single-pass denoising recovery on the low-resolution latent representation, generating the reconstructed high-resolution latent representation. Its single-step reconstruction process is represented as follows: in, This represents the cumulative signal retention coefficient at time step t during the diffusion process, and is a preset diffusion process parameter. This represents the velocity field predicted by the single-step diffusion Transformer model.

[0034] By applying the constraint of latent spatial consistency loss, the reconstructed high-resolution latent representation is made consistent with the high-resolution latent representation corresponding to the high-resolution video samples. The loss function is expressed as: in, This indicates a potential loss of spatial consistency. This represents the mathematical expectation operation. Representation in low-resolution latent representation For input, diffusion parameters are The single-step reconstruction function corresponding to the single-step diffusion Transformer model outputs the high-resolution latent representation reconstructed above. ; This represents the high-resolution latent representation corresponding to a high-resolution video sample.

[0035] In some specific embodiments of this application, fractional distillation of the classifier is used in the second-stage training process. In the second-stage training process, a pre-trained diffusion model is introduced as a teacher model. Different diffusion noises are added to the high-resolution latent representation output by the student model to obtain intermediate states under different diffusion noise levels. The intermediate states are input into the teacher model to perform inference under conditional input and unconditional input respectively, and the noise prediction results under conditional input and unconditional input are obtained accordingly. The feature difference between the noise prediction results under conditional input and the noise prediction results under unconditional input is used as the classifier fractional distillation loss. The classifier fractional distillation loss is used as the main loss for the second stage of training. Backpropagation is used to update the network parameters of the student model, and the final training of the single-step diffusion Transformer model is completed.

[0036] For example, the second stage is classifier fractional distillation. A pre-trained diffusion model is introduced as the teacher model, utilizing its semantic guidance signal provided by the difference between conditional and unconditional inputs to guide the single-step diffusion Transformer model in learning the distribution prior of high-quality videos, thereby enhancing the semantic consistency and realism of the generated results. Classifier fractional distillation includes: adding diffusion noise to the reconstructed high-resolution latent representation and inputting it into the pre-trained diffusion model to obtain noise prediction results under conditional and unconditional inputs, and constructing a semantic guidance direction based on the difference between the two noise prediction results; simultaneously, combining the classifier's free guidance mechanism, the semantic guidance direction is weighted and adjusted to enhance the degree to which the generated results follow the conditional information. Specifically, in the second stage training process, a pre-trained diffusion model is introduced as the teacher model to further optimize the student model obtained in the first stage training. First, diffusion noise is added to the high-resolution latent representation generated by the student model to construct intermediate states under different noise levels; then, it is input into the teacher model to obtain noise prediction results under conditional and unconditional inputs. The semantic direction corresponding to the conditional information is extracted using the difference between the two prediction results, and a semantic supervision signal is constructed using the classifier's free guidance mechanism. in, This represents the conditional prediction output after weighting adjustment using the classifier free guidance mechanism (CFG). This represents the predicted output under unconditional input. For purely conditional prediction results, To guide the weights.

[0037] Correspondingly, this ultimately affects the parameters of the single-step diffusion Transformer model. gradient for: in, For reconstructing a high-resolution latent representation for time step t Mathematical expectation operation of semantic condition C; The semantic supervision signal guides the student model to gradually approximate the high-quality video distribution learned by the teacher model. The fractional distillation loss of the classifier is expressed as: in, For semantic conditions, The noisy latent state of the student model at time step t is represented by a high-resolution latent representation with added diffuse noise. ; Noise-bearing hidden state Predict the corresponding semantic conditions The probability of; This enhances the texture details, visual realism, and semantic consistency of the generated results.

[0038] In some specific embodiments of this application, in the second-stage training process, in addition to the main loss of the second-stage training, an auxiliary loss is also introduced. The main loss and the auxiliary loss of the second-stage training are weighted and summed as the overall optimization objective of the second-stage training. The auxiliary loss includes at least one of perceptual loss and temporal consistency loss.

[0039] In some specific embodiments of this application, the temporal consistency loss is obtained as follows: the inter-frame variation of each adjacent frame pair in the real high-resolution video sample and the reconstructed high-resolution video sequence is calculated respectively, the Euclidean distance between the inter-frame variation of the corresponding adjacent frame pairs in the real high-resolution video sample and the reconstructed high-resolution video sequence is extracted, and the average value of the Euclidean distances corresponding to all adjacent frame pairs is taken to obtain the temporal consistency loss.

[0040] For example, to further ensure the temporal continuity of the generated high-resolution video sequence, this application designs an inter-frame difference consistency constraint / temporal consistency constraint. This constraint compares the changes between adjacent frames of the generated high-resolution video sequence and the real high-resolution video sample, enabling the single-step diffusion Transformer model to learn the motion patterns and temporal variation laws in the real high-resolution video sample. Its expression is as follows: in, For temporal consistency loss, n is the number of real high-resolution video sample frames, and i represents the index of the real high-resolution video sample frame. This represents the i-th frame of a real high-resolution video sample. This represents the i-th frame of the generated high-resolution video sequence; The embodiments described above in this application calculate the differences in changes between adjacent video frames and constrain the reconstructed high-resolution video sequence to maintain consistency with the motion changes of the real high-resolution video samples in the time dimension, thereby reducing inter-frame flicker and enhancing the continuity of video motion, effectively suppressing video flickering, and improving cross-frame motion consistency and temporal stability.

[0041] In some specific embodiments of this application, the perceptual loss is specifically as follows: within the depth feature space of the single-step diffusion Transformer model, the structural and texture differences between the real high-resolution video samples and the reconstructed high-resolution video sequence are calculated, and the arithmetic mean of the structural and texture differences is taken to obtain the perceptual loss.

[0042] For example, perceptual loss is used to construct perceptual quality constraints, which improve the visual realism and detail fidelity of the reconstruction results by measuring the structural and texture differences between real high-resolution video samples and reconstructed high-resolution video sequences in the deep feature space.

[0043] In order to enhance the perceptual quality of the reconstruction results, the embodiments of this application introduce a perceptual loss based on deep feature space. By constraining the similarity between the reconstructed video and the high-quality video (real high-resolution video sample) at the level of structural and texture features, the single-step diffusion Transformer model can recover more natural and realistic high-frequency details, thereby improving the overall visual quality.

[0044] The single-step diffusion Transformer model employs a multi-objective joint optimization strategy, with its overall training objective consisting of a weighted combination of various loss functions: the total loss function includes the latent space consistency loss. Classifier fractional distillation loss Perceived loss and timing consistency loss The loss terms collectively constrain the model's learning of structural consistency, semantic authenticity, visual quality, and temporal consistency. The final model's overall loss function is: in, Indicates perceived loss. , and This represents the weighting coefficient corresponding to each loss term.

[0045] Furthermore, by using a decoder to recover high-resolution video sequences, the computational complexity of traditional multi-step diffusion models is significantly reduced and inference efficiency is improved.

[0046] In the embodiments described above, the multi-step iterative sampling process required by traditional diffusion models is completely omitted during the inference phase. The video super-resolution reconstruction method of this application performs only a single diffusion denoising process during the inference phase: after encoding, the input low-resolution video sequence only needs to undergo a single forward propagation through a single-step diffusion Transformer model to generate a high-resolution latent representation from the low-resolution latent representation, and then directly recover the high-resolution video sequence through the decoder. Compared to traditional diffusion video super-resolution methods that require dozens or even hundreds of iterative denoising processes, this application only requires one denoising calculation to complete video super-resolution reconstruction, significantly reducing computational complexity and inference latency.

[0047] This application employs a single-step diffusion reconstruction framework to replace the traditional multi-step diffusion sampling process. While maintaining the powerful generative and detail recovery capabilities of the diffusion model, it significantly improves inference efficiency and reduces computational resource consumption. The proposed two-stage distillation strategy of "latent space consistency distillation + classifier fractional distillation" fully utilizes the structural and semantic priors inherent in the pre-trained video diffusion model, enabling the single-step model to learn high-quality video distribution features and achieve excellent reconstruction quality while reducing sampling steps. This application is applicable to various scenarios such as video enhancement, video restoration, high-definition video generation, streaming media transmission optimization, and mobile terminal video processing, and can achieve high-quality and high-efficiency video super-resolution reconstruction under limited computational resources.

[0048] This application's video super-resolution reconstruction method uses a pre-trained 3D variational autoencoder (3D-VAE) as a fixed encoder and decoder to encode the input low-resolution video sequence into a latent space representation. Within this latent space, a diffusion Transformer network (DiT) based on single-step denoising is constructed to directly predict the high-resolution latent representation. The DiT model is initialized from a pre-trained text-to-video diffusion model and optimized using a two-stage distillation strategy, updating only the parameters of the DiT while keeping the encoder and decoder fixed. In the first stage, latent space pixel consistency distillation is employed to minimize the mean square error between low-resolution and high-resolution videos in the latent space, achieving structural information alignment and basic mapping learning. In the second stage, classifier fractional distillation (CSD) and classifier free guidance (CFG) mechanisms are introduced. The difference between conditional and unconditional noise predictions from the pre-trained diffusion model is used as a semantic gradient supervision signal to guide the DiT model to learn the semantic prior of high-quality video distributions. Simultaneously, DISTS perceptual loss is combined to improve texture realism, and inter-frame difference constraints are introduced to enhance temporal consistency and motion coherence. Finally, joint optimization is achieved by weighted fusion of multiple loss functions, thereby enabling efficient reconstruction of low-resolution video to high-resolution video within a single-step inference framework. This application significantly reduces the computational overhead of multi-step sampling in the diffusion model while effectively improving the structural fidelity, perceptual quality, and temporal consistency of the reconstruction results, making it suitable for high-quality video super-resolution reconstruction tasks in resource-constrained environments.

[0049] The present application will be further described below with reference to specific embodiments in order to better understand the above technical solutions of the present application. It should be understood that the following are only some examples and are not intended to limit the present application.

[0050] Example 1: Refer to Figure 2 and Figure 3 As shown, the flowchart of the efficient video super-resolution method based on single-step diffusion is as follows: S101: Construction of a Frozen 3D-VAE Spatiotemporal Encoding and Decoding Module: A spatiotemporal representation module based on a 3D variational autoencoder is constructed, consisting of an encoder and a decoder. The encoder maps the input low-resolution video sequence to the latent feature space, obtaining a low-quality latent variable representation; the decoder restores the latent variables from the feature space to the pixel space, thereby generating a high-resolution video sequence output. Before training begins, the encoder and decoder parameters are frozen to remain fixed during subsequent training, not participating in gradient updates, and are used only as a stable spatiotemporal feature transformation and reconstruction module.

[0051] S102: Latent Space Structure Learning of the Single-Step Diffusion Transformer Model (First Stage Training): A single-step diffusion Transformer model is constructed, using a pre-trained multi-step video diffusion model as the initialization source. This model is then transformed into a single-step denoising structure to directly map low-quality videos to high-quality videos in the latent space. Low-resolution video samples and their corresponding high-resolution video samples are encoded into the latent space and then input into the single-step diffusion Transformer model for learning. By minimizing the difference between low-quality latent variables (i.e., low-resolution latent representations) and high-quality latent variables (high-resolution latent representations), the single-step diffusion Transformer model learns the fundamental structural alignment capability and spatiotemporal consistency mapping relationship. The goal of this first stage training is to enable the single-step diffusion Transformer model to first possess stable structural recovery capabilities, thereby providing reliable initialization for subsequent semantic distillation.

[0052] S103: Semantic Enhancement Training Based on Classifier Fractional Distillation (Second Stage Training): Building upon the first stage training, a pre-trained diffusion model is introduced as the teacher model to perform classifier fractional distillation training on the single-step diffusion Transformer model, thereby enhancing its semantic expressiveness and visual detail generation capabilities. By perturbing latent variables within a set diffusion time range and constructing a classifier free guidance signal using the teacher model's conditional and unconditional prediction results, semantic-level supervision constraints are applied to the student model's output, gradually bringing its generated results closer to the distribution of high-quality videos. Simultaneously, multiple auxiliary constraints are introduced during training: inter-frame consistency constraints, which constrain the consistency of changes between adjacent frames to ensure the continuity and stability of the videos generated by the single-step diffusion Transformer model in the temporal dimension, reducing flickering and jitter; and perceptual quality constraints, which constrain the structural and textural similarity between the generated results and real high-quality videos in the deep feature space, thereby improving visual naturalness and detail realism. Through the joint optimization of these multiple constraints, the model simultaneously possesses structural recovery capabilities, semantic alignment capabilities, and temporal consistency modeling capabilities.

[0053] S104: Single-Step Latent Space Inference and Video Reconstruction Output: During the inference phase, the input low-resolution video sequence is mapped to the latent space through a frozen encoder to obtain a low-resolution latent representation. This low-resolution latent representation is then input into the trained single-step diffusion Transformer model for one-time denoising, directly generating a high-resolution latent representation. Subsequently, the generated high-resolution latent representation is input into the decoder for reconstruction, outputting the final high-resolution video sequence. This process eliminates the need for the multi-step iterative sampling process of traditional diffusion models, completing video enhancement through a single forward inference, thus significantly improving inference efficiency and reducing computational overhead while maintaining structural consistency and semantic quality.

[0054] The experimental results of the single-step diffusion Transformer model described in the above embodiments are evaluated below. In the quantitative evaluation, this application uses high-quality video datasets such as REDS, DAVIS, and HQ-VSR for training to cover training samples with different resolutions, motion complexities, and content distributions, thereby improving the model's generalization ability. In the testing phase, validation was performed on several classic video super-resolution benchmark datasets, including UDM10, SPMCS, YouHQ40, and the real-world dataset VideoLQ. These datasets cover both synthetic and realistic degradation scenes, enabling a comprehensive evaluation of the model's performance in terms of structural recovery, texture detail reconstruction, and temporal consistency.

[0055] The evaluation metrics adopt standard objective metrics from video super-resolution tasks, including PSNR and SSIM to measure structural fidelity. LPIPS and DISTS are introduced to evaluate perceptual quality, CLIP-IQA and MUSIQ to measure semantic consistency and visual quality, and BRISQUE and DOVER, among other no-reference quality metrics, are used to assess visual naturalness in real-world scenarios and overall video quality. Furthermore, to evaluate temporal consistency, the Ewarp metric, based on motion consistency, is introduced to comprehensively measure the model's overall performance in both spatial and temporal dimensions.

[0056] To verify the effectiveness of the proposed single-step diffusion model, the proposed single-step diffusion VSR method is compared with several representative methods, including traditional non-diffusion video super-resolution methods (DBVSR, BasicVSR++, RealBasicVSR), diffusion video super-resolution methods (MGLD-VSR, Upscale-A-Video, STAR), and lightweight super-resolution methods (PiSA-SR). Most of the compared methods rely on multi-step inference or complex temporal propagation structures, while the proposed method employs a single-step diffusion inference mechanism, significantly reducing computational complexity while maintaining high-quality reconstruction capabilities.

[0057] Table 1. Comparison of the objective quality of the video super-resolution method of this application with other deep learning video super-resolution models: As shown in Table 1, the experimental results demonstrate that the proposed method achieves superior or near-optimal performance on multiple datasets. On synthetic degradation datasets, the proposed method exhibits stable performance on PSNR and SSIM metrics, indicating that the model effectively recovers image structural information. In terms of perceptual quality metrics, the proposed method demonstrates significant advantages on LPIPS and DISTS metrics, indicating superior texture detail recovery and visual realism compared to the contrasting methods. Furthermore, the proposed method achieves leading results on multiple datasets on CLIP-IQA and MUSIQ metrics, demonstrating stronger advantages in semantic consistency and overall visual quality. On the real-world dataset VideoLQ, the proposed method performs excellently on the no-reference quality evaluation metrics DOVER and BRISQUE, demonstrating stable visual reconstruction capabilities and good naturalness even in complex degradation environments. In addition, the proposed method outperforms the Ewarp metric in temporal consistency evaluation, indicating stronger capabilities in maintaining motion continuity between video frames and suppressing flicker.

[0058] In terms of visual effects, such as Figure 4 and Figure 5 As shown, the method of this application can better recover complex textures and fine-grained structures, such as animal fur, text details, and complex edge structures. In real-world scenes, for blurred text and low-light areas, the method of this application can more accurately recover semantic information and structural details. Compared with the common problems of oversmoothing, loss of detail, and temporal flicker in existing methods, the method of this application shows superior overall performance in both spatial detail recovery and temporal stability.

[0059] Reference Figure 6 As shown, based on the same inventive concept, another embodiment of this application provides a video super-resolution reconstruction system based on a single-step diffusion model. The video super-resolution reconstruction system 100 includes: Encoding module 110 is used to input the low-resolution video sequence into the encoder in the three-dimensional variational autoencoder of the frozen parameters, and the encoder maps the low-resolution video sequence to the latent space to obtain the low-resolution latent representation. The diffusion module 120 is used to input the low-resolution latent representation into the trained single-step diffusion Transformer model, and perform separate denoising prediction under the condition of fixed diffusion time steps to generate the corresponding high-resolution latent representation. The single-step diffusion Transformer model is built based on the pre-trained diffusion model, and a two-stage distillation strategy including latent space consistency distillation and classifier fractional distillation is adopted during the training process. The reconstruction module 130 is used to input the high-resolution latent representation into the decoder in the three-dimensional variational autoencoder with frozen parameters. The decoder restores the high-resolution latent representation to the pixel space and reconstructs the final high-resolution video sequence.

[0060] It should be noted that the modules in the video super-resolution reconstruction system based on the single-step diffusion model provided in the above embodiments of this application correspond to the steps of the video super-resolution reconstruction method based on the single-step diffusion model in any of the above embodiments. Those skilled in the art can refer to the step features of the video super-resolution reconstruction method based on the single-step diffusion model to implement the corresponding modules in the video super-resolution reconstruction system based on the single-step diffusion model, which will not be elaborated here.

[0061] In another embodiment of this application, an electronic device is also provided, including a memory and a processor; the memory is used to store program instructions; the processor is used to call the program instructions stored in the memory and execute the steps of the video super-resolution reconstruction method based on the single-step diffusion model according to the obtained program instructions.

[0062] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs and functional modules that implement the above methods), computer instructions, etc., and the aforementioned computer programs and computer instructions can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0063] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0064] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0065] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0066] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0067] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0068] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0069] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0070] The preferred features in the above embodiments can be used individually in any embodiment, or in any combination thereof, provided they do not conflict with each other. Furthermore, parts not described in detail in the embodiments can be implemented using existing technologies.

[0071] The foregoing has described some specific embodiments of this application. It should be understood that this application is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this application. The above-described preferred features can be used in any combination without conflict.

Claims

1. A video super-resolution reconstruction method based on a single-step diffusion model, characterized in that, include: A low-resolution video sequence is input into the encoder in a three-dimensional variational autoencoder with frozen parameters. The encoder maps the low-resolution video sequence to the latent space to obtain a low-resolution latent representation. The low-resolution latent representation is input into the trained single-step diffusion Transformer model, and separate denoising prediction is performed under the condition of fixed diffusion time steps to generate the corresponding high-resolution latent representation; wherein, the single-step diffusion Transformer model is built based on the pre-trained diffusion model, and a two-stage distillation strategy including latent space consistency distillation and classifier fractional distillation is adopted during the training process. The high-resolution latent representation is input into the decoder in the three-dimensional variational autoencoder with frozen parameters. The decoder restores the high-resolution latent representation to the pixel space and reconstructs the final high-resolution video sequence.

2. The video super-resolution reconstruction method based on a single-step diffusion model according to claim 1, characterized in that, The separate denoising prediction includes: The low-resolution latent representation is used as an intermediate state of the diffusion process at a specific noise level. The low-resolution latent representation is input into a trained single-step diffusion Transformer model, and the denoising direction corresponding to the intermediate state is predicted by the single-step diffusion Transformer model in a velocity prediction mode. The corresponding high-resolution latent representation is reconstructed according to the preset diffusion process parameters and the denoising direction.

3. The video super-resolution reconstruction method based on a single-step diffusion model according to claim 1, characterized in that, The latent space consistency distillation is used in the first stage of training. In the first stage of training, pairs of low-resolution video samples and corresponding high-resolution video samples are acquired. The parameters of the encoder and decoder in the 3D variational autoencoder are frozen. The encoder with frozen parameters maps the low-resolution video samples and corresponding high-resolution video samples to a unified latent space, respectively, to obtain low-resolution latent representations and true high-resolution latent representations. The low-resolution latent representation is used as input, and the corresponding high-resolution latent representation is predicted and generated by the single-step diffusion Transformer model. The feature difference between the predicted high-resolution latent representation and the true high-resolution latent representation is used as the latent space consistency loss. The latent space consistency loss is used as the main loss for the first stage of training. The single-step diffusion Transformer model is trained in the first stage until convergence. After training, a student model is obtained.

4. The video super-resolution reconstruction method based on a single-step diffusion model according to claim 3, characterized in that, The classifier fractional distillation is used in the second-stage training process. In the second-stage training process, a pre-trained diffusion model is introduced as the teacher model. Different diffusion noises are added to the high-resolution latent representation output by the student model to obtain intermediate states under different diffusion noise levels. The intermediate states are input into the teacher model to perform inference under conditional input and unconditional input respectively, and the noise prediction results under conditional input and unconditional input are obtained accordingly. The feature difference between the noise prediction results under conditional input and the noise prediction results under unconditional input is used as the classifier fractional distillation loss. The classifier fractional distillation loss is used as the main loss for the second stage of training. Backpropagation is used to update the network parameters of the student model, and the final training of the single-step diffusion Transformer model is completed.

5. The video super-resolution reconstruction method based on a single-step diffusion model according to claim 4, characterized in that, In the second stage of training, in addition to the main loss of the second stage training, an auxiliary loss is also introduced. The main loss and the auxiliary loss of the second stage training are weighted and summed as the overall optimization objective of the second stage training. The auxiliary loss includes at least one of perceptual loss and temporal consistency loss.

6. The video super-resolution reconstruction method based on a single-step diffusion model according to claim 5, characterized in that, The timing consistency loss is obtained as follows: The inter-frame variation of each adjacent frame pair in the real high-resolution video sample and the reconstructed high-resolution video sequence is calculated respectively. The Euclidean distance between the inter-frame variation of the corresponding adjacent frame pairs in the real high-resolution video sample and the reconstructed high-resolution video sequence is extracted. The average value of the Euclidean distances corresponding to all adjacent frame pairs is taken to obtain the temporal consistency loss.

7. The video super-resolution reconstruction method based on a single-step diffusion model according to claim 5, characterized in that, The perceived loss includes: Within the deep feature space of the single-step diffusion Transformer model, the structural and texture differences between real high-resolution video samples and reconstructed high-resolution video sequences are calculated, and the arithmetic mean of the structural and texture differences is used to obtain the perceptual loss.

8. A video super-resolution reconstruction system based on a single-step diffusion model, characterized in that, include: The encoding module is used to input a low-resolution video sequence into the encoder in the three-dimensional variational autoencoder of the frozen parameters, and the encoder maps the low-resolution video sequence to the latent space to obtain a low-resolution latent representation. The diffusion module is used to input the low-resolution latent representation into the trained single-step diffusion Transformer model, and perform separate denoising prediction under the condition of fixed diffusion time steps to generate the corresponding high-resolution latent representation; wherein, the single-step diffusion Transformer model is built based on the pre-trained diffusion model, and a two-stage distillation strategy including latent space consistency distillation and classifier fractional distillation is adopted during the training process. The reconstruction module is used to input the high-resolution latent representation into the decoder in the three-dimensional variational autoencoder with frozen parameters, and the decoder restores the high-resolution latent representation to the pixel space to reconstruct the final high-resolution video sequence.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1-7.

10. An electronic device, characterized in that, include: At least one memory for storing program instructions; At least one processor is configured to invoke program instructions stored in the memory and execute the steps of the method as described in any one of claims 1-7 according to the obtained program instructions.

Images