Distillation method, image enhancement method, device, equipment and product of diffusion model

Abstract

The disclosure relates to a diffusion model distillation method, a diffusion model-based image enhancement method, an apparatus, an electronic device and a computer program product, and belongs to the technical field of image processing. The diffusion model distillation method comprises: obtaining a multi-step diffusion model as a teacher model, and sampling to obtain first sample images and second sample images at different time steps on the same diffusion trajectory; inputting the first sample images and the second sample images into a student model to obtain predicted first target sample images and second target sample images, and obtaining a consistency matching loss; constructing a discriminator based on the teacher model, and inputting target sample images and real sample images into the discriminator to obtain an adversarial loss; obtaining an overall loss according to the consistency matching loss and the adversarial loss, and training model parameters in the student model according to the overall loss to obtain a single-step diffusion model. The disclosure can realize single-step reasoning to improve efficiency while maintaining high-quality image generation effect.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and more specifically, to a distillation method based on a diffusion model, an image enhancement method based on a diffusion model, a distillation apparatus based on a diffusion model, an image enhancement apparatus based on a diffusion model, an electronic device, and a computer program product. Background Technology

[0002] With the rapid development of artificial intelligence technology, video quality enhancement technology has been widely applied in scenarios such as video streaming, video surveillance, and virtual reality. Video quality enhancement technology aims to improve the visual expressiveness of videos, achieving higher resolution, clearer detail, and more realistic color reproduction while preserving the integrity of the original content. Current video enhancement methods mainly rely on complex image processing algorithms. These algorithms are complex in structure and inefficient, making them unsuitable for the real-time processing needs of large-scale video data, especially in high-resolution video streaming scenarios.

[0003] In recent years, deep learning-based diffusion models have achieved significant breakthroughs in image and video enhancement. Diffusion models, by progressively reversing the image or video generation process, can generate high-quality images or videos, demonstrating exceptional capabilities, particularly in denoising and super-resolution tasks. However, the inference speed of diffusion models is relatively slow, especially in video enhancement tasks, where the need for complex calculations on each frame leads to poor real-time performance, failing to meet the demands of practical applications.

[0004] Therefore, there is an urgent need in this field for a method that can improve inference efficiency while ensuring the quality of image or video generation.

[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The purpose of this disclosure is to provide a distillation method based on a diffusion model, an image enhancement method based on a diffusion model, a distillation apparatus based on a diffusion model, an image enhancement apparatus based on a diffusion model, an electronic device, and a computer program product, thereby improving inference efficiency and ensuring the generation effect of images or videos to at least a certain extent.

[0007] According to a first aspect of this disclosure, a distillation method based on a diffusion model is provided, comprising: A pre-trained multi-step diffusion model is obtained as a teacher model, and first and second sample images at different time steps on the same diffusion trajectory are obtained by sampling through the teacher model. The first sample image and the second sample image are input into the student model to obtain the predicted first target sample image and the second target sample image, and the consistency matching loss is obtained based on the first target sample image and the second target sample image. A discriminator is constructed based on the teacher model, and the predicted target sample image and the real sample image are input into the discriminator. The adversarial loss is obtained based on the output of the discriminator. The target sample image is either the first target sample image or the second target sample image. The overall loss is obtained based on the consistency matching loss and the adversarial loss, and the model parameters in the student model are trained based on the overall loss. A single-step diffusion model is obtained based on the trained student model.

[0008] In one exemplary embodiment of this disclosure, the step of inputting the first sample image and the second sample image into the student model to obtain the predicted first target sample image and the second target sample image includes: The prediction target of the student model is reparameterized to predict the average velocity from any time step to the target sample image on the diffusion trajectory; Input the first sample image and the second sample image into the student model to obtain the predicted average velocity of the first sample and the average velocity of the second sample. The first target sample image is obtained based on the average velocity of the first sample, and the second target sample image is obtained based on the average velocity of the second sample.

[0009] In one exemplary embodiment of this disclosure, obtaining the consistency matching loss based on the first target sample image and the second target sample image includes: The consistency matching loss is obtained based on the distance parameter between the first target sample image and the second target sample image.

[0010] In one exemplary embodiment of this disclosure, constructing a discriminator based on the teacher model includes: The discriminator is constructed based on the intermediate features of the teacher model during the multi-step diffusion process.

[0011] In one exemplary embodiment of this disclosure, obtaining the adversarial loss based on the output of the discriminator includes: The target sample image or the real sample image is mixed with the intermediate features to obtain an adversarial sample image; The adversarial sample image and interpolation coefficients are input into the discriminator to obtain the corresponding discrimination score, and the adversarial loss is obtained based on the discrimination score.

[0012] According to a second aspect of this disclosure, an image enhancement method based on a diffusion model is provided, comprising: The first video frame image to be processed is obtained, and the first video frame image is input into the single-step diffusion model, which is obtained by the distillation method of the diffusion model as described above. The first video frame image is predicted step-by-step using the single-step diffusion model to obtain an enhanced second video frame image, wherein the image quality of the second video frame image is higher than that of the first video frame image.

[0013] According to a third aspect of this disclosure, a distillation apparatus based on a diffusion model is provided, comprising: The sample image sampling module is configured to acquire a pre-trained multi-step diffusion model as a teacher model, and to sample first and second sample images at different time steps on the same diffusion trajectory through the teacher model. The matching loss determination module is configured to input the first sample image and the second sample image into the student model to obtain the predicted first target sample image and the second target sample image, and to obtain the consistency matching loss based on the first target sample image and the second target sample image; The adversarial loss determination module is configured to construct a discriminator based on the teacher model, input the predicted target sample image and the real sample image into the discriminator, and obtain the adversarial loss based on the output of the discriminator; wherein the target sample image is either the first target sample image or the second target sample image; The model parameter training module is configured to perform an overall loss based on the consistency matching loss and the adversarial loss, train the model parameters in the student model based on the overall loss, and obtain a single-step diffusion model based on the trained student model.

[0014] In one exemplary embodiment of this disclosure, the matching loss determination module includes: The reparameterization unit is configured to perform the reparameterization of the student model's prediction target to predict the average velocity from any time step to the target sample image on the diffusion trajectory. The sample average velocity determination unit is configured to input the first sample image and the second sample image into the student model to obtain the predicted first sample average velocity and the second sample average velocity. The target sample image determination unit is configured to perform the following operations: obtaining the first target sample image based on the first sample average speed, and obtaining the second target sample image based on the second sample average speed.

[0015] In one exemplary embodiment of this disclosure, the matching loss determination module further includes: The matching loss determination unit is configured to perform a consistency matching loss based on the distance parameter between the first target sample image and the second target sample image.

[0016] In one exemplary embodiment of this disclosure, the adversarial loss determination module includes: The discriminator construction unit is configured to execute a structure that constructs the discriminator based on intermediate features of the teacher model during a multi-step diffusion process.

[0017] In one exemplary embodiment of this disclosure, the adversarial loss determination module further includes: The adversarial sample image determination unit is configured to perform the operation of mixing the target sample image or the real sample image with the intermediate features to obtain an adversarial sample image; The adversarial loss determination unit is configured to input the adversarial sample image and interpolation coefficients into the discriminator to obtain the corresponding discrimination score, and to obtain the adversarial loss based on the discrimination score.

[0018] According to a fourth aspect of this disclosure, an image enhancement apparatus based on a diffusion model is provided, comprising: The image input module is configured to acquire a first video frame image to be processed and input the first video frame image into a single-step diffusion model, which is obtained by the distillation method of the diffusion model as described above. The image prediction module is configured to perform a single-step prediction on the first video frame image using the single-step diffusion model to obtain an enhanced second video frame image, wherein the image quality of the second video frame image is higher than that of the first video frame image.

[0019] According to a fifth aspect of this disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement a distillation method of a diffusion model as described in any of the preceding claims.

[0020] According to a sixth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements a distillation method using the diffusion model described in any one of the preceding claims.

[0021] The exemplary embodiments disclosed herein can have the following beneficial effects: In the distillation method of the diffusion model in the exemplary implementation of this disclosure, the teacher model samples sample images at different time steps on the same diffusion trajectory, the student model predicts the target sample image and calculates the consistency matching loss, and combines the adversarial loss generated by the discriminator built based on the teacher model to train the student model parameters to obtain an efficient single-step diffusion model. This method can efficiently distill a multi-step diffusion model into a single-step model, significantly reducing computational overhead and inference latency. At the same time, the consistency matching loss and adversarial loss ensure the generation quality, avoiding the problems of missing details and noise. The generated image still maintains the image quality and detail performance approximately consistent with the original multi-step inference model, achieving high-quality generation effect while achieving single-step inference, thereby meeting the needs of real-time video processing and achieving a balance between high speedup and high image fidelity.

[0022] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0024] Figure 1 A schematic flowchart of a distillation method for a diffusion model according to an exemplary embodiment of this disclosure is shown. Figure 2 A schematic diagram of a consistency model predicting a target is shown in one specific embodiment of this disclosure; Figure 3 A schematic diagram illustrating the process of obtaining a target sample image according to an exemplary embodiment of this disclosure is shown; Figure 4 A schematic diagram of a training method for a consistency model according to a specific embodiment of this disclosure is shown; Figure 5 A schematic diagram of a training method for a reparameterized consistency model according to a specific embodiment of this disclosure is shown. Figure 6 A schematic flowchart of a distillation method using a diffusion model according to a specific embodiment of this disclosure is shown. Figure 7 A schematic flowchart of an image enhancement method based on a diffusion model according to an exemplary embodiment of this disclosure is shown; Figure 8 A block diagram of a distillation apparatus for a diffusion model according to an exemplary embodiment of the present disclosure is shown; Figure 9A block diagram of a diffusion-based image enhancement apparatus according to an exemplary embodiment of the present disclosure is shown; Figure 10 A schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present disclosure is shown. Detailed Implementation

[0025] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0026] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein.

[0027] The following exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0028] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0029] In some related embodiments, model compression and acceleration techniques can be used to improve the inference speed of diffusion models. Distillation is an effective model optimization technique, with trajectory distillation and distribution distillation being two important methods. Trajectory distillation improves inference efficiency by passing intermediate states from the teacher model's inference process to the student model, helping the student model learn more refined inference trajectories. Distribution distillation, on the other hand, matches the distributional differences between the teacher and student models, enabling the student model to better approximate the teacher model's performance.

[0030] Accelerating diffusion video enhancement models requires balancing speed and accuracy. Speeding up inference often necessitates simplifying or compressing the model structure, which can lead to a decrease in accuracy. Therefore, it's crucial to maintain the high-quality output required for video enhancement tasks while accelerating the model. A joint optimization method combining trajectory distillation and distribution distillation promises to preserve accuracy to the greatest extent possible while maintaining inference speed, thus providing a more efficient and accurate video enhancement solution for practical applications.

[0031] The video enhancement method based on the diffusion model in the relevant embodiments performs well in terms of generation quality, but it faces significant performance bottlenecks in actual deployment, mainly in the following two aspects: 1. The diffusion model generates image or video frames using a stepwise denoising method. A typical implementation usually requires 50 or more iterations of sampling to recover the target image from the noise. This means that high-frequency repetitive inference needs to be performed for each frame in the video, resulting in a huge computational load and inference latency.

[0032] 2. The generative capability of diffusion models relies on a massive network structure and a huge number of parameters, typically reaching the level of 1 billion (1B) or more. Furthermore, the numerous spatial and temporal attention operations within the model further increase the computational cost of single-step inference, making the real-time application of diffusion models in video enhancement tasks extremely difficult.

[0033] To address the aforementioned bottlenecks, the acceleration strategies mainly fall into two categories: 1. "One-step inference" methods that reduce the number of inference steps: such as DOVE (Deformable Offset Volume-based Exploration) and FastVSR (Fast Video Super-Resolution). These methods load the parameters of a pre-trained video diffusion generation model and then directly train a video enhancement model that can perform single-step inference, thereby completely skipping the multi-step diffusion sampling process and achieving inference speedup of tens of times.

[0034] 2. Structural optimization methods to reduce the computational cost of single-step inference: such as FlashVSR, which introduce sparse causal attention or other efficient attention structures to reduce the computational overhead of the attention module in the diffusion model, thereby shortening the execution time of single-step inference.

[0035] While the two approaches mentioned above have made some progress in accelerating the diffusion model, they still have the following significant shortcomings: 1. One-step inference methods suffer significant loss in generation capability: These methods do not explicitly reproduce the multi-step denoising trajectory of the diffusion model during training, resulting in a significant weakening of generation capability despite achieving over 20 times inference speedup. When faced with complex and varied video content in real-world business scenarios, they often fail to maintain the high-fidelity and robust video enhancement effects that the diffusion model should possess.

[0036] 2. Limited acceleration effect of structure optimization methods: Structure optimization such as sparsification attention can reduce the amount of computation to a certain extent and maintain high generation quality, but the overall speedup is usually only 10%~20%, which cannot meet the requirements of real-time performance and throughput in production environments, especially for high-resolution video enhancement tasks.

[0037] In summary, the video enhancement methods based on diffusion models in the relevant embodiments cannot simultaneously meet the requirements of "significant acceleration" and "maintaining generation quality," and therefore still have considerable room for optimization.

[0038] To address the aforementioned issues, this exemplary implementation first provides a distillation method based on a diffusion model. (See reference...) Figure 1 As shown, the distillation method of the above diffusion model may include the following steps: Step S110. Obtain a pre-trained multi-step diffusion model as a teacher model, and obtain the first sample image and the second sample image at different time steps on the same diffusion trajectory through sampling using the teacher model.

[0039] Step S120. Input the first sample image and the second sample image into the student model to obtain the predicted first target sample image and the second target sample image, and obtain the consistency matching loss based on the first target sample image and the second target sample image.

[0040] Step S130. Construct a discriminator based on the teacher model, and input the predicted target sample image and the real sample image into the discriminator. Obtain the adversarial loss based on the output of the discriminator.

[0041] The target sample image is either the first target sample image or the second target sample image.

[0042] Step S140. Obtain the overall loss based on the consistency matching loss and the adversarial loss, and train the model parameters in the student model based on the overall loss. Obtain the single-step diffusion model based on the trained student model.

[0043] In the distillation method of the diffusion model in the exemplary implementation of this disclosure, the teacher model samples sample images at different time steps on the same diffusion trajectory, the student model predicts the target sample image and calculates the consistency matching loss, and combines the adversarial loss generated by the discriminator built based on the teacher model to train the student model parameters to obtain an efficient single-step diffusion model. This method can efficiently distill a multi-step diffusion model into a single-step model, significantly reducing computational overhead and inference latency. At the same time, the consistency matching loss and adversarial loss ensure the generation quality, avoiding the problems of missing details and noise. The generated image still maintains the image quality and detail performance approximately consistent with the original multi-step inference model, achieving high-quality generation effect while achieving single-step inference, thereby meeting the needs of real-time video processing and achieving a balance between high speedup and high image fidelity.

[0044] Below, in conjunction with Figures 2 to 6 The steps described above in this example implementation will be explained in more detail.

[0045] In step S110, a pre-trained multi-step diffusion model is obtained as a teacher model, and first and second sample images at different time steps on the same diffusion trajectory are obtained by sampling through the teacher model.

[0046] In this example implementation, the teacher model refers to a pre-trained, high-performance, but computationally expensive multi-step diffusion model capable of generating images or videos with high image quality. The diffusion trajectory refers to a series of intermediate image states experienced during the inverse denoising process of the diffusion model, gradually recovering from pure noise to a higher-resolution image. Different time steps on this trajectory correspond to different degrees of denoising. A time step refers to a discretized point in time during the denoising process of the diffusion model; at different time steps, the noise level and detail of the image vary. A sample image refers to an image with a specific noise level extracted from the diffusion trajectory.

[0047] In this example implementation, trajectory distillation of the diffusion model can be performed based on the consistency model principle. The consistency model is a method of sampling step distillation for a diffusion model, and its goal is to enable the model to directly perform single-step inference to obtain the same target image from samples at different time steps on the same diffusion trajectory.

[0048] Figure 2 A schematic diagram illustrating the consistency model predicting the target in a specific embodiment of this disclosure is shown. Here, t represents the time step in which the diffusion model gradually diffuses from a clean image to noise. This represents the sample image at time t. This represents the neural network that needs to be trained, i.e., the student model.

[0049] The teacher model can sample first and second sample images at different time steps along the diffusion trajectory. The first and second sample images can be two sample images from adjacent time steps, for example... and .

[0050] In step S120, the first sample image and the second sample image are input into the student model to obtain the predicted first target sample image and the second target sample image, and the consistency matching loss is obtained based on the first target sample image and the second target sample image.

[0051] In this example implementation, the student model is typically a simpler neural network with fewer parameters. The target sample image refers to the output predicted by the student model based on the input sample image, which aims to approximate a real image with high resolution. This target sample image is the key output for the student model to learn denoising capabilities.

[0052] When the first and second sample images are input into the student model, the model attempts to denoise or transform them to output the corresponding predicted image. To measure the consistency between these predictions, the difference between the first and second target sample images can be calculated.

[0053] In this example implementation, as Figure 3 As shown, inputting the first sample image and the second sample image into the student model to obtain the predicted first target sample image and the second target sample image can specifically include the following steps: Step S310. Reparameterize the predicted target of the student model as the average velocity from any time step to the target sample image on the predicted diffusion trajectory.

[0054] In this example implementation, the form of the student model's output can be changed so that it no longer directly predicts the final target sample image, but instead predicts an intermediate quantity related to the diffusion process, namely, the average velocity. The average velocity can be understood as the average rate of change or direction of change from the current time step to the target sample image along the diffusion trajectory.

[0055] Figure 4 A schematic diagram of a training method for a consistency model according to a specific embodiment of this disclosure is shown. The training method for the consistency model involves sampling two sample images on the same diffusion trajectory using a teacher model, for example... and The student model is used to predict the target image from these two sample points, and the two prediction results are constrained to be the same.

[0056] However, current diffusion models typically use the Rectified-flow diffusion framework, with the diffusion speed of samples along the diffusion trajectory as the prediction target. Directly applying consistency model methods to distill multi-step diffusion video enhancement models leads to insufficient utilization of pre-trained weights, resulting in high training difficulty and high error. Therefore, reparameterization can be performed to reduce the gap between students and teachers in predicting the target and to reduce training difficulty.

[0057] Figure 5 A schematic diagram of a training method for a reparameterized consistency model according to a specific embodiment of this disclosure is shown. and The average speed from the first sample image and the second sample image to the target sample image.

[0058] By reparameterizing the goal of consistency prediction from "directly predicting the target image" to "predicting the average velocity from any time point to the target image along the diffusion trajectory," this reparameterization technique makes the student model's prediction target more numerically and semantically closer to the teacher's multi-step output, thus significantly improving the convergence and final quality of consistency distillation. After the reparameterization improvement, trajectory distillation of the diffusion video enhancement model can be completed, and the number of diffusion sampling steps can be successfully reduced without significant image quality loss, for example, from 25 steps to 1 step, accelerating the diffusion sampling speed by 25 times.

[0059] Step S320. Input the first sample image and the second sample image into the student model to obtain the predicted average velocity of the first sample and the average velocity of the second sample.

[0060] The student model receives the first and second sample images as input. Instead of directly outputting the target sample image, it outputs the average velocity corresponding to these sample images. These average velocities are the model's predicted targets after reparameterization.

[0061] Step S330. Obtain the first target sample image based on the average velocity of the first sample, and obtain the second target sample image based on the average velocity of the second sample.

[0062] Since the student model no longer directly predicts the target image, the target sample image can be derived from the predicted average speed through a computational process.

[0063] In this example implementation, the prediction target of the student model is reparameterized. Specifically, the student model no longer directly predicts the target sample image on the diffusion trajectory, but is instead configured to predict the average velocity from any time step to the target sample image. When the first sample image and the second sample image are input into the student model, the model outputs the corresponding first sample average velocity and second sample average velocity, respectively. Subsequently, using these predicted average velocities and combining them with the mathematical principles of the diffusion model, the corresponding first target sample image and second target sample image can be derived.

[0064] In this example implementation, the consistency matching loss can be obtained based on the distance parameter between the first target sample image and the second target sample image.

[0065] In this example implementation, the consistency matching loss is a loss function that measures the consistency of the student model's predictions across different time steps. By minimizing this loss, it can be ensured that the student model produces consistent and logically coherent outputs when processing images with different noise levels along the same diffusion trajectory. The distance parameter can be used to quantify the degree of difference between two images.

[0066] The formula for calculating the consistency matching loss is as follows:

[0067] in, It is a weighting function related to the time step. These represent the first target sample image and the second target sample image, respectively.

[0068] In this example implementation, a consistency model is introduced and reparameterization is improved in the trajectory distillation stage. By predicting the average velocity from any time point to the target image, the student model can more accurately approximate the teacher model's multi-step denoised trajectory in single-step prediction, thereby significantly reducing the training difficulty and ensuring the overall structural and semantic consistency, and achieving inference acceleration without significant image quality loss.

[0069] Trajectory distillation has compressed multi-step diffusion sampling into single-step prediction, but single-step models are still prone to image quality degradation in detailed areas, especially in high-frequency textures such as hair and eyebrows, which often exhibit noise, breaks, and jagged edges. This phenomenon stems from the fact that while the single-step student model approximates the teacher model in trajectory space, it still has statistical biases at the generation distribution level, failing to fully reproduce the detail modeling capabilities of multi-step diffusion in high-dimensional distributions. Therefore, this example implementation proposes a distribution distillation method based on diffusion adversarial distillation, which constrains the single-step model at the distribution level, further reducing the difference between the single-step student model and the multi-step teacher model at the generation distribution level. Simultaneously, trajectory distillation and distribution distillation are optimized as part of the joint loss during training, ensuring that the single-step model achieves pixel-level fidelity in output image quality compared to the teacher model, while optimizing the detail generation quality of the single-step model and compensating for the shortcomings of single-step inference in high-frequency texture information recovery.

[0070] In step S130, a discriminator is constructed based on the teacher model, and the predicted target sample image and the real sample image are input into the discriminator. The adversarial loss is obtained based on the output of the discriminator.

[0071] In this example implementation, the discriminator is a standalone neural network whose task is to distinguish whether the input image comes from the predictions of the student model (i.e., the predicted target sample image) or from images in the real dataset (i.e., real sample images). The adversarial loss is the loss generated during adversarial training between the discriminator and the student model. This loss prompts the student model to generate images that are visually closer to reality and more difficult for the discriminator to distinguish, thereby improving the generation quality of the student model.

[0072] In this example implementation, the discriminator structure can be constructed based on the intermediate features of the teacher model in the multi-step diffusion process.

[0073] By constructing the discriminator's structural framework using intermediate features from the teacher model during the multi-step diffusion process, the discriminator can learn fine-grained differences in the true distribution using the diffusion feature space as a metric. The discriminator receives multi-step generation results from the teacher model and single-step outputs from the student model, driving the student model to statistically approximate the teacher model through adversarial learning. Unlike traditional GAN ​​(Generative Adversarial Network) discriminators, the discriminator in this example implementation relies on diffusion features for construction. Its discrimination criteria include not only the visual consistency in the image space but also the semantic structure, detail frequencies, and noise distribution progressively encoded by the teacher model in the diffusion trajectory. Therefore, it can more effectively constrain the high-frequency details of the single-step generation model.

[0074] In this example implementation, the target sample image or real sample image can be mixed with intermediate features to obtain an adversarial sample image; the adversarial sample image and interpolation coefficients are input into the discriminator to obtain the corresponding discrimination score, and the adversarial loss is obtained based on the discrimination score.

[0075] During training, the discriminator receives images generated by the student model and real images, and outputs a discrimination score. This score reflects the probability that the discriminator considers the input image to be real. The adversarial loss is calculated based on these discrimination scores. Through this adversarial training, the student model learns to generate images that are highly similar to the distribution of real data.

[0076] The formula for calculating adversarial losses is as follows:

[0077] in, Let represent the adversarial example image, s represent the interpolation coefficients, and Disc represent the discriminator. By mixing the target or real sample image with intermediate features, the aim is to combine the visual content of the image with the deep semantic or structural information extracted by the teacher model during multi-step diffusion. In this way, adversarial examples with richer information and more complex patterns can be generated, thereby increasing the difficulty and effectiveness of the discriminator's learning.

[0078] In this example implementation, by introducing a GAN-based distribution distillation method and trajectory distillation for joint optimization, and using the discriminator design constructed from the features of the teacher's multi-step model, the student model can better recover high-frequency textures and details under single-step sampling conditions, suppressing noise and jagged edges, thereby maintaining image quality and detail fidelity close to that of a multi-step model while accelerating the process.

[0079] In step S140, the overall loss is obtained based on the consistency matching loss and the adversarial loss, and the model parameters in the student model are trained based on the overall loss. The single-step diffusion model is obtained based on the trained student model.

[0080] In this example implementation, the overall loss is the total loss function used to train the student model during the distillation process, which can be a weighted combination of consistency matching loss and adversarial loss. By optimizing this overall loss, the student model can simultaneously learn trajectory consistency and distribution fidelity. A single-step diffusion model refers to a student model trained through distillation that can complete image denoising or generation tasks within a single inference step, improving processing efficiency.

[0081] In this example implementation, the overall loss can be obtained based on the consistency matching loss and the adversarial loss. The formula for calculating the overall loss can be, for example:

[0082] in, It is a weighting coefficient related to the resistance loss.

[0083] In this example implementation, distribution distillation and trajectory distillation are optimized in a unified training process using a joint loss. Trajectory distillation aligns the student model's overall prediction trend along the diffusion trajectory, while distribution distillation compensates for the lack of high-frequency texture modeling capabilities in the single-step model at the distribution level. The combination of these two methods not only ensures that the single-step model maintains the structural fidelity of the teacher model but also significantly improves its performance in detail recovery, noise suppression, and edge continuity, enabling the single-step model to achieve visual quality close to that of a multi-step model while being significantly accelerated.

[0084] In this example implementation, a joint optimization acceleration method combining trajectory distillation and distribution distillation is proposed. This method explicitly reproduces and compresses the multi-step denoising trajectory of the diffusion model during the training phase, while simultaneously approximating the generative characteristics of the teacher model at the distribution level. This significantly reduces the number of diffusion sampling steps while maintaining the model's high-quality video generation capability. Compared to other publicly available diffusion video enhancement model acceleration algorithms, the joint optimization strategy not only improves the training stability and convergence efficiency of single-step distillation but also achieves higher fidelity and detail recovery in the generated quality.

[0085] like Figure 6 The diagram shown is a schematic flowchart of a distillation method using a diffusion model according to a specific embodiment of this disclosure. It illustrates the steps described above in this example embodiment and includes the following two main collaborative modules: 1. Trajectory distillation based on a consistency model is used to compress multi-step diffusion sampling into single-step prediction to achieve a high speedup ratio.

[0086] 2. Distribution distillation based on diffusion adversarial distillation: By introducing adversarial learning and constructing a discriminator skeleton with the features of a multi-step teacher model, the image quality of a single-step student model is compensated and refined at the distribution level.

[0087] The two modules are jointly optimized during training. Trajectory distillation ensures that the student model approximates the teacher model in the overall generated trajectory, while distribution distillation is responsible for recovering high-frequency details and suppressing the noise and glitches common in single-step predictions. This significantly reduces the number of sampling steps while maintaining or approaching the visual quality of the original multi-step model.

[0088] In this example implementation, the inference efficiency of the diffusion video enhancement model is significantly improved without sacrificing the enhancement effect, enabling it to meet the high throughput and low latency video processing requirements of the production environment. This provides a feasible, efficient and stable acceleration solution for the actual deployment of the diffusion model in large-scale video enhancement services.

[0089] This example implementation also provides an image enhancement method based on a diffusion model. (See references) Figure 7 As shown, the above-mentioned image enhancement method based on the diffusion model may include the following steps: Step S710. Obtain the first video frame image to be processed and input the first video frame image into the single-step diffusion model.

[0090] The single-step diffusion model is obtained by distillation of the above diffusion model.

[0091] Step S720. Perform single-step prediction on the first video frame image using a single-step diffusion model to obtain an enhanced second video frame image, wherein the image quality of the second video frame image is higher than that of the first video frame image.

[0092] By introducing a single-step diffusion model jointly optimized and trained by trajectory distillation and distribution distillation, high-quality enhanced images are directly output during single-step prediction, avoiding the frame-by-frame iterative calculation process of traditional multi-step diffusion models. This achieves the effect of significantly improving inference efficiency while maintaining image enhancement quality. Specifically, because this single-step diffusion model constrains the logical correlation of prediction results at different time steps on the same diffusion trajectory through consistency matching loss during the training phase, and uses adversarial loss to approximate the generation distribution characteristics of the teacher model, the model can completely reproduce the inherent structure and distribution characteristics of the multi-step denoising trajectory during single-step inference. Therefore, when processing low-quality video frames, this method does not need to perform time-consuming multi-step iterations; it only requires one forward propagation to generate high-resolution, detailed enhancement results, effectively solving the key problem of balancing real-time performance and generation quality in video enhancement tasks.

[0093] It should be noted that although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0094] Furthermore, this disclosure also provides a distillation apparatus based on a diffusion model. (See reference...) Figure 8 As shown, the distillation apparatus for this diffusion model may include a sample image sampling module 810, a matching loss determination module 820, an adversarial loss determination module 830, and a model parameter training module 840. Wherein: The sample image sampling module 810 is configured to acquire a pre-trained multi-step diffusion model as a teacher model, and to obtain first and second sample images at different time steps on the same diffusion trajectory through the teacher model sampling. The matching loss determination module 820 is configured to input the first sample image and the second sample image into the student model to obtain the predicted first target sample image and the second target sample image, and to obtain the consistency matching loss based on the first target sample image and the second target sample image; The adversarial loss determination module 830 is configured to construct a discriminator based on the teacher model, and input the predicted target sample image and the real sample image into the discriminator, and obtain the adversarial loss based on the output of the discriminator; wherein the target sample image is either a first target sample image or a second target sample image; The model parameter training module 840 is configured to perform the following: obtain the overall loss based on the consistency matching loss and the adversarial loss; train the model parameters in the student model based on the overall loss; and obtain the single-step diffusion model based on the trained student model.

[0095] In some exemplary embodiments of this disclosure, the matching loss determination module 820 may include a reparameterization unit, a sample average velocity determination unit, and a target sample image determination unit. Wherein: The reparameterization unit is configured to perform the reparameterization of the student model's predicted target to the average velocity from any time step to the target sample image on the predicted diffusion trajectory. The sample average velocity determination unit is configured to input the first sample image and the second sample image into the student model to obtain the predicted first sample average velocity and the second sample average velocity. The target sample image determination unit is configured to perform the following operations: obtaining a first target sample image based on the first sample average speed and obtaining a second target sample image based on the second sample average speed.

[0096] In some exemplary embodiments of this disclosure, the matching loss determination module 820 may further include a matching loss determination unit configured to perform a consistency matching loss based on a distance parameter between the first target sample image and the second target sample image.

[0097] In some exemplary embodiments of this disclosure, the adversarial loss determination module 830 may include a discriminator construction unit configured to perform a structure for constructing a discriminator based on intermediate features of the teacher model during a multi-step diffusion process.

[0098] In some exemplary embodiments of this disclosure, the adversarial loss determination module 830 may further include an adversarial sample image determination unit and an adversarial loss determination unit. Wherein: The adversarial example image determination unit is configured to perform the operation of mixing a target sample image or a real sample image with intermediate features to obtain an adversarial example image; The adversarial loss determination unit is configured to input the adversarial sample image and interpolation coefficients into the discriminator to obtain the corresponding discrimination score, and to obtain the adversarial loss based on the discrimination score.

[0099] Furthermore, this disclosure also provides an image enhancement apparatus based on a diffusion model. (Reference) Figure 9 As shown, the image enhancement device based on the diffusion model may include an image input module 910 and an image prediction module 920. Wherein: The image input module 910 is configured to acquire a first video frame image to be processed and input the first video frame image into a single-step diffusion model, which is obtained by the distillation method of the above-mentioned diffusion model. The image prediction module 920 is configured to perform a single-step prediction on the first video frame image using a single-step diffusion model to obtain an enhanced second video frame image, wherein the image quality of the second video frame image is higher than that of the first video frame image.

[0100] The specific details of each module / unit in the distillation apparatus of the above diffusion model and the image enhancement apparatus based on the diffusion model have been described in detail in the corresponding method embodiment section, and will not be repeated here.

[0101] Figure 10 A schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present disclosure is shown.

[0102] It should be noted that, Figure 10 The computer system 1000 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0103] like Figure 10As shown, the computer system 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 1002 or programs loaded from storage section 1008 into random access memory (RAM) 1003. The RAM 1003 also stores various programs and data required for system operation. The CPU 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0104] The following components are connected to I / O interface 1005: an input section 1006 including a keyboard, mouse, etc.; an output section 1007 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to I / O interface 1005 as needed. A removable medium 1011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 1010 as needed so that computer programs read from it can be installed into storage section 1008 as needed.

[0105] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1009, and / or installed from removable medium 1011. When the computer program is executed by central processing unit (CPU) 1001, it performs various functions defined in the system of this disclosure.

[0106] Exemplary embodiments of this disclosure also provide a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the distillation method of the diffusion model described above.

[0107] In one implementation, the computer program product can be a tangible product containing a computer program, such as a computer-readable storage medium storing the computer program. The readable storage medium can be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, or other signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory, hard disk drive (HDD), solid-state drive (SSD), etc. For example, the computer program product can be implemented as a non-volatile storage medium storing a computer program, such as read-only memory, NAND flash memory, etc.

[0108] In one implementation, the computer program product can be an intangible product containing a computer program. For example, the computer program product can be implemented as a virtual digital product, such as an executable file, installation package, or other digital file storing the computer program.

[0109] Computer program code can be written in one or more programming languages. Examples of programming languages ​​include C, Java, and C++. Program code can execute entirely on the user's computing device, partially on the user's computing device, or as a standalone software package. It can also execute partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, such as a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via an internet connection provided by a mobile network operator).

[0110] Computer programs can be carried or transmitted via signals such as electricity, magnetism, light, electromagnetic fields, and infrared radiation. Electronic devices can convert signals carrying computer programs into digital signals, thereby running the computer programs. When a computer program runs on an electronic device, its code is used to cause the electronic device to execute (more specifically, to execute by the processor of the electronic device) the method steps of various exemplary embodiments of this disclosure, such as the distillation method of the diffusion model described above.

[0111] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0112] It should be noted that although several modules for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided and embodied by multiple modules.

[0113] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein.

[0114] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A distillation method using a diffusion model, characterized in that, include: A pre-trained multi-step diffusion model is obtained as a teacher model, and first and second sample images at different time steps on the same diffusion trajectory are obtained by sampling through the teacher model. The first sample image and the second sample image are input into the student model to obtain the predicted first target sample image and the second target sample image, and the consistency matching loss is obtained based on the first target sample image and the second target sample image. A discriminator is constructed based on the teacher model, and the predicted target sample image and the real sample image are input into the discriminator. The adversarial loss is obtained based on the output of the discriminator. The target sample image is either the first target sample image or the second target sample image. The overall loss is obtained based on the consistency matching loss and the adversarial loss, and the model parameters in the student model are trained based on the overall loss. A single-step diffusion model is obtained based on the trained student model.

2. The distillation method of the diffusion model according to claim 1, characterized in that, The step of inputting the first sample image and the second sample image into the student model to obtain the predicted first target sample image and the second target sample image includes: The prediction target of the student model is reparameterized to predict the average velocity from any time step to the target sample image on the diffusion trajectory; Input the first sample image and the second sample image into the student model to obtain the predicted average velocity of the first sample and the average velocity of the second sample. The first target sample image is obtained based on the average velocity of the first sample, and the second target sample image is obtained based on the average velocity of the second sample.

3. The distillation method of the diffusion model according to claim 1, characterized in that, The step of obtaining the consistency matching loss based on the first target sample image and the second target sample image includes: The consistency matching loss is obtained based on the distance parameter between the first target sample image and the second target sample image.

4. The distillation method of the diffusion model according to claim 1, characterized in that, The discriminator constructed based on the teacher model includes: The discriminator is constructed based on the intermediate features of the teacher model during the multi-step diffusion process.

5. The distillation method of the diffusion model according to claim 4, characterized in that, The process of obtaining the adversarial loss based on the output of the discriminator includes: The target sample image or the real sample image is mixed with the intermediate features to obtain an adversarial sample image; The adversarial sample image and interpolation coefficients are input into the discriminator to obtain the corresponding discrimination score, and the adversarial loss is obtained based on the discrimination score.

6. An image enhancement method based on a diffusion model, characterized in that, include: A first video frame image to be processed is acquired, and the first video frame image is input into a single-step diffusion model, wherein the single-step diffusion model is obtained by the distillation method of the diffusion model as described in any one of claims 1 to 5; The first video frame image is predicted step-by-step using the single-step diffusion model to obtain an enhanced second video frame image, wherein the image quality of the second video frame image is higher than that of the first video frame image.

7. A distillation apparatus based on a diffusion model, characterized in that, include: The sample image sampling module is configured to acquire a pre-trained multi-step diffusion model as a teacher model, and to sample first and second sample images at different time steps on the same diffusion trajectory through the teacher model. The matching loss determination module is configured to input the first sample image and the second sample image into the student model to obtain the predicted first target sample image and the second target sample image, and to obtain the consistency matching loss based on the first target sample image and the second target sample image; The adversarial loss determination module is configured to construct a discriminator based on the teacher model, input the predicted target sample image and the real sample image into the discriminator, and obtain the adversarial loss based on the output of the discriminator; wherein the target sample image is either the first target sample image or the second target sample image; The model parameter training module is configured to perform an overall loss based on the consistency matching loss and the adversarial loss, train the model parameters in the student model based on the overall loss, and obtain a single-step diffusion model based on the trained student model.

8. An image enhancement device based on a diffusion model, characterized in that, include: The image input module is configured to acquire a first video frame image to be processed and input the first video frame image into a single-step diffusion model, wherein the single-step diffusion model is obtained by the distillation method of the diffusion model as described in any one of claims 1 to 5; The image prediction module is configured to perform a single-step prediction on the first video frame image using the single-step diffusion model to obtain an enhanced second video frame image, wherein the image quality of the second video frame image is higher than that of the first video frame image.

9. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the distillation method of the diffusion model as described in any one of claims 1 to 5 or the image enhancement method based on the diffusion model as described in claim 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the distillation method of the diffusion model as described in any one of claims 1 to 5 or the image enhancement method based on the diffusion model as described in claim 6.

Images