Method, device and equipment for training image generation model, medium and product

Abstract

The embodiment of the invention relates to a method and device for training an image generation model, equipment, a medium and a product. The method comprises: based on a randomly sampled first time step and a first noise-added image corresponding to the first time step, performing distillation training on a second model by using a trained first model in a first time step interval, the first time step interval being an interval formed by the first time step and a final time step. The method further includes determining whether a number of distillation trainings performed for the first time step interval satisfies a predetermined condition. The method further comprises the steps that in response to the fact that the number of times meets the preset condition, based on the second time step and a second noise adding image corresponding to the second time step, distillation training is carried out on the second model through the trained first model in a second time step interval, and the second time step interval is an interval formed by the second time step and the first time step. Through the method, double-interval continuous distillation training from the noise end to the target data end can be realized, the model generation consistency is ensured, the training stability and the generation quality are improved, the reasoning delay is remarkably reduced, the overall convergence efficiency of the model is improved, and the user experience is improved.

Description

Technical Field

[0001] The embodiments of this disclosure generally relate to the field of model training, and more specifically to methods, apparatuses, devices, media, and products for training image generation models. Background Technology

[0002] With the continuous development of deep learning and large-scale models, diffusion-generative models have become an important direction in the field of image and video generation. These models achieve high-quality visual content generation through a gradual noise removal process, and have made significant progress in multiple areas such as image synthesis, video generation, and 3D scene reconstruction. With the optimization of model structure and the improvement of computing power, diffusion-generative models have continued to improve in terms of resolution, generation speed, and visual consistency, driving the overall development of generative content production.

[0003] In recent years, image and video generation technologies based on diffusion generation models have been widely applied in practical industrial scenarios. These technologies have been extensively deployed in downstream fields such as camera movement, material generation, creative design, film and television production, and digital marketing, providing high-quality, highly controllable automated solutions for visual content creation. Meanwhile, ongoing research combining multimodal input and generation control strategies has further promoted the diversified application of diffusion generation technology in many fields. Summary of the Invention

[0004] Embodiments of this disclosure provide a method, apparatus, device, medium, and product for training an image generation model.

[0005] According to a first aspect of this disclosure, a method for training an image generation model is provided. The method includes performing distillation training on a second model using a trained first model within a first time step interval, based on a randomly sampled first time step and a first noisy image corresponding to the first time step, wherein the first time step interval is the interval formed by the first time step and a final time step. The method further includes determining whether the number of distillation training operations performed on the second model satisfies the predetermined condition. The method further includes, in response to the number of operations satisfying the predetermined condition, performing distillation training on the second model using the trained first model within a second time step interval, based on a second time step and a second noisy image corresponding to the second time step, wherein the second time step interval is the interval formed by the second time step and the first time step.

[0006] According to a second aspect of this disclosure, an apparatus for training an image generation model is provided. The apparatus includes a first time-step interval distillation training module configured to perform distillation training on a second model within a first time-step interval using a trained first model, based on a randomly sampled first time-step and a first noisy image corresponding to the first time-step, wherein the first time-step interval is the interval formed by the first time-step and the final time-step; a training count determination module configured to determine whether the number of distillation training operations performed on the second model meets a predetermined condition; and a second time-step interval distillation training module configured to, in response to the number of training operations meeting the predetermined condition, perform distillation training on the second model within a second time-step interval using the trained first model, based on a second time-step and a second noisy image corresponding to the second time-step, wherein the second time-step interval is the interval formed by the second time-step and the first time-step.

[0007] In a third aspect of this disclosure, an electronic device is provided, including at least one processor; and a storage device for storing at least one program, which, when executed by the at least one processor, causes the at least one processor to implement the method according to the first aspect of this disclosure.

[0008] In a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the method according to a first aspect of this disclosure.

[0009] In a fifth aspect of this disclosure, a computer program product is provided. This computer program product includes a computer program that, when executed by a processor, implements the method according to a first aspect of this disclosure.

[0010] It should be understood that the content described in this section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The above and other objects, features and advantages of this disclosure will become more apparent from the accompanying drawings, in which like reference numerals generally denote like parts.

[0012] Figure 1 The illustration shows a schematic diagram of an example environment in which some embodiments of the present disclosure may be implemented;

[0013] Figure 2 The illustration shows a schematic diagram of an example method for training an image generation model according to some embodiments of the present disclosure;

[0014] Figure 3 The illustration shows a flowchart of an example process for training a second model in a first time step interval according to some embodiments of the present disclosure;

[0015] Figure 4 The illustration shows a flowchart of an example process for training a second model in a second time step interval according to some embodiments of the present disclosure;

[0016] Figure 5 The illustration shows a flowchart of an example flow of an image reasoning process according to some embodiments of the present disclosure;

[0017] Figure 6 The illustration shows a schematic block diagram of an apparatus for training an image generation model according to some embodiments of the present disclosure;

[0018] Figure 7 A schematic block diagram of an example device suitable for implementing various embodiments of the present disclosure is illustrated. Detailed Implementation

[0019] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0020] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0021] For example, upon receiving a user's proactive request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0022] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0023] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0024] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0025] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0026] In current image and video generation tasks, mainstream diffusion-based methods typically rely on multi-step iterative denoising to achieve progressive sampling. One bottleneck of these diffusion models lies in their poor computational performance and large inference latency due to the iterative denoising nature of the process. This limits deployment scenarios where computational resources are scarce and timeliness is crucial. Generally, a model trained with flow matching can achieve 50-100 steps of denoising and inference. According to current advanced consistency models, the original 50-step model can be compressed to 2-4 inference time steps. However, single-step model generation still suffers from poor generation quality and susceptibility to local distortions, significantly limiting the model's application. Furthermore, for consistency models, the algorithm requires discretizing a very close point t_(n+1) at time step t_n. Determining the scale (how small) of this very close point is controversial and leads to discretization errors.

[0027] To address this, embodiments of this disclosure propose a method for training an image generation model. In this method, a computing device utilizes a randomly sampled first time step and a first noisy image corresponding to the first time step to perform distillation training on a second model using a trained first model within a first time step interval, where the first time step interval is the interval formed by the first time step and the final time step. The computing device then determines whether the number of distillation training iterations performed on the second model meets a predetermined condition. Then, in response to the predetermined condition, the computing device further utilizes a second time step and a second noisy image corresponding to the second time step to perform distillation training on the second model using the trained first model within a second time step interval, where the second time step interval is the interval formed by the second time step and the first time step. This method improves the learning efficiency and generation stability of the model across different time intervals, achieves continuous optimization from the noise source to the target data source, simplifies the distillation training scheduling process, significantly improves the quality and consistency of the generated results, and enhances the user experience.

[0028] The embodiments of this disclosure will now be described in further detail with reference to the accompanying drawings. Figure 1 The illustration shows an example environment in which the devices and / or methods of embodiments of the present disclosure may be implemented. In environment 100, computing device 102 may generate a corresponding target image or video based on random noise and text prompts.

[0029] Examples of computing device 102 include, but are not limited to, personal computers, server computers, handheld or laptop devices, mobile devices (such as mobile phones, personal digital assistants (PDAs), media players, etc.), multiprocessor systems, consumer electronics, minicomputers, mainframe computers, and distributed computing environments that include any of the above systems or devices.

[0030] like Figure 1As shown, computing device 102 is used to perform image or video generation tasks based on a diffusion model and to complete distillation learning between the teacher model and the student model under time-step driven conditions. Each time step is a stage division in the model's gradual noise removal process during generation, representing a state point in the generation process. The first time step 104 and the first noisy image 106 together constitute the training input, simulating the initial state of the image under noise interference. Computing device 102 uses the trained first model 108 (i.e., the teacher model) to infer the input and obtain a "velocity field" prediction corresponding to the current time step. The velocity field can be understood as the direction and rate of change of the model's movement in the image space, used to guide the second model (student model) in learning the dynamic features of the generation path. Subsequently, the distillation training module 110 passes the prediction results of the first model to the second model 112, enabling the latter to learn the same generation rules as the teacher model in fewer time steps, thereby establishing a continuous consistency constraint across time steps.

[0031] After completing the distillation of the first time step interval, the computing device 102 determines at box 114 whether the current training iteration count meets a predetermined condition, such as whether the number of distillation training loops has reached a predetermined number or threshold. This step is equivalent to a dynamic scheduling mechanism for the model training progress, used to balance training stability and computational efficiency. When the condition is not met, the computing device continues to perform distillation within the time step interval formed by the randomly sampled time step and the final time step to enhance the model's fitting ability to the target end (i.e., the region near the real data); when the condition is met, the training logic automatically switches to the second time step interval formed by the smaller time step and the first time step, realizing training near the noise end. Through this condition control mechanism, the computing device can train alternately in different intervals, thereby ensuring a more balanced learning distribution of the model throughout the diffusion path and avoiding overfitting to a single interval.

[0032] Upon switching to the second time step interval, the computing device 102 reconstructs the input sample based on the second time step 116 and the second noisy image 118, and uses the first model 108 to guide the second model 112 in distillation. The core of this stage lies in "boundary consistency constraints," that is, ensuring the stability of the generated path under high noise conditions. At this time, the first model determines a target result by predicting the velocity and combining the gradient of the second model's measurement results with respect to the time step, and optimizes the second model by combining the second model's prediction results for the input sample. Through continuous optimization, the second model can not only maintain high-fidelity generation in the low-noise stage, but also has strong reversibility and stability in the noisy stage, significantly improving the overall quality of the generated result.

[0033] Within the entire dual-interval distillation framework, the alternating training of the first and second time-step intervals constitutes a complete optimization loop. Through this "dual-end optimization" strategy, the computing device 102 continuously compresses the multi-step diffusion process of the teacher model in the student model, thereby enabling the generation of high-quality images or videos through single-step or few-step inference.

[0034] This method reduces inference latency and memory overhead, improves the robustness and generalization performance of the model in multi-scenario generation tasks, and can meet the requirements of high-efficiency and high-fidelity generation.

[0035] The above combination Figure 1 The following is a schematic diagram illustrating an example environment in which some embodiments of this disclosure may be implemented, in conjunction with... Figure 2 A schematic diagram illustrating an example method for training an image generation model according to some embodiments of the present disclosure. Figure 2 The method in can be derived from Figure 1 The computing device 102 or any suitable device in the system can be used for execution.

[0036] like Figure 2 As shown, in example method 200, at box 202, computing device 102 performs distillation training on a second model using a trained first model within a first time step interval, based on a randomly sampled first time step and a first noisy image corresponding to the first time step. The first time step interval is the interval formed by the first time step and the final time step. Here, a time step represents the discrete steps into which the continuous change is decomposed during the forward noise addition and reverse denoising process, with each step corresponding to a specific change state in noise intensity or sample sharpness. The first time step interval, formed by the first time step and the final time step, describes the continuous interval from the intermediate state to the target image state. By performing distillation training within this interval, the computing device enables the second model to learn the generation rules of the first model in a multi-step diffusion process with a small number of steps.

[0037] The computing device 102 first determines the image and corresponding noise samples for training. The computing device generates samples with different noise intensities by applying Gaussian noise to the original image to simulate the noise addition process of the diffusion model. Therefore, the computing device can generate a first noisy image, i.e., the noisy image state at a specified time step, based on the image, noise, and a first time step. This image serves as an input sample, which is then processed by the generative model to generate a prediction result.

[0038] During the distillation training process, the computing device 102 performs paired learning using a first model (teacher model) and a second model (student model). First, based on the first model, a first noisy image, and a first time step, a first velocity is determined for the first model; this velocity represents the speed at which the image evolves from noise to a clear image along the diffusion path. Then, based on the first time step and the first noisy image, the computing device uses the second model to generate a first prediction result. Next, by calculating the gradient of the first prediction result and combining it with the first velocity, the computing device determines a first target result, reflecting the ideal or target result generated by the teacher model at that time step. Finally, based on the deviation between the first prediction result and the first target result, the computing device adjusts the parameters of the second model to ensure that its generation path aligns with the continuous velocity field of the teacher model.

[0039] In the above process, the computing device 102 can also calculate the difference between the first prediction result and the first target result to obtain the first distillation loss, and use it as the optimization objective. This loss is used to measure the gap between the student model and the teacher model in terms of generation consistency within the current time step interval. The computing device uses this loss to update the parameters of the second model, so that while reducing the number of computation steps, it can still maintain a generation effect comparable to the multi-step diffusion model. Through repeated iterations of the above process, the student model gradually acquires the ability to generate high-quality images in a single step.

[0040] In some embodiments, the first model can be an image generation model trained through multi-step diffusion, such as a flow matching or diffusion transformer (DiT) model, capable of predicting velocity fields or noisy mappings at multiple time steps. The second model is a target consistency model, designed to compress the multi-step diffusion process of the first model through distillation learning, thereby generating high-fidelity images in a single step or very few steps during the inference phase. Through this training process, the computing device not only achieves a significant acceleration of model inference but also constructs stable consistency constraints in the continuous time domain, making the model smoother, more controllable, and physically meaningful during the generation process.

[0041] At box 204, computing device 102 determines whether the number of distillation training iterations performed on the second model meets predetermined conditions. This step is a dynamic control step in the entire training process, used to determine whether training needs to continue in the current time interval or switch to a new time step interval. The computing device dynamically adjusts the number of training iterations in different time frequency intervals by monitoring the number of distillation training iterations.

[0042] The predetermined conditions may include whether the number of distillation training iterations is an integer multiple of a threshold number. After each predetermined number of training iterations, the computing device triggers a transition to another time step interval for training. This mechanism effectively prevents overfitting caused by overtraining, while ensuring the rhythm and stability of the model's distillation learning across different time intervals. If the computing device detects that the current number of training iterations has not reached the threshold, it continues to perform distillation based on randomly sampled time steps to further enhance the consistency of the second model's generation within that interval. If the condition is not met, further refinement and optimization continue between the randomly sampled time steps and the final time step. This adaptive judgment mechanism makes the entire training process more efficient and stable, ensuring that the second model can fully inherit the generation capabilities of the first model in the multi-step diffusion process, while maintaining high-fidelity image generation performance while reducing the number of iterations.

[0043] At box 206, in response to the predetermined condition being met, the computing device 102 performs distillation training on the second model within the second time step interval using the trained first model, based on the second time step and the second noisy image corresponding to the second time step. The second time step interval is the interval formed by the second time step and the first time step. By performing distillation training within this interval, the computing device can establish constraints in the generation boundary region of the model, enabling the second model to remain convergent and stable even under high noise input, thereby improving the overall generation consistency.

[0044] In some embodiments, when the number of iterations for training the second model meets a predetermined condition, the computing device 102 can obtain a new third time step through random sampling to dynamically adjust the training step size. The third time step is typically less than a preset threshold time step, for example, less than 0.5. Then, based on the sampled third time step, the computing device can calculate a second time step according to a predetermined rule and generate a corresponding second noisy image based on the second time step and the original image.

[0045] In some embodiments, the computing device 102 calculates a second velocity for the first model at this stage based on the first model, the second noisy image, and the second time step. This "velocity" is the rate of change of the image from a noisy state to a target state during diffusion. Subsequently, the computing device generates a second prediction result using the second model (i.e., the student model) based on the second time step and the second noisy image. Next, the computing device 102 can further determine the gradient of the second prediction result, and then further utilize the second velocity and the gradient of the second prediction result to determine a second target result, which serves as the ideal reference output of the teacher model at this time step. The core of this step is introducing gradient information to construct the target signal. Subsequently, the computing device calculates a second distillation loss based on the deviation between the second prediction result and the second target result. This loss reflects the degree of consistency between the student model and the teacher model's velocity field prediction within the current time step interval.

[0046] In some embodiments, the computing device 102 uses the calculated second distillation loss to backpropagate and update the parameters of the second model to reduce prediction bias. Unlike the training in the first time step interval, the second time step interval is mainly used to strengthen the boundary consistency of the model at the noise level, so that it can still maintain the correct generation direction and gradient smoothness when there is high noise input. Through this dual-interval alternating optimization, the student model can not only maintain the stability of the generated details and structure at the target level, but also have strong reversibility and generalization ability at the noise level.

[0047] The computing device 102 can also perform an inference phase after completing the dual-interval distillation. The computing device acquires a text prompt and uses the second model to perform single-step inference on random noise based on the prompt, thereby generating the target image or video. Through distillation training, the second model has learned the velocity field features at any time step, so only one forward computation is needed during inference to obtain a high-quality generated result.

[0048] This method not only significantly reduces inference costs but also ensures the clarity, stability, and physical consistency of the generated content, thus improving the user experience.

[0049] The above combination Figure 2 A schematic diagram illustrating example methods for training image generation models according to some embodiments of this disclosure is provided. The implementation scheme is further described below. In mathematical modeling, to better achieve continuous form modeling, the computing device defines the consistency function as the following formula (1): (1)

[0050] in This represents the velocity field predicted by the neural network, reflected in the displacement. The displacement point that can be reached after a time interval (t - r) from the starting point. The velocity field is composed of... The prediction, because we want to model a single-step model, means that the velocity field here is actually the average velocity field of the path, and the ultimate goal is to optimize the neural network representing this average velocity field. It enables data to be retrieved from any point in time. This generates the final image. To simplify the training of this velocity field model for a single step, the average velocity from the original timestamp 0 to timestamp 1 can be divided into two segments, namely (0, ...). t ) section and ( t , 1) segment.

[0051] exist( t When optimizing segment 1), segment (t, 1) can be substituted into the consistency function to obtain... , for time step Find the total derivative and the uniformity function. The total derivative is 0 (because the uniform function is identical and independent of t, which is reflected in the total derivative being 0).

[0052] The training objectives for this section are summarized as follows: ,in It is a multi-step diffusion generative teacher model in time Point velocity field prediction or construction from data ), It is a neural network pair The total derivative can be obtained using the JVP operation operator. The final trained... It's the one on the right side of the equation. That is, the final optimization loss is .

[0053] For the optimization of the (0, t) segment, since the input is closer to noise the closer to time 0, we can first select a relatively small time segment. This allows them to reach the boundary. That is, writing a consistent function Here, time As the research object, make it meet the following requirements. The boundary on top. Similarly, it can be made to be on... The total derivative is 0, therefore... Similarly, update the left side of the equation using the right side. That is, update the loss: For the losses obtained in the two segments mentioned above, at an update frequency ( Alternately update the two losses, such as updating segment (t,1). After this, the (r, t) segment is updated once. Then, the decay for r can also be determined.

[0054] For the segment (0, t) The decay process can be defined as a linear decrease to time 0 with each training epoch. Specifically, a small time interval is fixed initially. , without going through Update after one epoch. .

[0055] The following is combined with Figure 3 A flowchart describing an example process for training a second model in a first time step interval according to some embodiments of the present disclosure. Figure 3 Example method 300 in the example can be derived from Figure 1 The computing device 102 or any suitable device in the system can be used for processing.

[0056] like Figure 3 As shown in Example Method 300, the structure of the computational device performing distillation training on the image generation model during the optimization phase from t to 1 is illustrated. This process optimizes the average velocity field of the single-step generation model through continuous-time modeling, enabling the model to return to the target data source, i.e., generate the final image, at any time step.

[0057] In the initial training phase, the computing device first receives a randomly determined first time step 304 and the corresponding first noisy image 302, and inputs them as samples into the first model 306 (i.e., the teacher model). The first model predicts a velocity output at this time step, describing the direction and rate of the image's evolution from its current state towards the real data endpoint; this output is denoted as the first velocity 308. This velocity field is obtained through the diffusion inference process of the teacher model and can be considered as an instantaneous motion vector extracted from the multi-step diffusion model, used to guide the learning of the single-step consistency model.

[0058] Simultaneously, the second model 310 (student model) generates a first prediction result 312 at the same time step and input to simulate its own generation behavior at that moment. The computing device then determines a first target result 314 based on the gradient information between the first velocity 308 and the first prediction result 312. Here, the gradient reflects the trend of the model output with respect to time step t, used to capture the evolution direction of the model in a continuous temporal space. The parameter adjustment module 316 then performs backpropagation updates based on the difference between the first prediction result and the first target result, thereby optimizing the parameters of the second model so that it gradually approximates the generation path of the teacher model.

[0059] For example, in the training initialization phase, the student model is initialized. Teacher Model Data loading module, etc., maximum iteration loop is The computing device sets the current iteration loop to i=1 and selects the current sample data for training from the data loading module. These sample images, representing real data, will provide alignment targets for the model throughout the training process, guiding the student model to gradually learn generation paths consistent with the real distribution.

[0060] During the optimization phase at (t, 1), the computing device obtains the current time step t (usually located in the high time step region close to 1) and random noise through random sampling. And use the following formula (2) for linear interpolation to construct the current input. (2)

[0061] This step simulates the noise mixing state of the diffusion generation model at intermediate time points, enabling training to cover multiple noise distributions in continuous time and space. Subsequently, the computing device will... Input the time step t into the teacher model The teacher model predicts the velocity field at the current moment, and the velocity field is shown in Equation 3 below: (3)

[0062] The velocity field It describes the dynamic change direction and rate of the image during the diffusion process, reflects the generation trajectory of the teacher model in continuous space, and serves as an important reference target for student model learning.

[0063] Next, the student model Predictions are made for the same input to simulate the student model's generation performance at that time step. To obtain the dynamic response of the student model in the time dimension, the computing device calculates the model's output value and the total derivative with respect to the time step using the JVP (Jacobian-VectorProduct) operator. The formula for calculating the total derivative is shown in formula (4) below: (4)

[0064] in This represents a unit perturbation in the time dimension. This process can efficiently calculate the rate of change of the model with respect to time, helping students establish a stable velocity prediction capability for their models over continuous time. This represents the gradient value.

[0065] After obtaining the above results, the computing device calculates the distillation loss function for segment (t, 1) according to formula (5): (5)

[0066] This objective combines the instantaneous velocity prediction of the teacher model with the time response of the student model, thus forming a continuous optimization constraint describing the average velocity field, enabling the student model to approximate the actual generated trajectory of the teacher model in the later time series.

[0067] Finally, the computing device backpropagates the loss function and updates the student model parameters using gradients. After multiple rounds of iterative training, the student model can gradually learn the velocity distribution characteristics of the teacher model in the interval (t, 1), achieving dynamic consistency modeling of the generated path in high-time-step regions.

[0068] This method effectively eliminates the problem of inter-step error accumulation caused by traditional discrete training, enabling the model to converge stably in continuous time and space, and achieving high-quality image generation through single-step inference.

[0069] The above combination Figure 3 A flowchart describing an example process for training a second model in a first time step interval according to some embodiments of this disclosure is described below; in conjunction with Figure 4 A flowchart describing an example process for training a second model within a second time step interval according to some embodiments of the present disclosure. Figure 4 Example process 400 can be generated by Figure 1 The computing device 102 shown or any suitable device may be used for execution.

[0070] The second model can be optimized during the second time step region (0, t). In this stage, the computing device performs a structured process of continuous consistency distillation training on the image generation model. This stage complements the (t, 1) segment and is mainly used to optimize the generation consistency of the model in the low time step region (close to the completely noisy state), thereby enabling the model to maintain a smooth dynamic convergence trajectory throughout the time interval 0 to 1.

[0071] In the initial training phase, the computing device first obtains input sample images and noise from the dataset, and randomly samples a small time step s (s<0.5) according to a random sampling strategy in the (0, t) interval, and determines... , where i is the iteration number. Since the input approaches pure noise as the time step r gets closer to 0, it is necessary to start from a smaller time interval r to ensure that it reaches the boundary. At this point, the computing device constructs the following formula (6): (6)

[0072] And r is provided as input to the teacher model The teacher model derives the velocity prediction formula (7) for the current moment through reasoning: (7)

[0073] This velocity field reflects directional information during the transition from a purely noisy state to the target data space, and is used to guide the student model's generative learning under extreme noise conditions.

[0074] Next, the student model Receive the same input The time step is r, and the target time t is used as the upper bound to predict the velocity field response in the interval (r, t). In order to capture the continuous change of the student model in the time dimension, the computing device calculates the total derivative of the output value with time using the JVP (Jacobian-Vector Product) operator, as shown in the following formula (8): (8)

[0075] in This represents a unit perturbation in the time direction. This process describes the local response characteristics of the model at time r and provides gradient constraints for subsequent loss calculations.

[0076] Finally, the distillation loss in the (r, t) segment is calculated as shown in the following formula (9): (9)

[0077] The computing device then performs backpropagation, updating the model parameters θ with this loss as the optimization objective, thereby optimizing the student model's mean velocity field prediction in the low time step region.

[0078] like Figure 4 As shown, in example flow 400, the first model 408 and the second model 412 operate collaboratively during training. By processing the second noisy image 404 and the second time step 406, the first model (teacher model) outputs a second velocity 410, providing an accurate dynamic reference signal; the second model outputs a second prediction result 414, which, by comparing it with the second target result 416, enters the parameter adjustment module 418 to perform gradient updates. This architecture ensures that the knowledge of the teacher model on the multi-step diffusion trajectory can be effectively distilled into the student model through consistency constraints.

[0079] In terms of training scheduling, the computing device adopts an alternating update strategy, that is, under a set update frequency freq, it first performs (t, 1) optimization freq times, and then performs (r, t) optimization 1 time, in order to balance the learning stability of the model in high noise and low noise regions.

[0080] Ultimately, the computing device achieves continuous and consistent modeling throughout the [0, 1] time space by alternating updates within the two intervals (r, t) and (t, 1). During this process, the computing device performs an optimization step of (r, t), which includes randomly sampling time steps s (s<0.5) and calculating the value of r, constructing the input, predicting the velocity, calculating the derivative, calculating the loss, and backpropagation. The training process continues until the model parameters θ converge or the termination condition is met.

[0081] In this way, the model not only establishes stable denoising direction perception at the high-noise end, but also achieves continuous smooth convergence in the low time step region, thus possessing global consistency and high-fidelity output capability across the entire generated trajectory.

[0082] like Figure 5 As shown, Example 500 illustrates a single-step generation process during the inference phase. The computing device first receives an input text prompt in box 502, which can consist of a natural language description input by the user, defining the semantic content of the image or video to be generated. Subsequently, in box 504, the system randomly samples initial latent variables in the latent space, using them as the starting point for generation. These latent variables are essentially high-dimensional noise vectors representing the initial state of the model in the generation space, driving the subsequent generation process.

[0083] In box 506, the computing device invokes a trained single-step generative model to perform inference operations. The model receives initial latent variables and text prompts as input and outputs the corresponding latent space update direction. Mathematically, this can be represented by the following formula 10: (10) in The prediction function of the model. Let P be the initial noise latent variable and P be the input text prompt. Essentially, this step predicts the displacement from the noise space to the semantic space through a forward propagation, i.e., the average velocity field from the noise to the generated result.

[0084] Next, in box 508, the computing device updates the latent variables and calculates and generates latent variable samples, as shown in Formula 11 below: (11)

[0085] This update adjusts the latent variables semantically, gradually bringing them closer to the latent space representation of the target image or video. Unlike traditional diffusion models that require multi-step denoising, this scheme achieves the transition from noise to the latent image representation through a single-step update, significantly reducing the computational cost of inference.

[0086] Finally, in box 510, the computing device will update the latent variables. The input is fed into the variational autoencoder (VAE) decoder for reconstruction, and the final visual output is obtained by executing the following formula (12). The output can be a static image or a dynamic video, depending on the generation type of the model. At this stage, the VAE decoder is responsible for restoring the latent space features to the pixel space, realizing the mapping from abstract semantics to visible content. (12)

[0087] This method can output high-quality results with only one forward inference, which greatly improves the generation efficiency and ensures the excellent performance of the generated results in terms of semantic consistency and detail fidelity.

[0088] Figure 6 The illustration shows a schematic block diagram of an apparatus for training an image generation model according to some embodiments of the present disclosure. Figure 6 As shown, device 600 can Figure 1 The device 600 is implemented in computing device 102, and includes a first time step interval distillation training module 602, configured to perform distillation training on a second model using a trained first model within a first time step interval based on a randomly sampled first time step and a first noisy image corresponding to the first time step, wherein the first time step interval is the interval formed by the first time step and the final time step; a training count determination module 604, configured to determine whether the number of distillation trainings performed on the second model meets a predetermined condition; and a second time step interval distillation training module 606, configured to, in response to the number of trainings meeting the predetermined condition, perform distillation training on the second model using a trained first model within a second time step interval based on a second time step and a second noisy image corresponding to the second time step, wherein the second time step interval is the interval formed by the second time step and the first time step.

[0089] In some embodiments, the apparatus 600 further includes: an image and noise determination module configured to determine an image and noise corresponding to a first time step; and a first noisy image generation module configured to determine a first noisy image based on the image, noise, and the first time step.

[0090] In some embodiments, the first timestep interval distillation training module 602 includes: a first velocity determination module configured to determine a first velocity for the first model based on a first model, a first noisy image, and a first timestep; a first prediction result determination module configured to obtain a first prediction result using a second model based on the first timestep and the first noisy image; a first target result determination module configured to determine a first target result based on the gradient of the first velocity and the first prediction result; and a second model parameter first step adjustment module configured to adjust the parameters of the second model based on the first prediction result and the first target result.

[0091] In some embodiments, the second model parameter first step adjustment module includes: a first distillation loss calculation module, configured to calculate a first distillation loss in a first time step interval based on a first prediction result and a first target result; and a second model parameter first step determination module, configured to adjust the parameters of the second model based on the first distillation loss.

[0092] In some embodiments, the first model is an image generation model obtained through multi-step diffusion training.

[0093] In some embodiments, the apparatus 600 further includes a condition-triggered distillation training execution module, configured to continue distillation training for the second model within the time step interval formed by the random sampling time step and the final time step in response to a predetermined condition not being met.

[0094] In some embodiments, the predetermined condition includes a number of times that is an integer multiple of a threshold number.

[0095] In some embodiments, the apparatus 600 further includes: a third time step determination module configured to obtain a third time step by random sampling, wherein the third time step is less than a threshold time step; a second time step determination module configured to determine a second time step based on the third time step and a predetermined rule; and a second noisy image determination module configured to determine a second noisy image based on the image and the second time step.

[0096] In some embodiments, the second timestep interval distillation training module 606 includes: a second velocity determination module configured to determine a second velocity for the first model based on a first model, a second noisy image, and a second timestep; a second prediction result determination module configured to obtain a second prediction result using the second model based on the second timestep and the second noisy image; a second target result determination module configured to determine a second target result based on the gradient of the second velocity and the second prediction result; and a second model parameter second step adjustment module configured to adjust the parameters of the second model based on the second prediction result and the second target result.

[0097] In some embodiments, the second model parameter second step adjustment module includes: a second distillation loss determination module, configured to calculate the second distillation loss in the second time step interval based on the second prediction result and the second target result; and a second model parameter second step determination module, configured to adjust the parameters of the second model based on the second distillation loss.

[0098] In some embodiments, the apparatus 600 further includes: a text prompt acquisition module configured to acquire text prompts; and a target image or video generation module configured to generate an image or video based on the text prompts by performing single-step inference on random noise using a second model.

[0099] Figure 7 A schematic block diagram of an example device 700 that can be used to implement embodiments of the present disclosure is shown. Figure 1 The computing device 102 can be implemented using device 700. As shown, device 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes based on computer program instructions stored in read-only memory (ROM) 702 or loaded from storage unit 708 into random access memory (RAM) 703. RAM 703 can also store various programs and data required for the operation of device 700. CPU 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 707 is also connected to bus 704.

[0100] Multiple components in device 700 are connected to I / O interface 707, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0101] The various processes and handling described above, such as method 200, can be executed by processing unit 701. For example, in some embodiments, method 200 can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by CPU 701, one or more actions of the example method 200 described above can be performed.

[0102] This disclosure can be a method, apparatus, system, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0103] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0104] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0105] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0106] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0107] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0108] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0109] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0110] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for training an image generation model, comprising: performing, by a second model, distillation training in a first time step interval using a first trained model based on a randomly sampled first time step and a first noisy image corresponding to the first time step, the first time step interval being formed by the first time step and a final time step; determining whether a number of times of performing the distillation training by the second model satisfies a predetermined condition; and in response to the number of times satisfying the predetermined condition, performing, by the second model, distillation training in a second time step interval using the first trained model based on a second time step and a second noisy image corresponding to the second time step, the second time step interval being formed by the second time step and the first time step.

2. The method of claim 1, further comprising: determining an image and a noise corresponding to the first time step; and determining the first noisy image based on the image, the noise, and the first time step.

3. The method of claim 1, wherein performing, by a second model, distillation training in a first time step interval using a first trained model comprises: determining a first speed for the first model based on the first model, the first noisy image, and the first time step; obtaining a first prediction result by the second model based on the first time step and the first noisy image; determining a first target result based on the first speed and a gradient of the first prediction result; and adjusting a parameter of the second model based on the first prediction result and the first target result.

4. The method of claim 3, wherein adjusting a parameter of the second model based on the first prediction result and the first target result comprises: calculating a first distillation loss in the first time step interval based on the first prediction result and the first target result; and adjusting the parameter of the second model based on the first distillation loss.

5. The method of claim 1, wherein the first model is an image generation model trained by multi-step diffusion.

6. The method of claim 1, further comprising: in response to the number of times not satisfying the predetermined condition, continuing the distillation training for the second model in a time step interval formed by a randomly sampled time step and a final time step.

7. The method of claim 1, wherein the predetermined condition comprises the number of times being an integer multiple of a threshold number of times.

8. The method of claim 1, further comprising: obtaining a third time step smaller than a threshold time step by random sampling; determining the second time step based on the third time step and a predetermined rule; and determining the second noisy image based on the image and the second time step.

9. The method of claim 1, wherein performing, by a second model, distillation training in a second time step interval using a first trained model comprises: determining a second speed for the first model based on the first model, the second noisy image, and the second time step; ​ ​ ​ ​ obtaining a second prediction result by using the second model based on the second time step and the second noisy image; determining a second target result based on the second speed and a gradient of the second prediction result; and adjusting parameters of the second model based on the second prediction result and the second target result.

10. The method of claim 9, wherein adjusting the parameters of the second model based on the second prediction result and the second target result comprises: calculating a second distillation loss in the second time step interval based on the second prediction result and the second target result; and adjusting the parameters of the second model based on the second distillation loss.

11. The method of claim 1, further comprising: obtaining a text prompt; and performing single-step inference by using the second model on random noise to generate an image or a video based on the text prompt.

12. An apparatus for training an image generation model, comprising: a first time step interval distillation training module configured to perform distillation training on a second model by using a trained first model in a first time step interval based on a first time step randomly sampled and a first noisy image corresponding to the first time step, the first time step interval being an interval formed by the first time step and a final time step; a training number determination module configured to determine whether a number of times of distillation training performed on the second model satisfies a predetermined condition; and a second time step interval distillation training module configured to perform distillation training on the second model by using the trained first model in a second time step interval based on a second time step and a second noisy image corresponding to the second time step in response to the number satisfying the predetermined condition, the second time step interval being an interval formed by the second time step and the first time step.

13. An electronic device, comprising: at least one processor; and a memory storing at least one program, when the at least one program is executed by the at least one processor, causing the at least one processor to implement the method according to any one of claims 1-11.

14. A computer-readable storage medium having a computer program stored thereon, the computer program, when executed by a processor, implementing the method according to any one of claims 1-11.

15. A computer program product comprising a computer program, the computer program, when executed by a processor, implementing the method according to any one of claims 1-11. ​ ​ ​ ​ ​

Images