Image generation model training method and device, storage medium and program product
By using momentum modeling and progressive knowledge distillation training with curve integral links, the complex curve trajectory of the teacher model is accurately fitted, solving the problem of quality degradation in diffusion model generation and achieving efficient image generation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING ELECTRONIC DIGITAL INTELLIGENCE TECHNOLOGY CO LTD
- Filing Date
- 2026-04-07
- Publication Date
- 2026-07-03
AI Technical Summary
Existing diffusion models, in the process of image generation, use straight lines to approximate the denoising trajectory, which leads to a decrease in generation quality and makes it difficult to balance generation efficiency and quality.
By employing momentum modeling and line integral calculation links, and through progressive knowledge distillation training, the complex curve-like generation path of the teacher model is accurately fitted, and the denoising operation of the student model is optimized.
While reducing the number of denoising steps, it maintains the quality of image generation, improves generation efficiency, and preserves image details and contour integrity.
Smart Images
Figure CN122336464A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of image generation technology, and in particular to an image generation model training method, apparatus, storage medium, and program product. Background Technology
[0002] Diffusion models, as a mainstream AI image generation technology, generate clear images by starting with random noisy images and iteratively denoising them through multiple rounds. They are widely used in content creation, advertising design, and other fields. Existing efficient generation solutions often employ knowledge distillation, transferring the generation knowledge from pre-trained multi-step diffusion models to lightweight models with fewer steps, thereby improving inference speed.
[0003] Traditional distillation methods often approximate the denoising trajectory of the original model with straight lines, which cannot accurately fit complex curved generation paths and is prone to trajectory deviation. After significantly reducing the number of denoising steps, the generation quality such as image details and contour integrity is significantly reduced, making it difficult to balance generation efficiency and quality. Summary of the Invention
[0004] In view of this, the present disclosure provides an image generation model training method, apparatus, storage medium, and program product, which can accurately calculate the denoising trajectory with few steps through momentum modeling and line integral, and make the student model highly fit the teacher model trajectory with the help of progressive knowledge distillation, thereby significantly reducing the number of denoising steps while ensuring the quality of image generation, and balancing efficiency and effectiveness.
[0005] In a first aspect, embodiments of this disclosure provide an image generation model training method, employing the following technical solution: Obtain a teacher model and a student model, wherein the teacher model performs more denoising steps than the student model during image generation; Momentum modeling is performed on each denoising operation of the student model in the image generation process to construct a curved trajectory representing the image generation process, wherein the curved trajectory is composed of a weighted mixture of multiple sub-trajectories; Perform integral modeling on the curve trajectory and construct the curve integral calculation link; The teacher model and the student model are trained by progressive knowledge distillation. During each training session, the total displacement of the latent variables of the teacher model during the image generation process is obtained. The output projection layer of the student model is used to predict the dynamic parameters of each sub-trajectory. The dynamic parameters of the sub-trajectory are input into the curve integral calculation link to obtain the total latent displacement of the student model during the image generation process; Based on the total shift of the latent variables of the teacher model and the total shift of the latent variables of the student model, optimize the student model; When the progressive knowledge distillation training is complete, the trained learning model is used as the target image generation model.
[0006] Optionally, the step of performing momentum modeling on each denoising operation of the student model in the image generation process to construct a curve trajectory representing the image generation process includes: Define each denoising operation corresponding to each image feature type as a basic velocity vector; The momentum factor is set to characterize the evolution and decay of the basic velocity vector as the denoising process progresses; Based on the basic velocity vector and the momentum factor, sub-trajectories corresponding to each image feature type are constructed respectively; Set the gating probability for each sub-trajectory; Based on the gating probability, a weighted sum is performed on each sub-trajectory to obtain the curved trajectory.
[0007] Optionally, the step of performing integral modeling on the curve trajectory and constructing a curve integral calculation link includes: Based on the momentum factor of the sub-trajectory and the start and end time steps of each denoising operation in the image generation process of the student model, the momentum integral coefficient of the sub-trajectory in each denoising operation is obtained. The basic velocity vector and corresponding momentum integral coefficient of the sub-trajectory in each denoising operation are multiplied, and then weighted and summed through the gating probability of the sub-trajectory to obtain the curve integral calculation link of each denoising operation of the student model in the process of generating the image.
[0008] Optionally, the momentum integral coefficient of the sub-trajectory at each denoising operation is obtained based on the momentum factor of the sub-trajectory and the start and end time steps of each denoising operation in the image generation process of the student model, including: When the momentum factor of the sub-trajectory is 1, the momentum integral coefficient of the sub-trajectory in each denoising operation is equal to the difference between the start time step and the end time step of each denoising operation. When the momentum factor of the sub-trajectory is not 1, the exponential time evolution relationship of the momentum factor is analytically integrated based on the start time step and the end time step of each denoising operation to obtain the momentum integral coefficient of the sub-trajectory in each denoising operation.
[0009] Optionally, the dynamic parameters include the basic velocity vector, momentum factor, and gating probability; The output projection layer is a multi-branch mapping structure set between the backbone network and the output layer of the student model, used to map the high-dimensional features output by the backbone network to the basic velocity vector, momentum factor and gating probability.
[0010] Optionally, the step of inputting the dynamic parameters of the sub-trajectory into the curve integral calculation link to obtain the total latent displacement of the student model during the image generation process includes: The dynamic parameters of each sub-trajectory predicted by the output projection layer in each denoising operation are input into the corresponding curve integral calculation link to obtain the latent variable displacement of each denoising operation. The total latent variable shift of the student model during the image generation process is obtained by summing up the latent variable shifts of each denoising operation.
[0011] Optionally, optimizing the student model based on the total shift of the latent variables of the teacher model and the total shift of the latent variables of the student model includes: Obtain the mean square error between the total latent variable shift of the teacher model and the total latent variable shift of the student model; The student model is optimized with the goal of minimizing the mean square error.
[0012] Secondly, this disclosure also provides an image generation model training system, which adopts the following technical solution: The model acquisition module is used to acquire a teacher model and a student model, wherein the teacher model performs more denoising steps than the student model during the image generation process; The curve trajectory construction module is used to perform momentum modeling on each denoising operation of the student model in the image generation process, and construct a curve trajectory representing the image generation process, wherein the curve trajectory is composed of a weighted mixture of multiple sub-trajectories; The integral modeling module is used to perform integral modeling on the curve trajectory and construct the curve integral calculation link; The knowledge distillation training module is used to perform progressive knowledge distillation training on the teacher model and the student model. During each training session, the total displacement of the latent variables of the teacher model during the image generation process is obtained, and the output projection layer of the student model is used to predict the dynamic parameters of each sub-trajectory. The total displacement acquisition module is used to input the dynamic parameters of the sub-trajectory into the curve integral calculation link to obtain the total latent variable displacement of the student model during the image generation process; The student model optimization module is used to optimize the student model based on the total shift of the latent variables of the teacher model and the total shift of the latent variables of the student model. The target model determination module is used to use the trained learning model as the target image generation model when the progressive knowledge distillation training is completed.
[0013] Thirdly, this disclosure also provides a computer device, which adopts the following technical solution: The computer device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform any of the image generation model training methods described above.
[0014] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium storing computer instructions for causing a computer to execute any of the image generation model training methods described above.
[0015] Fifthly, embodiments of this disclosure also provide a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of any of the methods described above.
[0016] The image generation model training method provided in this disclosure performs momentum modeling on each denoising operation of the student model and constructs a curved trajectory using a weighted mixture of multiple sub-trajectories. This accurately depicts the dynamic evolution of latent variables during the denoising process, fully preserving the nonlinear characteristics of the original model trajectory. Simultaneously, by constructing a curve integral calculation link, the dynamic parameters predicted by the output projection layer are transformed into quantifiable latent variable displacements, enabling accurate calculation and reconstruction of complex curved trajectories. This avoids trajectory distortion caused by straight-line approximation, thereby achieving accurate fitting of the complex curved generation path of the teacher model. This scheme employs progressive knowledge distillation training, using the total latent variable displacement of the teacher model (multi-step denoising) as the supervision target. It efficiently transfers the high-precision trajectory knowledge of the teacher model to the student model (few-step denoising). Furthermore, through momentum modeling and the curve integral link, it ensures that the student model can still accurately replicate the trajectory features of the teacher model in a few-step denoising scenario. Furthermore, the student model only needs to perform a few denoising operations, which significantly reduces inference overhead and improves generation efficiency. Moreover, through accurate trajectory fitting, it effectively preserves image details and contour integrity, achieving a dual improvement in generation efficiency and generation quality.
[0017] The above description is merely an overview of the technical solution disclosed herein. In order to better understand the technical means of this disclosure and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description
[0018] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 A schematic flowchart illustrating the image generation model training method provided in this embodiment of the disclosure; Figure 2 A flowchart illustrating the curve trajectory construction method provided in this embodiment of the disclosure; Figure 3 A flowchart illustrating the curve integral calculation link construction method provided in this embodiment of the disclosure; Figure 4 This is a block diagram illustrating the principle of an image generation model training system provided in an embodiment of this disclosure. Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present disclosure. Detailed Implementation
[0020] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.
[0021] It should be understood that the following specific examples illustrate the implementation of this disclosure, and those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. This disclosure can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.
[0022] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.
[0023] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. The drawings only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.
[0024] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.
[0025] Reference Figure 1 This disclosure provides a method for training an image generation model, including the following steps: S1: Obtain the teacher model and student model, wherein the teacher model performs more denoising steps than the student model during image generation; S2: Momentum modeling is performed on each denoising operation of the student model in the process of image generation, and a curved trajectory representing the image generation process is constructed. The curved trajectory is composed of a weighted mixture of multiple sub-trajectories. S3: Perform integral modeling on the curve trajectory and construct the curve integral calculation link; S4: Perform progressive knowledge distillation training on the teacher model and the student model. During each training session, obtain the total displacement of the latent variables of the teacher model in the image generation process, and use the output projection layer of the student model to predict the dynamic parameters of each sub-trajectory. S5: Input the dynamic parameters of the sub-trajectory into the curve integral calculation link to obtain the total latent displacement of the student model during the image generation process; S6: Optimize the student model based on the total shift of latent variables in the teacher model and the total shift of latent variables in the student model; S7: When the progressive knowledge distillation training is complete, the trained learning model will be used as the target image generation model.
[0026] The image generation model training method disclosed herein performs momentum modeling on each denoising operation of the student model and constructs a curve trajectory by weighted mixing of multiple sub-trajectories. This method can accurately characterize the dynamic evolution of latent variables in the denoising process and fully preserve the nonlinear characteristics of the original model trajectory. At the same time, by constructing a curve integral calculation link, the dynamic parameters predicted by the output projection layer are transformed into quantifiable latent variable displacements, enabling accurate calculation and reconstruction of complex curve trajectories. This avoids trajectory distortion caused by straight-line approximation and thus achieves accurate fitting of the complex curve generation path of the teacher model.
[0027] This scheme employs progressive knowledge distillation training, using the total displacement of the latent variables in the teacher model (multi-step denoising) as the supervised target. It efficiently transfers the high-precision trajectory knowledge of the teacher model to the student model (few-step denoising). Simultaneously, through momentum modeling and curve integral links, it ensures that the student model can accurately replicate the trajectory features of the teacher model even in few-step denoising scenarios. Furthermore, the student model only needs to perform a few denoising operations, significantly reducing inference overhead and improving generation efficiency. Through accurate trajectory fitting, it effectively preserves image details and contour integrity, achieving a dual improvement in generation efficiency and quality.
[0028] In S1, the teacher model is a pre-trained image generation model, and the student model is an image generation model to be trained. The teacher model is a multi-step diffraction model, while the student model is a short-step model. Both models input noisy images and text prompts, and both output clear images, differing only in the number of denoising steps. For example, the teacher model is a 40-step model, requiring 40 denoising operations to transform noise into a clear image, resulting in high quality but low efficiency. The student model is a 2-4 step model, performing only 2-4 denoising operations to transform noise into a clear image. Initial parameters are randomly initialized, and the teacher model's generation patterns need to be learned through distillation. This step lays the groundwork for subsequent distillation training by determining the basic attributes and division of labor between the teacher and student models.
[0029] In S2, refer to Figure 2 The flowchart illustrating the method for constructing the curve trajectory demonstrates that "momentum modeling is performed on each denoising operation of the student model during image generation to construct a curve trajectory representing the image generation process," which includes the following steps: S21: Define each denoising operation corresponding to each image feature type as a basic velocity vector; S22: Set the momentum factor to characterize the evolution and decay of the basic velocity vector as the denoising process progresses; S23: Based on the basic velocity vector and momentum factor, construct sub-trajectories corresponding to each image feature type; S24: Set the gating probability for each sub-trajectory; S25: Based on the gating probability, the weighted summation of each sub-trajectory is performed to obtain the curved trajectory.
[0030] The denoising process of the diffusion model is essentially solving motion evolution equations, which can be viewed as a continuous state transition from a noisy image to a clear image. The direction and magnitude of each state update correspond to a velocity vector, and the denoising operation is equivalent to predicting this velocity vector. Therefore, each denoising operation in the image generation process of the student model is abstracted and structurally modeled. A single-step denoising operation is abstracted into a basic velocity vector (velocity field). This vector represents the direction and magnitude of the denoising update, i.e., denoising direction + denoising magnitude = basic velocity vector. The vector direction represents the correction direction of image features, and the vector magnitude represents the corresponding correction intensity. The velocities between adjacent denoising operations are correlated and not completely independent. Therefore, a momentum factor is introduced to model the inertial correlation of velocities between adjacent denoising steps, describing the temporal decay law of the velocity vector. This allows the velocity to maintain inertia at a certain proportion, so that the current velocity is jointly determined by the inertia of the newly predicted velocity and the historical velocity. This allows the denoising process to form a continuous and smooth curved trajectory, avoiding sudden velocity changes and trajectory jitter. This is used to construct the basic mathematical structure of the student model's curved trajectory (at this point, it is only a framework, and the model parameters have not yet been trained).
[0031] Given that different feature types in an image (such as edge contours, texture details, and color gradations) have their own independent variation patterns during denoising, a divide-and-conquer strategy is adopted to split the overall curve trajectory into multiple sub-trajectories according to feature type. This allows the evolution patterns of different features to be independently described and accurately modeled. Each sub-trajectory is only responsible for learning the denoising speed variation pattern of one type of image feature, and each sub-trajectory can independently output a sub-velocity vector for the corresponding type of image feature, used to independently describe the denoising direction and intensity of that type of image feature. A corresponding basic velocity vector and momentum factor are set for each sub-trajectory, and a gating probability is introduced to control the contribution weight of each sub-trajectory in the overall trajectory. By weighting and fusing the dynamic components of each sub-trajectory through the gating probability, the continuous curve trajectory of the student model's overall denoising can be equivalently represented. This framework replaces the straight-line approximation in traditional few-step generation with a smooth curve trajectory, which can effectively reduce geometric approximation errors and improve the accuracy and stability of few-step generation. Moreover, the sub-trajectories constituting the curve trajectory can achieve divide-and-conquer, allowing the denoising patterns of different types of image features to be learned independently.
[0032] The momentum factor describes the exponential evolution of the fundamental velocity vector along the time axis. For the sub-trajectory, from the initial time step to any time step, its velocity evolution satisfies the momentum transfer law, which is expressed as follows: In the formula, Represents the conditional expectation operator; Indicates the first The conditional desired velocity of the strip trajectory, expressed as a whole in the known first... The strip trajectory at the initial time step initial velocity Momentum factor Under the condition that the sub-trajectory is at any subsequent time step speed Statistical expectation; Indicates the first The strip trajectory at the initial time step The fundamental velocity vector; Indicates the first Implicit variable indication of the strip trajectory; Indicates the start time step Latent variables; Indicates any subsequent time step The latent variables are the core carriers describing state transitions in the flow matching framework.
[0033] Since the overall velocity field is a probabilistic mixture of all sub-trajectories, the velocities of each sub-trajectory are weighted and summed according to gated probabilities to obtain the complete mixed velocity field, i.e., the curved trajectory. The expression for the curved trajectory is as follows: In the formula, Represents the trajectory of a curve; Indicates the total number of sub-trajectories; Indicates the first Gating probability of the strip trajectory and ; Indicates the first The fundamental velocity vector of the strip trajectory, , express 3D real space, Indicate the dimension of latent variables; Indicates the first The momentum factor of the strip trajectory. , Represent the set of positive real numbers; This represents the momentum modulation term, indicating the modulating effect of the momentum factor on velocity. When, the speed increases with time (acceleration mode), when When, the speed decreases with time (damped mode), when At that time, it degenerates into a uniform linear flow, ensuring compatibility with traditional flow matching. The strip trajectories work together to characterize complex nonlinear generated trajectories.
[0034] In S3, refer to Figure 3 The flowchart illustrating the curve integral calculation link construction method shows that "modeling the curve trajectory integrally and constructing the curve integral calculation link" includes the following steps: S31: Based on the momentum factor of the sub-trajectory and the start and end time steps of each denoising operation in the student model during image generation, obtain the momentum integral coefficient of the sub-trajectory in each denoising operation. S32: Multiply the base velocity vector and the corresponding momentum integral coefficient of the sub-trajectory in each denoising operation, and then perform a weighted summation through the gating probability of the sub-trajectory to obtain the curve integral calculation link of each denoising operation in the process of image generation by the student model.
[0035] In S31-S32, traditional flow matching / diffusion models require multiple small steps to perform stepwise numerical integration of the Probability Flow Ordinary Differential Equation (PF-ODE). Each step introduces discrete errors, which accumulate and lead to significant generation bias. To achieve efficient and accurate transitions from the initial state to the target state and avoid the gradual accumulation of errors in traditional methods, this scheme utilizes the mathematical properties of curved trajectories to derive a closed-form analytical solution from the starting point to the ending point of the denoising operation. Only one calculation is needed to directly obtain the ending state of the denoising operation, completely avoiding the gradual accumulation of discrete errors. During the inference phase, given the initial noisy image and the number of target denoising operations (usually 2-4), the system can complete the equivalent trajectory integration in multiple steps through a single forward propagation of the neural network. This achieves extreme compression from the traditional 40-step generation to a mere 2-4 steps, and the numerical accuracy of the analytical integration ensures that the quality of the few-step generation is comparable to the original multi-step model.
[0036] For a single sampling step from the start time step to the end time step, the analytical trajectory transfer operator is constructed by performing a closed-form integral of the PF-ODE using the mixed velocity field formula. (Analytic Transition Operator) is used to characterize a single denoising operation from the initial state of the latent variables. To the target state displacement The core computational logic is as follows: For each sub-trajectory, based on its momentum factor, start time step, and end time, the corresponding momentum integral coefficient is calculated. Then, the basic velocity vector of each sub-trajectory is multiplied by the corresponding momentum integral coefficient, and then a weighted sum is performed using the gating probability of the sub-trajectory. Finally, a complete curve integral computation chain is formed, which directly solves for the latent variable displacement. This completes a precise single-step jump from the starting point to the ending point. The expression for calculating the link using line integral is as follows: In the formula, This represents the analytical trajectory transition operator, used to characterize the complete migration pattern of latent variables from the initial state to the target state; This indicates the starting time step of the single-step denoising operation on the student model; This indicates the termination time step of the single-step denoising operation on the student model; Represents the network parameters of the student model; Define the symbols to represent equivalence relations; This represents the latent variable displacement of the student model in a single-step denoising operation. It is the output of the curve integral calculation link and is used to directly update the latent variable state until the final image is obtained. Indicates the first The momentum integral coefficients of the strip trajectory in the single-step denoising operation are closed-form coefficients. Their function is to transform the exponential evolution law of velocity into scalar scaling coefficients, directly completing the time-domain integration without step-by-step calculation.
[0037] Furthermore, when the momentum factor of the sub-trajectory is 1, the momentum integral coefficient of the sub-trajectory in each denoising operation is equal to the difference between the start and end time steps of each denoising operation. When the momentum factor of the sub-trajectory is not 1, the momentum integral coefficient of the sub-trajectory in each denoising operation is obtained by analytically integrating the exponential time evolution relationship of the momentum factor based on the start and end time steps of each denoising operation. The formula for calculating the momentum integral coefficient is as follows: In the formula, The essence is The closed solution.
[0038] In S4, during the progressive knowledge distillation training process, lightweight parameter fine-tuning enables rapid adaptation of the student model. This significantly reduces training costs while avoiding full parameter retraining. All parameters of the student model's backbone network are frozen. The backbone network is the main structure for feature extraction in the student model, used to encode high-dimensional features of input latent variables and text prompts while keeping pre-trained knowledge unchanged. At the same time, a low-rank adaptation module (LoRA adapter) is injected into the feedforward layer of the student model. This module is a lightweight trainable structure with fewer than 5% of the parameters of the original model. It is used to make small adaptive adjustments to high-dimensional features without changing the backbone network, making the features more suitable for subsequent dynamic parameter prediction.
[0039] The dynamic parameters include the basic velocity vector, momentum factor, and gating probability. The output projection layer is a multi-branch mapping structure set between the backbone network and the output layer of the student model. It is used to map the high-dimensional features output by the backbone network to the basic velocity vector, momentum factor, and gating probability. This output projection layer contains multiple decoding branches (Decoder). Two additional decoding branches are added to the original decoding branch that outputs the basic velocity vector, respectively used to predict the momentum factor and gating probability, forming a multi-branch mapping structure. Through lightweight fine-tuning, the output projection layer has the ability to independently predict dynamic parameters for each sub-trajectory in each denoising step, thereby outputting the basic velocity vector, momentum factor, and gating probability required for curve trajectory modeling, providing support for subsequent analytical integral calculations step by step of denoising.
[0040] In S5, the latent variables of the student model's single-step denoising operation at the final time step are represented by the following formula: In the formula, This represents the latent variable at the final time step of the single-step denoising operation in the student model; This represents the latent variable at the initial time step of the student model's single-step denoising operation.
[0041] As can be seen from the above formula, the latent variables are updated as the denoising operation progresses. Therefore, the complete computational chain involves the student model outputting the dynamic parameters of the current denoising operation for each image feature type during a single forward propagation of the neural network. By calculating the momentum integral coefficients and then weighting and summing them according to the gating probability, the latent variable displacement of the current denoising operation can be obtained. The latent variables are then updated based on this displacement until the final denoising operation is completed. Using this principle, the dynamic parameters of each sub-trajectory predicted by the output projection layer in each denoising operation are input into the corresponding curve integral computation chain to obtain the latent variable displacement of each denoising operation. Accumulating the latent variable displacements of each denoising operation yields the total latent variable displacement of the student model during the image generation process.
[0042] In S6, the mean squared error between the total latent variable displacements of the teacher model and the student model is obtained. The student model is optimized with the goal of minimizing the mean squared error. The total latent variable displacements of the teacher model serve as the supervision baseline, and the mean squared error calculated from them is used as the loss function for model optimization. Gradients are calculated through backpropagation, and only the LoRA module parameters and output projection layer parameters in the student model are updated, while the backbone network parameters remain frozen. By minimizing the aforementioned error, the dynamic parameters and overall generated trajectory output by the student model are aligned with the high-precision trajectory of the teacher model.
[0043] In S7, once the progressive knowledge distillation training is complete, the trained student model is used as the final target image generation model, based on the teacher-student collaborative learning mechanism employed during training. This teacher-student collaborative learning mechanism follows a progressive guidance strategy. In the early stages of training, the teacher model (the original multi-step model) is the core, primarily relying on the high-precision latent variable total displacement and trajectory information output by the teacher model to provide guidance, ensuring that the generated trajectory of the student model is always in the correct direction and avoiding oscillations and instability in the early stages of training. During the training process, the guidance weight of the teacher model is gradually reduced, while the autonomous decision-making power of the student model (the few-step model) is increased simultaneously, continuously cultivating its ability to independently generate high-precision trajectories, ultimately achieving smooth convergence of model training. After training, only the optimized LoRA module parameters and output projection layer parameters in the student model are retained, while the backbone network parameters remain unchanged from the original pre-training state. This preserves the pre-training knowledge of the original model and avoids catastrophic forgetting. Furthermore, by updating only a small number of parameters, the number of training parameters is reduced by more than 95%, significantly reducing memory usage and computational overhead during training. At the same time, the trained target image generation model can directly replace the original model without modifying the original inference framework, achieving lightweight, efficient, and high-quality image generation inference. This successfully and efficiently transfers the knowledge of the pre-trained large model to the small model with fewer steps, achieving the core goal of progressive knowledge distillation training.
[0044] After training, i.e., in the actual use phase of the target image generation model, there is no need to adjust the output projection layer to its original state. Its structure remains unchanged, and the trained parameters are continuously retained. In the actual image generation process, the target image generation model completes image generation according to the following specific steps: First, input noise and corresponding text prompts. The backbone network extracts features from the input noise and text prompts to generate high-dimensional features. Then, retaining the trained parameters, the output projection layer maps the extracted high-dimensional features into three types of core parameters (basic velocity vector, momentum factor, and gating probability). These three types of parameters are then input into the curve integral calculation link to first calculate the potential corresponding to the current denoising step. The displacement of variables is then fused with gating weights to obtain the total displacement of this denoising step. Based on this total displacement, the latent variable state is updated, and the intermediate image corresponding to the current denoising step is generated. After that, the model repeats the above denoising process: the updated latent variables and text prompts are input again, the backbone network continues to extract features, the output projection layer (keeping the trained parameters unchanged) is mapped again to generate 3 types of core parameters, the input integral is used to calculate the link to obtain the new latent variable displacement, the latent variable state is updated and a new intermediate image is generated. This process is repeated in a loop, repeating 2 to 4 preset denoising operations until all denoising steps are completed, and finally the output layer outputs a clear target image.
[0045] Reference Figure 4 This disclosure provides an image generation model training system, comprising: The model acquisition module 101 is used to acquire the teacher model and the student model, wherein the teacher model performs more denoising steps than the student model during the image generation process; The curve trajectory construction module 102 is used to perform momentum modeling on each denoising operation of the student model in the process of generating images, and to construct a curve trajectory that represents the image generation process. The curve trajectory is composed of a weighted mixture of multiple sub-trajectories. The integral modeling module 103 is used to perform integral modeling on the curve trajectory and construct the curve integral calculation link; The knowledge distillation training module 104 is used to perform progressive knowledge distillation training on the teacher model and the student model. During each training session, the total displacement of the latent variables of the teacher model during the image generation process is obtained, and the output projection layer of the student model is used to predict the dynamic parameters of each sub-trajectory. The total displacement acquisition module 105 is used to input the dynamic parameters of the sub-trajectory into the curve integral calculation link to obtain the total displacement of the latent variable of the student model during the image generation process. The student model optimization module 106 is used to optimize the student model based on the total shift of the latent variables of the teacher model and the total shift of the latent variables of the student model. The target model determination module 107 is used to use the trained learning model as the target image generation model when the progressive knowledge distillation training is completed.
[0046] The various variations and specific examples of the image generation model training method provided above are also applicable to the image generation model training system provided in this disclosure. Through the foregoing detailed description of the image generation model training method, those skilled in the art can clearly understand the implementation method of the image generation model training system. For the sake of brevity, they will not be described in detail here.
[0047] A computer device according to embodiments of the present disclosure includes a memory and a processor. The memory is used to store non-transitory computer-readable instructions. Specifically, the memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, etc.
[0048] The processor may be a central processing unit (CPU) or other processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the computer device to perform desired functions. In one embodiment of this disclosure, the processor is used to execute computer-readable instructions stored in the memory, causing the computer device to perform all or part of the steps of the image generation model training methods of the foregoing embodiments of this disclosure.
[0049] Those skilled in the art will understand that, in order to solve the technical problem of how to achieve a good user experience, this embodiment may also include well-known structures such as communication buses and interfaces, and these well-known structures should also be included within the protection scope of this disclosure.
[0050] like Figure 5 This is a schematic diagram of a computer device provided for an embodiment of the present disclosure. It illustrates a structural schematic diagram suitable for implementing the computer device in the embodiments of the present disclosure. Figure 5 The computer device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.
[0051] like Figure 5 As shown, a computer device may include a processor (such as a central processing unit, graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) or programs loaded from storage devices into random access memory (RAM). The RAM also stores various programs and data required for the operation of the computer device. The processor, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.
[0052] Typically, the following devices can be connected to the I / O interface: input devices, such as sensors or visual information acquisition devices; output devices, such as displays; storage devices, such as magnetic tapes or hard drives; and communication devices. Communication devices allow the computer device to communicate wirelessly or wiredly with other devices (such as edge computing devices) to exchange data. Although Figure 5 A computer apparatus with various devices is shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or included alternatively.
[0053] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from ROM. When the computer program is executed by a processor, all or part of the steps of the image generation model training method of embodiments of this disclosure are performed.
[0054] For a detailed description of this embodiment, please refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.
[0055] A computer-readable storage medium according to embodiments of the present disclosure stores non-transitory computer-readable instructions. When these non-transitory computer-readable instructions are executed by a processor, all or part of the steps of the image generation model training methods described in the foregoing embodiments of the present disclosure are performed.
[0056] The aforementioned computer-readable storage media include, but are not limited to: optical storage media (e.g., CD-ROM and DVD), magneto-optical storage media (e.g., MO), magnetic storage media (e.g., magnetic tape or portable hard drive), media with built-in rewritable non-volatile memory (e.g., memory card), and media with built-in ROM (e.g., ROM cartridge).
[0057] For a detailed description of this embodiment, please refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.
[0058] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.
[0059] In this disclosure, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The block diagrams of devices, apparatuses, devices, and systems involved in this disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as "comprising," "including," "having," etc., are open-ended terms meaning "including but not limited to," and are used interchangeably with them. The terms "or" and "and" as used herein refer to the terms "and / or," and are used interchangeably with them unless the context clearly indicates otherwise. The term "such as" as used herein refers to the phrase "such as but not limited to," and is used interchangeably with it.
[0060] Additionally, as used herein, the "or" used in a list of items beginning with "at least one" indicates a separate list, such that a list of, for example, "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not imply that the described example is preferred or better than other examples.
[0061] It should also be noted that in the systems and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.
[0062] Various changes, substitutions, and modifications can be made to the technology described herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufactures, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Therefore, the appended claims include such processes, machines, manufactures, events, means, methods, or actions within their scope.
[0063] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.
[0064] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.
Claims
1. A method for training an image generation model, characterized in that, include: Obtain a teacher model and a student model, wherein the teacher model performs more denoising steps than the student model during image generation; Momentum modeling is performed on each denoising operation of the student model in the image generation process to construct a curved trajectory representing the image generation process, wherein the curved trajectory is composed of a weighted mixture of multiple sub-trajectories; Perform integral modeling on the curve trajectory and construct the curve integral calculation link; The teacher model and the student model are trained by progressive knowledge distillation. During each training session, the total displacement of the latent variables of the teacher model during the image generation process is obtained. The output projection layer of the student model is used to predict the dynamic parameters of each sub-trajectory. The dynamic parameters of the sub-trajectory are input into the curve integral calculation link to obtain the total latent displacement of the student model during the image generation process; Based on the total shift of the latent variables of the teacher model and the total shift of the latent variables of the student model, optimize the student model; When the progressive knowledge distillation training is complete, the trained learning model is used as the target image generation model. 2.The image generation model training method of claim 1, wherein, The step of performing momentum modeling on each denoising operation of the student model in the image generation process, and constructing a curve trajectory representing the image generation process, includes: Define each denoising operation corresponding to each image feature type as a basic velocity vector; The momentum factor is set to characterize the evolution and decay of the basic velocity vector as the denoising process progresses; Based on the basic velocity vector and the momentum factor, sub-trajectories corresponding to each image feature type are constructed respectively; Set the gating probability for each sub-trajectory; Based on the gating probability, a weighted sum is performed on each sub-trajectory to obtain the curved trajectory. 3.The image generation model training method of claim 2, wherein, The step of performing integral modeling on the curve trajectory and constructing the curve integral calculation link includes: Based on the momentum factor of the sub-trajectory and the start and end time steps of each denoising operation in the image generation process of the student model, the momentum integral coefficient of the sub-trajectory in each denoising operation is obtained. The basic velocity vector and corresponding momentum integral coefficient of the sub-trajectory in each denoising operation are multiplied, and then weighted and summed through the gating probability of the sub-trajectory to obtain the curve integral calculation link of each denoising operation of the student model in the process of generating the image. 4.The image generation model training method of claim 3, wherein, The momentum integral coefficients of the sub-trajectory at each denoising operation are obtained based on the momentum factor of the sub-trajectory and the start and end time steps of each denoising operation in the image generation process of the student model, including: When the momentum factor of the sub-trajectory is 1, the momentum integral coefficient of the sub-trajectory in each denoising operation is equal to the difference between the start time step and the end time step of each denoising operation. When the momentum factor of the sub-trajectory is not 1, the exponential time evolution relationship of the momentum factor is analytically integrated based on the start time step and the end time step of each denoising operation to obtain the momentum integral coefficient of the sub-trajectory in each denoising operation. 5.The image generation model training method of claim 3, wherein, The dynamic parameters include the basic velocity vector, momentum factor, and gating probability; The output projection layer is a multi-branch mapping structure set between the backbone network and the output layer of the student model, used to map the high-dimensional features output by the backbone network to the basic velocity vector, momentum factor and gating probability.
6. The image generation model training method of claim 5, wherein, The step of inputting the dynamic parameters of the sub-trajectory into the curve integral calculation link to obtain the total latent displacement of the student model during the image generation process includes: The dynamic parameters of each sub-trajectory predicted by the output projection layer in each denoising operation are input into the corresponding curve integral calculation link to obtain the latent variable displacement of each denoising operation. The total latent variable shift of the student model during the image generation process is obtained by summing the latent variable shifts of each denoising operation.
7. The image generation model training method according to claim 1, characterized in that, The optimization of the student model based on the total shift of the latent variables of the teacher model and the total shift of the latent variables of the student model includes: Obtain the mean square error between the total latent variable shift of the teacher model and the total latent variable shift of the student model; The student model is optimized with the goal of minimizing the mean squared error.
8. A computer device, characterized in that, The computer device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the image generation model training method according to any one of claims 1-7.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the image generation model training method according to any one of claims 1-7.
10. A computer program product comprising computer instructions, characterized in that, When executed by a processor, the computer instructions implement the steps of the image generation model training method according to any one of claims 1-7.