Image generation model training method and apparatus, and non-transitory storage medium

Abstract

This application discloses an image generation model training method, apparatus, and non-volatile storage medium. The method includes: during distillation training of the image generation model to be trained, acquiring a first denoised image obtained by the image generation model to be trained after denoising a first noisy image based on a first descriptive text, and first intermediate layer features from the denoising process; comparing the image features of the first initial image and the first intermediate layer features to obtain a representation alignment loss function value for the image generation model to be trained; weightedly fusing the representation alignment loss function value and the distillation loss function value to obtain a comprehensive loss function value for the image generation model to be trained, and updating the model parameters of the image generation model to be trained based on the comprehensive loss function value. This application solves the technical problem of high training cost and low training efficiency caused by excessive resource consumption during image generation model training in related technologies.

Description

Technical Field

[0001] This application relates to the field of image generation, and more specifically, to an image generation model training method, apparatus, and non-volatile storage medium. Background Technology

[0002] In related technologies, training image generation models typically requires a significant investment of hardware resources and human resources, and the training cycle is quite long.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides an image generation model training method, apparatus, and non-volatile storage medium to at least solve the technical problem of high training cost and low training efficiency caused by excessive resources consumed when training image generation models in related technologies.

[0005] According to one aspect of the embodiments of this application, an image generation model training method is provided, comprising: during the distillation training of the image generation model to be trained, obtaining a first denoised image obtained by the image generation model to be trained after denoising a first noisy image based on a first descriptive text, and a first intermediate layer feature during the denoising process, wherein the first noisy image is an image obtained by adding noise to a first initial image, the first teacher model is a model with frozen parameters, and the second teacher model is a model with iteratively updated parameters during the distillation training process; comparing the image features of the first initial image and the first intermediate layer feature to obtain the representation alignment loss function value of the image generation model to be trained; determining the distillation loss function value of the image generation model to be trained based on the first denoised image; performing weighted fusion of the representation alignment loss function value and the distillation loss function value to obtain the comprehensive loss function value of the image generation model to be trained, and updating the model parameters of the image generation model to be trained based on the comprehensive loss function value.

[0006] Optionally, comparing the image features of the first initial image with the features of the first intermediate layer to obtain the representation alignment loss function value of the image generation model to be trained includes: extracting image features from the first initial image using a pre-trained visual model; determining the image feature similarity between the first intermediate layer features and the image features; and determining the alignment loss function value based on the image feature similarity.

[0007] Optionally, determining the image feature similarity between the first intermediate layer features and the image features includes: dividing the first intermediate layer features into blocks to obtain a first block, and dividing the image features into blocks to obtain a second block; determining the second block corresponding to the first block based on the position information of the first block in the first intermediate layer features and the position information of the second block in the image features; processing the first block using a linear mapping layer to obtain a third block, and determining the second block corresponding to the third block based on the second block corresponding to the first block, wherein the third block and the second block have the same dimension; calculating the block similarity between each third block and the corresponding second block, and determining the image feature similarity based on the block similarity.

[0008] Optionally, weighted fusion of the characterization alignment loss function value and the distillation loss function value includes: determining the current iteration number; determining the weighting coefficient of the characterization alignment loss function value based on the current iteration number, wherein the magnitude of the current iteration number and the magnitude of the weighting coefficient are negatively correlated; and performing a weighted summation of the characterization alignment loss function value and the distillation loss function value based on the weighting coefficient.

[0009] Optionally, distillation training of the image generation model to be trained includes using a first teacher model and a second teacher model for distillation training, wherein the first teacher model is a model with frozen parameters, and the second teacher model is a model with iteratively updated parameters during the distillation training process; determining the distillation loss function value of the image generation model to be trained based on the first denoised image includes: adding noise to the first denoised image to obtain a second noisy image; inputting the first noisy image and the second noisy image into the first teacher model and the second teacher model respectively to obtain a first fractional function output by the first teacher model and a second fractional function output by the second teacher model; determining the distribution matching loss function value based on the deviation information between the first fractional function and the second fractional function; determining the generative adversarial loss function value through the second teacher model; and summing the generative adversarial loss function value and the distribution matching loss function value to obtain the distillation loss function value.

[0010] Optionally, determining the generative adversarial loss function value through the second teacher model includes: adding noise to the second initial image to obtain a third noisy image; using the second teacher model to denoise the third noisy image and obtaining the second intermediate layer features of the second teacher model in the process of denoising the third noisy image; and using a pre-trained discriminator to process the second intermediate layer features to obtain the generative adversarial loss function value.

[0011] Optionally, the method further includes: inputting a first noisy image and noise step information corresponding to the first noisy image into a second teacher model, and obtaining a second denoised image output by the second teacher model; determining the diffusion loss function value of the second teacher model based on the first denoised image and the second denoised image; and updating the model parameters of the second teacher model based on the diffusion loss function value, wherein the model parameters of the second teacher model are updated a preset number of times each time the model parameters of the image to be trained generation model are updated.

[0012] According to another aspect of the embodiments of this application, an image generation model training apparatus is also provided, comprising: a first processing module, configured to, during the distillation training of the image generation model to be trained, acquire a first denoised image obtained by the image generation model to be trained after denoising a first noisy image based on a first descriptive text, and a first intermediate layer feature during the denoising process, wherein the first noisy image is an image obtained by adding noise to a first initial image; a second processing module, configured to compare the first initial image and the first intermediate layer feature to obtain a representation alignment loss function value of the image generation model to be trained; a third processing module, configured to determine the distillation loss function value of the image generation model to be trained based on the first denoised image; and a fourth processing module, configured to perform a weighted summation of the representation alignment loss function value and the distillation loss function value to obtain a comprehensive loss function value of the image generation model to be trained, and update the model parameters of the image generation model to be trained based on the comprehensive loss function value.

[0013] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, wherein a program is stored in the non-volatile storage medium, and the program controls the device where the non-volatile storage medium is located to execute an image generation model training method when it runs.

[0014] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the program executes an image generation model training method during runtime.

[0015] In this embodiment, during the distillation training of the image generation model to be trained, the following steps are taken: First, a denoised image is obtained after the image generation model denoises a first noisy image based on a first descriptive text, along with first intermediate layer features from the denoising process. The first noisy image is an image obtained by adding noise to a first initial image. The image features of the first initial image and the first intermediate layer features are compared to obtain the representation alignment loss function value of the image generation model to be trained. The distillation loss function value of the image generation model to be trained is determined based on the first denoised image. The representation alignment loss function value and the distillation loss function value are weighted and fused to obtain the comprehensive loss function value of the image generation model to be trained. The model parameters of the image generation model to be trained are then updated based on the comprehensive loss function value. By adding representation alignment loss during training to optimize the training process, the resource consumption of the training process is reduced, thereby achieving the technical effect of reducing training costs. This solves the technical problem of high training costs and low training efficiency caused by excessive resource consumption when training image generation models in related technologies. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0017] Figure 1 This is a schematic diagram of the structure of a computer terminal (or mobile device) according to an embodiment of this application;

[0018] Figure 2 This is a flowchart illustrating an image generation model training method according to an embodiment of this application;

[0019] Figure 3 This is a schematic diagram of a model training process provided according to an embodiment of this application;

[0020] Figure 4 This is a schematic diagram of a model training architecture provided according to an embodiment of this application;

[0021] Figure 5 This is a schematic diagram of a model-generated image process according to an embodiment of this application;

[0022] Figure 6 This is a comparative schematic diagram of a generated image provided according to an embodiment of this application;

[0023] Figure 7 This is a schematic diagram of the structure of an image generation model training device provided according to an embodiment of this application. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] To better understand the embodiments of this application, the technical terms involved in the embodiments of this application are explained below:

[0027] DiT: Diffusion Transformer (DiT) is a generative model network that deeply integrates the Transformer architecture with diffusion models. It replaces the U-Net architecture in traditional diffusion models and is specifically designed for image generation tasks. DiT completely abandons the convolution-dominated structure of U-Net, instead consisting of multiple stacked Transformer blocks. It captures global semantic relationships and local detail features of images through a multi-head self-attention mechanism, while introducing image spatial location encoding to adapt to visual task characteristics and supporting cross-layer interaction of multi-scale features. Currently, DiT has become a core component of mainstream text-to-image diffusion models such as SDXL and DMD2, and is also a key carrier for diffusion distillation acceleration technology, widely used in high-resolution image generation, real-time content creation, and other scenarios.

[0028] Discriminator: The discriminator is one of the core components of a generative adversarial network (GAN). Together with the generator, it forms a dual-entity in adversarial training and is essentially a classification model. Its core task is to distinguish the source of input samples: it receives real data or fake samples generated by the generator as input and outputs a probability of 0-1. The closer the value is to 1, the more likely it is a real sample; the closer it is to 0, the more likely it is a fake sample. During training, the discriminator optimizes its parameters through gradient descent, aiming to distinguish between real and fake samples as accurately as possible.

[0029] Self-supervised models: Self-supervised models are machine learning models that do not require manual labeling and are trained by mining supervision signals from the data itself. Their core objective is to learn general feature representations of the data. In the field of computer vision, they learn the core features of images, such as semantics and structure, autonomously through pre-tasks like masked image reconstruction, image patch order prediction, and contrastive learning, rather than relying on labeled data. Compared to supervised models, self-supervised models can utilize massive amounts of unlabeled data for training, resulting in more generalizable features that can be quickly transferred to downstream tasks such as classification, segmentation, and detection. The DINO V2 model used in this paper is a Transformer model specifically designed for visual representation. Based on the ViT architecture, it significantly improves the discriminativeness and robustness of features by optimizing self-supervised contrastive loss and using high-resolution images for training.

[0030] The score function, essentially the gradient of the logarithm of the probability distribution, is used in diffusion models to quantify the degree and direction in which noisy samples deviate from the true data distribution. It is a core signal guiding the model's denoising process. (Regarding the data distribution...) The score function is defined as ,in It is about data The gradient is calculated. The gradient direction points to the region with higher probability in the data distribution, and the gradient magnitude reflects the degree of deviation.

[0031] Distribution Matching Distillation (DMD) refers to the use of a diffusion model to parameterize the logarithmic gradients of two fractional functions, i.e., different data distributions.

[0032] With the continuous development of image generative models, related technologies have shown remarkable performance in generating high-resolution, high-fidelity images. Diffusion models, in particular, optimize noisy samples through multiple rounds of denoising iterations during the inference phase, generating images that possess both realism and diversity, significantly outperforming traditional models such as generative adversarial networks. However, the multi-round denoising of diffusion models requires dozens to hundreds of neural network forward propagations, and the generation time for a single image typically reaches the second level, making it difficult to meet the demands of real-time interaction and high-concurrency generation scenarios.

[0033] Against this backdrop, there is an urgent need for a rapid diffusion model generation scheme that balances efficiency and high quality.

[0034] To address the aforementioned issues, this application provides relevant solutions, which are detailed below.

[0035] According to an embodiment of this application, a method embodiment for training an image generation model is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0036] The methods and embodiments provided in this application can be executed on mobile terminals, computer terminals, or similar computing devices. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing an image generation model training method is shown. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0037] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0038] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the image generation model training method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the above-mentioned image generation model training method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0039] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0040] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0041] Under the above operating environment, embodiments of this application provide an image generation model training method, such as... Figure 2 As shown, the method includes the following steps:

[0042] Step S202: During the distillation training of the image generation model to be trained, the first denoised image obtained by the image generation model to be trained after denoising the first noisy image according to the first descriptive text, and the first intermediate layer features in the denoising process are obtained, wherein the first noisy image is the image obtained after adding noise to the first initial image.

[0043] In some embodiments of this application, distillation training of the image generation model to be trained includes training the image generation model using a first teacher model and a second teacher model. The first teacher model is a model with frozen parameters, while the second teacher model is a model whose parameters are iteratively updated during distillation training. The first teacher model, the second teacher model, and the image generation model to be trained have the same basic model architecture; for example, they may all include multiple cascaded DIS denoising blocks, and each model contains the same number of DIS denoising blocks, for example, five.

[0044] In some embodiments of this application, the parameters of the first teacher model can be used to initialize the parameters of the student model, and also to initialize the parameters of the second teacher model. That is, the parameters of the first teacher model are set as the initial parameters of the image generation model to be trained and the second teacher model. Furthermore, in embodiments of this application, the first teacher model is also referred to as the true diffusion model, and the second teacher model is also referred to as the spurious diffusion model.

[0045] As an alternative implementation, the original images (i.e., the first initial images) can be loaded in batches. Then, noise is added to these initial images to obtain the first noisy image. The noise addition process can be expressed by the following formula:

[0046]

[0047] In the above formula, It is noise, following a standard Gaussian distribution. (Coefficients) and The timetable is derived from a predefined timetable in the diffusion model. This represents the first initial image. This represents the first noisy image. The probability distribution is represented by t, which represents the t-th time of the diffusion process. These are predefined time table parameters for the diffusion model, corresponding to different levels of noise.

[0048] Then the first noisy image can be... The input is given to the image generation model to be trained, which then denoises the image according to the following formula:

[0049]

[0050] in, It is a student model. This is the first denoised image.

[0051] Another point to note is that... The first intermediate layer features of the model need to be preserved during the denoising process. This feature requires further mapping.

[0052] Step S204: Compare the image features of the first initial image with the features of the first intermediate layer to obtain the representation alignment loss function value of the image generation model to be trained;

[0053] In the technical solution provided in step S204, comparing the image features of the first initial image and the features of the first intermediate layer to obtain the representation alignment loss function value of the image generation model to be trained includes: extracting image features from the first initial image using a pre-trained visual model; determining the image feature similarity between the first intermediate layer features and the image features; and determining the alignment loss function value based on the image feature similarity.

[0054] As an optional implementation, the step of determining the image feature similarity between the first intermediate layer feature and the image feature includes: determining the image feature similarity between the first intermediate layer feature and the image feature by: performing block processing on the first intermediate layer feature to obtain a first block, and performing block processing on the image feature to obtain a second block; determining the second block corresponding to the first block based on the position information of the first block in the first intermediate layer feature and the position information of the second block in the image feature; processing the first block using a linear mapping layer to obtain a third block, and determining the second block corresponding to the third block based on the second block corresponding to the first block, wherein the third block and the second block have the same dimension; calculating the block similarity between each third block and the corresponding second block, and determining the image feature similarity based on the block similarity.

[0055] In some embodiments of this application, the linear mapping layer described above can be a lightweight projection layer. The pre-trained visual model described above can be any existing visual model, such as DINOv2, MoCov3, etc.

[0056] Optionally, the image features extracted by the pre-trained visual model and the first intermediate layer features obtained after processing the first intermediate layer features can be aligned using the following formula:

[0057]

[0058]

[0059] Pre-trained visual models such as DINO V2 first divide the image into blocks and then extract features. In the formula above, It is the number of the image block. The first real image extracted by the pre-trained self-supervised model The representation of each block is the target alignment feature of the REPA loss. It is the hidden state (i.e., the first intermediate layer feature) inside the diffusion model after processing the noise input. The Middle The characteristics of each block It is a lightweight projection head (such as a 3-layer MLP), whose function is to project... Dimension mapping to Consistent. Among them, It is a predefined cosine similarity calculation function. The closer the value is to 1, the more aligned the features of the two blocks are.

[0060] Optionally, by determining similarity through block segmentation, the internal representation of the DMD2 generator (i.e. the image generation model to be trained) can be forced to maintain semantic consistency with the feature representation of the self-supervised model, ensuring that the model does not lose key visual information (such as object outlines, texture details, and semantic categories) during the denoising process.

[0061] Step S206: Determine the distillation loss function value of the image generation model to be trained based on the first denoised image;

[0062] In the technical solution provided in step S206, the step of determining the distillation loss function value of the image generation model to be trained based on the first denoised image includes: adding noise to the first denoised image to obtain a second noisy image; inputting the first noisy image and the second noisy image into the first teacher model and the second teacher model respectively to obtain the first fractional function output by the first teacher model and the second fractional function output by the second teacher model; determining the distribution matching loss function value based on the deviation information between the first fractional function and the second fractional function; determining the generative adversarial loss function value through the second teacher model; and summing the generative adversarial loss function value and the distribution matching loss function value to obtain the distillation loss function value.

[0063] In some embodiments of this application, a pre-trained real diffusion model can be used. (First Teacher Model) Denoises the image to obtain the first fractional function. :

[0064]

[0065] Using the spurious diffusion model again (That is, the second teacher model) performs denoising on the same image, resulting in a false score function. :

[0066]

[0067] Then the distribution difference can be calculated:

[0068]

[0069] The gradient difference reflected by the distribution difference can then be used to update the model parameters of the image generation model to be trained.

[0070] As an optional implementation, the step of determining the generative adversarial loss function value through the second teacher model includes: adding noise to the second initial image to obtain a third noisy image; using the second teacher model to denoise the third noisy image and obtaining the second intermediate layer features of the second teacher model in the process of denoising the third noisy image; and using a pre-trained discriminator to process the second intermediate layer features to obtain the generative adversarial loss function value.

[0071] In some embodiments of this application, the pre-trained discriminator can be trained using generative adversarial loss according to the following formula:

[0072]

[0073] in, It is a discriminator. It is a forward diffusion process. The discriminator is responsible for measuring the true distribution. and generator Generate distribution The discriminator can also be used to measure the difference between the distributions of the spurious diffusion model and the true diffusion model without requiring paired data and is independent of the sampling trajectories of the first and second teacher models. The true distribution refers to the distribution determined based on real images.

[0074] Step S208: The representation alignment loss function value and the distillation loss function value are weighted and fused to obtain the comprehensive loss function value of the image generation model to be trained, and the model parameters of the image generation model to be trained are updated according to the comprehensive loss function value.

[0075] In the technical solution provided in step S208, the step of weighted fusion of the characterization alignment loss function value and the distillation loss function value includes: determining the current iteration number; determining the weighting coefficient of the characterization alignment loss function value based on the current iteration number, wherein the magnitude of the current iteration number and the magnitude of the weighting coefficient are negatively correlated; and performing a weighted summation of the characterization alignment loss function value and the distillation loss function value based on the weighting coefficient.

[0076] It is important to note that during DMD2 distillation training, while using only the DMD2 distillation loss (including distribution matching loss and GAN loss) can improve generation speed, it can also lead to insufficient representation learning in the student model, resulting in slow training speed and generated images with missing details and semantic biases. Therefore, this application provides a dynamic weighted coordination mechanism to achieve a precise balance between the two types of losses: an adaptive dynamic weighting coefficient is used, which is dynamically adjusted based on the number of training iterations. In the early stages of training, the REPA loss (representation alignment loss) is weighted more heavily to strengthen its guiding role and help the student model quickly establish a high-quality representation foundation. In the middle stages of training, the weighting coefficient is smoothly reduced through an exponential decay function to balance the optimization priorities of the two types of losses. In the later stages of training, the dominant position of the DMD2 distillation loss is emphasized to ensure that the student model fully adapts to the distribution matching requirements of the distillation framework.

[0077] In some embodiments of this application, the process of training the image generation model to be trained is as follows: Figure 3 As shown, it includes the following steps:

[0078] The first step is to initialize the training configuration: determine the core components required for training, including the first teacher model with frozen parameters, the second teacher model with iteratively updatable parameters, and the image generation model to be trained. Use the parameters of the first teacher model to initialize the initial parameters of the image generation model to be trained and the second teacher model; prepare the paired dataset D (containing multiple sets of "first description text - first initial image" samples); configure the adjustment rules for the REPA dynamic weighting coefficients (dynamically changing based on the number of training iterations).

[0079] The second step is batch data loading and noise addition: the first initial image (original clean image) is loaded in batches, and noise following a standard Gaussian distribution is added to it to generate the first noisy image; at the same time, a batch of second initial images is selected separately, and noise is added to them to obtain the third noisy image, which is used for subsequent calculation of adversarial loss.

[0080] The third step is to denoise and extract features from the image generation model to be trained: the first noisy image and the corresponding first descriptive text are input into the image generation model to be trained. The model denoises the first noisy image and outputs the first denoised image. At the same time, the first intermediate layer features in the denoising process are retained for subsequent representation alignment loss calculation.

[0081] The fourth step is to calculate the representation alignment loss function value: a pre-trained visual model (such as DINOv2 or MoCov3) is used to extract image features from the first initial image; the first intermediate layer features and the image features are processed into blocks to obtain the first block and the second block; the correspondence between the first block and the second block is determined based on the position information of the blocks in their respective features; the first block is processed through a linear mapping layer to obtain the third block with the same dimension as the second block; the block similarity between each corresponding third block and the second block is calculated to determine the image feature similarity, and finally the representation alignment loss function value is obtained.

[0082] Step 5, Distillation Loss Function Calculation: Add noise again to the first denoised image to generate a second noisy image; input the first noisy image into the first teacher model to output the first fractional function; input the second noisy image into the second teacher model to output the second fractional function; calculate the distribution matching loss function value based on the deviation information between the two fractional functions; input the third noisy image into the second teacher model for denoising, extract the second intermediate layer features in the process, process the features through a pre-trained discriminator to obtain the generative adversarial loss function value; sum the distribution matching loss function value and the generative adversarial loss function value to obtain the distillation loss function value.

[0083] Step 6, updating the parameters of the second teacher model: input the first noisy image and its corresponding noise step information into the second teacher model, and output the second denoised image; calculate the diffusion loss function value of the second teacher model based on the first denoised image and the second denoised image; update the parameters of the second teacher model according to the rule that the parameters of the model generated by the image to be trained are updated a preset number of times (e.g., 5 times) for each update of the parameters of the model generated by the image to be trained.

[0084] Comprehensive loss calculation and model update: Determine the current training iteration number, and based on the rule that the iteration number is negatively correlated with the weighting coefficient, determine the weighting coefficient of the representation alignment loss function value; use this weighting coefficient to perform a weighted summation of the representation alignment loss function value and the distillation loss function value to obtain the comprehensive loss function value; based on the comprehensive loss function value, update the model parameters of the image generation model to be trained through the backpropagation algorithm.

[0085] Iterative training until completion: Repeat steps two through seven, performing batch data processing, various loss calculations, and model parameter updates in a loop until the preset total number of iterations M is reached. Training ends, and the trained image generation model is output.

[0086] In some embodiments of this application, the method further includes: inputting a first noisy image and noise step information corresponding to the first noisy image into a second teacher model, and obtaining a second denoised image output by the second teacher model; determining the diffusion loss function value of the second teacher model based on the first denoised image and the second denoised image; updating the model parameters of the second teacher model based on the diffusion loss function value, wherein the model parameters of the image to be trained generation model are updated a preset number of times each time the model parameters are updated.

[0087] Optionally, the above diffusion loss function value can be expressed as the following formula, which represents the standard diffusion loss:

[0088]

[0089] in It is a spurious diffusion model. This is the denoising result of the image generation model to be trained. During actual training, the parameters of the image generation model to be trained are updated each time... The parameters need to be updated a preset number of times, such as five times.

[0090] In some embodiments of this application, the following are also provided: Figure 4 The model training architecture is shown below. Figure 4 As can be seen from this, the training process includes:

[0091] First, noise is added to the initial image to obtain a first noisy image. This first noisy image is then input into the image generation model to be trained (the student model). The image generation model denoises the image based on the first descriptive text, outputting a first denoised image while retaining the features of the first intermediate layer during the denoising process. The added noise is random noise of varying degrees.

[0092] Subsequently, noise is added again to the first denoised image to generate a second noisy image. The first noisy image and the second noisy image are then input into a first teacher model (real diffusion model) with frozen parameters and a second teacher model (false diffusion model) with iteratively updated parameters, respectively. The first teacher model outputs a first fractional function, and the second teacher model outputs a second fractional function. The distribution matching loss is obtained by calculating the deviation information between the two.

[0093] Alternatively, another batch of initial images can be taken and noise can be added directly to obtain a third noisy image. The third noisy image is then input into the second teacher model for denoising and the second intermediate layer features are extracted during the process. The second intermediate layer features are then processed using a pre-trained discriminator to obtain the generative adversarial loss. The above distribution matching loss and generative adversarial loss are used together as the distillation loss function value to guide the parameter update of the image generation model to be trained.

[0094] Furthermore, the pre-trained visual model (such as DINOv2) extracts image features from the first initial image and splits it into second blocks according to image patches. The first intermediate layer features of the image generation model to be trained are also divided into blocks after being processed by a lightweight linear mapping layer. The correspondence is determined based on the position information of the blocks in their respective features. The representation alignment loss function value is obtained by calculating the cosine similarity of each corresponding block. This representation alignment loss function value and the distillation loss function value are weighted and fused through dynamic weighting coefficients to form a comprehensive loss function value. Finally, the parameters of the image generation model to be trained are updated based on the comprehensive loss function value. This not only alleviates the problem of inaccurate representation estimation in the training of the second teacher model, but also ensures that the model does not lose key visual information such as object contours and texture details during accelerated training.

[0095] In some embodiments of this application, the process by which the trained image generation model actually generates images after training is as follows: Figure 5 As shown in the figure. It can be seen that the model trained using the model training method provided in this application, even with a large amount of disordered training resources, only requires four forward propagation steps to obtain high-quality images during runtime. The comparison results between images generated by the model trained using the method provided in this application and images generated by the teacher model are shown below. Figure 6 As shown. Among them Figure 6 The two images on the left show the generated images from the trained student model, while the one on the right shows the generated images from the teacher model. It's worth noting that, because the student model trained in this application only requires four forward propagations to obtain the desired image during actual runtime, its image generation speed significantly outpaces that of the teacher model.

[0096] By employing a method that, during the distillation training of the image generation model to be trained, obtains the first denoised image obtained by the model after denoising a first noisy image based on a first descriptive text, and the first intermediate layer features during the denoising process (where the first noisy image is the image obtained by adding noise to a first initial image), the representation alignment loss function value of the image generation model to be trained is obtained by comparing the image features of the first initial image and the first intermediate layer features, the distillation loss function value of the image generation model to be trained is determined based on the first denoised image, the representation alignment loss function value and the distillation loss function value are weighted and fused to obtain the comprehensive loss function value of the image generation model to be trained, and the model parameters of the image generation model to be trained are updated based on the comprehensive loss function value, the training process is optimized by adding representation alignment loss during training, thereby reducing the resource consumption of the training process and achieving the technical effect of reducing training costs. This solves the technical problem of high training costs and low training efficiency caused by excessive resource consumption when training image generation models in related technologies.

[0097] This application provides an image generation model training device. Figure 7 This is a schematic diagram of the device. From Figure 7 As can be seen from the diagram, the device includes: a first processing module 70, used to acquire, during the distillation training process of the image generation model to be trained, a first denoised image obtained by the image generation model to be trained after denoising a first noisy image based on a first descriptive text, and a first intermediate layer feature during the denoising process, wherein the first noisy image is an image obtained by adding noise to a first initial image; a second processing module 72, used to compare the first initial image and the first intermediate layer feature to obtain the representation alignment loss function value of the image generation model to be trained; a third processing module 74, used to determine the distillation loss function value of the image generation model to be trained based on the first denoised image; and a fourth processing module 76, used to perform a weighted summation of the representation alignment loss function value and the distillation loss function value to obtain the comprehensive loss function value of the image generation model to be trained, and update the model parameters of the image generation model to be trained based on the comprehensive loss function value.

[0098] In some embodiments of this application, the step of the second processing module 72 comparing the image features of the first initial image and the first intermediate layer features to obtain the representation alignment loss function value of the image generation model to be trained includes: extracting image features from the first initial image using a pre-trained visual model; determining the image feature similarity between the first intermediate layer features and the image features; and determining the alignment loss function value based on the image feature similarity.

[0099] In some embodiments of this application, the step of the second processing module 72 in determining the image feature similarity between the first intermediate layer features and the image features includes: performing block processing on the first intermediate layer features to obtain a first block, and performing block processing on the image features to obtain a second block; determining the second block corresponding to the first block based on the position information of the first block in the first intermediate layer features and the position information of the second block in the image features; processing the first block using a linear mapping layer to obtain a third block, and determining the second block corresponding to the third block based on the second block corresponding to the first block, wherein the third block and the second block have the same dimension; calculating the block similarity between each third block and the corresponding second block, and determining the image feature similarity based on the block similarity.

[0100] In some embodiments of this application, distillation training of the image generation model to be trained includes distillation training using a first teacher model and a second teacher model, wherein the first teacher model is a model with frozen parameters, and the second teacher model is a model with iteratively updated parameters during the distillation training process; the step of the third processing module 74 determining the distillation loss function value of the image generation model to be trained based on the first denoised image includes: adding noise to the first denoised image to obtain a second noisy image; inputting the first noisy image and the second noisy image into the first teacher model and the second teacher model respectively to obtain a first fractional function output by the first teacher model and a second fractional function output by the second teacher model; determining the distribution matching loss function value based on the deviation information between the first fractional function and the second fractional function; determining the generative adversarial loss function value through the second teacher model; and summing the generative adversarial loss function value and the distribution matching loss function value to obtain the distillation loss function value.

[0101] In some embodiments of this application, the step of the third processing module 74 determining the generative adversarial loss function value through the second teacher model includes: adding noise to the second initial image to obtain a third noisy image; using the second teacher model to denoise the third noisy image and obtaining the second intermediate layer features of the second teacher model in the process of denoising the third noisy image; and using a pre-trained discriminator to process the second intermediate layer features to obtain the generative adversarial loss function value.

[0102] In some embodiments of this application, the step of the fourth processing module 76 in weightedly fusing the characterization alignment loss function value and the distillation loss function value includes: determining the current iteration number; determining the weighting coefficient of the characterization alignment loss function value based on the current iteration number, wherein the magnitude of the current iteration number and the magnitude of the weighting coefficient are negatively correlated; and performing a weighted summation of the characterization alignment loss function value and the distillation loss function value based on the weighting coefficient.

[0103] In some embodiments of this application, the fourth processing module 76 is further configured to: input the first noisy image and the noise step information corresponding to the first noisy image into the second teacher model, and obtain the second denoised image output by the second teacher model; determine the diffusion loss function value of the second teacher model based on the first denoised image and the second denoised image; update the model parameters of the second teacher model based on the diffusion loss function value, wherein the model parameters of the training image generation model are updated a preset number of times each time the model parameters are updated.

[0104] It should be noted that each module in the above-mentioned image generation model training device can be a program module (e.g., a set of program instructions to implement a certain function) or a hardware module. For the latter, it can be manifested in the following forms, but is not limited to them: each of the above modules is manifested as a processor, or the functions of each of the above modules are implemented by a processor.

[0105] According to an embodiment of this application, a non-volatile storage medium is provided, which stores a program. During program execution, the device containing the non-volatile storage medium executes the following image generation model training method: During distillation training of the image generation model to be trained, a first denoised image obtained by the image generation model to be trained after denoising a first noisy image based on a first descriptive text, and first intermediate layer features during the denoising process are obtained. The first noisy image is an image obtained by adding noise to a first initial image, the first teacher model is a model with frozen parameters, and the second teacher model is a model whose parameters are iteratively updated during distillation training. The image features of the first initial image and the first intermediate layer features are compared to obtain the representation alignment loss function value of the image generation model to be trained. Based on the first denoised image, the distillation loss function value of the image generation model to be trained is determined. The representation alignment loss function value and the distillation loss function value are weighted and fused to obtain the comprehensive loss function value of the image generation model to be trained, and the model parameters of the image generation model to be trained are updated based on the comprehensive loss function value.

[0106] According to an embodiment of this application, an electronic device is provided, including a memory and a processor. The processor is used to run a program stored in the memory. During program execution, the following image generation model training method is performed: In the process of distillation training of the image generation model to be trained, a first denoised image obtained by the image generation model to be trained after denoising a first noisy image based on a first descriptive text, and first intermediate layer features during the denoising process are obtained. The first noisy image is an image obtained by adding noise to a first initial image, the first teacher model is a model with frozen parameters, and the second teacher model is a model whose parameters are iteratively updated during distillation training. The image features of the first initial image and the first intermediate layer features are compared to obtain the representation alignment loss function value of the image generation model to be trained. Based on the first denoised image, the distillation loss function value of the image generation model to be trained is determined. The representation alignment loss function value and the distillation loss function value are weighted and fused to obtain the comprehensive loss function value of the image generation model to be trained, and the model parameters of the image generation model to be trained are updated based on the comprehensive loss function value.

[0107] According to an embodiment of this application, an electronic device is provided, including a memory and a processor. The processor is used to run a program stored in the memory. During program execution, the following image generation model training method is performed: In the process of distillation training of the image generation model to be trained, a first denoised image obtained by the image generation model to be trained after denoising a first noisy image based on a first descriptive text, and first intermediate layer features during the denoising process are obtained. The first noisy image is an image obtained by adding noise to a first initial image, the first teacher model is a model with frozen parameters, and the second teacher model is a model whose parameters are iteratively updated during distillation training. The image features of the first initial image and the first intermediate layer features are compared to obtain the representation alignment loss function value of the image generation model to be trained. Based on the first denoised image, the distillation loss function value of the image generation model to be trained is determined. The representation alignment loss function value and the distillation loss function value are weighted and fused to obtain the comprehensive loss function value of the image generation model to be trained, and the model parameters of the image generation model to be trained are updated based on the comprehensive loss function value.

[0108] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0109] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0110] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0111] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0112] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to related technologies, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0113] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for training an image generation model, characterized in that, include: During the distillation training of the image generation model to be trained, the first denoised image obtained by the image generation model to be trained after denoising the first noisy image according to the first descriptive text, and the first intermediate layer features in the denoising process are obtained, wherein the first noisy image is the image obtained by adding noise to the first initial image. By comparing the image features of the first initial image with the features of the first intermediate layer, the representation alignment loss function value of the image generation model to be trained is obtained; The distillation loss function value of the image generation model to be trained is determined based on the first denoised image; The representation alignment loss function value and the distillation loss function value are weighted and fused to obtain the comprehensive loss function value of the image generation model to be trained, and the model parameters of the image generation model to be trained are updated based on the comprehensive loss function value.

2. The image generation model training method according to claim 1, characterized in that, By comparing the image features of the first initial image with the features of the first intermediate layer, the representation alignment loss function value of the image generation model to be trained is obtained as follows: The image features are extracted from the first initial image using a pre-trained visual model; Determine the image feature similarity between the first intermediate layer features and the image features, and determine the alignment loss function value based on the image feature similarity.

3. The image generation model training method according to claim 2, characterized in that, Determining the image feature similarity between the first intermediate layer features and the image features includes: The first intermediate layer features are divided into blocks to obtain a first block, and the image features are divided into blocks to obtain a second block; Based on the position information of the first block in the first intermediate layer features and the position information of the second block in the image features, the second block corresponding to the first block is determined; The first block is processed using a linear mapping layer to obtain the third block, and the second block corresponding to the third block is determined based on the second block corresponding to the first block. Calculate the block similarity between each of the third blocks and the corresponding second blocks, and determine the image feature similarity based on the block similarity.

4. The image generation model training method according to claim 1, characterized in that, The weighted fusion of the characterization alignment loss function value and the distillation loss function value includes: Determine the current iteration number; Based on the current iteration number, the weighting coefficients representing the alignment loss function value are determined, wherein the magnitude of the current iteration number and the magnitude of the weighting coefficients are negatively correlated. Based on the weighting coefficients, the value of the characterization alignment loss function and the value of the distillation loss function are weighted and summed.

5. The image generation model training method according to claim 1, characterized in that, Distillation training of the image generation model to be trained includes distillation training of the image generation model to be trained using a first teacher model and a second teacher model, wherein the first teacher model is a model with frozen parameters, and the second teacher model is a model with iteratively updated parameters during the distillation training process; determining the distillation loss function value of the image generation model to be trained based on the first denoised image includes: Add noise to the first denoised image to obtain a second noisy image; The first noise image and the second noise image are respectively input into the first teacher model and the second teacher model to obtain the first fractional function output by the first teacher model and the second fractional function output by the second teacher model; The distribution matching loss function value is determined based on the deviation information between the first fractional function and the second fractional function; The value of the adversarial loss function is determined using the second teacher model; The distillation loss function value is obtained by summing the generative adversarial loss function value and the distribution matching loss function value.

6. The image generation model training method according to claim 5, characterized in that, The values ​​of the generative adversarial loss function determined using the second teacher model include: Add noise to the second initial image to obtain a third noisy image; The second teacher model is used to denoise the third noisy image, and the second intermediate layer features of the second teacher model in the process of denoising the third noisy image are obtained. The second intermediate layer features are processed using a pre-trained discriminator to obtain the value of the generative adversarial loss function.

7. The image generation model training method according to claim 5, characterized in that, The method further includes: The first noise image and the noise step information corresponding to the first noise image are input into the second teacher model, and the second denoised image output by the second teacher model is obtained; Based on the first denoised image and the second denoised image, determine the diffusion loss function value of the second teacher model; The model parameters of the second teacher model are updated based on the diffusion loss function value, wherein the model parameters of the second teacher model are updated a preset number of times each time the model parameters of the image generation model to be trained are updated.

8. An image generation model training device, characterized in that, include: The first processing module is used to obtain, during the distillation training process of the image generation model to be trained, a first denoised image obtained by the image generation model to be trained after denoising the first noisy image according to the first descriptive text, and a first intermediate layer feature in the denoising process, wherein the first noisy image is an image obtained by adding noise to the first initial image. The second processing module is used to compare the first initial image and the first intermediate layer features to obtain the representation alignment loss function value of the image generation model to be trained. The third processing module is used to determine the distillation loss function value of the image generation model to be trained based on the first denoised image; The fourth processing module is used to perform a weighted summation of the representation alignment loss function value and the distillation loss function value to obtain the comprehensive loss function value of the image generation model to be trained, and to update the model parameters of the image generation model to be trained based on the comprehensive loss function value.

9. A non-volatile storage medium, characterized in that, The non-volatile storage medium stores a program, wherein when the program is executed, it controls the device where the non-volatile storage medium is located to execute the image generation model training method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, include: A memory and a processor, the processor being configured to run a program stored in the memory, wherein the program, when running, executes the image generation model training method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the image generation model training method according to any one of claims 1 to 7.

Images