Image generation method and device, electronic equipment and storage medium

Abstract

The invention provides an image generation method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining a target text; and inputting the target text into an image generation model to obtain a target image output by the image generation model, the image generation model being obtained by adversarial training on a student model based on a sample noise-added image and a corresponding sample text, the student model is obtained by performing progressive distillation training on the teacher model based on the sample noise-added image and the corresponding sample text. Progressive distillation training and adversarial training are combined, so that the image generation model obtained based on student model training can generate details and definition closer to those of an image output by a teacher model, and the quality of the generated image is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an image generation method, device, electronic device and storage medium. Background Art

[0002] Text-to-image generation aims to generate an image given a text description. Diffusion models, also known as score-based generation models, have shown excellent performance in generation tasks such as image generation, audio generation, and video generation. A key feature of diffusion models is the iterative sampling process that gradually removes noise from the random initial vector.

[0003] In the related technology, the pre-trained diffusion model is refined into a model capable of few-step reasoning, namely model distillation, and the teacher model is converted into an efficient, few-step student model, which is used to generate images based on text.

[0004] However, in the above-mentioned related technologies, the student model usually suffers losses when learning the paired noise-to-image mapping of the teacher model, so that the student model cannot have the same performance as the teacher model, thereby reducing the quality of the images generated by the student model. Summary of the invention

[0005] The present invention provides an image generation method, device, electronic device and storage medium, which are used to solve the defect of reducing the quality of images generated by student models in the prior art.

[0006] The present invention provides an image generating method, comprising the following steps.

[0007] Get the target text; The target text is input into an image generation model to obtain a target image output by the image generation model, wherein the image generation model is obtained by adversarial training of a student model based on a sample noisy image and a corresponding sample text, and the student model is obtained by progressive distillation training of a teacher model based on the sample noisy image and a corresponding sample text.

[0008] According to an image generation method provided by the present invention, the image generation model is trained based on the following method: Performing noise processing on the sample image to obtain the sample noisy image; Based on the sample noisy image and the corresponding sample text, the teacher model is progressively distilled for a preset number of steps to obtain a first predicted noisy image corresponding to the last step; Inputting the sample noisy image and the corresponding sample text into the student model to obtain a first predicted noisy image output by the student model, and determining a second predicted noisy image corresponding to the last step based on the first predicted noisy image and the sample noisy image; Determining a distillation loss based on the first predicted noisy image and the second predicted noisy image; Determining an adversarial loss based on the sample noisy image and the first predicted noisy image; Based on the distillation loss and the adversarial loss, the model parameters of the student model are updated to obtain the image generation model.

[0009] According to an image generation method provided by the present invention, determining the adversarial loss based on the sample noisy image and the first predicted noisy image includes: Determine a third predicted noisy image corresponding to the student model based on the sample noisy image and the first predicted noisy image; Inputting the third predicted noisy image into a pre-trained feature network of a discriminator to obtain a first visual feature output by the pre-trained feature network; Inputting the first visual feature into a discriminator head of the discriminator to obtain a first prediction probability output by the discriminator head, wherein the first prediction probability is a probability that the third predicted noisy image is a true image; Based on the first predicted probability, the adversarial loss is determined.

[0010] According to an image generation method provided by the present invention, the method performs progressive distillation of a preset number of steps on the teacher model based on the sample noisy image and the corresponding sample text to obtain a first predicted noisy image corresponding to the last step, including: Inputting the sample noisy image and the sample text into the teacher model to obtain a second predicted noise image corresponding to the first step output by the teacher model; Determine the sample noisy image corresponding to the first step based on the second predicted noise image and the sample noisy image; The sample noisy image corresponding to the first step and the sample text are input into the teacher model to obtain the third predicted noisy image corresponding to the second step output by the teacher model, and based on the third predicted noisy image and the sample noisy image corresponding to the first step, the sample noisy image corresponding to the second step is determined, and the above steps are repeated until the first predicted noisy image corresponding to the last step is obtained.

[0011] According to an image generation method provided by the present invention, the method further includes: Inputting the sample image into the pre-trained feature network to obtain a second visual feature output by the pre-trained feature network; Inputting the second visual feature into the discriminator head to obtain a second prediction probability output by the discriminator head, wherein the second prediction probability is a probability that the sample image is a real image; Determining a discrimination loss based on the sample image, the third predicted noisy image, the first predicted probability, and the second predicted probability; The network parameters of the discriminator head are updated based on the discriminant loss to obtain the discriminator.

[0012] According to an image generation method provided by the present invention, the model parameters of the student model are updated based on the distillation loss and the adversarial loss to obtain the image generation model, including: Obtaining a weight of the adversarial loss; Based on the weight of the adversarial loss, the distillation loss and the adversarial loss, the model parameters of the student model are updated to obtain the image generation model.

[0013] The present invention also provides an image generating device, comprising: An acquisition unit, used for acquiring a target text; A generation unit is used to input the target text into an image generation model to obtain a target image output by the image generation model, wherein the image generation model is obtained by adversarial training of a student model based on a sample noisy image and a corresponding sample text, and the student model is obtained by progressive distillation training of a teacher model based on the sample noisy image and the corresponding sample text.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any of the above-mentioned image generation methods is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the image generating method described in any one of the above is implemented.

[0016] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the image generating method described above is implemented.

[0017] The image generation method, device, electronic device and storage medium provided by the present invention input the acquired target text into the image generation model to obtain the target image output by the image generation model, wherein the image generation model is obtained by adversarial training of a student model based on a sample noisy image and a corresponding sample text, and the student model is obtained by progressive distillation training of a teacher model based on a sample noisy image and a corresponding sample text. It can be seen that the present invention combines progressive distillation training with adversarial training, so that the image generation model obtained by training the student model can produce details and clarity closer to the output image of the teacher model, thereby improving the quality of the generated image. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 It is a flowchart of an image generating method provided by an embodiment of the present invention.

[0020] Figure 2 This is one of the schematic diagrams of the training process of the image generation model provided by an embodiment of the present invention.

[0021] Figure 3 This is the second schematic diagram of the training process of the image generation model provided by an embodiment of the present invention.

[0022] Figure 4 It is a schematic diagram of the overall flow of the image generation method provided by an embodiment of the present invention.

[0023] Figure 5 It is a schematic diagram of the structure of an image generating device provided by an embodiment of the present invention.

[0024] Figure 6 It is a schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] Accelerating the inference process of diffusion models has always been a key focus in this field. Sampling from diffusion models is equivalent to solving the probability flow ordinary differential equation (PF-ODE) associated with the diffusion process. Therefore, most of the current research aimed at improving the sampling efficiency of diffusion models focuses on the ordinary differential equation (ODE) formula. There are two mainstream methods: The first approach, the fast diffusion sampler, aims to advance numerical solvers for PF-ODEs, with the goal of enabling solutions to the ODE with larger discretization scales, ultimately achieving fewer necessary inference steps, for example, from 1000 inference steps to 20-50 inference steps. However, the inherent trade-off between sampling step size and accuracy remains. Given the highly nonlinear and complex trajectory of PF-ODE, it would be extremely difficult to reduce the number of required inference steps to a minimal level. Even state-of-the-art solvers can only generate images within 10 to 20 inference steps, and further reducing the number of inference steps leads to a significant degradation in image quality.

[0027] The second approach is to extract PF-ODE trajectories from pre-trained diffusion models, that is, distillation of diffusion models, which has become a way to further improve sampling speed. Diffusion distillation aims to train a student model to extract the multi-inference step output of the original diffusion model into a single inference step. However, an important challenge is that it is expensive to run the full denoising trajectory implemented by each loss function. To address this problem, progressive distillation trains a series of student models that halve the number of sampling steps of the previous model. Consistency Distillation (CD) trains a student model so that the output of the trained student model is self-consistent at any time step of the ODE trajectory, that is, consistent with the output of the teacher model, thereby significantly reducing the number of solution steps. However, due to training errors, existing distillation models have difficulty in generating images with both clarity and detail complexity at the same time. The quality of images generated in a single step or even the minimum step (for example, 4 or 8 steps) is still significantly behind the teacher model.

[0028] Based on this, the present invention proposes an image generation method that combines progressive distillation training and adversarial training, so that the image generation model obtained based on student model training can produce details and clarity closer to the output image of the teacher model, thereby improving the quality of the generated image.

[0029] Combine the following Figure 1-Figure 4 The image generation method of the present invention is described. The execution subject of the image generation method can be an electronic device such as a terminal, a tablet computer, a computer, or an image generation device set in the electronic device. The image generation device can be implemented by software, hardware, or a combination of the two.

[0030] Figure 1 is a flow chart of an image generation method provided by an embodiment of the present invention, such as Figure 1 As shown, the image generation method includes the following steps: Step 101: Obtain target text.

[0031] For example, when a user needs to convert text into an image, the text to be converted can be input into an electronic device, so that the electronic device obtains the target text that the user needs to convert; or the user inputs the content to be converted through voice, and when the electronic device receives the content in voice form, it converts the content in voice form into text to obtain the target text.

[0032] Step 102: input the target text into an image generation model to obtain a target image output by the image generation model, wherein the image generation model is obtained by adversarial training of a student model based on a sample noisy image and a corresponding sample text, and the student model is obtained by progressive distillation training of a teacher model based on the sample noisy image and a corresponding sample text.

[0033] For example, when the target text is obtained, the target text is input into the trained image generation model, and the noise in the pure noise image is gradually removed based on the target text by the image generation model, and finally the target image corresponding to the target text is output. Among them, the image generation model is trained by combining progressive distillation training and adversarial training, so that the trained image generation model can produce details and clarity closer to the output image of the teacher model.

[0034] The image generation method provided by the present invention inputs the acquired target text into the image generation model to obtain the target image output by the image generation model, wherein the image generation model is obtained by adversarial training of a student model based on a sample noisy image and a corresponding sample text, and the student model is obtained by progressive distillation training of a teacher model based on a sample noisy image and a corresponding sample text. It can be seen that the present invention combines progressive distillation training with adversarial training, so that the image generation model obtained by training the student model can produce details and clarity closer to the output image of the teacher model, thereby improving the quality of the generated image.

[0035] In one embodiment, Figure 2 is one of the training flow diagrams of the image generation model provided by the embodiment of the present invention, such as Figure 2 As shown, the image generation model is trained based on the following method: Step 201: Perform noise processing on the sample image to obtain the sample noisy image.

[0036] For example, a data set is obtained, the data set includes a plurality of real data, and the real data includes sample images and sample image The corresponding sample text, that is, the real image and the corresponding real text, for each sample image in the data set , the forward diffusion (noise) process can be modeled as a widely used stochastic differential equation (SDE), and the sample noisy image at any time step t can be obtained through the stochastic differential equation ,in, represents noise, 0,1), that is It follows a standard normal distribution. and is a predefined noise schedule, and .

[0037] Step 202: Perform progressive distillation of a preset number of steps on the teacher model based on the sample noisy image and the corresponding sample text to obtain a first predicted noisy image corresponding to the last step.

[0038] For example, after obtaining the sample noisy image, the sample noisy image, the sample text corresponding to the sample image, and the time step t are input into the teacher model, so that the teacher model predicts the noise in the sample noisy image based on the sample text and the time step t, and finally the teacher model outputs the predicted noise, and then generates the sample noisy image based on the predicted noise output by the teacher model, repeats the above steps, and uses the teacher model to perform a preset number of inference steps n, and finally obtains the first predicted noisy image corresponding to the last step ,in, represents the step length, Represents the time step The corresponding first predicted noisy image is the first predicted noisy image corresponding to the last step.

[0039] Step 203: input the sample noisy image and the corresponding sample text into the student model to obtain a first predicted noisy image output by the student model, and determine a second predicted noisy image corresponding to the last step based on the first predicted noisy image and the sample noisy image.

[0040] For example, after obtaining a sample noisy image, the sample noisy image, the sample text corresponding to the sample image, and the time step t are input into the student model, so that the student model predicts the noise in the sample noisy image based on the sample text and the time step t, and finally the student model outputs the predicted noise , predict noise It can be expressed by the following formula (1), and then the second predicted noisy image corresponding to the last step is generated based on the predicted noise output by the student model: ,from Directly pointing , It can be expressed by the following formula (2): in, represents the model parameters of the student model, and Represents the time step Corresponding noise scheduling.

[0041] Step 204: Determine a distillation loss based on the first predicted noisy image and the second predicted noisy image.

[0042] For example, when a first predicted noisy image corresponding to the teacher model and a second predicted noisy image corresponding to the student model are obtained, the distance between the first predicted noisy image and the second predicted noisy image is determined, and the distillation loss is determined based on the distance. Specifically, the distillation loss can be expressed by the following formula (3): : in, Represents the Mean Squared Error (MSE) loss.

[0043] Step 205: Determine an adversarial loss based on the sample noisy image and the first predicted noisy image.

[0044] For example, when the sample noisy image and the first predicted noisy image are obtained, the adversarial loss may also be determined based on the sample noisy image and the first predicted noisy image.

[0045] Step 206: Based on the distillation loss and the adversarial loss, the model parameters of the student model are updated to obtain the image generation model.

[0046] For example, when the distillation loss and the adversarial loss are obtained, the model parameters of the student model can be updated based on the distillation loss and the adversarial loss until the convergence condition is reached, and finally the image generation model is obtained.

[0047] In this embodiment, the image generation model is trained by combining adversarial training and progressive distillation training, which effectively alleviates the image quality loss and structural degradation caused by the model distillation process, thereby improving the quality of images generated by the image generation model.

[0048] In one embodiment, Figure 3 FIG. 2 is a second schematic diagram of the training process of the image generation model provided by an embodiment of the present invention. Figure 3 As shown, the above step 205 determines the adversarial loss based on the sample noisy image and the first predicted noisy image, which can be specifically implemented by the following steps: Step 301: Determine a third predicted noisy image corresponding to the student model based on the sample noisy image and the first predicted noisy image.

[0049] For example, a Generative Adversarial Network (GAN) is introduced to improve the quality of the student model's inference results. Adversarial training involves the minimax optimization between the discriminator (aimed at identifying generated samples from real samples) and the generator (aimed at deceiving the discriminator). The generator is the student model, and the first predicted noise image output by the student model is used. And the sample noisy image Calculate the third predicted noisy image inferred by the student model , the third predicted noisy image can be expressed by the following formula (4): : Step 302: input the third predicted noisy image into a pre-trained feature network of a discriminator to obtain a first visual feature output by the pre-trained feature network.

[0050] For example, the third predicted noisy image of the sample image (real image) is and sample image Real and predicted images decoded to pixel space by Variational Autoencoder (VAE) , and then the real image and the predicted image The structure of the discriminator is mainly composed of a frozen pre-trained feature network F and a set of trainable lightweight discriminator heads. Composition, among which, represents the network parameters of the discriminator head, Represents the number of discriminator heads. For the pre-trained feature network F, the pre-trained DINOv2 visual Transformer (ViT-S) architecture can be used as the backbone network to encode the visual features of the real image or the generated image (predicted image), that is, the third predicted noisy image is input into the pre-trained feature network of the discriminator to obtain the first visual feature output by the pre-trained feature network.

[0051] Step 303: input the first visual feature into the discriminator head of the discriminator to obtain a first prediction probability output by the discriminator head, wherein the first prediction probability is the probability that the third predicted noisy image is a true image.

[0052] For example, the output features of different layers of the pre-trained feature network F are (k=1,2,...,K), for example, K=5, the output features of different layers are input into the corresponding trainable discriminator head Therefore, for each feature layer of the pre-trained feature network F, the first visual feature output by the feature layer is input into the corresponding discriminator head, and the first prediction probability output by the discriminator head to determine that the third predicted noisy image is a real image is obtained. For example, if the first prediction probability is 0, it means that the third predicted noisy image is not a real image, but a predicted noisy image corresponding to the student model.

[0053] Step 304: Determine the adversarial loss based on the first predicted probability.

[0054] For example, when the first predicted probability output by the discriminator head is obtained, the adversarial loss can be determined based on the following formula (5): : in, represents the first The first visual feature output by the feature layer is Indicates The first prediction probability output by the discriminator head.

[0055] In this embodiment, a generative adversarial network is introduced to promote the sampling results of the student model to be close to the image output by the teacher model in the pixel space. The introduction of adversarial training enables the image generation model finally trained to produce details and clarity closer to the image output by the teacher model, thereby improving the quality of the generated image.

[0056] In one embodiment, the step 202 performs progressive distillation of a preset number of steps on the teacher model based on the sample noisy image and the corresponding sample text to obtain a first predicted noisy image corresponding to the last step, which can be specifically implemented by the following steps: Inputting the sample noisy image and the sample text into the teacher model to obtain a second predicted noise image corresponding to the first step output by the teacher model; Determine the sample noisy image corresponding to the first step based on the second predicted noise image and the sample noisy image; The sample noisy image corresponding to the first step and the sample text are input into the teacher model to obtain the third predicted noisy image corresponding to the second step output by the teacher model, and based on the third predicted noisy image and the sample noisy image corresponding to the first step, the sample noisy image corresponding to the second step is determined, and the above steps are repeated until the first predicted noisy image corresponding to the last step is obtained.

[0057] For example, the progressive distillation of a preset number of steps of the teacher model based on the sample noisy image and the corresponding sample text can be specifically expressed by the following formulas (6) to (11): in, represents the second predicted noise image corresponding to time step t, that is, the second predicted noise image corresponding to the first step, represents the model parameters of the teacher model (freezable), represents the sample noisy image corresponding to the first step, Represents the time step The corresponding third predicted noise image, that is, the third predicted noise image corresponding to the second step, represents the sample noisy image corresponding to the second step, Indicates the last step The corresponding predicted noise image, Represents the first predicted noisy image corresponding to the last step.

[0058] Assuming n is 2, the original teacher model of N inference steps is gradually distilled into a student model of N / 2. Once the student model converges, the student model continues to be used as the teacher model and the distillation process is repeated until the student model can reach the expected number of inference steps, such as 2 steps, 4 steps, or 8 steps.

[0059] In this embodiment, through progressive distillation, the student model is gradually promoted to use fewer reasoning steps than the teacher model to achieve reasoning results close to or the same as the teacher model, thereby improving the image generation speed and image generation quality of the image generation model trained based on the student model.

[0060] In one embodiment, the image generation method further comprises the following steps: The sample image is input into the pre-trained feature network to obtain a second visual feature output by the pre-trained feature network; the second visual feature is input into the discriminator head to obtain a second prediction probability output by the discriminator head, wherein the second prediction probability is the probability that the sample image is a real image; based on the sample image, the third predicted noisy image, the first prediction probability and the second prediction probability, a discriminant loss is determined; based on the discriminant loss, the network parameters of the discriminator head are updated to obtain the discriminator.

[0061] For example, the discriminant loss can be expressed as follows: : in, represents the first The second visual feature output by the feature layer, Indicates The second prediction probability output by the discriminator head, for example, if the second prediction probability is 1, it means that the sample image is a real image. represents the first The first visual feature output by the feature layer is Indicates The first prediction probability output by the discriminator head is Represents a constant parameter, and R1 represents the R1 gradient penalty loss, which is conducive to stabilizing the generation of adversarial training process. Among them, the distillation loss is calculated in the latent space, while the adversarial loss is calculated in the pixel space. And due to the unique visual Transformer structure in the pre-trained feature network of the discriminator, the reasoning structure of the student model can achieve the same clarity and detail richness of the images output by the teacher model in both the foreground and the background.

[0062] It should be noted that in order to improve the performance of the discriminator, the information of real data (text embedding and image embedding ) as a condition for the discriminator to constrain the discriminator, where the text embedding is composed of sample images The sample text is obtained by pre-training the Chinese-Contrastive Language-Image Pre-training (CN-CLIP) network, and the image embedding is obtained by the sample image The image is obtained through a pre-trained (Depth-Induced Noise for Online Video Self-Supervision, Version 2, DINOv2) network, which is not limited in the present invention.

[0063] In one embodiment, the above step 206 updates the model parameters of the student model based on the distillation loss and the adversarial loss to obtain the image generation model, which can be specifically implemented by the following steps: Obtaining the weight of the adversarial loss; and updating the model parameters of the student model based on the weight of the adversarial loss, the distillation loss, and the adversarial loss to obtain the image generation model.

[0064] For example, when the adversarial loss and the distillation loss are obtained, the total loss can be determined based on the following formula (13): , based on the total loss The model parameters of the student model are updated until the convergence conditions are reached, and finally the image generation model is obtained.

[0065] in, represents the weight of adversarial loss, The larger the value, the more detailed the sampling results of the student model, but at the same time, too large This will lead to the generation of irrelevant details. That is, it is close to the original distillation process. After analyzing and comparing the experimental results, Set to 0.25.

[0066] In this embodiment, based on the weight of the adversarial loss, the distillation loss and the adversarial loss, the model parameters of the student model are updated so that the final image generation model can produce details and clarity closer to the output image of the teacher model, further improving the quality of the generated image.

[0067] Figure 4 is a schematic diagram of the overall flow of the image generation method provided by an embodiment of the present invention, such as Figure 4 As shown in Figure 1, there are mainly three networks involved: the teacher model is a pre-trained diffusion model that requires more inference steps. The model parameters of the teacher model are and is frozen; the student model is composed of parameters The trainable student model initialized by the teacher model has model parameters θ; the trainable discriminator has parameters , which is used to distinguish the authenticity of the sample image and the predicted image corresponding to the student model. The discriminator mainly consists of a frozen pre-trained feature network F and a set of trainable lightweight discriminator heads Composition. The present invention adopts the basic framework of progressive distillation to ensure that the teacher model follows the same ODE process, distills the teacher model into a student model that can be inferred in 8 steps or 4 steps, and trains the student model to better learn the knowledge and patterns of the teacher model. At the same time, the advantages of generative adversarial networks (GANs) in single-step generation are used to introduce adversarial training into the progressive distillation process, namely adversarial progressive distillation. A Wensheng graph model reasoning acceleration algorithm based on adversarial progressive distillation is proposed, the goal of which is to generate high-quality images in as few reasoning steps as possible, improve the impact of the distillation process on clarity, detail richness, etc. on image quality, alleviate the losses caused by the distillation process, and thus improve the speed and quality of image generation by the image generation model. The idea of ​​combining progressive distillation and adversarial training used in the present invention can also be applied to the accelerated training of other model reasoning that do not meet real-time requirements.

[0068] The image generating device provided by the present invention is described below. The image generating device described below and the image generating method described above can be referred to each other.

[0069] Figure 5 is a schematic diagram of the structure of an image generating device provided by an embodiment of the present invention, such as Figure 5 As shown, the image generating device 500 includes an acquiring unit 501 and a generating unit 502; wherein: An acquisition unit 501 is used to acquire a target text; A generation unit 502 is used to input the target text into an image generation model to obtain a target image output by the image generation model, wherein the image generation model is obtained by adversarial training of a student model based on a sample noisy image and a corresponding sample text, and the student model is obtained by progressive distillation training of a teacher model based on the sample noisy image and a corresponding sample text.

[0070] The image generation device provided by the present invention inputs the acquired target text into the image generation model to obtain the target image output by the image generation model, wherein the image generation model is obtained by adversarial training of a student model based on a sample noisy image and a corresponding sample text, and the student model is obtained by progressive distillation training of a teacher model based on a sample noisy image and a corresponding sample text. It can be seen that the present invention combines progressive distillation training with adversarial training, so that the image generation model obtained by training the student model can produce details and clarity closer to the output image of the teacher model, thereby improving the quality of the generated image.

[0071] Based on any of the above embodiments, the image generation model is trained based on the following method: Performing noise processing on the sample image to obtain the sample noisy image; Based on the sample noisy image and the corresponding sample text, the teacher model is progressively distilled for a preset number of steps to obtain a first predicted noisy image corresponding to the last step; Inputting the sample noisy image and the corresponding sample text into the student model to obtain a first predicted noisy image output by the student model, and determining a second predicted noisy image corresponding to the last step based on the first predicted noisy image and the sample noisy image; Determining a distillation loss based on the first predicted noisy image and the second predicted noisy image; Determining an adversarial loss based on the sample noisy image and the first predicted noisy image; Based on the distillation loss and the adversarial loss, the model parameters of the student model are updated to obtain the image generation model.

[0072] Based on any of the foregoing embodiments, determining the adversarial loss based on the sample noisy image and the first predicted noisy image includes: Determine a third predicted noisy image corresponding to the student model based on the sample noisy image and the first predicted noisy image; Inputting the third predicted noisy image into a pre-trained feature network of a discriminator to obtain a first visual feature output by the pre-trained feature network; Inputting the first visual feature into a discriminator head of the discriminator to obtain a first prediction probability output by the discriminator head, wherein the first prediction probability is a probability that the third predicted noisy image is a true image; Based on the first predicted probability, the adversarial loss is determined.

[0073] Based on any of the above embodiments, the step of performing progressive distillation of a preset number of steps on the teacher model based on the sample noisy image and the corresponding sample text to obtain a first predicted noisy image corresponding to the last step includes: Inputting the sample noisy image and the sample text into the teacher model to obtain a second predicted noise image corresponding to the first step output by the teacher model; Determine the sample noisy image corresponding to the first step based on the second predicted noise image and the sample noisy image; The sample noisy image corresponding to the first step and the sample text are input into the teacher model to obtain the third predicted noisy image corresponding to the second step output by the teacher model, and based on the third predicted noisy image and the sample noisy image corresponding to the first step, the sample noisy image corresponding to the second step is determined, and the above steps are repeated until the first predicted noisy image corresponding to the last step is obtained.

[0074] Based on any of the above embodiments, the image generating device 500 further includes: Inputting the sample image into the pre-trained feature network to obtain a second visual feature output by the pre-trained feature network; Inputting the second visual feature into the discriminator head to obtain a second prediction probability output by the discriminator head, wherein the second prediction probability is a probability that the sample image is a real image; Determining a discrimination loss based on the sample image, the third predicted noisy image, the first predicted probability, and the second predicted probability; The network parameters of the discriminator head are updated based on the discriminant loss to obtain the discriminator.

[0075] Based on any of the above embodiments, the updating of the model parameters of the student model based on the distillation loss and the adversarial loss to obtain the image generation model includes: Obtaining a weight of the adversarial loss; Based on the weight of the adversarial loss, the distillation loss and the adversarial loss, the model parameters of the student model are updated to obtain the image generation model.

[0076] Figure 6 is a schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention, such as Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other via the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the image generation method, which includes: obtaining the target text; The target text is input into an image generation model to obtain a target image output by the image generation model, wherein the image generation model is obtained by adversarial training of a student model based on a sample noisy image and a corresponding sample text, and the student model is obtained by progressive distillation training of a teacher model based on the sample noisy image and a corresponding sample text.

[0077] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0078] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored in a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the image generation method provided by the above methods, the method includes: obtaining a target text; The target text is input into an image generation model to obtain a target image output by the image generation model, wherein the image generation model is obtained by adversarial training of a student model based on a sample noisy image and a corresponding sample text, and the student model is obtained by progressive distillation training of a teacher model based on the sample noisy image and a corresponding sample text.

[0079] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, the computer program being implemented when executed by a processor to execute the image generation method provided by the above methods, the method comprising: obtaining a target text; The target text is input into an image generation model to obtain a target image output by the image generation model, wherein the image generation model is obtained by adversarial training of a student model based on a sample noisy image and a corresponding sample text, and the student model is obtained by progressive distillation training of a teacher model based on the sample noisy image and a corresponding sample text.

[0080] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0081] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0082] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image generation method, characterized in that: include: Get the target text; The target text is input into an image generation model to obtain a target image output by the image generation model, wherein the image generation model is obtained by adversarial training of a student model based on a sample noisy image and a corresponding sample text, and the student model is obtained by progressive distillation training of a teacher model based on the sample noisy image and a corresponding sample text.

2. The image generation method according to claim 1, characterized in that: The image generation model is trained based on the following method: Performing noise processing on the sample image to obtain the sample noisy image; Based on the sample noisy image and the corresponding sample text, the teacher model is progressively distilled for a preset number of steps to obtain a first predicted noisy image corresponding to the last step; Inputting the sample noisy image and the corresponding sample text into the student model to obtain a first predicted noisy image output by the student model, and determining a second predicted noisy image corresponding to the last step based on the first predicted noisy image and the sample noisy image; Determining a distillation loss based on the first predicted noisy image and the second predicted noisy image; Determining an adversarial loss based on the sample noisy image and the first predicted noisy image; Based on the distillation loss and the adversarial loss, the model parameters of the student model are updated to obtain the image generation model.

3. The image generation method according to claim 2, characterized in that: The determining the adversarial loss based on the sample noisy image and the first predicted noisy image includes: Determine a third predicted noisy image corresponding to the student model based on the sample noisy image and the first predicted noisy image; Inputting the third predicted noisy image into a pre-trained feature network of a discriminator to obtain a first visual feature output by the pre-trained feature network; Inputting the first visual feature into a discriminator head of the discriminator to obtain a first prediction probability output by the discriminator head, wherein the first prediction probability is a probability that the third predicted noisy image is a true image; Based on the first predicted probability, the adversarial loss is determined.

4. The image generation method according to claim 2, characterized in that: The step of performing a preset number of steps of progressive distillation on the teacher model based on the sample noisy image and the corresponding sample text to obtain a first predicted noisy image corresponding to the last step includes: Inputting the sample noisy image and the sample text into the teacher model to obtain a second predicted noise image corresponding to the first step output by the teacher model; Determine the sample noisy image corresponding to the first step based on the second predicted noise image and the sample noisy image; The sample noisy image corresponding to the first step and the sample text are input into the teacher model to obtain the third predicted noisy image corresponding to the second step output by the teacher model, and based on the third predicted noisy image and the sample noisy image corresponding to the first step, the sample noisy image corresponding to the second step is determined, and the above steps are repeated until the first predicted noisy image corresponding to the last step is obtained.

5. The image generation method according to claim 3, characterized in that: The method further comprises: Inputting the sample image into the pre-trained feature network to obtain a second visual feature output by the pre-trained feature network; Inputting the second visual feature into the discriminator head to obtain a second prediction probability output by the discriminator head, wherein the second prediction probability is a probability that the sample image is a real image; Determining a discrimination loss based on the sample image, the third predicted noisy image, the first predicted probability, and the second predicted probability; The network parameters of the discriminator head are updated based on the discriminant loss to obtain the discriminator.

6. The image generation method according to claim 2, characterized in that: The updating of the model parameters of the student model based on the distillation loss and the adversarial loss to obtain the image generation model includes: Obtaining a weight of the adversarial loss; Based on the weight of the adversarial loss, the distillation loss and the adversarial loss, the model parameters of the student model are updated to obtain the image generation model.

7. An image generating device, characterized in that: include: An acquisition unit, used for acquiring a target text; A generation unit is used to input the target text into an image generation model to obtain a target image output by the image generation model, wherein the image generation model is obtained by adversarial training of a student model based on a sample noisy image and a corresponding sample text, and the student model is obtained by progressive distillation training of a teacher model based on the sample noisy image and the corresponding sample text.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the image generating method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image generating method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the image generating method according to any one of claims 1 to 6 is implemented.

Images