Training method of image generation model and image generation method and device

Abstract

The invention relates to a training method of an image generation model and an image generation method and device.The method comprises the steps that a first sample noise adding image and a first sample description text are obtained, the first sample description text is input into a preset text encoder for text encoding, and a first global semantic feature and a first character semantic feature are obtained; generating a first sample fusion condition feature based on the first global semantic feature and the sample time step; inputting the first character semantic feature, the first sample fusion condition feature and the first sample noise adding image into a preset noise predictor for noise prediction processing to obtain first prediction noise; performing denoising processing on the first sample noise-added image based on a denoising network to obtain a predicted image; and according to the difference between the first prediction noise and the sample noise label and the difference between the prediction image and the first sample image label, training the preset image generation model to obtain the target image generation model, thereby improving the semantic following capability and accuracy of the image.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a training method for an image generation model, an image generation method and a device. Background Art

[0002] The text-generated graph model can generate a target image corresponding to the description text by understanding the input description text. The text-generated graph model in the prior art can be based on a diffusion model, by predicting noise and then denoising the random noise image to obtain the target image. It can also be based on a generative adversarial model, in which the generator and the discriminator jointly determine the target image. However, no matter which image generation method is used, the text needs to be understood. However, most of the existing text encoders are pre-trained and cannot be fully adapted to the text-generated graph task, resulting in poor semantic following ability and insufficient accuracy of the images generated by the trained image generation model. Summary of the invention

[0003] The present disclosure provides a training method, an image generation method and a device for an image generation model, so as to at least solve the problem that the image generated by the image generation model in the related art has poor semantic following ability and insufficient accuracy. The technical solution of the present disclosure is as follows:

[0004] According to a first aspect of an embodiment of the present disclosure, a method for training an image generation model is provided, comprising:

[0005] Acquire a first sample noisy image and a first sample description text, wherein the first sample description text is a description text of the first sample image obtained by denoising the first sample noisy image based on sample noise; the first sample noisy image is annotated with a sample noise label and a first sample image label;

[0006] Inputting the first sample description text into a preset text encoder of a preset image generation model for text encoding to obtain a first global semantic feature and a first character semantic feature corresponding to the first sample description text;

[0007] Generate a first sample fusion condition feature based on the first global semantic feature and the sample time step corresponding to the sample noise; the sample time step is the time step corresponding to the sample noise in the sample time step sequence; the sample time step sequence is used to gradually superimpose noise to change the clear image into a random noise image;

[0008] Inputting the first character semantic feature, the first sample fusion condition feature and the first sample noisy image into a preset noise predictor of the preset image generation model for noise prediction processing to obtain a first predicted noise;

[0009] Based on a denoising network of the preset image generation model, denoising the first sample noisy image according to the first predicted noise to obtain a predicted image;

[0010] The preset image generation model is trained according to the difference between the first predicted noise and the sample noise label and the difference between the predicted image and the first sample image label, and the preset image generation model at the end of the training is determined as the target image generation model.

[0011] In an exemplary embodiment, the training of the preset image generation model according to the difference between the first predicted noise and the sample noise label and the difference between the predicted image and the first sample image label, and determining the preset image generation model at the end of the training as the target image generation model includes:

[0012] Determining a comprehensive loss value according to a difference between the first predicted noise and the sample noise label and a difference between the predicted image and the first sample image label;

[0013] Based on the comprehensive loss value, the preset text encoder and the preset noise predictor are trained, and the preset text encoder at the end of the training is determined as a target text encoder, and the preset noise predictor at the end of the training is determined as a first target noise predictor;

[0014] The target text encoder, the first target noise predictor and the denoising network are combined to obtain the target image generation model.

[0015] In an exemplary embodiment, the step of training the preset text encoder and the preset noise predictor based on the comprehensive loss value, determining the preset text encoder at the end of the training as a target text encoder, and determining the preset noise predictor at the end of the training as a first target noise predictor includes:

[0016] Based on the comprehensive loss value, the preset text encoder is updated to obtain an updated text encoder, and the updated text encoder is re-determined as the preset text encoder; and based on the comprehensive loss value, the preset noise predictor is updated to obtain an updated noise predictor, and the updated noise predictor is re-determined as the preset noise predictor;

[0017] Jump to the step of inputting the first sample description text into a preset text encoder for text encoding to obtain a first global semantic feature and a first character semantic feature corresponding to the first sample description text;

[0018] When a preset training condition is met, the preset text encoder at the end of training is determined as the target text encoder, and the preset noise predictor at the end of training is determined as the first target noise predictor.

[0019] In an exemplary embodiment, determining the comprehensive loss value according to the difference between the first predicted noise and the sample noise label and the difference between the predicted image and the first sample image label includes:

[0020] Determining a first loss value according to a difference between the first predicted noise and the sample noise label;

[0021] Determining a second loss value according to a difference between the predicted image and the first sample image label;

[0022] The comprehensive loss value is determined based on the first loss value and the second loss value.

[0023] In an exemplary embodiment, after determining the preset text encoder at the end of training as the target text encoder and determining the preset noise predictor at the end of training as the first target noise predictor, the method further includes:

[0024] Acquire a second sample noisy image and a second sample description text, wherein the second sample description text is a description text of the second sample image obtained by denoising the second sample noisy image based on the sample noise; the second sample noisy image is annotated with a sample noise label; and the resolution of the second sample noisy image is greater than that of the first sample noisy image;

[0025] Freezing the parameters of the target text encoder, and inputting the second sample description text into the target text encoder for text encoding to obtain a second global semantic feature and a second character semantic feature corresponding to the second sample description text;

[0026] Generate a second sample fusion condition feature based on the second global semantic feature and the sample time step corresponding to the sample noise;

[0027] Inputting the second character semantic feature, the second sample fusion condition feature and the second sample noisy image into the first target noise predictor for noise prediction processing to obtain a second predicted noise;

[0028] training the first target noise predictor according to the difference between the second predicted noise and the sample noise label, and determining the first target noise predictor at the end of the training as the second target noise predictor;

[0029] The target text encoder, the second target noise predictor and the denoising network are combined to obtain the target image generation model.

[0030] In an exemplary embodiment, generating a first sample fusion condition feature based on the first global semantic feature and the sample time step corresponding to the sample noise includes:

[0031] Performing position encoding processing on the sample time step to obtain initial sample time step features;

[0032] Acquire image shape information corresponding to the first sample noisy image, and perform position encoding processing on the image shape information to obtain initial sample shape features;

[0033] The first sample fusion condition feature is generated based on the initial sample time step feature, the first global semantic feature and the initial sample shape feature.

[0034] In an exemplary embodiment, the acquiring image shape information corresponding to the first sample noisy image and performing position encoding processing on the image shape information to obtain initial sample shape features includes:

[0035] Acquire sample width information and sample height information of the first sample noisy image, and target width information and target height information of a target image; the target image represents an image generated by the target image generation model;

[0036] Acquire clipping coordinate information; the clipping coordinate information represents the position coordinates of the preset vertices of the target image in the first sample noisy image;

[0037] Performing position encoding processing on the sample width information and the sample height information to obtain sample width and height features;

[0038] Performing position encoding processing on target width information and target height information of the target image to obtain target width and height features;

[0039] Performing position encoding processing on the clipping coordinate information to obtain clipping features;

[0040] The sample width and height features, the target width and height features and the cropping features are subjected to feature fusion processing to obtain the initial sample shape features.

[0041] In an exemplary embodiment, generating the first sample fusion condition feature based on the initial sample time step feature, the first global semantic feature and the initial sample shape feature includes:

[0042] Mapping the initial sample time step feature to a preset dimension to obtain a sample time step feature;

[0043] Performing feature concatenation processing on the initial sample shape feature and the first global semantic feature to obtain an initial condition feature;

[0044] Mapping the initial condition feature to the preset dimension to obtain a sample condition feature;

[0045] The sample condition feature and the sample time step feature are subjected to feature fusion processing to obtain the first sample fusion condition feature.

[0046] In an exemplary embodiment, the method further comprises:

[0047] Acquire a sample time step sequence, wherein the sample time step sequence includes a plurality of sample time steps that increase in sequence;

[0048] For each of the sample time steps in the sample time step sequence, determine the noise intensity parameter corresponding to each of the sample time steps according to a preset mapping relationship; the preset mapping relationship represents a correspondence between a preset time step and a preset noise intensity parameter;

[0049] Determining the sample noise according to the noise intensity parameter;

[0050] The sample noise is superimposed on the first sample image to obtain the first sample noisy image corresponding to the sample time step.

[0051] According to a second aspect of an embodiment of the present disclosure, there is provided an image generating method, comprising:

[0052] Acquire a target description text; the target description text is used to prompt the target image generation model to output a target image corresponding to the target description text;

[0053] The random noise image and the target description text are input into a target image generation model for image generation processing to obtain a target image corresponding to the target description text; the target image generation model is trained according to the image generation model training method described in the first aspect.

[0054] In an exemplary embodiment, the target image generation model includes a target text encoder, a second target noise predictor and a denoising network; the random noise image and the target description text are input into the target image generation model for image generation processing to obtain a target image corresponding to the target description text, including:

[0055] Obtain a target time step sequence, wherein the target time step sequence includes multiple time steps, and the multiple time steps are arranged from large to small;

[0056] Determine the random noise image as an initial image, and determine the first time step in the time step sequence as a target time step;

[0057] Inputting the target time step, the target description text and the initial image into the second target noise predictor for noise prediction processing to obtain target prediction noise;

[0058] Based on the denoising network, the initial image is denoised according to the target predicted noise to obtain an intermediate image, and the intermediate image is re-determined as the initial image, and the next time step in the time step sequence is determined as the target time step;

[0059] When the target time step is greater than the preset time step, jump to the step of inputting the target time step, the target description text, the target shape information and the initial image into the second target noise predictor for noise prediction processing to obtain target prediction noise;

[0060] When the target time step is equal to the preset time step, the intermediate image corresponding to the preset time step is determined as the target image corresponding to the target description text.

[0061] In an exemplary embodiment, the inputting the target time step, the target description text and the initial image into the second target noise predictor for noise prediction processing to obtain target predicted noise includes:

[0062] Inputting the target description text into the target text encoder for text encoding to obtain the target global semantic features and target character semantic features corresponding to the target description text;

[0063] Generate a target fusion condition feature based on the target global semantic feature and the target time step;

[0064] The target character semantic feature, the target fusion condition feature and the initial image are input into the second target noise predictor for noise prediction processing to obtain the target predicted noise.

[0065] According to a third aspect of an embodiment of the present disclosure, a training device for an image generation model is provided, comprising:

[0066] A first acquisition module is configured to acquire a first sample noisy image and a first sample description text, wherein the first sample description text is a description text of the first sample image obtained by denoising the first sample noisy image based on sample noise; the first sample noisy image is annotated with a sample noise label and a first sample image label;

[0067] A text encoding module is configured to perform text encoding by inputting the first sample description text into a preset text encoder of a preset image generation model to obtain a first global semantic feature and a first character semantic feature corresponding to the first sample description text;

[0068] A fusion module is configured to generate a first sample fusion condition feature based on the first global semantic feature and a sample time step corresponding to the sample noise; the sample time step is a time step corresponding to the sample noise in a sample time step sequence; the sample time step sequence is used to gradually superimpose noise to change a clear image into a random noise image;

[0069] A noise prediction module is configured to perform noise prediction processing by inputting the first character semantic feature, the first sample fusion condition feature and the first sample noisy image into a preset noise predictor of the preset image generation model to obtain a first predicted noise;

[0070] a denoising module, configured to execute a denoising network based on the preset image generation model, and to denoise the first sample noisy image according to the first predicted noise to obtain a predicted image;

[0071] The target image generation model determination module is configured to perform training on the preset image generation model according to the difference between the first predicted noise and the sample noise label and the difference between the predicted image and the first sample image label, and determine the preset image generation model at the end of the training as the target image generation model.

[0072] According to a fourth aspect of an embodiment of the present disclosure, there is provided an image generating device, including:

[0073] A first target description text acquisition module is configured to execute acquisition of a target description text; the target description text is used to prompt the target image generation model to output a target image corresponding to the target description text;

[0074] The image generation module is configured to execute image generation processing by inputting a random noise image and the target description text into a target image generation model to obtain a target image corresponding to the target description text; the target image generation model is trained according to the training method of the image generation model described in the first aspect.

[0075] According to a fifth aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0076] processor;

[0077] a memory for storing instructions executable by the processor;

[0078] Wherein, the processor is configured to execute the instructions to implement the training method of the image generation model or the image generation method as described above.

[0079] According to a sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the training method of the image generation model or the image generation method as described above.

[0080] According to a seventh aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program / instruction, which, when executed by a processor, implements the training method of the image generation model or the image generation method as described above.

[0081] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0082] In the disclosed embodiment, a first sample noisy image and a first sample description text are obtained, and then the first sample description text is input into a preset text encoder of a preset image generation model for text encoding, so as to obtain a first global semantic feature and a first character semantic feature corresponding to the first sample description text; the first character semantic feature is used to represent the text feature of the description text, and then a first sample fusion condition feature is generated based on the first global semantic feature and the sample time step corresponding to the sample noise, so that a preset noise predictor can perform noise prediction according to the main constraint of the first character semantic feature and the additional constraint of the first sample fusion condition feature, and then, based on a denoising network of the preset image generation model, the first sample noisy image is denoised according to the first predicted noise to obtain a predicted image; according to the difference between the first predicted noise and the sample noise label and the difference between the predicted image and the first sample image label, the preset image generation model is trained as a whole, and the preset image generation model at the end of the training is determined as the target image generation model. The method of the disclosed embodiment not only trains the noise predictor, but also trains the text encoder, so that the text encoder can not only learn how to understand the meaning of the descriptive text itself, but also learn how to encode the text more efficiently so that the subsequent noise predictor can predict more accurate noise based on the features output by the text encoder, thereby making the trained image generation model more accurate and improving the semantic following ability and accuracy of the generated image.

[0083] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0085] Figure 1 It is an application environment diagram of a training method for an image generation model according to an exemplary embodiment.

[0086] Figure 2 The figure is a flowchart of a method for training an image generation model according to an exemplary embodiment.

[0087] Figure 3 The flowchart of obtaining a first sample noisy image is shown according to an exemplary embodiment.

[0088] Figure 4 It is a flow chart of a method for generating first sample fusion condition features according to an exemplary embodiment.

[0089] Figure 5 It is a flow chart of a method for training a preset image generation model according to an exemplary embodiment.

[0090] Figure 6 The present invention is a flow chart of a method for training a preset text encoder and a preset noise predictor based on a comprehensive loss value according to an exemplary embodiment.

[0091] Figure 7 It is a schematic diagram of a method for training a target image generation model according to an exemplary embodiment.

[0092] Figure 8 The figure is a flowchart of an image generating method according to an exemplary embodiment.

[0093] Fig. 9 The figure is a flowchart of an image generating method according to an exemplary embodiment.

[0094] Fig.10 It is a schematic structural diagram of a target image generation model according to an exemplary embodiment.

[0095] Fig.11 The figure is a block diagram of a training device for an image generation model according to an exemplary embodiment.

[0096] Fig.12 The figure is a block diagram of an image generating device according to an exemplary embodiment.

[0097] Fig.13 It is a block diagram of an electronic device according to an exemplary embodiment.

[0098] Fig.14 It is a block diagram of an electronic device for training an image generation model according to an exemplary embodiment. DETAILED DESCRIPTION

[0099] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0100] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0101] Figure 1 FIG. 1 is an application environment diagram of a training method for an image generation model according to an exemplary embodiment. Figure 1 As shown, the application environment may include a server 01 and a client 02 .

[0102] Specifically, in the disclosed embodiment, the server 01 may include an independently operated server, or a distributed server, or a server cluster composed of multiple servers, and may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Server 01 may include a network communication unit, a processor, a memory, and the like. Specifically, the server 01 may be used to train a target image generation model.

[0103] Specifically, in the disclosed embodiment, the client 02 may include physical devices such as smart phones, desktop computers, tablet computers, laptop computers, digital assistants, smart wearable devices, and vehicle-mounted terminals, and may also include software running in physical devices, such as web pages provided by some service providers to users, or applications provided by these service providers to users. Specifically, the client 02 may be used to generate a model based on a target image, obtain a target description text, and thus quickly generate a target image corresponding to the target description text.

[0104] Figure 2FIG. 1 is a flowchart of a method for training an image generation model according to an exemplary embodiment. Figure 2 As mentioned above, this method can be applied to Figure 1 The server 01 shown may include the following steps:

[0105] In step S201, a first sample noisy image and a first sample description text are obtained;

[0106] The first sample description text is a description text of the first sample image obtained by denoising the first sample noisy image based on the sample noise; the first sample noisy image is annotated with a sample noise label and a first sample image label.

[0107] In one example, it is assumed that the first sample noisy image may be an image after random noise is superimposed on a clear image with the content of "blue sky and white clouds". The superimposed random noise may be sample noise, and the clear image may be the first sample image. After the first sample noisy image is denoised by sample noise, the first sample image may be obtained, and the corresponding first sample description text may be "blue sky and white clouds".

[0108] In step S203, the first sample description text is input into a preset text encoder of a preset image generation model for text encoding to obtain a first global semantic feature and a first character semantic feature corresponding to the first sample description text.

[0109] The preset text encoder is used to perform text encoding on the first sample description text to achieve semantic understanding, so that the image generation model outputs an image corresponding to the first sample description text.

[0110] In the disclosed embodiment, the preset text encoder can be a Chinese Large Language Model (LLM). Compared with the text noise problem and the lack of long text training corpus caused by the text encoding using the Contrastive Language-Image Pretraining (CLIP) text encoder in the traditional solution, the Chinese Large Language Model is introduced in the disclosed embodiment, which can support longer Chinese text lengths, can further understand more detailed and complex description texts, and can significantly improve the image generation model's ability to understand the complex semantics of long texts and its ability to follow complex semantics. It should be noted that the disclosed embodiment does not limit the specific model name of the Chinese Large Language Model.

[0111] The first global semantic feature can be understood as a sentence-level semantic feature. In an example, assuming that the first sample description text is "red apple", then the first global semantic feature is the feature obtained by extracting features from the entire first sample description text; the first character semantic feature can be understood as a character-level semantic feature. Still taking the first sample description text as "red apple" as an example, it is first divided into multiple characters (Tokens), such as "red", "color", "of", "apple" and "fruit", and then these characters are encoded respectively to obtain intermediate character codes, and then the intermediate character codes are fused with position features according to the positions of the characters in the first sample description text, thereby obtaining the first character semantic feature. It should be noted that the character division method and the specific implementation method of obtaining the first character semantic feature in the disclosed embodiment are not limited.

[0112] In step S205, a first sample fusion condition feature is generated based on the first global semantic feature and the sample time step corresponding to the sample noise; the sample time step is the time step corresponding to the sample noise in the sample time step sequence; the sample time step sequence is used to gradually superimpose noise to change the clear image into a random noise image.

[0113] In the disclosed embodiment, the sample noise corresponds to a time step (TimeStep). By introducing the time step, the image generation model can understand the position of the first sample noisy image in the process of gradually adding noise to change the clear image into a random noise image.

[0114] It should be noted that the first sample image can be regarded as the image corresponding to time step 0, that is, the clear image, and the random noise image can be regarded as the image corresponding to time step 1000, that is, the random noise image. At different time steps, the corresponding sample noise is different. The larger the time step, the greater the noise intensity of the sample noise, and the corresponding first sample noisy image is closer to the random noise image.

[0115] A first sample fusion condition feature can be generated based on the first global semantic feature and the sample time step corresponding to the sample noise, which is used to guide a subsequent preset noise predictor to predict the noise corresponding to the first sample fusion condition feature, thereby generating a corresponding image.

[0116] In step S207, the first character semantic feature, the first sample fusion condition feature and the first sample noisy image are input into a preset noise predictor of a preset image generation model for noise prediction processing to obtain a first predicted noise.

[0117] In one embodiment, the preset noise predictor may be a U-shaped Network (UNet), which outputs the first predicted noise by extracting features from the first character semantic feature, the first sample fusion condition feature, and the first sample noisy image and performing upsampling.

[0118] In one embodiment, the first character semantic feature and the first sample fusion condition feature can be fused with the feature of the first sample noisy image through a cross attention structure, and then a preset noise predictor can be used to perform noise prediction based on the fused features.

[0119] In step S209, based on a denoising network of a preset image generation model, the first sample noisy image is denoised according to the first predicted noise to obtain a predicted image.

[0120] The denoising network can be understood as a subtraction network. Based on the denoising network, the first prediction noise can be removed from the first sample noisy image to obtain a predicted image.

[0121] In step S211, the preset image generation model is trained according to the difference between the first predicted noise and the sample noise label and the difference between the predicted image and the first sample image label, and the preset image generation model at the end of the training is determined as the target image generation model.

[0122] According to the difference between the first predicted noise output by the preset image generation model and the sample noise label, and the difference between the predicted image and the first sample image label, the preset image generation model can be trained based on the back propagation algorithm to obtain a target image generation model. The target image generation model can generate a target image corresponding to any description text based on the description text.

[0123] In the disclosed embodiment, a first sample noisy image and a first sample description text are obtained, and then the first sample description text is input into a preset text encoder of a preset image generation model for text encoding, so as to obtain a first global semantic feature and a first character semantic feature corresponding to the first sample description text; the first character semantic feature is used to represent the text feature of the description text, and then a first sample fusion condition feature is generated based on the first global semantic feature and the sample time step corresponding to the sample noise, so that a preset noise predictor can perform noise prediction according to the main constraint of the first character semantic feature and the additional constraint of the first sample fusion condition feature, and then, based on a denoising network of the preset image generation model, the first sample noisy image is denoised according to the first predicted noise to obtain a predicted image; according to the difference between the first predicted noise and the sample noise label and the difference between the predicted image and the first sample image label, the preset image generation model is trained as a whole, and the preset image generation model at the end of the training is determined as the target image generation model. The method of the disclosed embodiment not only trains the noise predictor, but also trains the text encoder, so that the text encoder can not only learn how to understand the meaning of the descriptive text itself, but also learn how to encode the text more efficiently so that the subsequent noise predictor can predict more accurate noise based on the features output by the text encoder, thereby making the trained image generation model more accurate and improving the semantic following ability and accuracy of the generated image.

[0124] Figure 3 The flowchart of obtaining a first sample noisy image is shown according to an exemplary embodiment.

[0125] In step S301 , a sample time step sequence is obtained.

[0126] Wherein, the sample time step sequence includes a plurality of sample time steps which increase in sequence;

[0127] The introduction of the sample time step sequence can be used to simulate the process of gradually adding noise disturbance to the clear image to obtain a random noise image. This process can be understood as the forward diffusion process of the diffusion model.

[0128] In one example, assuming that for a clear image, a sample time step sequence is set to include K sample time steps, and a certain intensity of noise is added to each step. If K is large enough, the clear image can be converted into a random noise image. In one example, K can be 1000, and the sample time steps are 0, 1, 2...999 from small to large. It should be noted that the specific value of K is not limited in this application.

[0129] In step S303, for each sample time step in the sample time step sequence, a noise intensity parameter corresponding to each sample time step is determined according to a preset mapping relationship.

[0130] The preset mapping relationship represents the corresponding relationship between the preset time step and the preset noise intensity parameter.

[0131] Since the sample noises all conform to the Gaussian distribution, a corresponding relationship between the preset time step and the preset noise intensity parameter, ie, a preset mapping relationship, can be established. The preset noise intensity parameters can be two parameters, the mean and the variance.

[0132] It can be understood that, in the preset mapping relationship, the larger the preset time step is, the larger the preset noise intensity parameter is.

[0133] In step S305, the sample noise is determined according to the noise intensity parameter.

[0134] Since the sample noises all conform to the Gaussian distribution, after determining the noise intensity parameter, the corresponding Gaussian distribution, ie, the sample noise, can be determined.

[0135] In step S307, the sample noise is superimposed on the first sample image to obtain a first sample noisy image corresponding to the sample time step.

[0136] In one embodiment, the first sample noisy image may be determined by the following formula:

[0137]

[0138] ε~N(0,1)

[0139] =1-βt

[0140]

[0141] Among them, x t represents the first sample noisy image, x0 represents the first sample image, ε represents a Gaussian distribution with a mean of 0 and a variance of 1, t represents the sample time step, β is a hyperparameter, and a is a hyperparameter.

[0142] In the disclosed embodiment, the forward diffusion process is used to realize the gradual noise addition process of the clear image, so that the subsequent noise prediction model can learn the characteristics of the sample noise and the sample noise image corresponding to each time step, so that the target image can be gradually restored from the random noise image in the application stage of the model. On the other hand, the superimposed noise method also realizes the expansion of the sample image, enhancing the prediction ability and image generation ability of the model.

[0143] Figure 4 It is a flow chart of a method for generating first sample fusion condition features according to an exemplary embodiment.

[0144] In step 401, position encoding is performed on the sample time step to obtain initial sample time step features.

[0145] The sample time step is position-encoded (position embedding) so that the subsequent preset noise predictor can understand different time steps and predict the predicted noise at the time step. In an example, assuming that the time step is 14, the initial sample time step feature obtained by position encoding can be an embedding vector of "0.12, 0.31, 0.34...0.02". By introducing the sample time step, the model learns how to denoise the sample first sample noisy image at each time step, and uses time step encoding to help understand the current denoising state.

[0146] It should be noted that the present application does not limit the specific implementation method of position coding processing, for example, it can be sine-cosine coding, Fourier coding, etc.

[0147] In step 403, image shape information corresponding to the first sample noisy image is obtained, and position encoding processing is performed on the image shape information to obtain initial sample shape features.

[0148] In the disclosed embodiments, the relationship between the image size used for training and the size of the model output image is further considered, so that the trained target image generation model can learn the output requirements of different image sizes.

[0149] In some embodiments, step 403 may include:

[0150] First, the sample width information and sample height information of the first sample noisy image, the target width information and target height information of the target image are obtained; the target image represents an image generated by the target image generation model. Next, the cropping coordinate information is obtained; the cropping coordinate information represents the position coordinates of the preset vertex of the target image in the first sample noisy image.

[0151] The sample width information and sample height information are the width and height of the first sample image or the first sample noisy image used for training. Then, when applying, the user may require the output of target images of different sizes. Therefore, the disclosed embodiment further introduces the target width and target height of the target image. It should be noted that the target width and target height may be different from the sample width and sample height, or they may be the same. When the two are different, the position coordinates of the preset vertices of the target image in the first sample noisy image are determined as the cropping coordinate information, so that the image features of the cropped first sample noisy image are subsequently learned according to the cropping coordinate information and denoising is performed on the cropped first sample noisy image based on the first predicted noise.

[0152] The preset vertex may be the upper left vertex of the target image, or may be other vertices.

[0153] Then, the sample width information and sample height information are position-encoded to obtain the sample width and height features. The target width information and target height information of the target image are position-encoded to obtain the target width and height features. The cropping coordinate information is position-encoded to obtain the cropping features. The sample width and height features, target width and height features, and cropping features are feature-fused to obtain the initial sample shape features.

[0154] Similar to obtaining the initial sample time step features by position encoding the time step, the disclosed embodiment can obtain the sample width and height features, the target width and height features, and the cropping features, and then perform feature fusion on these three features to obtain the initial sample shape features. In one example, the feature fusion method can be feature splicing of the sample width and height features, the target width and height features, and the cropping features.

[0155] In the embodiment of the present disclosure, by introducing image shape information, the generation of images with different resolutions is taken into account, thereby meeting the diverse image size generation requirements.

[0156] In step 405, a first sample fusion condition feature is generated based on the initial sample time step feature, the first global semantic feature and the initial sample shape feature.

[0157] In some embodiments, step S405 may include:

[0158] First, the initial sample time step features are mapped to a preset dimension to obtain the sample time step features. The initial sample time step features can be mapped to the preset dimension by linear mapping to facilitate subsequent feature concatenation and fusion.

[0159] Then, the initial sample shape feature and the first global semantic feature are subjected to feature concatenation to obtain an initial condition feature. The initial condition feature is also mapped to a preset dimension to obtain a sample condition feature.

[0160] Then, the sample conditional features and the sample time step features are fused to obtain the first sample fusion conditional features. By performing feature splicing processing on the initial sample shape features and the first global semantic features, the generation of images with different resolutions and the global semantic information of the input description text are taken into account during the training process. After that, the sample conditional features and the sample time step features are fused to generate the first sample fusion conditional features as constraints to control the image generation process.

[0161] In the disclosed embodiment, the first sample fusion condition feature is generated by the initial sample time step feature, the first global semantic feature and the initial sample shape feature, and the conditional restrictions such as the learning time step, semantics and image shape are implemented so that the noise predictor can output the predicted noise under the condition according to the first sample fusion condition feature, thereby achieving accurate prediction of the noise.

[0162] Figure 5 It is a flow chart of a method for training a preset image generation model according to an exemplary embodiment.

[0163] In step 501, a comprehensive loss value is determined based on the difference between the first predicted noise and the sample noise label and the difference between the predicted image and the first sample image label.

[0164] In some embodiments, determining a comprehensive loss value based on the difference between the first predicted noise and the sample noise label and the difference between the predicted image and the first sample image label may include: determining a first loss value based on the difference between the first predicted noise and the sample noise label; determining a second loss value based on the difference between the predicted image and the first sample image label; and determining a comprehensive loss value based on the first loss value and the second loss value.

[0165] It should be noted that the mean square error loss (MSE) can be calculated based on the first prediction noise and the sample noise label to obtain a first loss value to regress the first prediction noise. Similarly, the mean square error loss can be determined based on the prediction image and the first sample image label to obtain a second loss value. Then, the first loss value and the second loss value are weighted and summed according to the preset weights to obtain a comprehensive loss value.

[0166] In the disclosed embodiment, comprehensive loss information is determined quickly and accurately through the difference between the first predicted noise and the sample noise label and the difference between the predicted image and the first sample image label, thereby facilitating model training.

[0167] In step 503, based on the comprehensive loss value, the preset text encoder and the preset noise predictor are trained, and the preset text encoder at the end of the training is determined as the target text encoder, and the preset noise predictor at the end of the training is determined as the first target noise predictor.

[0168] In step 505, the target text encoder, the first target noise predictor and the denoising network are combined to obtain a target image generation model.

[0169] The disclosed embodiment can be regarded as a complete training stage, by simultaneously training the preset text encoder and the preset noise predictor, and using the comprehensive loss value to update the parameters of the preset text encoder and the preset noise predictor, so that the training is more closed, and then the target text encoder, the first target noise predictor and the denoising network trained in this training stage are combined to obtain a target image generation model, thereby improving the semantic following ability of the target image generation model, so that the generated target image is more consistent with the description text.

[0170] Figure 6 The present invention is a flowchart of a method for training a preset text encoder and a preset noise predictor based on a comprehensive loss value according to an exemplary embodiment. In some embodiments, based on the comprehensive loss value, the preset text encoder and the preset noise predictor are trained, and the preset text encoder at the end of the training is determined as the target text encoder, and the preset noise predictor at the end of the training is determined as the first target noise predictor, including:

[0171] In step 601, the preset text encoder is updated based on the comprehensive loss value to obtain an updated text encoder, and the updated text encoder is re-determined as the preset text encoder; and the preset noise predictor is updated based on the comprehensive loss value to obtain an updated noise predictor, and the updated noise predictor is re-determined as the preset noise predictor.

[0172] Based on the comprehensive loss value, the preset text encoder is updated by using the back propagation algorithm to obtain an updated text encoder, and the preset noise predictor is updated to obtain an updated noise predictor.

[0173] In step 603, the process jumps to the step of inputting the first sample description text into a preset text encoder for text encoding to obtain the first global semantic feature and the first character semantic feature corresponding to the first sample description text, that is, step S203.

[0174] Since the parameters of the preset text encoder have been updated at this time, the first sample description text needs to be re-encoded after each parameter update to obtain the updated first global semantic feature and first character semantic feature, and input them into the updated preset noise predictor for noise prediction.

[0175] In step 605, when the preset training condition is met, the preset text encoder at the end of the training is determined as the target text encoder, and the preset noise predictor at the end of the training is determined as the first target noise predictor.

[0176] In one example, the preset training condition may be that the comprehensive loss value is less than a preset threshold, or the first loss value is less than a preset threshold, or the second loss value is less than a preset threshold, and the training is judged to be finished. In one example, the preset training condition may also be that the number of training iterations reaches a preset number.

[0177] In the embodiment of the present disclosure, the preset text encoder and the preset noise predictor are trained simultaneously, so that the first global semantic feature and the first character semantic feature change with the preset text encoder after the parameters are updated. Compared with the traditional scheme of directly applying the pre-trained text encoder, the target text encoder provided by the embodiment of the present disclosure can learn during the training process not only how to accurately represent the description text itself, but also how to efficiently represent the description text so that the subsequent noise predictor can understand the description text, thereby improving the overall image generation effect of the image generation model.

[0178] Figure 7 It is a schematic diagram of a method for training a target image generation model according to an exemplary embodiment.

[0179] The disclosed embodiment further provides a two-stage image generation model training method. After determining the preset text encoder at the end of the training as the target text encoder, and determining the preset noise predictor at the end of the training as the first target noise predictor, the method provided by the disclosed embodiment further includes steps 701 to 711. Steps 701 to 711 can be regarded as the second stage of the training process.

[0180] In step 701, a second sample noisy image and a second sample description text are obtained.

[0181] The second sample description text is a description text of the second sample image obtained by denoising the second sample noisy image based on the sample noise; the second sample noisy image is annotated with a sample noise label; the resolution of the second sample noisy image is greater than that of the first sample noisy image;

[0182] In one example, assuming that the resolution of the first sample noisy image is 256*256, the resolution of the second sample noisy image may be 512*512. The acquisition process of the second sample noisy image is similar to that of the first sample noisy image, which will not be described in detail here.

[0183] In step 703, the parameters of the target text encoder are frozen, and the second sample description text is input into the target text encoder for text encoding to obtain the second global semantic features and second character semantic features corresponding to the second sample description text.

[0184] Since the text encoder has been trained using a large number of sample images with smaller resolutions in the first training phase to obtain the target text encoder, the target text encoder has learned how to represent text as encoding features that the noise predictor can understand. Therefore, in the second training phase, the parameters of the target text encoder are frozen and only the subsequent noise predictor is trained.

[0185] It should be noted that, since the parameters of the target text encoder are frozen, the second global semantic features and the second character semantic features corresponding to the second sample description text will not change and can be stored for subsequent calls.

[0186] In step 705, a second sample fusion condition feature is generated based on the second global semantic feature and the sample time step corresponding to the sample noise;

[0187] The process of generating the second sample fusion condition feature is similar to that of generating the first sample fusion condition, and will not be repeated here.

[0188] In step 707, the second character semantic feature, the sample fusion condition feature and the second sample noisy image are input into the first target noise predictor for noise prediction processing to obtain a second predicted noise;

[0189] In step 709 , the first target noise predictor is trained according to the difference between the second predicted noise and the sample noise label, and the first target noise predictor at the end of the training is determined as the second target noise predictor.

[0190] The second predicted noise is predicted based on the second sample noisy image. Through the difference between the second predicted noise and the sample noise label, the first target noise predictor is further trained on the basis of the first target noise predictor obtained in the first training stage, thereby obtaining the second target noise predictor. Compared with the first target noise predictor, the second target noise predictor has stronger noise prediction ability and more accurate prediction.

[0191] In step 711, the target text encoder, the second target noise predictor and the denoising network are combined to obtain a target image generation model.

[0192] In the disclosed embodiment, based on the above-mentioned first training stage, the target text encoder is frozen, and the noise predictor is further trained through the second sample noise image to obtain the second target noise predictor, so that the target image generation model finally obtained can further improve the image generation capability. At the same time, in the first training stage, the text encoder and the noise predictor are trained through samples with a smaller resolution, while in the second training stage, only the noise predictor is trained through samples with a larger resolution, which saves computing resources and improves the effect of model image generation.

[0193] In some embodiments, the second target noise predictor can be further trained using the third sample noisy image and the third sample description text to obtain a third target noise predictor, and the target text encoder, the third target noise predictor and the denoising network are combined to obtain a target image generation model. The resolution of the third sample noisy image is greater than that of the second sample noisy image. For example, the resolution of the third sample noisy image can be 1024*1024.

[0194] Figure 8 The figure is a flowchart of an image generating method according to an exemplary embodiment.

[0195] In step 801, a target description text is obtained;

[0196] The target description text is used to prompt the target image generation model to output a target image corresponding to the target description text.

[0197] In some embodiments, the target description text can be obtained by receiving text input in an input box; in some embodiments, the target description text can also be obtained by receiving voice input and then converting the voice input into text. The specific method of obtaining the target description text is not limited in the embodiments of the present disclosure.

[0198] In step 803, the random noise image and the target description text are input into the target image generation model for image generation processing to obtain a target image corresponding to the target description text;

[0199] Among them, the target image generation model is trained according to the training method of the image generation model in the above-mentioned public embodiment.

[0200] In an example, assuming that the target description text is “The Great Wall”, the target image generation model can output a target image with the picture content of “The Great Wall”.

[0201] The random noise image is an image formed by a Gaussian noise matrix. By taking the target description text as a constraint, the target image generation model can perform back diffusion on the basis of the random noise image to accurately obtain the target image.

[0202] Fig. 9 The figure is a flowchart of an image generating method according to an exemplary embodiment.

[0203] In some embodiments, the target image generation model includes a target text encoder, a second target noise predictor, and a denoising network;

[0204] Since the back diffusion process can be regarded as a process of stepwise iteration for the target time step, in some embodiments, the random noise image and the target description text are input into the target image generation model for image generation processing to accurately obtain the target image corresponding to the target description text, which may include steps 901 to 911.

[0205] In step 901, a target time step sequence is obtained;

[0206] The target time step sequence includes multiple time steps, and the multiple time steps are arranged from large to small;

[0207] In contrast to the forward diffusion process, in the application phase, multiple time steps in the target time step sequence are arranged from large to small, so that the random noise image corresponding to the time step K is gradually denoised to obtain a clear image with a time step of 0. It should be noted that the value of K is the same as the number of sample time steps contained in the sample time step sequence. In the target time step sequence, there may be preset intervals between time steps. In an example, the time steps contained in the target time step sequence may be 1000, 950, 900, 850…0. It should be noted that the embodiments of the present disclosure do not limit the specific values ​​of the preset intervals.

[0208] In step 903, the random noise image is determined as the initial image, and the first time step in the time step sequence is determined as the target time step;

[0209] Still taking the above target time step sequence as an example, the target time step may be 1000, indicating that the second target noise predictor needs to output the target predicted noise corresponding to a time step of 1000.

[0210] In step 905, the target time step, the target description text and the initial image are input into a second target noise predictor for noise prediction processing to obtain target prediction noise;

[0211] In some embodiments, step 905 may include:

[0212] The target description text is input into the target text encoder for text encoding to obtain the target global semantic features and target character semantic features corresponding to the target description text; the target fusion conditional features are generated based on the target global semantic features and the target time step; and the target character semantic features, the target fusion conditional features and the initial image are input into the second target noise predictor for noise prediction processing to obtain the target prediction noise.

[0213] It should be noted that the target text encoder has fixed parameters, and the text encoding process is similar to the training phase, which will not be described here. The target text encoder is used to encode the text to output semantic features that are easy for the subsequent second target noise predictor to understand, thereby improving the semantic following ability of the generated image.

[0214] In step 907, based on the denoising network, the initial image is denoised according to the target predicted noise to obtain an intermediate image, and the intermediate image is re-determined as the initial image, and the next time step in the time step sequence is determined as the target time step;

[0215] The intermediate image can be regarded as the image obtained by subtracting the target prediction noise from the initial image. For example, by denoising the random noise image (initial image) with the target prediction noise corresponding to time step 1000, an intermediate image can be obtained; this intermediate image is re-determined as the initial image, and the target time step is updated to 950 at this time, and then the initial image is denoised with the target prediction noise corresponding to time step 950, so as to iterate.

[0216] In step 909, when the target time step is greater than the preset time step, jump to step 905: input the target time step, the target description text, the target shape information and the initial image into the second target noise predictor for noise prediction processing to obtain the target prediction noise;

[0217] The value corresponding to the preset time step may be 0. When the target time step is greater than 0, the noise prediction and denoising process is repeated.

[0218] In step 911, when the target time step is equal to the preset time step, the intermediate image corresponding to the preset time step is determined as the target image corresponding to the target description text. When the target time step is the preset time step, the back diffusion ends, and the intermediate image at this time is the target image corresponding to the target description text, that is, the clear image with a time step of 0. Through the above multiple iterations, the model gradually recovers from the random noise image to the target image, thereby improving the accuracy of the image.

[0219] Through the above disclosed embodiment, by obtaining a first sample noisy image and a first sample description text, and then inputting the first sample description text into a preset text encoder of a preset image generation model for text encoding, a first global semantic feature and a first character semantic feature corresponding to the first sample description text are obtained; the first character semantic feature is used to represent the text feature of the description text, and then a first sample fusion condition feature is generated based on the first global semantic feature and the sample time step corresponding to the sample noise, so that the preset noise predictor can perform noise prediction according to the main constraint condition of the first character semantic feature and the additional constraint condition of the first sample fusion condition feature, and then based on the denoising network of the preset image generation model, the first sample noisy image is denoised according to the first predicted noise to obtain a predicted image; according to the difference between the first predicted noise and the sample noise label and the difference between the predicted image and the first sample image label, the preset image generation model is trained as a whole, and the preset image generation model at the end of the training is determined as the target image generation model. The method of the disclosed embodiment not only trains the noise predictor, but also trains the text encoder, so that the text encoder can not only learn how to understand the meaning of the descriptive text itself, but also learn how to encode the text more efficiently so that the subsequent noise predictor can predict more accurate noise based on the features output by the text encoder, thereby making the trained image generation model more accurate and improving the semantic following ability and accuracy of the generated image.

[0220] Fig.10 It is a schematic structural diagram of a target image generation model according to an exemplary embodiment.

[0221] like Fig.10 As shown in the figure, firstly, the time step, prompt word (Prompt) and image shape information are used as the input of the target image generation model. The image shape information includes image width and height, cropping coordinates and cropping width and height. After the time step is positionally encoded and time step encoded, the time step feature is obtained; the prompt word is input into the Chinese large language model for text encoding to obtain global features and character features, and then the global features are spliced ​​with the features after position encoding of the image shape information, and the spliced ​​features are conditionally encoded to obtain conditional features. The time step feature and the conditional feature are added correspondingly to obtain the conditional fusion feature. Then the character feature is further fused to obtain the text feature, and the text feature and the conditional fusion feature are input as constraints into the target noise predictor (UNet) to predict the noise, and the predicted noise is used as the input of the denoising network to gradually recover the target image corresponding to the prompt word from the random noise image.

[0222] Fig.11 is a block diagram of a training device for an image generation model according to an exemplary embodiment. Fig.11 , the device 1100 comprises:

[0223] The first acquisition module 1101 is configured to execute acquisition of a first sample noisy image and a first sample description text, wherein the first sample description text is a description text of the first sample image obtained by denoising the first sample noisy image based on sample noise; the first sample noisy image is annotated with a sample noise label and a first sample image label;

[0224] The text encoding module 1102 is configured to input the first sample description text into a preset text encoder of a preset image generation model to perform text encoding, so as to obtain a first global semantic feature and a first character semantic feature corresponding to the first sample description text;

[0225] The fusion module 1103 is configured to generate a first sample fusion condition feature based on the first global semantic feature and the sample time step corresponding to the sample noise; the sample time step is a time step corresponding to the sample noise in a sample time step sequence; the sample time step sequence is used to gradually superimpose noise to change a clear image into a random noise image;

[0226] The noise prediction module 1104 is configured to perform noise prediction processing by inputting the first character semantic feature, the first sample fusion condition feature and the first sample noisy image into a preset noise predictor of the preset image generation model to obtain a first predicted noise;

[0227] A denoising module 1105 is configured to execute a denoising network based on the preset image generation model, and denoise the first sample noisy image according to the first predicted noise to obtain a predicted image;

[0228] The target image generation model determination module 1106 is configured to perform training on the preset image generation model according to the difference between the first predicted noise and the sample noise label and the difference between the predicted image and the first sample image label, and determine the preset image generation model at the end of the training as the target image generation model.

[0229] In some embodiments, the target image generation model determination module may include:

[0230] a comprehensive loss determination submodule, configured to determine a comprehensive loss value according to a difference between the first predicted noise and the sample noise label and a difference between the predicted image and the first sample image label;

[0231] A training submodule, configured to perform training on the preset text encoder and the preset noise predictor based on the comprehensive loss value, and determine the preset text encoder as a target text encoder at the end of the training, and determine the preset noise predictor as a first target noise predictor at the end of the training;

[0232] The combination submodule is configured to combine the target text encoder, the first target noise predictor and the denoising network to obtain the target image generation model.

[0233] In some embodiments, the training submodule may include:

[0234] An updating unit is configured to update the preset text encoder based on the comprehensive loss value to obtain an updated text encoder, and redefine the updated text encoder as the preset text encoder; and update the preset noise predictor based on the comprehensive loss value to obtain an updated noise predictor, and redefine the updated noise predictor as the preset noise predictor;

[0235] A jump unit is configured to execute the jump to the step of inputting the first sample description text into a preset text encoder for text encoding to obtain a first global semantic feature and a first character semantic feature corresponding to the first sample description text;

[0236] The training ending unit is configured to determine the preset text encoder at the end of training as the target text encoder and the preset noise predictor at the end of training as the first target noise predictor when a preset training condition is met.

[0237] In some embodiments, the comprehensive loss determination submodule may include:

[0238] A first loss value determining subunit is configured to determine a first loss value according to a difference between the first prediction noise and the sample noise label;

[0239] A second loss value determining subunit is configured to determine a second loss value according to a difference between the predicted image and the first sample image label;

[0240] The comprehensive loss value determining subunit is configured to determine the comprehensive loss value based on the first loss value and the second loss value.

[0241] In some embodiments, the apparatus 1100 may further include:

[0242] A second acquisition module is configured to acquire a second sample noisy image and a second sample description text, wherein the second sample description text is a description text of the second sample image obtained by denoising the second sample noisy image based on the sample noise; the second sample noisy image is annotated with a sample noise label; and the resolution of the second sample noisy image is greater than that of the first sample noisy image;

[0243] A target text encoding module is configured to freeze the parameters of the target text encoder, and input the second sample description text into the target text encoder for text encoding to obtain a second global semantic feature and a second character semantic feature corresponding to the second sample description text;

[0244] A second fusion module is configured to generate a second sample fusion condition feature based on the second global semantic feature and the sample time step corresponding to the sample noise;

[0245] A second noise prediction module is configured to perform noise prediction processing by inputting the second character semantic feature, the second sample fusion condition feature and the second sample noisy image into the first target noise predictor to obtain a second predicted noise;

[0246] A second target noise predictor training module is configured to train the first target noise predictor according to the difference between the second predicted noise and the sample noise label, and determine the first target noise predictor at the end of the training as a second target noise predictor;

[0247] A combination module is configured to combine the target text encoder, the second target noise predictor and the denoising network to obtain the target image generation model.

[0248] In some embodiments, the fusion module may include:

[0249] The initial sample time step feature determination submodule is configured to perform position encoding processing on the sample time step to obtain the initial sample time step feature;

[0250] The initial sample shape feature determination submodule is configured to obtain image shape information corresponding to the first sample noisy image and perform position encoding processing on the image shape information to obtain initial sample shape features;

[0251] The first sample fusion condition feature determination submodule is configured to generate the first sample fusion condition feature based on the initial sample time step feature, the first global semantic feature and the initial sample shape feature.

[0252] In some embodiments, the initial sample shape feature determination submodule may include:

[0253] a width and height information acquisition unit, configured to acquire sample width information and sample height information of the first sample noisy image, and target width information and target height information of a target image; the target image represents an image generated by the target image generation model;

[0254] A clipping coordinate information acquisition unit is configured to acquire clipping coordinate information; the clipping coordinate information represents the position coordinates of the preset vertices of the target image in the first sample noisy image;

[0255] A sample width and height feature determination unit is configured to perform position encoding processing on the sample width information and the sample height information to obtain sample width and height features;

[0256] a target width and height feature determination unit, configured to perform position encoding processing on target width information and target height information of the target image to obtain target width and height features;

[0257] A clipping feature determination unit is configured to perform position encoding processing on the clipping coordinate information to obtain clipping features;

[0258] The shape feature fusion unit is configured to perform feature fusion processing on the sample width and height features, the target width and height features and the cropping features to obtain the initial sample shape features.

[0259] In some embodiments, the first sample fusion condition feature determination submodule may include:

[0260] A sample time step feature mapping unit is configured to map the initial sample time step feature to a preset dimension to obtain a sample time step feature;

[0261] A feature splicing unit is configured to perform feature splicing processing on the initial sample shape feature and the first global semantic feature to obtain an initial condition feature;

[0262] A sample condition feature mapping unit is configured to map the initial condition feature to the preset dimension to obtain a sample condition feature;

[0263] The conditional fusion unit is configured to perform feature fusion processing on the sample conditional feature and the sample time step feature to obtain the first sample fusion conditional feature.

[0264] In some embodiments, the apparatus 1100 may further include:

[0265] A sample time step sequence acquisition submodule is configured to execute acquisition of a sample time step sequence, wherein the sample time step sequence includes a plurality of sample time steps that increase in sequence;

[0266] The noise intensity parameter determination submodule is configured to determine the noise intensity parameter corresponding to each sample time step in the sample time step sequence according to a preset mapping relationship; the preset mapping relationship represents the corresponding relationship between the preset time step and the preset noise intensity parameter;

[0267] A sample noise determination submodule, configured to determine the sample noise according to the noise intensity parameter;

[0268] The noise superposition submodule is configured to perform superposition of the sample noise onto the first sample image to obtain the first sample noisy image corresponding to the sample time step.

[0269] Fig.12 is a block diagram of an image generating device according to an exemplary embodiment. Fig.12 , the apparatus 1200 may include:

[0270] The target description text acquisition module 1201 is configured to execute acquisition of a target description text; the target description text is used to prompt the target image generation model to output a target image corresponding to the target description text;

[0271] The image generation module 1202 is configured to execute image generation processing by inputting the random noise image and the target description text into a target image generation model to obtain a target image corresponding to the target description text; the target image generation model is trained according to the training method of the image generation model described in the first aspect.

[0272] In some embodiments, the target image generation model includes a target text encoder, a second target noise predictor, and a denoising network; the image generation module may include:

[0273] A target time step sequence acquisition submodule is configured to execute acquisition of a target time step sequence, wherein the target time step sequence includes a plurality of time steps, and the plurality of time steps are arranged from large to small;

[0274] An initial image determination submodule is configured to determine the random noise image as the initial image and determine the first time step in the time step sequence as the target time step;

[0275] A target prediction noise determination submodule is configured to input the target time step, the target description text and the initial image into the second target noise predictor for noise prediction processing to obtain target prediction noise;

[0276] an intermediate image determination submodule, configured to perform denoising processing on the initial image based on the denoising network according to the target predicted noise to obtain an intermediate image, and to re-determine the intermediate image as the initial image, and to determine the next time step in the time step sequence as the target time step;

[0277] A target jump submodule is configured to execute, when the target time step is greater than a preset time step, a step of jumping to inputting the target time step, the target description text, the target shape information and the initial image into the second target noise predictor for noise prediction processing to obtain target prediction noise;

[0278] The target image determination submodule is configured to determine the intermediate image corresponding to the preset time step as the target image corresponding to the target description text when the target time step is equal to the preset time step.

[0279] In some embodiments, the target prediction noise determination submodule may include:

[0280] A target semantic feature determination unit is configured to input the target description text into the target text encoder for text encoding, and obtain a target global semantic feature and a target character semantic feature corresponding to the target description text;

[0281] A target fusion condition feature determination unit is configured to generate a target fusion condition feature based on the target global semantic feature and the target time step;

[0282] The target prediction noise determination unit is configured to input the target character semantic feature, the target fusion condition feature and the initial image into the second target noise predictor for noise prediction processing to obtain the target prediction noise.

[0283] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0284] In an exemplary embodiment, an electronic device is also provided, comprising a processor; and a memory for storing instructions executable by the processor; wherein, when the processor is configured to execute the instructions stored in the memory, the image generation model training method or image generation method provided in any of the above embodiments is implemented.

[0285] The electronic device may be a terminal, a server or a similar computing device. For example, the electronic device is a server. Fig.13 is a block diagram of an electronic device according to an exemplary embodiment. Fig.13 As shown, the server 1300 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1310 (the processor 1310 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 1330 for storing data, and one or more storage media 1320 (such as one or more mass storage devices) for storing application programs 1323 or data 1322. Among them, the memory 1330 and the storage medium 1320 can be short-term storage or permanent storage. The program stored in the storage medium 1320 may include one or more modules, each of which may include a series of instruction operations on the server. Furthermore, the central processing unit 1310 can be configured to communicate with the storage medium 1320 and execute a series of instruction operations in the storage medium 1320 on the server 1300. The server 1300 may also include one or more power supplies 1360, one or more wired or wireless network interfaces 1350, one or more input and output interfaces 1340, and / or one or more operating systems 1321, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0286] The input / output interface 1340 may be used to receive or send data via a network. The specific example of the network may include a wireless network provided by a communication provider of the server 1300. In one example, the input / output interface 1340 includes a network adapter (Network Interface Controller, NIC), which may be connected to other network devices through a base station so as to communicate with the Internet. In one example, the input / output interface 1340 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0287] It can be understood by those skilled in the art that Fig.13 The structure shown is for illustration only and does not limit the structure of the above electronic device. Fig.13 More or fewer components as shown, or with Fig.13 Different configurations are shown.

[0288] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 1330 including instructions, and the instructions can be executed by the processor 1310 of the device 1300 to complete the above method. Alternatively, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0289] Fig.14 is a block diagram of an electronic device for training an image generation model according to an exemplary embodiment. The electronic device may be a terminal, and its internal structure diagram may be as shown in FIG. Fig.14 As shown. The electronic device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a training method for an image generation model is implemented. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covering the display screen, or a key, trackball or touchpad provided on the housing of the electronic device, or an external keyboard, touchpad or mouse, etc.

[0290] Those skilled in the art will understand that Fig.14 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present disclosure, and does not constitute a limitation on the electronic device to which the scheme of the present disclosure is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0291] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory including instructions, and the above instructions can be executed by a processor of an electronic device to complete the above-mentioned image generation model training method or image generation method. Optionally, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0292] In an exemplary embodiment, a computer program product is also provided, comprising a computer program / instruction, which, when executed by a processor, implements the above-mentioned training method of the image generation model or the image generation method.

[0293] In the disclosed embodiment, a first sample noisy image and a first sample description text are obtained, and then the first sample description text is input into a preset text encoder of a preset image generation model for text encoding, so as to obtain a first global semantic feature and a first character semantic feature corresponding to the first sample description text; the first character semantic feature is used to represent the text feature of the description text, and then a first sample fusion condition feature is generated based on the first global semantic feature and the sample time step corresponding to the sample noise, so that a preset noise predictor can perform noise prediction according to the main constraint of the first character semantic feature and the additional constraint of the first sample fusion condition feature, and then, based on a denoising network of the preset image generation model, the first sample noisy image is denoised according to the first predicted noise to obtain a predicted image; according to the difference between the first predicted noise and the sample noise label and the difference between the predicted image and the first sample image label, the preset image generation model is trained as a whole, and the preset image generation model at the end of the training is determined as the target image generation model. The method of the disclosed embodiment not only trains the noise predictor, but also trains the text encoder, so that the text encoder can not only learn how to understand the meaning of the descriptive text itself, but also learn how to encode the text more efficiently so that the subsequent noise predictor can predict more accurate noise based on the features output by the text encoder, thereby making the trained image generation model more accurate and improving the semantic following ability and accuracy of the generated image.

[0294] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0295] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0296] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A training method for an image generation model, characterized in that: include: Acquire a first sample noisy image and a first sample description text, wherein the first sample description text is a description text of the first sample image obtained by denoising the first sample noisy image based on sample noise; the first sample noisy image is annotated with a sample noise label and a first sample image label; Inputting the first sample description text into a preset text encoder of a preset image generation model for text encoding to obtain a first global semantic feature and a first character semantic feature corresponding to the first sample description text; Generate a first sample fusion condition feature based on the first global semantic feature and the sample time step corresponding to the sample noise; the sample time step is the time step corresponding to the sample noise in the sample time step sequence; the sample time step sequence is used to gradually superimpose noise to change the clear image into a random noise image; Inputting the first character semantic feature, the first sample fusion condition feature and the first sample noisy image into a preset noise predictor of the preset image generation model for noise prediction processing to obtain a first predicted noise; Based on a denoising network of the preset image generation model, denoising the first sample noisy image according to the first predicted noise to obtain a predicted image; The preset image generation model is trained according to the difference between the first predicted noise and the sample noise label and the difference between the predicted image and the first sample image label, and the preset image generation model at the end of the training is determined as the target image generation model.

2. The method according to claim 1, characterized in that The step of training the preset image generation model according to the difference between the first predicted noise and the sample noise label and the difference between the predicted image and the first sample image label, and determining the preset image generation model at the end of the training as the target image generation model, comprises: Determining a comprehensive loss value according to a difference between the first predicted noise and the sample noise label and a difference between the predicted image and the first sample image label; Based on the comprehensive loss value, the preset text encoder and the preset noise predictor are trained, and the preset text encoder at the end of the training is determined as a target text encoder, and the preset noise predictor at the end of the training is determined as a first target noise predictor; The target text encoder, the first target noise predictor and the denoising network are combined to obtain the target image generation model.

3. The method according to claim 2, characterized in that The method of training the preset text encoder and the preset noise predictor based on the comprehensive loss value, determining the preset text encoder as a target text encoder at the end of the training, and determining the preset noise predictor as a first target noise predictor at the end of the training, comprises: Based on the comprehensive loss value, the preset text encoder is updated to obtain an updated text encoder, and the updated text encoder is re-determined as the preset text encoder; and based on the comprehensive loss value, the preset noise predictor is updated to obtain an updated noise predictor, and the updated noise predictor is re-determined as the preset noise predictor; Jump to the step of inputting the first sample description text into a preset text encoder for text encoding to obtain a first global semantic feature and a first character semantic feature corresponding to the first sample description text; When a preset training condition is met, the preset text encoder at the end of training is determined as the target text encoder, and the preset noise predictor at the end of training is determined as the first target noise predictor.

4. The method according to claim 2, characterized in that: The determining of the comprehensive loss value according to the difference between the first predicted noise and the sample noise label and the difference between the predicted image and the first sample image label comprises: Determining a first loss value according to a difference between the first predicted noise and the sample noise label; Determining a second loss value according to a difference between the predicted image and the first sample image label; The comprehensive loss value is determined based on the first loss value and the second loss value.

5. The method according to claim 2, characterized in that: After determining the preset text encoder at the end of training as the target text encoder, and determining the preset noise predictor at the end of training as the first target noise predictor, the method further includes: Acquire a second sample noisy image and a second sample description text, wherein the second sample description text is a description text of the second sample image obtained by denoising the second sample noisy image based on the sample noise; the second sample noisy image is annotated with a sample noise label; and the resolution of the second sample noisy image is greater than that of the first sample noisy image; Freezing the parameters of the target text encoder, and inputting the second sample description text into the target text encoder for text encoding to obtain a second global semantic feature and a second character semantic feature corresponding to the second sample description text; Generate a second sample fusion condition feature based on the second global semantic feature and the sample time step corresponding to the sample noise; Inputting the second character semantic feature, the second sample fusion condition feature and the second sample noisy image into the first target noise predictor for noise prediction processing to obtain a second predicted noise; training the first target noise predictor according to the difference between the second predicted noise and the sample noise label, and determining the first target noise predictor at the end of the training as the second target noise predictor; The target text encoder, the second target noise predictor and the denoising network are combined to obtain the target image generation model.

6. The method according to claim 1, characterized in that The generating a first sample fusion condition feature based on the first global semantic feature and the sample time step corresponding to the sample noise includes: Performing position encoding processing on the sample time step to obtain initial sample time step features; Acquire image shape information corresponding to the first sample noisy image, and perform position encoding processing on the image shape information to obtain initial sample shape features; The first sample fusion condition feature is generated based on the initial sample time step feature, the first global semantic feature and the initial sample shape feature.

7. The method according to claim 6, characterized in that The acquiring of image shape information corresponding to the first sample noisy image and performing position encoding processing on the image shape information to obtain initial sample shape features includes: Acquire sample width information and sample height information of the first sample noisy image, and target width information and target height information of a target image; the target image represents an image generated by the target image generation model; Acquire clipping coordinate information; the clipping coordinate information represents the position coordinates of the preset vertices of the target image in the first sample noisy image; Performing position encoding processing on the sample width information and the sample height information to obtain sample width and height features; Performing position encoding processing on target width information and target height information of the target image to obtain target width and height features; Performing position encoding processing on the clipping coordinate information to obtain clipping features; The sample width and height features, the target width and height features and the cropping features are subjected to feature fusion processing to obtain the initial sample shape features.

8. The method according to claim 6, characterized in that The generating the first sample fusion condition feature based on the initial sample time step feature, the first global semantic feature and the initial sample shape feature comprises: Mapping the initial sample time step feature to a preset dimension to obtain a sample time step feature; Performing feature concatenation processing on the initial sample shape feature and the first global semantic feature to obtain an initial condition feature; Mapping the initial condition feature to the preset dimension to obtain a sample condition feature; The sample condition feature and the sample time step feature are subjected to feature fusion processing to obtain the first sample fusion condition feature.

9. The method according to claim 1, characterized in that: The method further comprises: Acquire a sample time step sequence, wherein the sample time step sequence includes a plurality of sample time steps that increase in sequence; For each of the sample time steps in the sample time step sequence, determine the noise intensity parameter corresponding to each of the sample time steps according to a preset mapping relationship; the preset mapping relationship represents a correspondence between a preset time step and a preset noise intensity parameter; Determining the sample noise according to the noise intensity parameter; The sample noise is superimposed on the first sample image to obtain the first sample noisy image corresponding to the sample time step.

10. An image generation method, characterized in that: include: Get the target description text; The target description text is used to prompt the target image generation model to output a target image corresponding to the target description text; The random noise image and the target description text are input into the target image generation model for image generation processing to obtain a target image corresponding to the target description text; the target image generation model is trained according to the training method of the image generation model described in any one of claims 1 to 9.

11. The method according to claim 10, characterized in that The target image generation model includes a target text encoder, a second target noise predictor and a denoising network; the random noise image and the target description text are input into the target image generation model for image generation processing to obtain a target image corresponding to the target description text, including: Obtain a target time step sequence, wherein the target time step sequence includes multiple time steps, and the multiple time steps are arranged from large to small; Determine the random noise image as an initial image, and determine the first time step in the time step sequence as a target time step; Inputting the target time step, the target description text and the initial image into the second target noise predictor for noise prediction processing to obtain target prediction noise; Based on the denoising network, the initial image is denoised according to the target predicted noise to obtain an intermediate image, and the intermediate image is re-determined as the initial image, and the next time step in the time step sequence is determined as the target time step; When the target time step is greater than the preset time step, jump to the step of inputting the target time step, the target description text, the target shape information and the initial image into the second target noise predictor for noise prediction processing to obtain target prediction noise; When the target time step is equal to the preset time step, the intermediate image corresponding to the preset time step is determined as the target image corresponding to the target description text.

12. The method according to claim 11, characterized in that The step of inputting the target time step, the target description text and the initial image into the second target noise predictor for noise prediction processing to obtain target prediction noise includes: Inputting the target description text into the target text encoder for text encoding to obtain target global semantic features and target character semantic features corresponding to the target description text; Generate a target fusion condition feature based on the target global semantic feature and the target time step; The target character semantic feature, the target fusion condition feature and the initial image are input into the second target noise predictor for noise prediction processing to obtain the target predicted noise.

13. A training device for an image generation model, characterized in that: include: A first acquisition module is configured to acquire a first sample noisy image and a first sample description text, wherein the first sample description text is a description text of the first sample image obtained by denoising the first sample noisy image based on sample noise; the first sample noisy image is annotated with a sample noise label and a first sample image label; A text encoding module is configured to perform text encoding by inputting the first sample description text into a preset text encoder of a preset image generation model to obtain a first global semantic feature and a first character semantic feature corresponding to the first sample description text; A fusion module is configured to generate a first sample fusion condition feature based on the first global semantic feature and a sample time step corresponding to the sample noise; the sample time step is a time step corresponding to the sample noise in a sample time step sequence; the sample time step sequence is used to gradually superimpose noise to change a clear image into a random noise image; A noise prediction module is configured to perform noise prediction processing by inputting the first character semantic feature, the first sample fusion condition feature and the first sample noisy image into a preset noise predictor of the preset image generation model to obtain a first predicted noise; a denoising module, configured to execute a denoising network based on the preset image generation model, and to denoise the first sample noisy image according to the first predicted noise to obtain a predicted image; The target image generation model determination module is configured to perform training on the preset image generation model according to the difference between the first predicted noise and the sample noise label and the difference between the predicted image and the first sample image label, and determine the preset image generation model at the end of the training as the target image generation model.

14. An image generating device, characterized in that: include: A first target description text acquisition module is configured to execute acquisition of a target description text; The target description text is used to prompt the target image generation model to output a target image corresponding to the target description text; An image generation module is configured to execute image generation processing by inputting a random noise image and the target description text into the target image generation model to obtain a target image corresponding to the target description text; the target image generation model is trained according to the training method of the image generation model described in any one of claims 1 to 9.

15. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the training method of the image generation model as described in any one of claims 1 to 9 or the image generation method as described in any one of claims 10 to 12.

16. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the training method of the image generation model as described in any one of claims 1 to 9 or the image generation method as described in any one of claims 10 to 12.

17. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the training method of the image generation model described in any one of claims 1 to 9 or the image generation method described in any one of claims 10 to 12 is implemented.

Images