Training method of image generation model, image generation method and related equipment

Abstract

The invention provides a training method for an image generation model, and the method comprises the steps: obtaining first training data of an Nth training process for the Nth training process; taking the description information of the first training image as the input of an image generation model, taking the first training image as a label, and training the image generation model; acquiring second description information; taking the second description information as the input of the image generation model after the Nth round of training process, and obtaining a first image output by the image generation model; performing fine processing on the first image by using an optimizer to obtain a second image output by the optimizer; and for the (N + 1) th round of training process, taking at least part of the second image and the description information of at least part of the second image as first training data of the (N + 1) th round of training process, and training the image generation model. According to the method, the image generation model is trained while the image quality of the training data is improved, and the image generation model capable of generating a high-quality image can be obtained through training.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a training method for an image generation model, an image generation method, a training device for an image generation model, an image generation device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] In an image generation scenario, users (such as business parties with image generation needs) can use image generation models to generate desired images. Therefore, training of image generation models is particularly important.

[0003] The quality of images generated by image generation models depends on the quality of training data. That is, using high-quality training data to train the model can produce an image generation model that can generate high-quality images. However, it is usually difficult to obtain a large amount of high-quality training data. How to use low-quality training data to train a high-quality image generation model has become an urgent problem to be solved. Summary of the invention

[0004] The present disclosure provides a method for training an image generation model. The method can train an image generation model that can generate high-quality images when only low-quality training data is available. The present disclosure also provides an image generation method corresponding to the above method, a training device for the image generation model, an image generation device, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] In a first aspect, the present disclosure provides a method for training an image generation model, the method comprising:

[0006] For an N-th round of training process, obtaining first training data of the N-th round of training process; wherein the first training data includes a first training image and description information of the first training image, and N is an integer greater than 0;

[0007] Using the description information of the first training image as input of an image generation model and using the first training image as a label to train the image generation model;

[0008] Obtaining second description information;

[0009] Using the second description information as input of the image generation model after the Nth round of training process, to obtain a first image output by the image generation model;

[0010] Using an optimizer to perform fine processing on the first image to obtain a second image output by the optimizer; wherein the image quality of the second image is better than the image quality of the first image;

[0011] For the N+1th round of training process, at least part of the second image and at least part of the description information of the second image are used as the first training data of the N+1th round of training process to train the image generation model.

[0012] In a second aspect, the present disclosure provides an image generation method, the method comprising:

[0013] Acquire first description information; wherein the first description information is used to describe the desired image;

[0014] The first description information is input into an image generation model to obtain a target image output by the image generation model; wherein the image generation model is trained by the method according to any one of claims 1 to 8.

[0015] In a third aspect, the present disclosure provides a training device for an image generation model, the device comprising:

[0016] An acquisition module, configured to acquire, for an N-th round of training process, first training data of the N-th round of training process; wherein the first training data includes a first training image and description information of the first training image, and N is an integer greater than 0;

[0017] A training module, configured to use the description information of the first training image as an input of an image generation model and use the first training image as a label to train the image generation model;

[0018] The acquisition module is further used to acquire second description information;

[0019] A generation module, used to use the second description information as an input of the image generation model after the Nth round of training process to obtain a first image output by the image generation model;

[0020] an optimization module, configured to perform fine processing on the first image using an optimizer to obtain a second image output by the optimizer; wherein the image quality of the second image is better than that of the first image;

[0021] The training module is also used to train the image generation model for the N+1th round of training process by using at least part of the second image and at least part of the description information of the second image as the first training data for the N+1th round of training process.

[0022] In a fourth aspect, the present disclosure provides an image generating device, the device comprising:

[0023] An acquisition module, used for acquiring first description information; wherein the first description information is used for describing a desired image;

[0024] A generation module is used to input the first description information into an image generation model to obtain a target image output by the image generation model; wherein the image generation model is trained by the training method of the image generation model in the first aspect or any one of the implementations of the first aspect.

[0025] In a fifth aspect, the present disclosure provides an electronic device, comprising a processor and a memory. The processor and the memory communicate with each other. The processor is used to execute instructions stored in the memory, so that the electronic device executes the training method of the image generation model in the first aspect or any one of the implementations of the first aspect, or executes the image generation method in the second aspect.

[0026] In a sixth aspect, the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, wherein the instructions instruct an electronic device to execute the training method of the image generation model described in the first aspect or any one of the implementations of the first aspect, or to execute the image generation method of the second aspect.

[0027] In a seventh aspect, the present disclosure provides a computer program product comprising instructions, which, when executed on an electronic device, enables the electronic device to execute the training method of the image generation model described in the first aspect or any one of the implementations of the first aspect, or to execute the image generation method described in the second aspect.

[0028] Based on the implementations provided in the above aspects, the present disclosure can be further combined to provide more implementations.

[0029] It can be seen from the above technical solutions that the present disclosure has the following advantages:

[0030] The present disclosure provides a training method for an image generation model. The method obtains the first training data of the Nth round of training process, wherein the first training data includes a first training image and description information of the first training image, and N is an integer greater than 0. The description information of the first training image is used as the input of the image generation model, and the first training image is used as a label to train the image generation model. Then, the second description information is obtained, and the second description information is used as the input of the image generation model after the Nth round of training process to obtain the first image output by the image generation model, and the first image is refined by an optimizer to obtain the second image output by the optimizer, wherein the image quality of the second image is better than the image quality of the first image. For the N+1th round of training process, at least part of the second image and at least part of the description information of the second image are used as the first training data of the N+1th round of training process to train the image generation model.

[0031] In this method, in the case of low-quality training data, the image generation model is first trained using low-quality training data (i.e., first training data), and then the first image output by the image generation model is optimized using an optimizer to improve the image quality and obtain a second image with higher image quality. In the next round of training, the image generation model is further trained using the second image with higher image quality. In this way, when there is only low-quality training data, the image generation model is trained while the image quality of the training data is improved, and an image generation model that can generate high-quality images is obtained through training. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical method of the embodiments of the present disclosure, the drawings required for use in the embodiments are briefly introduced below.

[0033] Figure 1 A flowchart of a method for training an image generation model provided in an embodiment of the present disclosure;

[0034] Figure 2 A schematic diagram of the structure of an image generation model provided by an embodiment of the present disclosure;

[0035] Figure 3 A flowchart of an image generation method provided by an embodiment of the present disclosure;

[0036] Figure 4 A schematic diagram of the structure of an image generation model provided by an embodiment of the present disclosure;

[0037] Figure 5 A schematic diagram of the structure of a training device for an image generation model provided in an embodiment of the present disclosure;

[0038] Figure 6 A schematic diagram of the structure of an image generating device provided by an embodiment of the present disclosure;

[0039] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0040] The terms "first" and "second" in the embodiments of the present disclosure are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0041] First, some technical terms and application scenarios involved in the embodiments of the present disclosure are introduced.

[0042] The image generation process can be understood as a process of automatically generating new images. Generally, the industry uses image generation models for image generation. For example, the image generation model can be a diffusion model based on deep learning.

[0043] The quality of images generated by image generation models depends on the quality of training data. That is, using high-quality training data for model training can produce an image generation model that can generate high-quality images. In related technologies, high-quality training data can be obtained by designers, purchased from high-quality image websites, or through data cleaning. However, the above methods require a large cost, and it is difficult to obtain a large amount of high-quality training data in scenarios such as model training.

[0044] In view of this, the present disclosure provides a training method for an image generation model. For the Nth round of training process, the method obtains the first training data for the Nth round of training process, wherein the first training data includes a first training image and description information of the first training image, and N is an integer greater than 0. The description information of the first training image is used as the input of the image generation model, and the first training image is used as a label to train the image generation model. Next, the second description information is obtained, and the second description information is used as the input of the image generation model after the Nth round of training process to obtain the first image output by the image generation model, and the first image is refined by an optimizer to obtain the second image output by the optimizer, wherein the image quality of the second image is better than the image quality of the first image. For the N+1th round of training process, at least part of the second image and at least part of the description information of the second image are used as the first training data for the N+1th round of training process to train the image generation model.

[0045] In this method, in the case of low-quality training data, the image generation model is first trained using low-quality training data (i.e., first training data), and then the first image output by the image generation model is optimized using an optimizer to improve the image quality and obtain a second image with higher image quality. In the next round of training, the image generation model is further trained using the second image with higher image quality. In this way, when there is only low-quality training data, the image generation model is trained while the image quality of the training data is improved, and an image generation model that can generate high-quality images is obtained through training.

[0046] To facilitate understanding of the technical solution provided by the embodiments of the present disclosure, the following is a description with reference to the accompanying drawings. Figure 1 The flowchart of a training method of an image generation model shown in FIG. 1 is a flowchart of a training method of an image generation model, and the method specifically includes:

[0047] S101: For an N-th round of training process, obtaining first training data of the N-th round of training process.

[0048] In the embodiment of the present disclosure, the training of the image generation model may include multiple training processes, and the model parameters of the image generation model may be updated once in each training process.

[0049] The Nth round of training process can be understood as any training process in the training of the image generation model, and N can be an integer greater than 0. The first training data of the Nth round of training process can be understood as the training data for training the image generation model in the Nth round of training process.

[0050] Specifically, the first training data may include a first training image and description information of the first training image, and the description information of the first training image may be used to describe the image content of the first training image. In the embodiment of the present disclosure, the description information of the first training image may be described in natural language to represent the natural language content, that is, the description information of the first training image may describe the image content of the first training image in the form of natural language.

[0051] In this way, the first training image and the description information of the first training image are used as the first training data, so that the image generation model can generate an image based on the text.

[0052] S102: Using the description information of the first training image as input of the image generation model and using the first training image as a label to train the image generation model.

[0053] In the disclosed embodiment, the training of the image generation model may be supervised training, and the label may be understood as the correct output result in the supervised training.

[0054] In a specific implementation, the description information of the first training image is input into the image generation model, and the image generation model generates an output image that matches the description information of the first training image. Then, based on the output image of the image generation model and the first training image as a label, a loss function value is calculated. The loss function value can characterize the degree of difference between the output image of the image generation model and the first training image. The model parameters of the image generation model are adjusted to minimize the loss function value, thereby completing the Nth round of training process for the image generation model.

[0055] S103: Obtain second description information.

[0056] After the Nth round of training for the image generation model is completed, the image generation model trained in the Nth round of training is used to generate the image. In other words, the second description information can be used as an input of the image generation model after the Nth round of training.

[0057] Similar to the description information of the first training image, the second description information may be described in natural language to represent the natural language content, that is, the second description information may describe the image content of the expected image in the form of natural language.

[0058] The disclosed embodiment does not limit the source of the second description information. For example, the second description information may be extracted from an existing image library. In another example, the second description information may be written by a user (eg, an annotator) based on a desired image.

[0059] S104: Using the second description information as input to the image generation model after the Nth round of training process to obtain a first image output by the image generation model.

[0060] like Figure 2 As shown, the input of the image generation model may include an initial noise image and second description information. The initial noise image may be understood as an image composed of initial noise (e.g., random numbers). Usually, the initial noise image has the same size as the first image. The image generation model may use the initial noise image as a reference and perform image generation based on the initial noise image.

[0061] It should be noted that the image generation model may also include more modules. For example, the image generation model may also include a variational autoencoder module, which includes an encoder and a decoder, wherein the encoder is used to downsample the initial noisy image (for example, compressing the 512×512 initial noisy image into a 16×16 image), and the decoder is used to upsample the image output by the image generation model (for example, restoring the 16×16 image output by the image generation model into a first image of 512×512).

[0062] Since the image generation model has undergone N rounds of training, after the second description information is input into the image generation model, the image generation model can input the first image that satisfies the second description information.

[0063] As mentioned above, since it is difficult to obtain a large amount of high-quality first training data during the training process, after the image generation model is trained in the Nth round of training using the first training data, the image quality of the first image generated by the image generation model may be poor. For example, the first image may have defects such as incomplete characters and traces of erasure, and the image effect is not good.

[0064] S105: Performing fine processing on the first image using an optimizer to obtain a second image output by the optimizer.

[0065] Continue as Figure 2As shown, in the embodiment of the present disclosure, for the first image generated by the image generation model after each round of training process, the optimizer is used to optimize the first image to obtain a second image, and the image quality of the second image can be better than the image quality of the first image. For example, the details of the second image are clearer, the objects in the second image are more complete, and the tone and style of the second image are more in line with the image generation requirements in the business scenario.

[0066] S106: For the N+1th round of training process, use at least part of the second image and at least part of the description information of the second image as first training data for the N+1th round of training process to train the image generation model.

[0067] Continue as Figure 2 As shown, in the disclosed embodiment, the optimizer is used to optimize the first image output by the image generation model after the Nth round of training process, and the optimized second image is used as the first training data of the N+1th round of training process, and the model training is continued for the image generation model. In this way, the quality of the first training data of each round of training process is improved compared with the quality of the first training data of the previous round of training process, and the training effect of the image generation model is gradually improved.

[0068] In this method, in the case of low-quality training data, the image generation model is first trained using low-quality training data (i.e., first training data), and then the first image output by the image generation model is optimized using an optimizer to improve the image quality and obtain a second image with higher image quality. In the next round of training, the image generation model is further trained using the second image with higher image quality. In this way, when there is only low-quality training data, the image generation model is trained while the image quality of the training data is improved, and an image generation model that can generate high-quality images is obtained through training.

[0069] The foregoing describes the training process of the image generation model provided in the embodiments of the present disclosure. The following will introduce the specific contents involved in the training process.

[0070] In the embodiment of the present disclosure, the first training data may be selected in combination with actual image generation requirements. Specifically, the first training data may be related to at least one of the following: target business, target content, or target style.

[0071] Among them, the target business can be understood as a specific business scenario with image generation requirements. For example, in business A, it is necessary to use an image generation model to generate a template image for advertising delivery. In this case, the target business is business A, and the first training data can include a first training image that meets the template image requirements and description information of the first training image.

[0072] The target content can be understood as a specific object in an image. For example, a user needs to use an image generation model to generate an image related to cartoon character B. In this case, cartoon character B is the target content, and the first training data may include a first training image containing cartoon character B and description information of the first training image.

[0073] The target style can be understood as a specific image style. For example, the user needs to use an image generation model to generate an image in an oil painting style. In this case, the target style is the oil painting style, and the first training data may include a first training image in the oil painting style and description information of the first training image.

[0074] In this way, the first training data is selected according to the actual image generation requirements, so that the image generation model can learn images that meet specific business, specific content or specific style, and improve the matching degree between the image generation model and the actual image generation requirements. At the same time, when the image generation requirements involve the above-mentioned target business, target content or target style, it is often difficult to obtain a large amount of high-quality training data. After applying the training method of the image generation model provided by the embodiment of the present disclosure, an image generation model with good reasoning performance can be obtained.

[0075] Similarly, the expected image described by the second description information may also be related to at least one of the target business, target content and target style. That is to say, after using the first training data to train the image generation model's image generation capability under a specific business, specific content or a specific style, the second description information under the same business, the same content or the same style can be obtained. On the one hand, the training effect of the image generation model after the Nth round of training process is verified, and on the other hand, the quality of the training data in subsequent training processes is improved.

[0076] Since the first training data and the second description information are both related to the same business, the same content or the same style, the second image output by the optimizer may also be related to the business, the content or the style, thereby improving the image quality of the image related to the business, the content or the style.

[0077] Model training is usually configured with an end condition, and when the end condition is met, the model training process ends. In the embodiment of the present disclosure, after the Mth round of training process, if the image quality of the first image output by the image generation model meets the first set condition, the training of the image generation model ends. Wherein M is an integer greater than 0.

[0078] That is to say, the end condition in the embodiment of the present disclosure may be "the image quality of the first image satisfies the first set condition". Different methods of determining image quality may correspond to different first set conditions. In some embodiments, the score characterizing the image quality of the first image is determined by a multimodal scoring model. In this case, the first set condition may be a score threshold. When the score characterizing the image quality of the first image is greater than the score threshold, it is determined that the image quality of the first image satisfies the first set condition. In other embodiments, the rating characterizing the image quality of the first image is determined by manual annotation. In this case, the first set condition may be a set level. When the rating characterizing the image quality of the first image is greater than the set level, it is determined that the image quality of the first image satisfies the first set condition.

[0079] In this way, when the image generation model can generate a first image with high image quality without relying on the optimizer, the training of the image generation model is terminated to ensure that the image generation model has good reasoning ability.

[0080] The following is a specific description of the process of optimizing the first image by the optimizer. In specific implementation, the first image is encoded to obtain the latent space encoding of the first image, the second description information and the latent space encoding of the first image are used as inputs of the optimizer, the latent space encoding of the second image output by the optimizer is obtained, and the latent space encoding of the second image is decoded to obtain the second image.

[0081] The process of obtaining the latent space encoding of the first image can be understood as the process of encoding the image data of the first image into a low-dimensional latent space. Through the latent space encoding technology, the image generation model can improve the quality and resolution of the first image while maintaining computational efficiency.

[0082] In the disclosed embodiment, the core principle of the optimizer is to optimize the image by performing denoising. The denoising process performed by the optimizer can be divided into multiple steps. In each step, the noise in the first image is gradually reduced to improve the image quality, so that the generated second image is more delicate. In this way, for the first image output by the image generation model, the optimizer performs denoising on the first image, so that the optimizer outputs a second image with better image quality.

[0083] The disclosed embodiments support image optimization using an optimizer in different ways. In some embodiments, the second description information and the latent space encoding of the first image are used as inputs to the optimizer, so that the optimizer performs a first denoising process on the latent space encoding of the first image to obtain the latent space encoding of the second image output by the optimizer.

[0084] In other embodiments, the latent space coding of the first image is denoised, and the second description information and the latent space coding of the first image after the denoising process are used as inputs to the optimizer, so that the optimizer performs a second denoising process on the latent space coding of the first image after the denoising process to obtain the latent space coding of the second image output by the optimizer.

[0085] That is to say, the optimizer can directly perform denoising on the first image output by the image generation model, or it can first perform denoising on the first image output by the image generation model, and then use the optimizer to perform denoising on the first image after denoising. In this way, in the method of directly performing denoising on the first image output by the image generation model, the optimizer reduces the noise in the first image, so that the second image blurs the background picture compared with the first image, the background is smoother, and the main object is more prominent. In the method of first performing denoising on the first image output by the image generation model and then performing denoising, more content is added to the first image through denoising, so that the second image has more image details and richer image information, and more textures and lines than the first image. At the same time, since the input of the optimizer also includes the second description information, the second image is more matched with the second description information than the first image, and is more in line with the user's image generation needs.

[0086] In some possible implementations, the first denoising process and the second denoising process may have different denoising degrees. For example, the first denoising process may be used to perform the last n% denoising, where n may be a real number greater than 0 and less than 100, and the second denoising process may be used to perform 100% denoising.

[0087] After the optimizer is used to refine the first image to obtain a second image with better image quality, at least part of the second image can be used as the first training image for the next round of training process (i.e., the N+1th round of training process). In specific implementation, the second target image whose image quality meets the second set condition is selected from the second image, and for the N+1th round of training process, the description information of the second target image is used as the input of the image generation model, and the second target image is used as a label to train the image generation model.

[0088] The second set condition can be understood as a screening condition for screening the second target image from the second image. That is, the second image that meets the second set condition is determined as the second target image, and the image generation model is trained for the N+1th round using the second target image and the description information of the second target image.

[0089] The embodiments of the present disclosure do not impose any restrictions on the second setting conditions. For example, the second setting conditions can be configured from different dimensions (such as detail clarity, object completeness, degree of matching with image generation requirements, etc.). When the second image meets the second setting conditions in all dimensions, the second image is determined as the second target image.

[0090] In this way, the second image optimized by the optimizer is screened, and a high-quality second target image in the second image is determined as the first training data in the N+1 round of training process, thereby improving the quality of the training data in the N+1 round of training process.

[0091] Furthermore, considering that the optimizer may have different processing methods for the first image, namely, a method of directly performing denoising on the first image output by the image generation model, or a method of first performing denoising on the first image output by the image generation model and then performing denoising, different processing methods may correspond to different image styles, namely, a style of blurred background or a style of enriched details. Therefore, in the embodiment of the present disclosure, the description information of the second target image may include an image style identifier, which may be used to identify the image style of the second target image. The image style of the second target image is determined based on the processing method of the first image by the optimizer.

[0092] That is to say, by adding an image style identifier to the first training data in the N+1th round of training process, the image generation model can learn different image styles. In this way, after completing the training of the image generation model, in the actual reasoning process, if the image generation model is required to generate an image of a certain image style, the image style identifier corresponding to the image style can be input into the image generation model, so that the image generation model has the ability to generate images of different image styles.

[0093] Based on the training method of the image generation model provided above, the embodiment of the present disclosure also provides an image generation method. Figure 3 A schematic flow chart of an image generation method is shown, the method specifically comprising:

[0094] S301: Obtain first description information.

[0095] The first description information may be used to describe a desired image. In other words, the first description information may represent the image generation requirements of the user.

[0096] The disclosed embodiment does not limit the source of the first description information. For example, the user may extract the first description information from existing advertising materials. For another example, in a template generation scenario, the user may also extract the first description information from video materials.

[0097] S302: Input the first description information into the image generation model to obtain a target image output by the image generation model.

[0098] The image generation model can be obtained by training the image generation model training method provided above. Specifically, Figure 4 As shown, unlike the training process, in the inference process, there is no need to add an optimizer. By inputting the first description information into the image generation model, the image generation model can generate a target image matching the first description information based on the original noise image, thereby realizing image generation.

[0099] In this method, since the quality of training data is gradually improved during the training process of the image generation model, the image generation model has good image generation capabilities and can generate high-quality images that meet the image generation requirements (ie, the first description information).

[0100] Combination of the above Figures 1 to 4 The training method of the image generation model and the image generation method provided in the embodiments of the present disclosure are introduced in detail. The apparatus and device provided in the embodiments of the present disclosure will be introduced in conjunction with the accompanying drawings.

[0101] See also Figure 5 The structural diagram of the training device of the image generation model shown in FIG. 50 includes:

[0102] An acquisition module 501 is used to acquire first training data of the Nth round of training process for the Nth round of training process; wherein the first training data includes a first training image and description information of the first training image, and N is an integer greater than 0;

[0103] A training module 502, configured to use the description information of the first training image as an input of an image generation model and use the first training image as a label to train the image generation model;

[0104] The acquisition module 501 is further used to acquire second description information;

[0105] A generating module 503 is used to use the second description information as an input of the image generation model after the Nth round of training process to obtain a first image output by the image generation model;

[0106] An optimization module 504 is used to perform a fine processing on the first image using an optimizer to obtain a second image output by the optimizer; wherein the image quality of the second image is better than the image quality of the first image;

[0107] The training module 502 is further used to train the image generation model for the N+1th round of training process by using at least part of the second image and at least part of the description information of the second image as the first training data for the N+1th round of training process.

[0108] In some possible implementations, the training module 502 is further configured to:

[0109] After the Mth round of training process, if the image quality of the first image output by the image generation model meets the first set condition, the training of the image generation model is terminated, where M is an integer greater than 0.

[0110] In some possible implementations, the first training data is related to at least one of the following: a target business, a target content, or a target style.

[0111] In some possible implementations, the optimization module 504 is specifically configured to:

[0112] Encoding the first image to obtain a latent space code of the first image;

[0113] Taking the second description information and the latent space code of the first image as inputs of the optimizer, and obtaining the latent space code of the second image output by the optimizer;

[0114] The latent space encoding of the second image is decoded to obtain a second image.

[0115] In some possible implementations, the optimization module 504 is specifically configured to:

[0116] The second description information and the latent space coding of the first image are used as inputs of the optimizer, so that the optimizer performs a first denoising process on the latent space coding of the first image to obtain the latent space coding of the second image output by the optimizer.

[0117] In some possible implementations, the optimization module 504 is specifically configured to:

[0118] Performing noise processing on the latent space coding of the first image;

[0119] The second description information and the latent space coding of the first image after noise processing are used as inputs of the optimizer, so that the optimizer performs a second denoising process on the latent space coding of the first image after noise processing to obtain the latent space coding of the second image output by the optimizer.

[0120] In some possible implementations, the training module 502 is specifically used to:

[0121] Selecting a second target image whose image quality satisfies a second set condition from the second image;

[0122] For the N+1th round of training process, the description information of the second target image is used as the input of the image generation model, and the second target image is used as a label to train the image generation model.

[0123] In some possible implementations, the description information of the second target image includes an image style identifier, and the image style identifier is used to identify the image style of the second target image. The image style of the second target image is determined based on the processing method of the optimizer on the first image.

[0124] The training device 50 for the image generation model according to the embodiment of the present disclosure may correspond to the method described in the embodiment of the present disclosure, and the above and other operations and / or functions of each module / unit of the training device 50 for the image generation model are respectively to achieve Figure 1 For the sake of brevity, the corresponding processes of each method in the illustrated embodiment are not described in detail here.

[0125] See also Figure 6 The schematic diagram of the structure of the image generating device shown in FIG. 60 includes:

[0126] The acquisition module 601 is used to acquire first description information; wherein the first description information is used to describe the expected image;

[0127] The generation module 602 is used to input the first description information into the image generation model to obtain the target image output by the image generation model; wherein the image generation model is trained by the training method of the aforementioned image generation model.

[0128] The image generating device 60 according to the embodiment of the present disclosure may correspond to executing the method described in the embodiment of the present disclosure, and the above and other operations and / or functions of each module / unit of the image generating device 60 are respectively to implement Figure 3 For the sake of brevity, the corresponding processes of each method in the illustrated embodiment are not described in detail here.

[0129] The present disclosure also provides an electronic device. The electronic device is specifically used to implement Figure 5 The function of the training device 50 of the image generation model in the embodiment shown, or, for realizing the following Figure 6 The function of the image generating device 60 in the embodiment shown.

[0130] Figure 7 A schematic diagram of the structure of an electronic device 700 is provided. Figure 7As shown, the electronic device 700 includes a bus 701, a processor 702, a communication interface 703 and a memory 704. The processor 702, the memory 704 and the communication interface 703 communicate with each other via the bus 701.

[0131] The bus 701 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0132] The processor 702 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0133] The communication interface 703 is used for communicating with the outside. For example, the communication interface 703 can be used for communicating with a terminal.

[0134] The memory 404 may include a volatile memory, such as a random access memory (RAM). The memory 704 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0135] The memory 704 stores executable codes, and the processor 702 executes the executable codes to perform the aforementioned image generation model training method or image generation method.

[0136] Specifically, in implementing Figure 5 or Figure 6 In the case of the embodiment shown, and Figure 5 The training device 50 or the image generation model described in the embodiment Figure 6 When each module or unit of the image generating device 60 described in the embodiment is implemented by software, the execution Figure 5 or Figure 6 The software or program code required for the functions of each module / unit in the system may be partially or completely stored in the memory 704. The processor 702 executes the program code corresponding to each unit stored in the memory 704 to execute the training method or image generation method of the aforementioned image generation model.

[0137] The present disclosure embodiment also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by the computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the training method of the image generation model of the training device 50 applied to the image generation model, or execute the image generation method applied to the image generation device 60.

[0138] The embodiment of the present disclosure also provides a computer program product, which includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, the process or function described in the embodiment of the present disclosure is generated in whole or in part.

[0139] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer or data center to another website, computer or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0140] When the computer program product is executed by a computer, the computer executes any of the aforementioned methods for training the image generation model or any of the aforementioned methods for generating images. The computer program product may be a software installation package, and when any of the aforementioned methods for training the image generation model or any of the aforementioned methods for generating images is needed, the computer program product may be downloaded and executed on a computer.

[0141] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to each embodiment of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0142] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit / module does not, in some cases, constitute a limitation on the unit itself.

[0143] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0144] In the context of the disclosed embodiments, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0145] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system or device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0146] It should be understood that in the present disclosure, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0147] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0148] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0149] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A training method for an image generation model, characterized in that: The method comprises: For an N-th round of training process, obtaining first training data of the N-th round of training process; wherein the first training data includes a first training image and description information of the first training image, and N is an integer greater than 0; Using the description information of the first training image as input of an image generation model and using the first training image as a label to train the image generation model; Obtaining second description information; Using the second description information as input of the image generation model after the Nth round of training process, to obtain a first image output by the image generation model; Using an optimizer to perform fine processing on the first image to obtain a second image output by the optimizer; wherein the image quality of the second image is better than the image quality of the first image; For the N+1th round of training process, at least part of the second image and at least part of the description information of the second image are used as the first training data of the N+1th round of training process to train the image generation model.

2. The method according to claim 1, characterized in that The method further comprises: After the Mth round of training process, if the image quality of the first image output by the image generation model meets the first set condition, the training of the image generation model is terminated, where M is an integer greater than 0.

3. The method according to claim 1, characterized in that: The first training data is related to at least one of the following: a target business, a target content, or a target style.

4. The method according to claim 1, characterized in that: The step of performing fine processing on the first image by using an optimizer to obtain a second image output by the optimizer includes: Encoding the first image to obtain a latent space code of the first image; Taking the second description information and the latent space code of the first image as inputs of the optimizer, and obtaining the latent space code of the second image output by the optimizer; The latent space encoding of the second image is decoded to obtain a second image.

5. The method according to claim 4, characterized in that The step of taking the second description information and the latent space encoding of the first image as inputs of the optimizer to obtain the latent space encoding of the second image output by the optimizer comprises: The second description information and the latent space coding of the first image are used as inputs of the optimizer, so that the optimizer performs a first denoising process on the latent space coding of the first image to obtain the latent space coding of the second image output by the optimizer.

6. The method according to claim 4, characterized in that The step of taking the second description information and the latent space encoding of the first image as inputs of the optimizer to obtain the latent space encoding of the second image output by the optimizer comprises: Performing noise processing on the latent space coding of the first image; The second description information and the latent space coding of the first image after noise processing are used as inputs of the optimizer, so that the optimizer performs a second denoising process on the latent space coding of the first image after noise processing to obtain the latent space coding of the second image output by the optimizer.

7. The method according to any one of claims 1 to 6, characterized in that: The step of using at least part of the second image and at least part of the description information of the second image as the first training data in the N+1th round of training process to train the image generation model includes: Selecting a second target image whose image quality satisfies a second set condition from the second image; For the N+1th round of training process, the description information of the second target image is used as the input of the image generation model, and the second target image is used as a label to train the image generation model.

8. The method according to claim 7, characterized in that The description information of the second target image includes an image style identifier, and the image style identifier is used to identify the image style of the second target image. The image style of the second target image is determined based on the processing method of the optimizer on the first image.

9. An image generation method, characterized in that: The method comprises: Acquire first description information; wherein the first description information is used to describe the desired image; The first description information is input into an image generation model to obtain a target image output by the image generation model; wherein the image generation model is trained by the method according to any one of claims 1 to 8.

10. A training device for an image generation model, characterized in that: The device comprises: An acquisition module, configured to acquire, for an N-th round of training process, first training data of the N-th round of training process; wherein the first training data includes a first training image and description information of the first training image, and N is an integer greater than 0; a training module, configured to use the description information of the first training image as an input of an image generation model and use the first training image as a label to train the image generation model; The acquisition module is further used to acquire second description information; A generation module, used to use the second description information as an input of the image generation model after the Nth round of training process to obtain a first image output by the image generation model; an optimization module, configured to perform fine processing on the first image using an optimizer to obtain a second image output by the optimizer; wherein the image quality of the second image is better than that of the first image; The training module is also used to train the image generation model for the N+1th round of training process by using at least part of the second image and at least part of the description information of the second image as the first training data for the N+1th round of training process.

11. An image generating device, characterized in that: The device comprises: An acquisition module, used for acquiring first description information; wherein the first description information is used for describing a desired image; A generation module, used to input the first description information into an image generation model to obtain a target image output by the image generation model; wherein the image generation model is trained by the method described in any one of claims 1 to 8.

12. An electronic device, characterized in that: The electronic device comprises a processor and a memory; The processor is configured to execute instructions stored in the memory, so that the electronic device performs the method according to any one of claims 1 to 8 or claim 9.

13. A computer-readable storage medium, characterized in that: The method comprises instructions, wherein the instructions instruct an electronic device to execute the method according to any one of claims 1 to 8 or claim 9.

14. A computer program product, characterized in that The computer program product comprises computer readable instructions for implementing the method of any one of claims 1 to 8 or claim 9.

Images