Image generation model training method and device, image generation method and device, equipment and medium

Abstract

The embodiment of the invention provides an image generation model training method and device, an image generation method and device, equipment and a medium, and the method comprises the steps: obtaining any first image description in an initial sample set which is a sample set used for training a target image generation model; based on the first image description, through a target text generation model, obtaining a plurality of prediction descriptions; based on each prediction description, through a target image generation model, obtaining at least two first generation images corresponding to each prediction description; based on the first image description and the at least two first generated images, determining an expanded image-text pair for training a target image generation model; and training the target image generation model by using the first generated image and the first image description in the expanded image-text pair, so as to improve the quality of the image generated by the target image generation model obtained by training, and improve the stability of the capability of generating the image with better quality.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to training of an image generation model, an image generation method, a device, a equipment and a medium. Background Art

[0002] In the field of image generation, image descriptions (also called text prompts) can be used to obtain images corresponding to the image descriptions through image generation models based on content generation technology. In the above process, the stronger the image generation capability of the image generation model, the better the quality of the image generated by the same image description.

[0003] Therefore, in order to ensure the quality of the generated images, it is necessary to train an image generation model with better image generation capabilities. Summary of the invention

[0004] One or more embodiments of the present disclosure provide a training of an image generation model, an image generation method, an apparatus, a device, and a medium to achieve a trained image generation model that can generate images with better quality.

[0005] According to a first aspect, a method for training an image generation model is provided, comprising:

[0006] Obtain a first image description in an initial sample set;

[0007] Based on the first image description, a plurality of predicted descriptions are obtained by using a target text generation model;

[0008] Based on each prediction description, at least two first generated images corresponding to each prediction description are obtained through the target image generation model;

[0009] Determining, based on the first image description and the at least two first generated images, an extended image-text pair for training the target image generation model;

[0010] The target image generation model is trained using the first generated image and the first image description in the extended image-text pair.

[0011] In an optional embodiment, the determination of the extended image-text pairs for training the target image generation model includes: determining the quality scores corresponding to each first description image pair based on a preset quality assessment algorithm, wherein each first description image pair includes the first image description and a first generated image; determining a number of target description image pairs whose corresponding quality scores meet the preset quality conditions from all first description image pairs based on the quality scores corresponding to each first description image pair; combining the first generated images in each target description image pair with the first image description respectively to obtain extended image-text pairs for training the target image generation model. By calculating the quality scores corresponding to each first description image pair, and then screening the first description image pairs based on the quality scores corresponding to each first description image pair, a target description image pair with better quality is obtained, and then the image in the target description image pair and the first image description are combined into an extended image-text pair to ensure that the quality of the determined extended image-text pair is better, and then the extended image-text pairs with better quality can be used to train the target image generation model to better improve the image generation capability of the target image generation model.

[0012] In an optional implementation, the quality score corresponding to each first description image pair is determined based on a preset quality assessment algorithm, including: determining the matching value corresponding to each first description image pair through a text-image association model based on the first image description and the first generated image in each first description image pair; determining the aesthetic score corresponding to each first description image pair through an aesthetic prediction model based on the first generated image in each first description image pair; and determining the quality score corresponding to each first description image pair based on the matching value and the aesthetic score corresponding to each first description image pair. The quality score corresponding to each first description image pair is determined by combining the matching value between the image and the description in each first description image pair and the aesthetic score of the image in each first description image pair, so as to ensure that first description image pairs with high matching degree (i.e., large matching value) and high image aesthetics (i.e., large aesthetic score) can be screened out, so as to ensure better quality of the expanded text-image pair.

[0013] In an optional implementation, the target description image pairs whose corresponding quality scores meet the preset quality conditions include: N first description image pairs whose corresponding quality scores are the largest, where N is a positive integer; or, the target description image pairs whose corresponding quality scores meet the preset quality conditions include: several first description image pairs whose corresponding quality scores exceed a preset quality score threshold. In this way, several target description image pairs with relatively best quality among the several first description image pairs are obtained, thereby ensuring that the quality of the obtained extended image-text pairs is better.

[0014] In an optional embodiment, before obtaining at least two first generated images corresponding to each prediction description through the target image generation model based on each prediction description, it also includes: obtaining a second image description and its corresponding sample image; obtaining a second generated image through the image generation model to be trained based on the second image description; determining the first prediction loss corresponding to the image generation model to be trained using the difference between the sample image and the second generated image; adjusting the parameters of the image generation model to be trained with the goal of minimizing the first prediction loss to obtain an image generation model that meets the first convergence condition as the target image generation model. The image generation model to be trained is trained through the second image description and its corresponding sample image, so that the target image generation model has the ability to generate images based on image descriptions.

[0015] In an optional embodiment, before obtaining multiple predicted descriptions through the target text generation model based on the first image description, it also includes: obtaining a third image description; using a preset occlusion algorithm to occlude part of the description of the third image description to obtain a fourth image description; based on the fourth image description, obtaining the output of the text generation model to be trained through the text generation model to be trained; using the output of the text generation model to be trained and the third image description, determining the second prediction loss corresponding to the text generation model to be trained; with the goal of minimizing the second prediction loss, adjusting the parameters of the text generation model to be trained to obtain the target text generation model that meets the second convergence condition. The text generation model to be trained is trained through the third image description and its corresponding fourth image description, so that the target text generation model has the ability to expand, complete and predict image descriptions.

[0016] In an optional embodiment, the first image description is multiple; the use of the first generated image and the first image description in the extended image-text pair to train the target image generation model includes: using the extended image-text pairs corresponding to each first image description to cyclically train the target image generation model. The target image generation model is cyclically trained using the extended image-text pairs with good quality corresponding to multiple first image descriptions, so that the target image generation model trained based on the extended image-text pairs can be cyclically used to perform data expansion to obtain extended image-text pairs with improved quality, and the extended image-text pairs with improved quality can be reused to cyclically train the trained target image generation model, so as to obtain extended image-text pairs with better quality and an image generation model with better image generation capability.

[0017] According to a second aspect, there is provided an image generation method, comprising:

[0018] Get the image description to be processed;

[0019] Based on the description of the image to be processed, a target image is obtained by using an image generation model trained by any of the image generation model training methods described in the first aspect.

[0020] According to a third aspect, a training device for an image generation model is provided, comprising:

[0021] A first acquisition module, configured to acquire a first image description in an initial sample set;

[0022] A first obtaining module is configured to obtain a plurality of predicted descriptions based on the first image description by using a target text generation model;

[0023] A second obtaining module is configured to obtain at least two first generated images corresponding to each prediction description respectively through the target image generation model based on each prediction description;

[0024] A first determination module, configured to determine an extended image-text pair for training the target image generation model based on the first image description and the at least two first generated images;

[0025] The training module is configured to train the target image generation model using the first generated image and the first image description in the extended image-text pair.

[0026] According to a fourth aspect, there is provided an image generating device, comprising:

[0027] A second acquisition module is configured to acquire a description of the image to be processed;

[0028] The third obtaining module is configured to obtain a target image based on the description of the image to be processed by using the image generation model trained by the image generation model training device described in the third aspect.

[0029] According to a fifth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method described in the first aspect or the second aspect.

[0030] According to a sixth aspect, an electronic device is provided, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in the first aspect or the second aspect is implemented.

[0031] According to the training, image generation method, apparatus, device and medium of the image generation model provided by the embodiments of the present disclosure, after obtaining the first image description in the initial sample set for training the target image generation model, multiple prediction descriptions are obtained based on the first image description through the target text generation model, and then at least two first generated images corresponding to each prediction description are obtained through the target image generation model based on each prediction description, so that the image quality of the first generated images generated is better than the image quality of the images generated based on the first image description. Then, based on the first image description and the at least two first generated images, an expanded image-text pair for training the target image generation model is determined, and the image-text pair can be expanded to obtain training data with better quality and more quantity for training the target image generation model. Then, the target image generation model is trained using the first generated image and the first image description in the expanded image-text pair, so that the target image generation model is trained using the expanded image-text pair with better quality and relatively more data, so as to better improve the quality of the images generated by the trained target image generation model and improve the stability of its ability to generate images with better quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0033] Figure 1 A schematic diagram of an implementation framework of an embodiment disclosed in the present disclosure;

[0034] Figure 2 A schematic diagram of a flow chart of a training method for an image generation model provided in an embodiment;

[0035] Figure 3 Another schematic flow chart of the image generation method provided in the embodiment;

[0036] Figure 4 A schematic block diagram of a training device for an image generation model provided in an embodiment;

[0037] Figure 5 Another schematic block diagram of an image generating device provided in an embodiment;

[0038] Figure 6 A schematic block diagram of an electronic device provided in an embodiment. DETAILED DESCRIPTION

[0039] The technical solution of the embodiment of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0040] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0041] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0042] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0043] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0044] The present disclosure discloses a method and device for expanding the training data of an image generation model. The application scenarios and technical concepts of the method are first introduced as follows:

[0045] As mentioned above, in the field of image generation, image descriptions (also called text prompts) can be used to obtain images corresponding to the image descriptions through image generation models based on content generation technology. In the above process, the stronger the image generation capability of the image generation model, the better the quality of the image generated by the same image description. Therefore, in order to ensure the quality of the generated images, it is necessary to train an image generation model with better image generation capabilities.

[0046] However, there is currently a lack of training data that can better improve the image generation capability of the image generation model (i.e., enable the image generation model to generate higher quality images). Therefore, it is crucial to obtain more training data that can better improve the image generation capability of the image generation model, and then use this type of training data to continue training the image generation model to improve the image generation capability of the image generation model.

[0047] In view of this, Figure 1A schematic diagram of an implementation scenario according to an embodiment disclosed in the present disclosure is shown. In the implementation scenario, a training system for training an image generation model is exemplarily shown, and the training system includes: a target text generation model and a target image generation model to be trained. The training system can first use the image description to perform data expansion through the target text generation model and the target image generation model to obtain expanded image-text pairs, and then use the expanded image-text pairs to continue training the target image generation model to improve the image generation capability of the target image generation model so that it can generate images with better quality.

[0048] Among them, the target image generation model can be an image generation model that has been pre-trained using a number of image-text pairs, and the target image generation model has the ability to generate images based on the input text (image description). Among them, each image-text pair includes an image description and an image corresponding to the image description. For the sake of clear description, the image in the image-text pair can be called the second sample image, and the image description in the image-text pair can be called the second image description. In one implementation, the image-text pair can be obtained from any specified public data that can provide a large number of image-text pairs or a specified artificially produced data set. The artificially produced data set includes: images obtained from specified channels and image descriptions manually annotated for the images.

[0049] The target text generation model can be a text generation model that has been pre-trained using each text and the occluded text generated after the occluded part of each text. Each text is an image description used to generate an image. For the sake of clarity, each text can be called a third image description. The target text generation network has the ability to predict and complete text.

[0050] In one implementation, the aforementioned image description may also be referred to as a text prompt, that is, a text prompt used to generate an image.

[0051] The following describes the process of training a target image generation model using the aforementioned training system, which may include:

[0052] A sample set for training the target image generation model is obtained, which is called an initial sample set. The initial sample set includes at least several image descriptions. Using the aforementioned training system and each image description in the initial sample set, data expansion is performed to obtain an expanded image-text pair for training the target image generation model, and then the expanded image-text pair is used to continue training the target image generation model. It can be understood that the process of using each image description in the initial sample set to perform data expansion and then using the expanded image-text pair obtained by data expansion to train the target image generation model is similar. The following takes any first image description in the initial sample set as an example to specifically describe the above process.

[0053] An image description is obtained from an initial sample set as a first image description; based on the first image description, multiple predicted descriptions are obtained through a target text generation model, that is, the first image description is input into the target text generation model to obtain multiple predicted descriptions corresponding to the first image description. Next, based on each predicted description, at least two first generated images corresponding to each predicted description are obtained through a target image generation model, that is, each predicted description is input into the target image generation model, and each predicted description is processed by the target image generation model to obtain at least two generated images corresponding to each predicted description. Next, based on the first image description and the at least two first generated images corresponding to each predicted description, an extended image-text pair for training the target image generation model is determined; and the target image generation model is trained using the first generated image and the first image description in the extended image-text pair.

[0054] In one implementation, in the process of training the target image generation model using the first generated image and the first image description in the extended image-text pair, the target image generation model in the aforementioned training system can be updated in real time to the target image generation model trained using the extended image-text pair, so that the image descriptions in the initial sample set or other sample sets used to train the target image generation model can continue to be used. Through the target image generation model and the target text generation model trained using the extended image-text pair, more and better quality extended image-text pairs can be obtained. Subsequently, the obtained extended image-text pairs can be used to continue training the target image generation model trained using the extended image-text pair, so as to better improve the image generation capability of the model.

[0055] In the above process, the image-text pairs can be expanded to obtain better quality training data for training the target image generation model, that is, expanded image-text pairs. The expanded image-text pairs can then be used to train the target image generation model, so that the image generation capability of the target image generation model is better.

[0056] The training process of the aforementioned image generation model may be repeated multiple times to obtain more expanded image-text pairs with better quality and to better improve the image generation capability of the image generation model.

[0057] The following is a detailed description of the training of the image generation model, the image generation method, the device, the equipment and the medium provided by the present disclosure in conjunction with specific embodiments. For a clear layout, the process of training the target image generation model and the target text generation model is first introduced.

[0058] Among them, the process of training to obtain the target image generation model may include: first obtaining a training data set S for training the image generation model to be trained, and the training data set S may include multiple pairs of image-text pairs, and one image-text pair includes an image description (called the second image description) X and its corresponding sample image T. In one case, the training data set S may be obtained from a specified public data set that can provide a large number of image-text pairs or an artificially produced data set. The artificially produced data set may include a captured image and its corresponding image description that is manually annotated for the image. In one case, the image description may also be called a text prompt, that is, a text prompt used to generate an image.

[0059] Then, the image generation model to be trained is obtained, and the second image description X in any image-text pair M in the training data set S is input into the image generation model to be trained, so that the image generation model processes the second image description X and generates a corresponding generated image, which is called the second generated image T'; the first prediction loss corresponding to the image generation model to be trained is determined by using the difference between the sample image T in the aforementioned image-text pair M and the second generated image T'. In one implementation, a cross entropy loss function or an MSE (mean square error) loss function or an MAE (mean absolute error) loss function can be used to determine the first prediction loss based on the difference between the sample image T and the second generated image T'.

[0060] Afterwards, the parameters of the image generation model to be trained are adjusted to minimize the first prediction loss, that is, to reduce the image difference between the sample image and the second generated image, until the image generation model reaches the first convergence condition, thereby obtaining a trained image generation model, and then obtaining a target image generation model. The first convergence condition may include, but is not limited to, the number of parameter adjustments is not less than the first adjustment threshold, or the calculated first prediction loss is less than the first loss threshold, etc.

[0061] Let's review the process of obtaining the target image generation model through the aforementioned training. The above embodiment is explained by taking an image-text pair (i.e., a pair of second image descriptions and their corresponding sample images) as an example. In another embodiment, the above process can also be performed on a batch of samples, i.e., multiple image-text pairs, to obtain the second generated images corresponding to the second image descriptions in each image-text pair, and then based on the second generated images corresponding to the second image descriptions in each image-text pair and the sample images in the image-text pair, the first prediction loss is jointly determined, and the parameters of the image generation model to be trained are adjusted with the goal of minimizing the first prediction loss. In this embodiment, the first prediction loss is determined for a batch of samples, and then the parameters of the image generation model are adjusted. This can reduce the number of parameter adjustments to the image generation model and make it easier to implement the above process.

[0062] In one implementation, the target image generation model may be an image generation model based on a Diffusion Model.

[0063] The process of training to obtain the target text generation model may include: constructing a sample description pair set D for training the text generation model to be trained, wherein the sample description pair set D may include a plurality of sample description pairs, wherein a sample description pair D i The construction process can be as follows: Get the third image description y i (For example, it can be expressed as {(a)(clear)(windless)(morning)}, or it can be expressed as {(a)(clear)(bright)(windless)(morning)(morning)}, where the content in each bracket represents the third image description y i The first representation is used as an example below), and it is classified into the sample description pair D i ; Using the preset occlusion algorithm, describe the third image y i The partial description (partial content) of is occluded to obtain the corresponding occluded third image description as the fourth image description x i (For example, it can be expressed as {(a)(?)(?)(of)(morning)}, where the question mark "?" is the blocked word), and it is classified into the sample description pair D i , that is, a sample description pair D i A third image description y is included i and its corresponding fourth image description x i . Among them, the preset masking algorithm can be any algorithm in the relevant technology that can realize text masking. In one implementation, using the preset masking algorithm, any content (or called words) at any other position except the first sentence (for example, the sentence before the first punctuation mark) of the third image description yi can be randomly masked; and words at any other position except the first a words of the fourth image description yi can also be randomly masked, where a is a positive integer.

[0064] In one case, the third image describes y i The image description may be generated to meet the preset conditions, wherein the preset conditions may include but are not limited to: the aesthetic score of the image is not less than a certain aesthetic score threshold, the clarity score of the image is not less than a certain clarity threshold, and / or the matching value between the image and the corresponding image description is not less than a certain matching threshold. In another case, the third image description y i It can be any image description obtained from the aforementioned public dataset or artificially created dataset.

[0065] After that, the text generation model to be trained is obtained, and then D is described based on any samplei The fourth image in the description x i , through the text generation model to be trained, the output of the text generation model to be trained is obtained, and based on the output, it can be determined that the text generation model to be trained is for the fourth image description x i The generated description is obtained by using the output of the text generation model to be trained and the third image description y i , determine the second prediction loss corresponding to the text generation model to be trained.

[0066] In one implementation, the output of the text generation model to be trained may include a first word probability set corresponding to each position in a plurality of positions, the first word probability set corresponding to each position includes a first probability that each word (token) in a preset word set appears in the position, the preset word set includes a set of words that can be predicted by the text generation model to be trained, and each third sample description yi may include a plurality of words in the preset word set. The specific number of the plurality of positions may be determined based on a specified parameter set for the text generation model to be trained, indicating the longest length of the output.

[0067] After obtaining the output of the text generation model to be trained, the third image description y i , from the output of the text generation model to be trained, determine the text generation model to be trained for the fourth image description x i Each blocked position in the image (can also be the fourth image description x i The third image description y appears at each position in i The first probability of the word at the corresponding position (for example, in the above example, the fourth image description x i The probability of the word "sunny" appearing at the first occluded position, i.e., the second position, in the fourth image description x i The probability of the word “no wind” appearing at the second blocked position, i.e., the third position in the image).

[0068] Then the cross entropy loss function is used to describe the fourth image x i Each occluded position in (or the fourth image description x i The third image description y appears at each position of i The first probability of the word at the corresponding position determines the second prediction loss. i Each occluded position in (or the fourth image description x i The third image description y appears at each position of i The greater the first probability of the word at the corresponding position, the smaller the second prediction loss, indicating that the image description completion and expansion capability of the text generation model to be trained is better. Correspondingly, in one case, it can be expressed by the following formula:

[0069] Among them, l represents the second prediction loss, p(y i |x i ) represents the text generation model to be trained for the fourth image description x i , output the probability of the third image description yi, p(y i |x i ) can be described based on the fourth image x i Each occluded position in (or the fourth image description x i The third image description y appears at each position of i The first probability of the word at the corresponding position is determined (for example, based on the fourth image description x i Each occluded position in (or the fourth image description x i The third image description y appears at each position of i The average or multiplication result of the sum of the first probabilities of the words at the corresponding positions), D represents the fourth image description x i and the third image description y i The sample description pair set D to which it belongs, It means to calculate logp(y i |x i )’s expected value.

[0070] In the above implementation, the target program can also determine the generated description corresponding to the fourth image description based on the output of the text generation model to be trained. Specifically, the generated description corresponding to the fourth image description is determined in combination with the output of the text generation model to be trained and the specified word determination rule, and then the second prediction loss is determined based on the difference between the generated description and the third image description (for example, the difference between the two is represented by the Euclidean distance between the two), wherein the smaller the difference between the generated description and the third image description, the smaller the second prediction loss. The specified word determination rule can be any rule in the related art that can determine the corresponding word according to the word probability set corresponding to each of the aforementioned several positions (the word at one / some positions can be empty). In one implementation, the parameters in the word determination rule can include pre-set parameters, and can also include parameters adjusted following the training process of the text generation model to be trained.

[0071] In another implementation, the output of the aforementioned text generation model to be trained may include a generated description corresponding to the fourth image description, and accordingly, the second prediction loss may be determined based on the difference between the generated description and the third image description.

[0072] Afterwards, to minimize the second prediction loss, that is, to maximize the text generation model to be trained for the fourth image description xi , output the probability of the third image description yi, that is, reduce the difference between the generated description and the third image description as the goal, adjust the parameters of the text generation model to be trained, until the text generation model to be trained reaches the second convergence condition, and obtain the target text generation model. Wherein, the second convergence condition may include but is not limited to, the number of parameter adjustments is not less than the second adjustment threshold, or the calculated second prediction loss is lower than the second loss threshold. The second adjustment threshold may be the same as or different from the aforementioned first adjustment threshold, and the second loss threshold may be the same as or different from the aforementioned first loss threshold.

[0073] Let's review the process of obtaining the target text generation model through the aforementioned training. The above embodiment is explained by taking a sample description pair (i.e., a pair of third image descriptions and their corresponding fourth image descriptions) as an example. In another embodiment, the above process can also be performed on a batch of samples, i.e., multiple sample description pairs, to obtain the output of the text generation model to be trained corresponding to the fourth image description in each sample description pair, and then determine the second prediction loss based on the output of the text generation model to be trained corresponding to the fourth image description in each sample description pair and the third image description in the sample description pair, and adjust the parameters of the text generation model to be trained with the goal of minimizing the second prediction loss. In this embodiment, the second prediction loss is determined for a batch of samples, and then the parameters of the text generation model are adjusted. This can reduce the number of parameter adjustments to the text generation model and make it easier to implement the above process.

[0074] In one implementation, the target text generation model may be a text generation model based on Transforme implementation.

[0075] After obtaining the target image generation model and the target text generation model, the two are combined to continue data expansion to obtain more expanded image-text pairs for further fine-tuning (training) the target image generation model, and then the expanded image-text pairs are used to further fine-tune (train) the target image generation model to improve the image generation capability of the image generation model.

[0076] Figure 2 The flowchart of the training method of the image generation model in one embodiment of the present disclosure is shown. The method can be implemented by a target program. The target program can be installed in an electronic device, and the electronic device can be implemented by any device, equipment, platform, device cluster, etc. with computing and processing capabilities. Figure 2 As shown, the method includes the following steps S210-S240:

[0077] In step S210, a first image description in an initial sample set is obtained. The initial sample set is a sample set used to train a target image generation model. The initial sample set may include at least a plurality of image descriptions, and the first image description is any image description among the plurality of image descriptions. In one implementation, the initial sample set may include the aforementioned image descriptions used in the process of training the target image generation model, such as the aforementioned training data set S.

[0078] In the process of first training the target image generation model based on the aforementioned training system, the target image generation model may be the target image generation model obtained by the aforementioned training, for example, the target image generation model obtained by training using the aforementioned training data set S and meeting the first convergence condition.

[0079] In the process of training the target image generation model based on the aforementioned training system which is not the first time, the target image generation model may be a target image generation model trained using a specified extended image-text pair. The specified extended image-text pair is an extended image-text pair (including an image description and its corresponding generated image) obtained by extension in the process of training the target image generation model based on the aforementioned training system before the current process of training the target image generation model based on the aforementioned training system.

[0080] Then, in step S220, based on the first image description, a target text generation model is used to obtain multiple predicted descriptions.

[0081] In this step, the target program can input the first image description into the target text generation model, and the target text generation model processes the first image description to obtain multiple model outputs corresponding to the first image description. In one implementation, the target text generation model is pre-set to output the multiple model outputs.

[0082] In one implementation, each model output may include a second word probability set corresponding to each of the several positions determined by the target text generation model based on the first image description, wherein the second word probability set for each position may include a second probability of each word in a preset word (token) set appearing at the position determined by the target text generation model for the first image description, and the preset word set is a set of words that can be predicted by the target text generation model. Based on the second word probability set corresponding to each of the several positions included in each model output, each predicted description can be determined. Specifically, the words corresponding to the corresponding positions can be determined in combination with the second word probability set corresponding to each of the several positions included in each model output and the specified word determination rules, that is, multiple predicted descriptions can be obtained, and each model output corresponds to a predicted description. In another implementation, each model output may directly include each predicted description.

[0083] In one case, the specific number m of the multiple predicted descriptions can be set according to demand. For example, the specific number m of the multiple predicted descriptions can be set to 4, that is, for a first image description, corresponding 4 predicted descriptions can be obtained through the target text generation model.

[0084] Next, in step S230, based on each prediction description, at least two first generated images corresponding to each prediction description are obtained through the target image generation model. In this step, the target program can input each prediction description into the target image generation model, and the target image generation model processes each prediction description to obtain at least two first generated images corresponding to each prediction description. In one implementation, the target image generation model is pre-set to output at least two images.

[0085] In one case, the specific number n of at least two first generated images corresponding to each prediction description can be set according to demand. For example, the specific number n of at least two first generated images can be set to 2 or more, that is, for one prediction description, at least two generated images corresponding to the prediction description can be obtained through the target image generation model.

[0086] After obtaining at least two first generated images corresponding to each predicted description, in step S240, based on the first image description and the at least two first generated images corresponding to each predicted description, an extended image-text pair for training the target image generation model is determined.

[0087] In one embodiment, in order to obtain a large number of training data for training the target image generation model, the target program can combine at least two first generated images corresponding to each predicted description with the first image description to obtain multiple image-text pairs, wherein each image-text pair includes the first image description and a first generated image, and the obtained multiple image-text pairs are determined as extended image-text pairs for training the target image generation model, i.e., training data for training the target image generation model.

[0088] In order to obtain an image generation model with better image generation capability, it is necessary to ensure both the quantity and quality of the training data for training the target image generation model. In view of this, in another embodiment, step S240 may include the following steps 11-13:

[0089] In step 11, based on a preset quality assessment algorithm, a quality score corresponding to each first description image pair is determined, wherein each first description image pair includes a first image description and a first generated image. The preset quality assessment algorithm may be any algorithm in the related art that can assess the quality of an image-text pair, and the quality of the image-text pair may be represented by the quality of the image in the image-text pair and / or the matching degree between the image and the image description in the image-text pair, wherein the better the quality of the image in the image-text pair, the stronger the matching degree between the image and the image description in the image-text pair, and the better the quality of the image-text pair. The quality of the image may be represented by the clarity of the image and / or the aesthetics of the image, wherein the higher the clarity of the image, the better the aesthetics of the image, and the better the quality of the image.

[0090] In one implementation, step 11 may include the following steps 111-113:

[0091] In step 111, based on the first image description and the first generated image in each first description image pair, the matching value corresponding to each first description image pair is determined through a text-image association model. In one implementation, the text-image association model may be a CLIP model (Contrastive Language-image pre-training). The text-image association model may be a model pre-trained based on sample text-image pairs and their corresponding labels (including matching value labels representing the matching between the image in the corresponding sample text-image pair and its corresponding image description). The text-image association model may output a matching value representing the matching between the image and the image description in the text-image pair for an input text-image pair (i.e., the image and the image description therein).

[0092] In this step, the target program can input the first image description and the first generated image in each first description image pair into the image-text association model to obtain the matching value between the first image description and the first generated image in each first description image pair, that is, the matching value corresponding to each first description image pair. The matching value can represent the matching degree, i.e., the correlation, between the first image description and the first generated image in the corresponding first description image pair. The higher the matching value, the higher the matching degree, i.e., the correlation, between the first image description and the first generated image in the corresponding first description image pair, and accordingly, the better the quality of the first description image pair.

[0093] Then in step 112, based on the first generated image in each first description image pair, the aesthetic score corresponding to each first description image pair is determined through an aesthetic prediction model. In one implementation, the aesthetic prediction model can be any model in the related art that can evaluate the aesthetics of an image. The aesthetic prediction model can be an aesthetic prediction model obtained by training based on any data set that can provide a number of images and aesthetic score labels corresponding to each image. The aesthetic prediction model can estimate the aesthetic score of an input image. In this step, the target program inputs the first generated image in each first description image pair into the aesthetic prediction model respectively to obtain the aesthetic score of the first generated image in each first description image pair, that is, the aesthetic score corresponding to each first description image pair.

[0094] In one case, a higher aesthetic score corresponding to the first description image pair can indicate a higher quality of the first generated image in the first description image pair, and accordingly, a higher quality of the first description image pair.

[0095] After obtaining the matching value and aesthetic score corresponding to each first description image pair through the above process, in step 113, the quality score corresponding to each first description image pair is determined based on the matching value and aesthetic score corresponding to each first description image pair.

[0096] In one implementation, the target program may determine the quality score corresponding to each first description image pair based on the matching value and the aesthetic score corresponding to each first description image pair, as well as the first weight value corresponding to the matching value and the second weight value corresponding to the aesthetic score. For example, for each first description image pair, such as the first description image pair j, the product of the matching value and the first weight value corresponding to the first description image pair j and the product of the aesthetic score and the second weight value corresponding to the first description image pair j may be determined as the quality score corresponding to the first description image pair j.

[0097] In yet another implementation, the target program may also directly determine the matching value or aesthetic score corresponding to each of the aforementioned first description image pairs as the quality score corresponding to each of the first description image pairs.

[0098] In another implementation, the target program may also determine the clarity score corresponding to the first generated image in each first description image pair based on a preset image clarity assessment model as the clarity score corresponding to each first description image pair, and then the clarity score, matching value and aesthetic score corresponding to each first description image pair may be combined to jointly determine the quality score corresponding to each first description image pair. The preset image clarity assessment model may be any model (algorithm) in the related art that can determine the clarity of an image, wherein the higher the clarity of the image, the greater the clarity score corresponding to the first description image pair containing the image.

[0099] After obtaining the quality scores corresponding to each first description image pair in the above manner, in step 12, based on the quality scores corresponding to each first description image pair, several target description image pairs whose corresponding quality scores meet the preset quality conditions are determined from all the first description image pairs.

[0100] In one implementation, several target description image pairs whose corresponding quality scores satisfy preset quality conditions include: N first description image pairs with the largest corresponding quality scores. That is to say, the target program can determine the N first description image pairs with the largest corresponding quality scores from all first description image pairs based on the quality scores corresponding to each first description image pair, as several target description image pairs. N is a positive integer, and its specific value can be determined according to actual needs. For example, N is set to a positive integer greater than 1. When the specific number m of multiple predicted descriptions is 4, and the specific number n of at least two first generated images is 2 (that is, one first image description can obtain 8 first generated images), the first description image pairs can be 8. Accordingly, N can be set to 2, that is, for one first image description, 2 target description image pairs can be obtained.

[0101] In another implementation, the target description image pairs whose corresponding quality scores satisfy the preset quality condition include: the first description image pairs whose corresponding quality scores exceed the preset quality score threshold. That is, the target program can determine the first description image pairs whose corresponding quality scores exceed the preset quality score threshold from all the first description image pairs based on the quality scores corresponding to the first description image pairs as the target description image pairs.

[0102] Through step 12, several target description image pairs with relatively good quality can be determined from all first description image pairs corresponding to the first image description. Then, in step 13, the first generated images in each target description image pair are respectively combined with the first image description to obtain an expanded image-text pair for training the target image generation model.

[0103] In this step, the target program can combine the first generated image in each target description image pair with the first image description to obtain a plurality of image-text pairs as extended image-text pairs for training the target image generation model. For example, the target description image pair includes target description image pair 1 and target description image pair 2, and correspondingly, the obtained extended image-text pairs include extended image-text pair 1 including the first generated image and the first image description in target description image pair 1, and extended image-text pair 1 including the first generated image and the first image description in target description image pair 2.

[0104] In another embodiment, after obtaining a plurality of target description image pairs, the target program may also use the first generated image in the plurality of target description image pairs and the predicted description corresponding to the first generated image as an expanded image-text pair for training the target image generation model.

[0105] After obtaining the extended image-text pair, in step S250, the target image generation model is trained using the first generated image and the first image description in the extended image-text pair. Specifically, the target program can use the first image description in any extended image-text pair obtained by the extension to obtain an image corresponding to the first image description through the target image generation model, which is called a predicted generated image, and then use a preset loss function (such as a cross entropy loss function, an MSE loss function, etc.) to use the difference between the predicted generated image and the first generated image in the extended image-text pair to determine the loss corresponding to the target image generation model, and then minimize the loss, that is, to reduce the difference between the predicted generated image and the first generated image in the extended image-text pair. The target image generation model is trained, that is, the parameters of the target image generation model are adjusted.

[0106] In one implementation, there are multiple first image descriptions obtained from the initial sample set, and step S250 may include: using the extended image-text pairs corresponding to each of the first image descriptions to cyclically train the target image generation model.

[0107] In one implementation, there are multiple first image descriptions, and the target program can use each first image description to execute steps S220-S240 through the aforementioned training system to obtain the expanded image-text pairs corresponding to each first image description. Thereafter, the target image generation model can be cyclically trained using the expanded image-text pairs corresponding to each first image description until the target image generation model reaches a preset convergence condition, thereby obtaining a trained target image generation model.

[0108] Among them, the preset convergence condition may include but is not limited to: the number of parameter adjustments is not less than a third adjustment threshold, or the calculated loss is lower than a third loss threshold, or the currently existing extended image-text pairs have been used to train the target image generation model, etc.

[0109] In another implementation, there are multiple first image descriptions, and the target program can first execute steps S220-S240 for one of the first image descriptions i through the aforementioned training system (including the target text generation model and the target image generation model) to obtain the extended image-text pair corresponding to the first image description i, and then use the extended image-text pair corresponding to the first image description i (that is, the first image description i and its first generated image) to train the target image generation model to obtain a new target image generation model; then use the new target image generation model to update the target image generation model in the aforementioned training system, that is, use the new target image generation model as the new target image generation model in the aforementioned training system.

[0110] Afterwards, the target program executes steps S220-S240 for another first image description j among the multiple first image descriptions through the updated training system (including the target text generation model and the new target image generation model) to obtain the extended text-image pair corresponding to the first image description j, and then uses the extended text-image pair corresponding to the first image description j (i.e., the first image description j and its first generated image) to continue training the new target image generation model to obtain a new target image generation model, and then uses the new target image generation model to update the new target image generation model in the training system again. By analogy, the target image generation model is cyclically trained using the extended text-image pairs corresponding to each first image description. This is to obtain more extended text-image pairs with better quality and an image generation model with better image generation capability, until the updated target image generation model reaches the aforementioned preset convergence condition, and a trained target image generation model is obtained.

[0111] In this embodiment, after obtaining the first image description in the initial sample set, multiple prediction descriptions are obtained according to the first image description through the target text generation model, and then at least two first generated images corresponding to each prediction description are obtained through the target image generation model based on each prediction description. The image quality of the first generated image generated in this way is better than the image quality of the image generated directly based on the first image description. Then, based on at least the first image description and at least two generated images corresponding to each prediction description, an expanded image-text pair for training the target image generation model is determined. Through the above process, the image-text pair can be expanded to obtain training data with better quality and more quantity for training the target image generation model, that is, expanded image-text pairs. Then, the expanded image-text pairs can be used to continue training and updating the target image generation model, so that the image generation capability of the target image generation model is better.

[0112] Furthermore, by calculating the quality scores corresponding to each first description image pair, and then based on the quality scores corresponding to each first description image pair, a target description image pair with better quality is screened out from all the first description image pairs, and then, the better quality image and the first image description in the target description image pair are used to obtain an extended image-text pair with better quality, so that the image generation capability of the target image generation model can be better optimized by using such an extended image-text pair. Such a target image generation model can obtain images with better quality for image descriptions that have not been expanded by the target text generation model, and can better provide users with high-quality image generation services, that is, generate images with better quality based on the image description input by the user.

[0113] Corresponding to the above method embodiment, the present disclosure embodiment provides an image generation method, which can be implemented by a first program. The first program can be the same physical program as the aforementioned target program, or can be a different physical program from the aforementioned target program, such as Figure 3 As shown, the method may include the following steps S310-S320:

[0114] In step S310, the image description to be processed is obtained. In one implementation, the image description to be processed may be any image description input by the user when there is a need to generate an image using the image description (for example, when generating corresponding illustrations for novels, news, etc.), and accordingly, the first program may obtain the image description input by the user as the image description to be processed.

[0115] In step S320, based on the description of the image to be processed, the target image is obtained by using the image generation model trained by the image generation model training method described in the above embodiment. In this step, the first program can input the description of the image to be processed into the image generation model trained by the image generation model training method described in the above embodiment, that is, the target image generation model that has completed the training (reached the preset convergence condition), to obtain the target image.

[0116] In the above process, by using the image generation model trained by the image generation model training method described in the above embodiment, a target image with better quality can be obtained to better improve the user experience.

[0117] The foregoing describes specific embodiments of the present disclosure, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims may be performed in an order different from that in the embodiments, and the desired results may still be achieved. In addition, the processes depicted in the accompanying drawings do not necessarily have to be performed in the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible, or may be advantageous.

[0118] Corresponding to the above method embodiment, the present disclosure embodiment provides a training device 400 for an image generation model, and its schematic block diagram is as follows: Figure 4 As shown, it is used to execute Figure 2 The method shown, the device 400 includes:

[0119] A first acquisition module 410 is configured to acquire a first image description in an initial sample set;

[0120] A first obtaining module 420 is configured to obtain a plurality of predicted descriptions based on the first image description by using a target text generation model;

[0121] A second obtaining module 430 is configured to obtain at least two first generated images corresponding to each prediction description through the target image generation model based on each prediction description;

[0122] A first determination module 440 is configured to determine an extended image-text pair for training the target image generation model based on the first image description and at least two first generated images corresponding to each predicted description;

[0123] The training module 450 is configured to train the target image generation model using the first generated image and the first image description in the extended image-text pair.

[0124] In an optional implementation, the first determining module 440 includes:

[0125] A first determining unit (not shown in the figure) is configured to determine a quality score corresponding to each first description image pair based on a preset quality assessment algorithm, wherein each first description image pair includes the first image description and one first generated image;

[0126] A second determining unit (not shown in the figure) is configured to determine, from all the first description image pairs, a number of target description image pairs whose corresponding quality scores meet a preset quality condition based on the quality scores corresponding to the first description image pairs;

[0127] A combining unit (not shown in the figure) is configured to combine the first generated image in each target description image pair with the first image description to obtain an expanded image-text pair for training the target image generation model.

[0128] In an optional implementation, the first determining unit is specifically configured to determine the matching value corresponding to each first description image pair through an image-text association model based on the first image description and the first generated image in each first description image pair;

[0129] Based on the first generated image in each first description image pair, determining the aesthetic score corresponding to each first description image pair through the aesthetic prediction model;

[0130] Based on the matching values ​​and aesthetic scores corresponding to the first description image pairs, the quality scores corresponding to the first description image pairs are determined.

[0131] In an optional implementation, the plurality of target description image pairs whose corresponding quality scores satisfy the preset quality condition include: N first description image pairs whose corresponding quality scores are the largest, where N is a positive integer;

[0132] Alternatively, the plurality of target description image pairs whose corresponding quality scores satisfy a preset quality condition include: a plurality of first description image pairs whose corresponding quality scores exceed a preset quality score threshold.

[0133] In an optional implementation, it also includes:

[0134] A third acquisition module (not shown in the figure) is configured to acquire a second image description and a corresponding sample image thereof before obtaining at least two first generated images corresponding to each prediction description respectively through the target image generation model based on each prediction description;

[0135] a fourth obtaining module (not shown in the figure), configured to obtain a second generated image based on the second image description by using an image generation model to be trained;

[0136] A second determination module (not shown in the figure) is configured to determine a first prediction loss corresponding to the image generation model to be trained by using a difference between the sample image and the second generated image;

[0137] The first adjustment module (not shown in the figure) is configured to adjust the parameters of the image generation model to be trained with the goal of minimizing the first prediction loss to obtain an image generation model that meets the first convergence condition as the target image generation model.

[0138] In an optional implementation, it also includes:

[0139] a fourth acquisition module (not shown in the figure), configured to acquire a third image description before obtaining a plurality of predicted descriptions based on the first image description through a target text generation model;

[0140] an occlusion module (not shown in the figure), configured to occlude part of the description of the third image description by using a preset occlusion algorithm to obtain a fourth image description;

[0141] a fifth obtaining module (not shown in the figure), configured to obtain an output of the text generation model to be trained based on the fourth image description through the text generation model to be trained;

[0142] A third determination module (not shown in the figure) is configured to determine a second prediction loss corresponding to the text generation model to be trained by using the output of the text generation model to be trained and the third image description;

[0143] The second adjustment module (not shown in the figure) is configured to adjust the parameters of the text generation model to be trained with the goal of minimizing the second prediction loss to obtain the target text generation model that meets the second convergence condition.

[0144] In an optional implementation, there are multiple first image descriptions; the training module 450 is specifically configured to use the extended image-text pairs corresponding to each first image description to cyclically train the target image generation model.

[0145] Corresponding to the above method embodiment, the present disclosure provides an image generating device 500, whose schematic block diagram is as follows: Figure 5 As shown, it is used to execute Figure 3 The method shown, the device 500 includes:

[0146] A second acquisition module 510 is configured to acquire a description of an image to be processed;

[0147] The third obtaining module 510 is configured to obtain a target image based on the description of the image to be processed by using the image generation model trained by the image generation model training method described in the above embodiment.

[0148] The above device embodiments correspond to the method embodiments. For specific descriptions, please refer to the description of the method embodiments, which will not be repeated here. The device embodiments are obtained based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. For specific descriptions, please refer to the corresponding method embodiments.

[0149] The present disclosure provides an electronic device, including a memory and a processor, wherein the memory stores an executable code, and when the processor executes the executable code, the following is implemented: Figure 2 or Figure 3 The method shown below. Figure 6 , which shows a schematic structural diagram of an electronic device suitable for implementing an embodiment of the present disclosure. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0150] like Figure 6As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In RAM 603, various programs and data required for the operation of the electronic device are also stored. The processing device 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0151] Typically, the following devices may be connected to the I / O interface 605: input devices 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, etc.; output devices 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 608 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Figure 6 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 6 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0152] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0153] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed in a computer, the computer is caused to execute the following steps provided in the present disclosure: Figure 2 The training method of the image generation model shown or Figure 3 The image generation method shown.

[0154] It should be noted that the computer-readable medium described in the embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0155] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains the first image description in the initial sample set; based on the first image description, obtains multiple prediction descriptions through the target text generation model; based on each prediction description, obtains at least two first generated images corresponding to each prediction description through the target image generation model; based on the first image description and the at least two first generated images corresponding to each prediction description, determines the extended image-text pair for training the target image generation model; trains the target image generation model using the first generated image and the first image description in the extended image-text pair. Alternatively, the electronic device: obtains the image description to be processed; based on the image description to be processed, obtains the target image using the image generation model trained by the image generation model training method described in the aforementioned embodiment.

[0156] Computer program code for performing the operations of the disclosed embodiments may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0157] Each embodiment in the present disclosure is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the storage medium and computing device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0158] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present invention may be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0159] The specific implementation methods described above further describe the purpose, technical solutions and beneficial effects of the embodiments of the present invention in detail. It should be understood that the above description is only a specific implementation method of the embodiments of the present invention and is not intended to limit the scope of protection of the present invention. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solution of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A training method for an image generation model, comprising: Obtain a first image description in an initial sample set; Based on the first image description, a plurality of predicted descriptions are obtained by using a target text generation model; Based on each prediction description, at least two first generated images corresponding to each prediction description are obtained through a target image generation model; Determining an extended image-text pair for training the target image generation model based on the first image description and at least two first generated images corresponding to each predicted description; The target image generation model is trained using the first generated image and the first image description in the extended image-text pair.

2. The method of claim 1, wherein: The step of determining the expanded image-text pairs for training the target image generation model comprises: Determine, based on a preset quality assessment algorithm, a quality score corresponding to each first description image pair, wherein each first description image pair includes the first image description and one of the first generated images; Based on the quality scores corresponding to the first description image pairs, determining a number of target description image pairs whose corresponding quality scores meet a preset quality condition from all the first description image pairs; The first generated image in each target description image pair is respectively combined with the first image description to obtain an expanded image-text pair for training the target image generation model.

3. The method of claim 2, wherein: The determining, based on a preset quality assessment algorithm, a quality score corresponding to each first description image pair comprises: Based on the first image description and the first generated image in each first description image pair, determining the matching value corresponding to each first description image pair through the image-text association model; Based on the first generated image in each first description image pair, determining the aesthetic score corresponding to each first description image pair through the aesthetic prediction model; Based on the matching values ​​and aesthetic scores corresponding to the first description image pairs, the quality scores corresponding to the first description image pairs are determined.

4. The method of claim 2, wherein: The target description image pairs whose corresponding quality scores meet the preset quality condition include: N first description image pairs whose corresponding quality scores are the largest, where N is a positive integer; Alternatively, the plurality of target description image pairs whose corresponding quality scores satisfy a preset quality condition include: a plurality of first description image pairs whose corresponding quality scores exceed a preset quality score threshold.

5. The method according to claim 1, before obtaining at least two first generated images corresponding to each prediction description respectively through the target image generation model based on each prediction description, further comprising: Obtaining a second image description and a corresponding sample image; Based on the second image description, obtaining a second generated image by using the image generation model to be trained; Determine a first prediction loss corresponding to the image generation model to be trained by using a difference between the sample image and the second generated image; With the goal of minimizing the first prediction loss, the parameters of the image generation model to be trained are adjusted to obtain an image generation model that meets the first convergence condition as the target image generation model.

6. The method according to claim 1, before obtaining a plurality of predicted descriptions through a target text generation model based on the first image description, further comprising: Obtaining a third image description; Using a preset occlusion algorithm, occluding part of the description of the third image description to obtain a fourth image description; Based on the fourth image description, obtaining an output of the text generation model to be trained through the text generation model to be trained; Determine a second prediction loss corresponding to the text generation model to be trained by using the output of the text generation model to be trained and the third image description; With the goal of minimizing the second prediction loss, the parameters of the text generation model to be trained are adjusted to obtain the target text generation model that meets the second convergence condition.

7. The method of claim 1, wherein: The first image is described as a plurality; The step of training the target image generation model by using the first generated image and the first image description in the extended image-text pair comprises: The target image generation model is cyclically trained using the extended image-text pairs corresponding to the first image descriptions.

8. A method for generating an image, comprising: Get the description of the image to be processed; Based on the description of the image to be processed, a target image is obtained by using an image generation model trained by the image generation model training method described in any one of claims 1 to 7.

9. A training device for an image generation model, comprising: A first acquisition module, configured to acquire a first image description in an initial sample set; A first obtaining module is configured to obtain a plurality of predicted descriptions based on the first image description by using a target text generation model; A second obtaining module is configured to obtain at least two first generated images corresponding to each prediction description respectively through the target image generation model based on each prediction description; A first determination module is configured to determine an extended image-text pair for training the target image generation model based on the first image description and at least two first generated images corresponding to each predicted description; The training module is configured to train the target image generation model using the first generated image and the first image description in the extended image-text pair.

10. An image generating device, comprising: A second acquisition module is configured to acquire a description of the image to be processed; The third obtaining module is configured to obtain a target image based on the description of the image to be processed by using the image generation model trained by the image generation model training device according to claim 9.

11. An electronic device comprising a memory and a processor, wherein: The memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 8 is implemented.

12. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 8.

Images