Text generation model training method and device, image generation method and device, equipment and medium

Abstract

The embodiment of the invention provides a text generation model training method and device, an image generation method and device, equipment and a medium. The method comprises the steps that any first image description in a training sample set is acquired; based on the first image description, a first generated image is obtained through a target image generation model; based on the first image description, through a text generation model, obtaining a prediction description; based on the prediction description, a second generated image is obtained through a target image generation model; determining a loss of the text generation model based on an image quality difference between the second generated image and the first generated image, and / or determining a loss of the text generation model based on a correlation between the second generated image and the prediction description; according to the method, the parameters of the text generation model are adjusted by taking the loss minimization as the target, so that the model capable of generating better image description is obtained through training, the image description with better quality is obtained, and the quality of the image generated based on the image description is improved.

Description

Technical Field

[0001] The present specification relates to the field of computer technology, and in particular to a text generation model training, method, device, equipment and medium, and an image generation method, device, equipment and medium. Background Art

[0002] In the field of image generation, image description (also called text prompt) can be used to guide image generation. Specifically, based on the image description, the image corresponding to the image description can be obtained through an image generation model based on content generation technology. From the above process, it can be seen that the quality of the image description is crucial to the quality of the generated image.

[0003] In order to ensure that the quality of the generated images is better, it is crucial to obtain better quality image descriptions. Summary of the invention

[0004] One or more embodiments of the present specification provide a text generation model training, method, apparatus, device and medium, as well as an image generation method, apparatus, device and medium, so as to train a model that can generate better quality image descriptions, so as to obtain better quality image descriptions, and thereby improve the quality of images generated based on the image descriptions.

[0005] According to a first aspect, a method for training a text generation model is provided, comprising:

[0006] Get any first image description in the training sample set;

[0007] Based on the first image description, a first generated image is obtained by using a target image generation model;

[0008] Based on the first image description, obtaining a predicted description through a text generation model;

[0009] Based on the prediction description, a second generated image is obtained by generating a model of the target image;

[0010] determining a loss of the text generation model based on a difference in image quality between the second generated image and the first generated image, and / or determining a loss of the text generation model based on a correlation between the second generated image and the predicted description;

[0011] With the goal of minimizing the loss, the parameters of the text generation model are adjusted.

[0012] According to a second aspect, there is provided an image generation method, comprising:

[0013] Get the description of the image to be processed;

[0014] Based on the image description to be processed, obtaining a target image description by using a text generation model trained by the text generation model training method described in the first aspect;

[0015] Based on the target image description, a target generated image is obtained through a target image generation model.

[0016] According to a third aspect, a training device for a text generation model is provided, comprising:

[0017] A first acquisition module, configured to acquire any first image description in a training sample set;

[0018] A first obtaining module is configured to obtain a first generated image through a target image generation model based on the first image description;

[0019] A second obtaining module is configured to obtain a predicted description based on the first image description through a text generation model;

[0020] A third obtaining module is configured to obtain a second generated image based on the prediction description through the target image generation model;

[0021] a first determination module configured to determine a loss of the text generation model based on an image quality difference between the second generated image and the first generated image, and / or to determine a loss of the text generation model based on a correlation between the second generated image and the predicted description;

[0022] The first adjustment module is configured to adjust the parameters of the text generation model with the goal of minimizing the loss.

[0023] According to a fourth aspect, there is provided an image generating device, comprising:

[0024] A second acquisition module is configured to acquire a description of the image to be processed;

[0025] A fourth obtaining module is configured to obtain a target image description based on the image description to be processed by using a text generation model trained by the training device for the text generation model described in the third aspect;

[0026] The fifth obtaining module is configured to obtain a target generated image based on the target image description through a target image generation model.

[0027] According to a fifth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method described in the first aspect or the second aspect.

[0028] According to a sixth aspect, an electronic device is provided, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in the first aspect or the second aspect is implemented.

[0029] According to the training of the text generation model, the image generation method, apparatus, device and medium provided in the embodiments of this specification, obtain an arbitrary first image description in the training sample set; based on the first image description, obtain a first generated image through a target image generation model; based on the first image description, obtain a predicted description through a text generation model; based on the predicted description, obtain a second generated image through a target image generation model; then determine the loss of the text generation model based on the image quality difference between the second generated image and the first generated image, and / or determine the loss of the text generation model based on the correlation between the second generated image and the predicted description; adjust the parameters of the text generation model with the goal of minimizing the loss of the text generation model.

[0030] In the above process, the idea of ​​reinforcement learning is adopted, and the first image description as the current environmental state is input into the text generation model to obtain a predicted description as an execution action, and the first image description is processed by the target image generation model to obtain a first generated image, and the predicted description is processed by the target image generation model to obtain a second generated image, and then based on the image quality difference between the second generated image and the first generated image, the current environmental feedback, i.e., the reward score, is determined, and / or the current environmental feedback is determined based on the correlation between the second generated image and the predicted description, and then the loss of the text generation model is determined, and the loss is fed back to the text generation model, that is, with the goal of minimizing the loss of the text generation model, the parameters of the text generation model are adjusted to realize the training of the text generation model in combination with the environmental feedback, with the goal of improving the quality of the image generated by the predicted description obtained by the text generation model, the parameters of the text generation model are adjusted to improve the generation ability of the image description of the text generation model, and obtain a better quality image description, thereby improving the quality of the image generated based on the image description. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0032] Figure 1 A schematic diagram of an implementation framework of an embodiment disclosed in this specification;

[0033] Figure 2A flowchart of a method for training a text generation model provided in an embodiment;

[0034] Figure 3 A schematic diagram of a flow chart of an image generation method provided in an embodiment;

[0035] Figure 4 A schematic block diagram of a training device for a text generation model provided in an embodiment;

[0036] Figure 5 A schematic block diagram of an image generating device provided in an embodiment;

[0037] Figure 6 A schematic block diagram of an electronic device provided in an embodiment. DETAILED DESCRIPTION

[0038] The technical solutions of the embodiments of this specification will be described in detail below with reference to the accompanying drawings.

[0039] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0040] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0041] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0042] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0043] The embodiments of this specification disclose a training method, device, equipment and medium for a text generation model, and an image generation method, device, equipment and medium. The application scenarios and technical concepts of the methods are first introduced as follows:

[0044] As mentioned above, in the field of image generation, image description (also called text prompt) can be used to guide image generation. Specifically, based on the image description, the image corresponding to the image description can be obtained through the image generation model based on content generation technology. From the above process, it can be seen that the quality of the image description is crucial to the quality of the generated image. In order to ensure the quality of the generated image is better, how to obtain a better quality image description is crucial.

[0045] In order to obtain better quality image description, Figure 1 A schematic diagram of an implementation scenario according to an embodiment disclosed in this specification is shown. In this implementation scenario, a training system for training (i.e., fine-tuning) a text generation model using the training method disclosed in the embodiment of the present disclosure is exemplified, and the training system includes: a text generation model to be trained (i.e., fine-tuned), a target image generation model, and a reward model. Through the training system, a text generation model that can generate better quality image descriptions is obtained through training.

[0046] Wherein, the text generation model may be a text generation model pre-trained using a plurality of text groups, each text group may include text and occluded text generated by the occluded portion of the text. The target image generation model may be an image generation model pre-trained based on a plurality of image-text pairs. The reward model is a model for determining feedback data (i.e., current environmental feedback, reward score) for fine-tuning the text generation model, and the feedback data is used to determine the loss for fine-tuning the text generation model. Wherein, the feedback data may include the image quality difference E1 between the second generated image and the first generated image mentioned later, and / or the difference E2 between the first correlation value determined based on the second generated image and the predicted description and the second correlation value determined based on the first generated image and the first image description mentioned later.

[0047] It should be noted that each of the aforementioned image-text pairs may include an image and its corresponding image description. For the sake of clarity, the image in the image-text pair may be referred to as a sample image, and the image description in the image-text pair may be referred to as a second image description. The aforementioned text may refer to an image description used for pre-training a text generation model, and for the sake of clarity, it may be referred to as a third image description.

[0048] The image generation model is trained using the image-text pairs, so that the image generation model has the ability to generate corresponding images according to the image description. The text generation model is pre-trained using each third image description and the occluded text generated by the occluded part of each third image description, so that the text generation model has the ability to predict and complete (expand) text.

[0049] The following is an introduction to the process of fine-tuning the text generation model (i.e., the training process of the text generation model). Specifically, a training sample set for fine-tuning the text generation model is obtained, wherein the training sample set includes multiple image descriptions. An arbitrary image description is obtained from the training sample set as a first image description; based on the first image description, a first generated image corresponding to the first image description is obtained through the target image generation model; based on the first image description, a predicted description is obtained through the text generation model (i.e., a pre-trained text generation model); based on the predicted description, a second generated image corresponding to the predicted description is obtained through the target image generation model.

[0050] Afterwards, the reward model is used to determine the image quality difference between the first generated image and the second generated image based on the first generated image and the second generated image; and / or, based on the second generated image and the predicted description, the correlation between the second generated image and the predicted description is determined, wherein the image quality difference between the second generated image and the first generated image can characterize the quality improvement of the predicted description obtained by the text generation model compared to the first image description; the correlation between the second generated image and the predicted description can characterize the relevance and quality of the second generated image and the predicted description.

[0051] Next, based on the image quality difference, the loss of the text generation model is determined, and / or based on the correlation between the second generated image and the predicted description, the loss of the text generation model is determined, that is, the correlation between the second generated image and the predicted description and / or the image quality difference is used to determine the loss of the text generation model. Accordingly, the loss can characterize the quality improvement of the predicted description obtained by the text generation model compared to the first image description, and / or the relevance and quality of the second generated image and the predicted description; then, with the goal of minimizing the loss, the parameters of the text generation model are adjusted, that is, the parameters of the text generation model are adjusted in the direction of increasing the quality improvement of the predicted description obtained by the text generation model compared to the first image description, and / or the direction of improving the relevance and quality of the second generated image and the predicted description, so as to improve the image description generation capability of the text generation model, improve the quality of the image description generated by it, and thus improve the quality of the image generated based on the image description.

[0052] In the above process, the idea of ​​reinforcement learning is adopted, and the first image description as the current environmental state is input into the text generation model to obtain a predicted description as an execution action, and the first image description is processed by the target image generation model to obtain a first generated image, and the predicted description is processed by the target image generation model to obtain a second generated image, and then based on the image quality difference between the second generated image and the first generated image, the current environmental feedback, i.e., the reward score, is determined, and / or the current environmental feedback is determined based on the correlation between the second generated image and the predicted description, and then the loss of the text generation model is determined, and the loss is fed back to the text generation model, that is, with the goal of minimizing the loss, the parameters of the text generation model are adjusted to realize training of the text generation model in combination with environmental feedback, with the goal of improving the quality of the image generated by using the predicted loss obtained by processing the text generation model, and the parameters of the text generation model are adjusted to improve the image description generation ability of the text generation model, so as to obtain a better quality image description, and then improve the quality of the image generated based on the image description.

[0053] Moreover, in the above process, there is no need to mark label data for the data in the training sample set, which can save labor costs while improving the ability of the text generation model to generate better quality image descriptions.

[0054] In combination with specific embodiments, the training method, device, equipment and medium of the text generation model and the image generation method, device, equipment and medium provided in this specification are described in detail below. For a clear layout, the pre-training process of the target image generation model and the text generation model are first introduced respectively.

[0055] The pre-training process of the target image generation model may include: obtaining the image generation model to be trained, obtaining a training data set S for training the image generation model, and the training data set S may include multiple pairs of image-text pairs, wherein any image-text pair M includes a sample image T and its corresponding image description (referred to as the second image description) X. In one case, the training data set S may be obtained from a public data set that can provide a large number of image-text pairs, or an artificially pre-manufactured data set. The process of artificially pre-manufactured data sets may include: collecting images, and annotating image descriptions on the images to obtain image-text pairs, thereby obtaining artificially pre-manufactured data sets.

[0056] In one case, the image description may also be referred to as a text prompt, that is, a text prompt used to generate an image.

[0057] Obtain any image-text pair M from the training data set S, that is, obtain the second image description X and the corresponding sample image T in the image-text pair M; based on the second image description X, obtain the generated image T' corresponding to the second image description X through the image generation model to be trained; then use the sample image T and the generated image T' to determine the loss corresponding to the image generation model to be trained as the first prediction loss. Specifically, in one case, the first prediction loss can be determined using the image difference between the sample image T and the generated image T'.

[0058] Afterwards, the parameters of the image generation model to be trained are adjusted with the goal of minimizing the first prediction loss, that is, reducing the image difference between the sample image T and the second generated image T', until the image generation model reaches the first convergence condition, thereby obtaining the target image generation model. The first convergence condition may include, but is not limited to, the number of parameter adjustments is not less than the first adjustment threshold, or the calculated first prediction loss is less than the first loss threshold, or the training time of the image generation model is not less than the preset training time.

[0059] Let's review the pre-training process of the aforementioned target image generation model. The above embodiment is explained by taking an image-text pair (i.e., a pair of second image descriptions and their corresponding sample images) as an example. In another embodiment, the above pre-training process can also be performed on a batch of samples, i.e., multiple image-text pairs, to obtain the generated images corresponding to the second image description in each image-text pair, and then based on the generated images corresponding to the second image description in each image-text pair and the sample images in each image-text pair, determine the first prediction loss, and adjust the parameters of the image generation model to be trained with the goal of minimizing the first prediction loss. In this embodiment, the first prediction loss is determined for a batch of samples, and then the parameters of the image generation model are adjusted. This can reduce the number of parameter adjustments to the image generation model and make it easier to implement the pre-training process.

[0060] In one implementation, the target image generation model may be an image generation model based on a Diffusion Model in the related art.

[0061] The pre-training process of the text generation model may include: obtaining a text generation model to be trained; constructing a sample description pair set D for pre-training the text generation model, wherein the sample description pair set D may include a plurality of sample description pairs, wherein the construction process of a sample description pair Di may be as follows: obtaining a third image description yi (for example, it may be expressed as {(a)(sunny)(windless)(of)(morning)}, or it may also be expressed as {(a)(a)(sunny)(bright)(without)(wind)(of)(morning)(up)}, wherein the content in each bracket represents the third image description y iA word in the expression (the first expression is taken as an example below), the third image description yi is included in the sample description pair Di.

[0062] Using a preset masking algorithm, mask part of the description (part of the content) of the third image description yi to obtain the corresponding masked third image description as the fourth image description xi (for example, it can be expressed as {(a)(?)(?)(of)(morning)}, where the question mark "?" is the masked word), and the fourth image description xi is included in the sample description pair Di, that is, a sample description pair Di includes a third image description yi and its corresponding fourth image description xi. The preset masking algorithm can be any algorithm in the related art that can implement text masking. In one implementation, using the preset masking algorithm, any content (or word) at any other position of the third image description yi except the first sentence (for example, the sentence before the first punctuation mark) can be randomly masked; words at any other position of the third image description yi except the first a words can also be randomly masked, and a can be a positive integer.

[0063] In one case, the third image description yi may be an image description that can generate an image that meets preset conditions, wherein the preset conditions may include but are not limited to: the aesthetic score of the image (which can represent the beauty of the image) is not less than a certain aesthetic score threshold, the clarity score of the image is not less than a certain clarity threshold, and / or the correlation value between the image and the corresponding image description is not less than a certain correlation threshold. In another case, the third image description yi may be an image description of any quality obtained from the aforementioned public data set that can provide a large number of image-text pairs.

[0064] Next, based on the fourth image description xi in any sample description pair Di, the output of the text generation model to be trained is obtained through the text generation model to be trained; using the output of the text generation model to be trained and the third image description yi in the sample description pair Di, the loss corresponding to the text generation model to be trained is determined, which is called the second prediction loss.

[0065] In one implementation, after obtaining the output of the text generation model to be trained, the probability of the words at the corresponding positions of the third image description yi appearing at each blocked position (or each position of the fourth image description xi) in the fourth image description xi can be determined from the output of the text generation model to be trained according to the third image description yi (for example, in the above example, the fourth image description x i The first occluded position in the fourth image description x i The probability of the word "sunny" appearing at the position of the second word, the fourth image description x iThe probability of the word “no wind” appearing at the second blocked position, i.e. the third word, in the sentence .

[0066] Then, the cross entropy loss function is used to determine the aforementioned second prediction loss based on the probability of the words at the corresponding positions of the third image description yi appearing at each occluded position in the fourth image description xi (or each position in the fourth image description xi). The greater the probability of the words at the corresponding positions of the third image description yi appearing at each occluded position in the fourth image description xi (or each position in the fourth image description xi), the smaller the second prediction loss, indicating that the image description generation ability of the text generation model to be trained is better. Correspondingly, in one case, the second prediction loss l can be expressed by the following formula:

[0067]

[0068] Wherein, p(yi|xi) represents the probability that the text generation model to be trained outputs the third image description yi for the fourth image description xi, and the p(yi|xi) can be determined based on the probability that the words at the corresponding positions of the third image description yi appear at each blocked position in the fourth image description xi (or each position in the fourth image description xi) (for example: based on the average value of the sum of the probabilities of the words at the corresponding positions of the third image description yi appearing at each blocked position in the fourth image description xi, or the result of the multiplication); D represents the aforementioned sample description pair set D to which the third image description yi and the fourth image description xi belong; It means calculating the expected value of logp(yi|xi).

[0069] In another implementation, the target program can determine the generated description corresponding to the fourth image description based on the output of the text generation model to be trained and the specified word determination rule, and then determine the second prediction loss based on the difference between the generated description corresponding to the fourth image description and the third image description (for example, the Euclidean distance), wherein the smaller the distance between the generated description and the third image description, the smaller the second prediction loss.

[0070] Afterwards, in order to minimize the second prediction loss, that is, to maximize the probability that the text generation model to be trained outputs the third image description yi for the fourth image description xi, that is, to reduce the difference between the generated description determined based on the output of the text generation model to be trained and the third image description, the parameters of the text generation model to be trained are adjusted until the text generation model reaches the second convergence condition, and a pre-trained text generation model is obtained. Among them, the second convergence condition may include, but is not limited to, the number of parameter adjustments is not less than the second adjustment threshold, or the calculated second prediction loss is lower than the second loss threshold. The second adjustment threshold may be the same as or different from the aforementioned first adjustment threshold, and the second loss threshold may be the same as or different from the first loss threshold of the signature.

[0071] Let's review the pre-training process of the aforementioned text generation model. The above embodiment is explained by taking a sample description pair (i.e., a pair of third image descriptions and their corresponding fourth image descriptions) as an example. In another embodiment, the above pre-training process can also be performed on a batch of samples, i.e., multiple sample description pairs, to obtain the output of the text generation model to be trained corresponding to the fourth image description in each sample description pair, and then based on the output of the text generation model to be trained corresponding to the fourth image description in each sample description pair and the third image description in each sample description pair, determine the second prediction loss, and adjust the parameters of the text generation model to be trained with the goal of minimizing the second prediction loss. In this embodiment, the second prediction loss is determined for a batch of samples, and then the parameters of the text generation model are adjusted. This can reduce the number of parameter adjustments to the text generation model and make it easier to implement the pre-training process.

[0072] In one implementation, the text generation model can be a text generation model based on transformer in the related art. The text generation model can expand and supplement the input image description to generate a richer image description.

[0073] After obtaining the pre-trained target image generation model and text generation model, the pre-trained text generation model is then fine-tuned (i.e., retrained) to improve its ability to generate high-quality image descriptions. Figure 2 The flowchart of the training method of the text generation model in one embodiment of the present specification is shown. The method can be implemented by a target program. The target program can be installed in an electronic device, and the electronic device can be implemented by any device, equipment, platform, device cluster, etc. with computing and processing capabilities. Figure 2 As shown, the method may include the following steps S210-S260:

[0074] In step S210, any first image description in the training sample set is obtained. The training sample set includes a plurality of first image descriptions. In one case, the first image description can be any type of image description used to generate an image.

[0075] Then, in step S220, based on the first image description, a first generated image is obtained through the target image generation model. In this step, the target program can input the first image description into the target image generation model, process the first image description through the target image generation model, and output an image corresponding to the first image description, i.e., the first generated image. Accordingly, the target program obtains the first generated image.

[0076] Then, in step S230, based on the first image description, a predicted description is obtained by using a text generation model. Then, in step S240, based on the predicted description, a second generated image is obtained by using a target image generation model.

[0077] In one implementation, the text generation model may include a text encoder and a text decoder. The target program inputs a first image description into the text generation model, encodes the first image description through the text encoder of the text generation model to obtain an encoding result corresponding to the first image description, and inputs the encoding result corresponding to the first image description into the text decoder of the text generation model to obtain a model output of the text generation model for the first image description.

[0078] The model output may include a word probability set corresponding to each of the several positions determined based on the first image description, the word probability set for each position includes the probability of each word in a preset word (token) set appearing at the position, and the preset word set is a set of words that can be predicted by the text generation model. Based on the word probability set corresponding to each of the several positions included in the model output and the specified word determination rule, the word corresponding to each position can be determined (the word corresponding to the position may be empty), that is, the description after the text generation model is expanded, that is, the predicted description, and accordingly, the target program obtains the predicted description.

[0079] The specified word determination rule may be any rule in the related art that can determine the corresponding word based on the word probability set corresponding to each of the aforementioned plurality of positions. The specific number of the plurality of positions may be determined based on the specified parameters set for the text generation model, and may refer to the longest length of the model output.

[0080] Next, the target program inputs the predicted description into the target image generation model, processes the predicted description through the target image generation model, and obtains a second generated image corresponding to the predicted description.

[0081] After the target program obtains the first generated image corresponding to the first image description, the predicted description and its corresponding second generated image, in step S250, the loss of the text generation model is determined based on the image quality difference between the second generated image and the first generated image, and / or the loss of the text generation model is determined based on the correlation between the second generated image and the predicted description.

[0082] In one implementation, the target program may determine the loss of the text generation model based on the image quality difference between the second generated image and the first generated image; or, determine the loss of the text generation model based on the correlation between the second generated image and the predicted description; or, jointly determine the loss of the text generation model based on the image quality difference between the second generated image and the first generated image and the correlation between the second generated image and the predicted description.

[0083] It is understandable that the purpose of fine-tuning (training) the text generation model is to improve the ability of the text generation model to generate image descriptions with better quality. In theory, the better the image quality of the second generated image obtained based on the predicted description generated by the text generation model is than the image quality of the first generated image generated based on the first image description, that is, the greater the difference in image quality between the second generated image and the first generated image, the better the quality of the predicted description is than the quality of the first image description, and accordingly, the better the ability of the text generation model to generate high-quality image descriptions. The higher the correlation between the second generated image and the predicted description, the better the correlation and quality of the predicted description and its second generated image, and accordingly, the better the ability of the text generation model to generate high-quality image descriptions.

[0084] In view of this, combined with the learning mechanism taking into account reinforcement learning, the current environmental feedback, i.e., the reward score, can be determined based on the image quality difference between the second generated image and the first generated image, and / or the correlation between the second generated image and the predicted description, and the loss of the text generation model can be determined based on the image quality difference between the second generated image and the first generated image, and / or, based on the correlation between the second generated image and the predicted description, the loss of the text generation model is determined, so that the loss of the text generation model is negatively correlated with the image quality difference between the second generated image and the first generated image, and / or, the loss of the text generation model is negatively correlated with the correlation between the second generated image and the predicted description.

[0085] In one embodiment, the loss of the text generation model may be determined by combining the correlation between the second generated image and the predicted description and the difference in correlation between the first generated image and the first image description. Specifically, in step S250, the following steps 11-13 may be specifically included:

[0086] In step 11, a first sub-loss is determined based on an image quality difference between the second generated image and the first generated image.

[0087] Before determining the first sub-loss in step 11, the target program may also include a process of determining image quality differences. The process of determining image quality differences may include the following steps 21-22: In step 21, using a preset quality assessment model, determine a first quality assessment value corresponding to the first generated image and a second quality assessment value corresponding to the second generated image.

[0088] In one implementation, the quality (i.e., quality evaluation value) of the second generated image and the first generated image may be determined from a variety of perspectives. For example, it may include but is not limited to: determining the second quality evaluation value (first quality evaluation value) based on the clarity of the second generated image (first generated image); determining the second quality evaluation value (first quality evaluation value) based on the aesthetics of the second generated image (first generated image); determining the second quality evaluation value (first quality evaluation value) by combining the clarity of the second generated image (first generated image) and the aesthetics of the second generated image (first generated image), etc.

[0089] Accordingly, in one embodiment, the aforementioned preset quality assessment model may include an aesthetic prediction model F for determining the aesthetics of an image. B , and / or an image gradient algorithm F for determining image sharpness T .

[0090] That is, the first quality assessment value may include: a first aesthetic score determined by the aesthetic prediction model based on the first generated image, and / or a first clarity score determined by an image gradient algorithm based on the first generated image; the second quality assessment value may include: a second aesthetic score determined by the aesthetic prediction model based on the second generated image, and / or a second clarity score determined by an image gradient algorithm based on the second generated image.

[0091] The aesthetic score may represent the aesthetics of the corresponding image, and a larger aesthetic score may represent a more aesthetically pleasing corresponding image and a better image quality. In one case, the aesthetic score may take a value between [0, 1].

[0092] The clarity score may represent the clarity of the corresponding image. A larger clarity score may represent that the corresponding image is clearer and the image quality is better. In one case, the clarity score may take a value between [0, 1].

[0093] It can be understood that the aforementioned aesthetic prediction model F BIt can be any model in the related art that can evaluate the aesthetics of an image, i.e., the aesthetic score. In one implementation, the aesthetic prediction model can be a model pre-trained based on an image and its corresponding aesthetic label value, and the aesthetic prediction model can output the aesthetic score of the image for an input image. The images and their corresponding aesthetic label values ​​used to train the aesthetic prediction model can be selected from any data set including a large number of images and their corresponding aesthetic score labels that characterize the aesthetics of the images.

[0094] The aforementioned image gradient algorithm F T It can be a Tenengrad gradient algorithm. The following takes the process of determining the first clarity score by the image gradient algorithm according to the first generated image as an example to illustrate the process of determining the clarity score of the image by the image gradient algorithm: Specifically, the target program can use the Sobel operator to calculate the horizontal gradient (which can be called the horizontal gradient) and the vertical gradient (which can be called the vertical gradient) of each pixel point (referred to as the first pixel point) of the first generated image, and then use the horizontal gradient and vertical gradient of each first pixel point to determine the total gradient of each first pixel point; then calculate the average value of the sum of the total gradients of each first pixel point as the first gradient of the first generated image. Then, the first gradient of the first generated image is used to determine the first clarity score of the first generated image. For example, the first gradient of the first generated image can be directly determined as the first clarity score of the first generated image; or the first gradient of the first generated image and the corresponding relationship between the preset gradient and the clarity score can be used to determine the first clarity score of the first generated image.

[0095] The larger the gradient of the first generated image (or the second generated image), the higher its clarity, and the larger its clarity score, that is, the better the image quality.

[0096] In one case, the total gradient of each first pixel point (for example, the jth first pixel point) can be determined by the following formula (1):

[0097]

[0098] Among them, T j represents the total gradient of the jth first pixel, t j1 represents the horizontal gradient of the jth first pixel, t j2 Represents the vertical gradient of the j-th first pixel.

[0099] A first quality assessment value corresponding to the first generated image and a second quality assessment value corresponding to the second generated image are determined using a preset quality assessment model. In step 22, an image quality difference is determined based on a difference between the second quality assessment value and the first quality assessment value.

[0100] Specifically, in one embodiment, the process of determining the image quality difference may include: determining the image quality difference based on the difference between the second aesthetic score and the first aesthetic score, and / or determining the image quality difference based on the difference between the second clarity score and the first clarity score.

[0101] Specifically, when the first quality assessment value includes a first aesthetic score and the second quality assessment value includes a second aesthetic score, the image quality difference can be determined based on the difference between the aesthetics of the second generated image and the first generated image, that is, the difference (e.g., the difference) between the second aesthetic score and the first aesthetic score.

[0102] When the first quality assessment value includes a first clarity score and the second quality assessment value includes a second clarity score, the image quality difference can be determined based on the difference between the clarity of the second generated image and the first generated image, i.e., the difference (e.g., the difference) between the second clarity score and the first clarity score.

[0103] When the first quality evaluation value includes a first aesthetic score and a first clarity score, and the second quality evaluation value includes a second aesthetic score and a second clarity score, an image quality difference is determined based on a difference between the second aesthetic score and the first aesthetic score, and a difference between the second clarity score and the first clarity score.

[0104] The greater the difference between the second aesthetic score and the first aesthetic score, the greater the difference between the second clarity score and the first clarity score, and the greater the image quality difference between the second generated image and the first generated image.

[0105] In one case, when the image quality difference is determined based on the difference between the second aesthetic score and the first aesthetic score, and the difference between the second clarity score and the first clarity score, the image quality difference can be expressed by the following formula (2):

[0106] E1=(F B (I P )-F B (I o ))+(F T (I P )-F T (I o ); (2)

[0107] Among them, E1 represents the image quality difference, F B (I P ) represents the second aesthetic score of the second generated image, F B (I o ) represents the first aesthetic score of the first generated image, F B(I P )-F B (I o )) represents the difference between the second aesthetic score and the first aesthetic score, F T (I P ) represents the second clarity score of the second generated image, F T (I o ) represents the first clarity score of the first generated image, (F T (I P )-F T (I o ) represents the difference between the second clarity score and the first clarity score, I P represents the second generated image, I o represents the first generated image.

[0108] Afterwards, the target program determines a first sub-loss based on an image quality difference between the second generated image and the first generated image, wherein the greater the image quality difference between the second generated image and the first generated image, the better the image description generation expansion capability characterizing the text generated image, and accordingly, the image quality difference is negatively correlated with the first sub-loss.

[0109] Next, in step 12, a second sub-loss is determined based on the difference between a first correlation value determined based on the second generated image and the predicted description and a second correlation value determined based on the first generated image and the first image description. The first correlation value may characterize the correlation between the second generated image and the predicted description, and the greater the correlation between the second generated image and the predicted description, the greater the first correlation value. Similarly, the second correlation value may characterize the correlation between the first generated image and the first image description, and the greater the correlation between the first generated image and the first image description, the greater the second correlation value. In one case, the correlation value may take a value between [0, 1].

[0110] The greater the difference between the first correlation value and the second correlation value, the better the image description generation expansion capability representing the text-generated image. Correspondingly, the difference (e.g., differential) between the first correlation value and the second correlation value is negatively correlated with the second sub-loss.

[0111] In one implementation, the first correlation value is determined by using the second generated image and the predicted description through the target image-text association model; the second correlation value is determined by using the first generated image and the first image description through the target image-text association model. G. Among them, the target image-text association model can be a model pre-trained based on a number of sample image-text pairs (each sample image-text pair includes an image and an image description) and their corresponding labels (including sample correlation values ​​that characterize the correlation of the corresponding sample image-text pairs). For example, the target image-text association model can be a CLIP model (Contrastive Language-Image Pre-training). The target image-text association model can determine the correlation between the input image and image description.

[0112] The difference E2 between the first correlation value determined based on the second generated image and the predicted description and the second correlation value determined based on the first generated image and the first image description can be expressed by the following formula (3):

[0113] E2=F G (I P , x2)-F G (I O , x1); (3)

[0114] Among them, F G (I P , x2) represents the image I generated according to the second P and the first correlation value determined by the prediction description x2, F G (I O , x1) represents the first generated image I O and a second correlation value determined by the first image description x1.

[0115] After obtaining the first sub-loss and / or the second sub-loss, in step 13, the loss of the text generation model is determined based on the first sub-loss and / or the second sub-loss. The loss of the text generation model may be positively correlated with the first sub-loss and the second sub-loss.

[0116] In one implementation, the aforementioned reward model includes an aesthetic prediction model F B , image gradient algorithm F T And the target image-text association model F G In the case of , the loss of the text generation model can be determined based on the first sub-loss and the second sub-loss. The loss of the text generation model can be expressed by the following formula (4):

[0117]

[0118] in, Indicates the calculation of expected value.

[0119] In yet another implementation, the aforementioned reward model includes an aesthetic prediction model F B , image gradient algorithm F T In the case of , the first sub-loss can be determined as the loss of the text generation model. G In the case of , the second sub-loss can be determined as the loss of the text generation model.

[0120] It can be understood that the loss of the text generation model can characterize the image description (i.e., text) generation expansion effect of the text generation model. The smaller the loss of the text generation model, the better the image description generation expansion effect of the text generation model. Correspondingly, in step S260, the parameters of the text generation model are adjusted with the goal of minimizing the loss of the text generation model. In this step, based on the loss of the text generation model, the adjustment gradient of the parameters of the text generation model is determined using the back propagation algorithm; the updated values ​​of the parameters of the text generation model are determined using the determined adjustment gradient and the current values ​​of the parameters of the text generation model; and then based on the updated values ​​of the parameters, the parameters of the text generation model are adjusted.

[0121] The adjustment gradient of the parameters of the text generation model is determined with the goal of minimizing the loss of the text generation model. The loss of the text generation model is minimized to improve the ability of the text generation model to generate high-quality image descriptions, that is, to improve (increase) the aesthetic score, clarity score (that is, increase the aforementioned image quality difference) and / or the correlation value between the second generated image and the predicted description (or increase the difference between the first correlation value determined based on the second generated image and the predicted description and the second correlation value determined based on the first generated image and the first image description).

[0122] Let's review the execution process of the aforementioned steps S210-S260. The above embodiment is explained by taking a first image description as a sample. In another embodiment, the above steps S210-S250 can also be performed for a batch of samples, i.e., multiple first image descriptions, to obtain the first generated image and predicted description corresponding to each first image description and the second generated image corresponding to the predicted description, respectively, and then in step S260, based on the image quality difference between the second generated image corresponding to each first image description and the first generated image, determine the loss of the text generation model, and / or, based on the correlation between the second generated image corresponding to each first image description and the predicted description, determine the loss of the text generation model, and adjust the parameters of the text generation model with the goal of minimizing the loss of the text generation model. In this embodiment, the loss of the text generation model is determined for a batch of samples, and then the parameters of the text generation model are adjusted, which can reduce the number of parameter adjustments to the text generation model and make it easier to implement the pre-training process.

[0123] Adjust the parameters of the text generation model until the text generation model reaches a preset convergence condition, thereby obtaining a trained text generation model. The preset convergence condition may include but is not limited to the number of parameter adjustments of the text generation model exceeding a preset number threshold, or the loss of the text generation model being lower than a preset generation loss threshold.

[0124] In this embodiment, the idea of ​​reinforcement learning is adopted, and the first image description as the current environmental state is input into the text generation model to obtain a predicted description as an execution action, and the first image description is processed by the target image generation model to obtain a first generated image, and the predicted description is processed by the target image generation model to obtain a second generated image, and then based on the image quality difference between the second generated image and the first generated image, the current environmental feedback, i.e., the reward score, is determined, and / or the current environmental feedback is determined based on the correlation between the second generated image and the predicted description, and then the loss of the text generation model is determined, and the loss is fed back to the text generation model, that is, with the goal of minimizing the loss, the parameters of the text generation model are adjusted to realize training of the text generation model in combination with environmental feedback, with the goal of improving the quality of the image generated by using the predicted loss obtained by processing the text generation model, the parameters of the text generation model are adjusted to improve the generation ability of the image description of the text generation model, so as to obtain an image description with better quality, and then improve the quality of the image generated based on the image description.

[0125] Corresponding to the above method embodiment, the embodiment of the present disclosure further provides an image generation method, which can be implemented by a first program. The first program can be the same physical program as the above target program, or can be a different physical program from the above target program. Figure 3 As shown, the method may include the following steps S310-S330:

[0126] In step S310, the image description to be processed is obtained. In one implementation, the image description to be processed may be any image description input by the user when there is a need to generate an image using the image description (for example, when generating corresponding illustrations for novels, news, etc.), and accordingly, the target program may obtain the image description input by the user as the image description to be processed.

[0127] Next, in step S320, based on the image description to be processed, the target image description is obtained by training the text generation model using the aforementioned text generation model training method. The process of obtaining the target image description can refer to the aforementioned process of obtaining the predicted description, which will not be described in detail here.

[0128] Then, in step S320, based on the target image description, a target generated image is obtained through a target image generation model. The target image generation model may be the target image generation model described in the aforementioned embodiment. It is understandable that, in step S320, based on the description of the image to be processed, a text generation model trained by the aforementioned text generation model training method can be used to obtain a target image description with better quality, and then based on the target image description, a target generated image with better quality can be obtained through the target image generation model, thereby improving the user experience.

[0129] The foregoing describes certain embodiments of the present specification, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in an order different from that in the embodiments, and the desired results may still be achieved. In addition, the processes depicted in the accompanying drawings do not necessarily have to be performed in the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0130] Corresponding to the above method embodiment, the present specification embodiment provides a text generation model training device 400, whose schematic block diagram is as follows: Figure 4 As shown, it is used to execute Figure 2 In the method flow shown, the device 400 includes:

[0131] A first acquisition module 410 is configured to acquire any first image description in a training sample set;

[0132] A first obtaining module 420 is configured to obtain a first generated image based on the first image description by using a target image generation model;

[0133] A second obtaining module 430 is configured to obtain a predicted description based on the first image description by using a text generation model;

[0134] A third obtaining module 440 is configured to obtain a second generated image based on the prediction description by using the target image generation model;

[0135] A first determination module 450, configured to determine a loss of the text generation model based on an image quality difference between the second generated image and the first generated image, and / or to determine a loss of the text generation model based on a correlation between the second generated image and the predicted description;

[0136] The first adjustment module 460 is configured to adjust the parameters of the text generation model with the goal of minimizing the loss.

[0137] In an optional implementation manner, the first determining module 450 includes:

[0138] A first determining unit (not shown in the figure), configured to determine a first sub-loss based on an image quality difference between the second generated image and the first generated image;

[0139] a second determining unit (not shown in the figure), configured to determine a second sub-loss based on a difference between a first correlation value determined based on the second generated image and the predicted description and a second correlation value determined based on the first generated image and the first image description;

[0140] A third determination unit (not shown in the figure) is configured to determine the loss of the text generation model based on the first sub-loss and / or the second sub-loss.

[0141] In an optional implementation manner, the first correlation value is determined by using the second generated image and the predicted description through a target image-text association model;

[0142] The second correlation value is determined by using the first generated image and the first image description through the target image-text association model.

[0143] In an optional implementation manner, the first determining module 450 further includes:

[0144] a fourth determining unit (not shown in the figure), configured to determine, before determining the second sub-loss, a first quality assessment value corresponding to the first generated image and a second quality assessment value corresponding to the second generated image using a preset quality assessment model;

[0145] A fifth determining unit (not shown in the figure) is configured to determine the image quality difference based on a difference between the second quality evaluation value and the first quality evaluation value.

[0146] In an optional embodiment, the preset quality assessment model includes an aesthetic prediction model and / or an image gradient algorithm;

[0147] The first quality assessment value comprises a first aesthetic score determined by the aesthetic prediction model based on the first generated image, and / or a first clarity score determined by the image gradient algorithm based on the first generated image;

[0148] The second quality assessment value comprises a second aesthetic score determined by the aesthetic prediction model based on the second generated image, and / or a first sharpness score determined by the image gradient algorithm based on the second generated image;

[0149] The fifth determination unit is specifically configured to determine the image quality difference according to a difference between the second aesthetic score and the first aesthetic score, and / or to determine the image quality difference according to a difference between the second clarity score and the first clarity score.

[0150] In an optional implementation, it also includes:

[0151] A third acquisition module (not shown in the figure), configured to acquire the second image description and its corresponding sample image;

[0152] a sixth obtaining module (not shown in the figure), configured to obtain a generated image corresponding to the second image description based on the second image description by using an image generation model to be trained;

[0153] A second determination module (not shown in the figure) is configured to determine a first prediction loss corresponding to the image generation model to be trained by using the sample image and the generated image corresponding to the second image description;

[0154] The second adjustment module (not shown in the figure) is configured to adjust the parameters of the image generation model to be trained with the goal of minimizing the first prediction loss to obtain the target image generation model that meets the first convergence condition.

[0155] In an optional implementation, it also includes:

[0156] a fourth acquisition module (not shown in the figure), configured to acquire a third image description before obtaining a predicted description based on the first image description through a text generation model;

[0157] an occlusion module (not shown in the figure), configured to occlude part of the description of the third image description by using a preset occlusion algorithm to obtain a fourth image description;

[0158] a seventh obtaining module (not shown in the figure), configured to obtain the output of the text generation model to be trained based on the fourth image description through the text generation model to be trained;

[0159] A third determination module (not shown in the figure) is configured to determine a second prediction loss corresponding to the text generation model to be trained by using the output of the text generation model to be trained and the third image description;

[0160] The third adjustment module (not shown in the figure) is configured to adjust the parameters of the text generation model to be trained with the goal of minimizing the second prediction loss to obtain the text generation model that meets the second convergence condition.

[0161] Corresponding to the above method embodiment, the present specification embodiment provides an image generating device 500, whose schematic block diagram is as follows: Figure 5 As shown, it is used to execute Figure 3 In the method flow shown, the device 500 includes:

[0162] A second acquisition module 510 is configured to acquire a description of an image to be processed;

[0163] A fourth obtaining module 520 is configured to obtain a target image description based on the image description to be processed by using the text generation model trained by the training device of the text generation model mentioned above;

[0164] The fifth obtaining module 530 is configured to obtain a target generated image based on the target image description through a target image generation model.

[0165] The above device embodiments correspond to the method embodiments. For specific descriptions, please refer to the description of the method embodiments, which will not be repeated here. The device embodiments are obtained based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. For specific descriptions, please refer to the corresponding method embodiments.

[0166] The present disclosure provides an electronic device, including a memory and a processor, wherein the memory stores an executable code, and when the processor executes the executable code, the following is implemented: Figure 2 or Figure 3 The method flow shown below. Figure 6 , which shows a schematic structural diagram of an electronic device suitable for implementing an embodiment of the present disclosure. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0167] like Figure 6 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In RAM 603, various programs and data required for the operation of the electronic device are also stored. The processing device 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0168] Typically, the following devices may be connected to the I / O interface 605: input devices 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, etc.; output devices 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 608 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Figure 6 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 6 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0169] In particular, according to an embodiment of the present specification, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present specification includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present specification are executed.

[0170] The embodiment of the present specification also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed in a computer, the computer is caused to execute the following steps provided in the present specification: Figure 2 The training method of the text generation model shown in Figure 3 The image generation method shown.

[0171] It should be noted that the computer-readable medium described in the embodiments of this specification may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiments of this specification, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the embodiments of this specification, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0172] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains any first image description in the training sample set; obtains a first generated image based on the first image description through the target image generation model; obtains a predicted description based on the first image description through the text generation model; obtains a second generated image based on the predicted description through the target image generation model; determines the loss of the text generation model based on the image quality difference between the second generated image and the first generated image, and / or determines the loss of the text generation model based on the correlation between the second generated image and the predicted description; adjusts the parameters of the text generation model with the goal of minimizing the loss. Alternatively, the electronic device: obtains a description of the image to be processed; obtains a target image description based on the image to be processed, through the text generation model trained by the aforementioned text generation model training method; obtains a target generated image based on the target image description through the target image generation model.

[0173] Computer program code for performing the operations of embodiments of the present specification may be written in one or more programming languages ​​or combinations thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0174] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the storage medium and computing device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0175] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present invention may be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0176] The specific implementation methods described above further describe the purpose, technical solutions and beneficial effects of the embodiments of the present invention in detail. It should be understood that the above description is only a specific implementation method of the embodiments of the present invention and is not intended to limit the scope of protection of the present invention. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solution of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A training method for a text generation model, comprising: Get any first image description in the training sample set; Based on the first image description, a first generated image is obtained by using a target image generation model; Based on the first image description, obtaining a predicted description through a text generation model; Based on the prediction description, a second generated image is obtained by generating a model of the target image; determining a loss of the text generation model based on a difference in image quality between the second generated image and the first generated image, and / or determining a loss of the text generation model based on a correlation between the second generated image and the predicted description; With the goal of minimizing the loss, the parameters of the text generation model are adjusted.

2. The method of claim 1, wherein: Determining the loss of the text generation model includes: determining a first sub-loss based on an image quality difference between the second generated image and the first generated image; determining a second sub-loss based on a difference between a first correlation value determined based on the second generated image and the predicted description and a second correlation value determined based on the first generated image and the first image description; The loss of the text generation model is determined based on the first sub-loss and / or the second sub-loss.

3. The method of claim 2, wherein: The first correlation value is determined by using the second generated image and the predicted description through a target image-text association model; The second correlation value is determined by using the first generated image and the first image description through the target image-text association model.

4. The method according to claim 2, before determining the second sub-loss, further comprising: Determine, by using a preset quality assessment model, a first quality assessment value corresponding to the first generated image and a second quality assessment value corresponding to the second generated image; The image quality difference is determined based on a difference between the second quality assessment value and the first quality assessment value.

5. The method of claim 4, wherein: The preset quality assessment model includes an aesthetic prediction model and / or an image gradient algorithm; The first quality assessment value comprises a first aesthetic score determined by the aesthetic prediction model based on the first generated image, and / or a first clarity score determined by the image gradient algorithm based on the first generated image; The second quality assessment value comprises a second aesthetic score determined by the aesthetic prediction model based on the second generated image, and / or a first sharpness score determined by the image gradient algorithm based on the second generated image; The determining the image quality difference comprises: The image quality difference is determined based on a difference between the second aesthetic score and the first aesthetic score, and / or the image quality difference is determined based on a difference between the second clarity score and the first clarity score.

6. The method of claim 1, further comprising: Obtaining a second image description and a corresponding sample image; Based on the second image description, obtaining a generated image corresponding to the second image description through an image generation model to be trained; Determine a first prediction loss corresponding to the image generation model to be trained by using the sample image and a generated image corresponding to the second image description; With the goal of minimizing the first prediction loss, the parameters of the image generation model to be trained are adjusted to obtain the target image generation model that meets the first convergence condition.

7. The method according to claim 1, before obtaining the predicted description based on the first image description by using a text generation model, further comprises: obtaining a third image description; Using a preset occlusion algorithm, occluding part of the description of the third image description to obtain a fourth image description; Based on the fourth image description, obtaining an output of the text generation model to be trained through the text generation model to be trained; Determine a second prediction loss corresponding to the text generation model to be trained by using the output of the text generation model to be trained and the third image description; With the goal of minimizing the second prediction loss, the parameters of the text generation model to be trained are adjusted to obtain the text generation model that meets the second convergence condition.

8. A method for generating an image, comprising: Get the description of the image to be processed; Based on the image description to be processed, obtaining a target image description by using a text generation model trained by the text generation model training method described in any one of claims 1 to 7; Based on the target image description, a target generated image is obtained through a target image generation model.

9. A training device for a text generation model, comprising: A first acquisition module, configured to acquire any first image description in a training sample set; A first obtaining module is configured to obtain a first generated image through a target image generation model based on the first image description; A second obtaining module is configured to obtain a predicted description based on the first image description through a text generation model; A third obtaining module is configured to obtain a second generated image based on the prediction description through the target image generation model; a first determination module configured to determine a loss of the text generation model based on an image quality difference between the second generated image and the first generated image, and / or to determine a loss of the text generation model based on a correlation between the second generated image and the predicted description; The first adjustment module is configured to adjust the parameters of the text generation model with the goal of minimizing the loss.

10. An image generating device, comprising: A second acquisition module is configured to acquire a description of the image to be processed; a fourth obtaining module configured to obtain a target image description based on the to-be-processed image description by using a text generation model trained by the training device for a text generation model according to claim 9; The fifth obtaining module is configured to obtain a target generated image based on the target image description through a target image generation model.

11. An electronic device comprising a memory and a processor, wherein: The memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 8 is implemented.

12. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 8.

Images