Text generation model training method and apparatus, image generation method and apparatus, device, and medium

Abstract

Embodiments of the present disclosure provide a text generation model training method and apparatus, an image generation method and apparatus, a device, and a medium. The training method comprises: obtaining any first image caption in a training sample set; on the basis of the first image caption, obtaining a first generated image by means of a target image generation model; on the basis of the first image caption, obtaining a predicted caption by means of a text generation model; on the basis of the predicted caption, obtaining a second generated image by means of a target image generation model; on the basis of the image quality difference between the second generated image and the first generated image, determining the loss of the text generation model, and / or, on the basis of the correlation between the second generated image and the predicted caption, determining the loss of the text generation model; and with the objective of minimizing the loss, adjusting a parameter of the text generation model.

Description

Text generation model training, image generation method, device, equipment and medium

[0001] This application claims priority to Chinese Patent Application No. 202311048571.8 filed on August 18, 2023, and the contents of the above-mentioned Chinese patent application disclosure are hereby incorporated by reference in their entirety as a part of this application. Technical Field

[0002] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a training method, apparatus, device, and medium for a text generation model, as well as an image generation method, apparatus, device, and medium. Background Art

[0003] In the field of image generation, image descriptions (also called text prompts, i.e., prompts) can be used to guide image generation.

[0004] Summary of the Invention

[0005] One or more embodiments of the present disclosure provide a method, apparatus, device, and medium for training a text generation model, as well as a method, apparatus, device, and medium for image generation, which can achieve a model that can generate better quality image descriptions through training, thereby obtaining better quality image descriptions, and thereby improving the quality of images generated based on the image descriptions.

[0006] According to a first aspect, an embodiment of the present disclosure provides a method for training a text generation model, comprising:

[0007] Get any first image description in the training sample set;

[0008] Based on the first image description, a first generated image is obtained by using a target image generation model;

[0009] Based on the first image description, obtain a predicted description through a text generation model;

[0010] Based on the prediction description, a second generated image is obtained by generating a model of the target image;

[0011] determining a loss of the text generation model based on a difference in image quality between the second generated image and the first generated image, and / or determining a loss of the text generation model based on a correlation between the second generated image and the predicted description;

[0012] With the goal of minimizing the loss, the parameters of the text generation model are adjusted.

[0013] According to a second aspect, an embodiment of the present disclosure provides an image generation method, comprising:

[0014] Get the description of the image to be processed;

[0015] Based on the image description to be processed, obtaining a target image description by using a text generation model trained using the text generation model training method described in the first aspect;

[0016] Based on the target image description, a target generated image is obtained through a target image generation model.

[0017] According to a third aspect, an embodiment of the present disclosure provides a training device for a text generation model, comprising:

[0018] A first acquisition module is configured to acquire any first image description in a training sample set;

[0019] A first generating module is configured to generate a first generated image based on the first image description by using a target image generating model;

[0020] a second generation module configured to obtain a predicted description based on the first image description through a text generation model;

[0021] a third generating module configured to generate a second generated image based on the predicted description and using the target image generation model;

[0022] a first determining module configured to determine a loss of the text generation model based on an image quality difference between the second generated image and the first generated image, and / or to determine a loss of the text generation model based on a correlation between the second generated image and the predicted description;

[0023] The first adjustment module is configured to adjust the parameters of the text generation model with the goal of minimizing the loss.

[0024] According to a fourth aspect, an embodiment of the present disclosure further provides an image generating device, comprising:

[0025] A second acquisition module is configured to acquire a description of the image to be processed;

[0026] a fourth generating module configured to obtain a target image description based on the image description to be processed by using a text generation model trained by the text generation model training device according to the third aspect;

[0027] The fifth generation module is configured to obtain a target generated image based on the target image description through a target image generation model.

[0028] According to a fifth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method described in the first aspect or the second aspect.

[0029] According to the sixth aspect, an embodiment of the present disclosure further provides an electronic device, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in the first aspect or the second aspect is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort.

[0031] FIG1 is a schematic diagram of an implementation framework of an embodiment disclosed in the present disclosure;

[0032] FIG2 is a flow chart of a method for training a text generation model according to an embodiment of the present disclosure;

[0033] FIG3 is a schematic diagram of a flow chart of an image generation method provided by an embodiment of the present disclosure;

[0034] FIG4 is a schematic block diagram of a training apparatus for a text generation model according to an embodiment of the present disclosure;

[0035] FIG5 is a schematic block diagram of an image generating device provided by an embodiment of the present disclosure;

[0036] FIG6 is a schematic block diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0037] The technical solutions of several embodiments of the present disclosure will be described in detail below in a schematic manner with reference to the accompanying drawings.

[0038] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0039] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0040] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0041] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0042] The embodiments of the present disclosure disclose a text generation model training method, apparatus, device, and medium, as well as an image generation method, apparatus, device, and medium. The application scenarios and technical concepts of the methods are first introduced below, specifically as follows:

[0043] In the field of image generation, image descriptions (also known as text prompts) can be used to guide image generation. For example, based on an image description, an image generation model based on content generation techniques can be used to generate the image corresponding to the description. As can be seen from the above process, the quality of the image description is crucial to the quality of the generated image. To ensure the quality of the generated image, it is crucial to determine how to obtain high-quality image descriptions.

[0044] To obtain higher-quality image descriptions, Figure 1 shows a schematic diagram of an implementation scenario according to an embodiment of the present disclosure. This implementation scenario illustrates a training system for training (i.e., fine-tuning) a text generation model using the training method provided by an embodiment of the present disclosure. The training system includes: a text generation model to be trained (i.e., fine-tuned), a target image generation model, and a reward model. This training system can be used to train a text generation model that can generate higher-quality image descriptions.

[0045] According to an embodiment of the present disclosure, the text generation model may be a text generation model pre-trained using several text groups, each text group may include text and occluded text generated by the occluded part of the text. The target image generation model may be an image generation model pre-trained based on several image-text pairs. The reward model is a model for determining feedback data (i.e., current environmental feedback, reward score) for fine-tuning the text generation model, and the feedback data is used to determine the loss for fine-tuning the text generation model. The feedback data may include the image quality difference E1 between the second generated image and the first generated image mentioned later, and / or the difference E2 between the first correlation value determined based on the second generated image and the predicted description and the second correlation value determined based on the first generated image and the first image description mentioned later.

[0046] It should be noted that each of the aforementioned image-text pairs may include an image and its corresponding image description. For clarity, the image in the image-text pair can be referred to as the sample image, and the image description in the image-text pair can be referred to as the second image description. The aforementioned text can refer to the image description used to pre-train the text generation model. For clarity, it can be referred to as the third image description.

[0047] The image-text pairs are used to train the image generation model, enabling it to generate corresponding images based on the image descriptions. The text generation model is pre-trained using the third image descriptions and the occluded text generated from the occluded portions of the third image descriptions, enabling it to predict and complete (expand) text.

[0048] The following describes the process of fine-tuning a text generation model (i.e., the training process of the text generation model). For example, a training sample set for fine-tuning the text generation model is obtained, where the training sample set includes multiple image descriptions; an arbitrary image description is obtained from the training sample set as a first image description; based on the first image description, a first generated image corresponding to the first image description is obtained using the target image generation model; based on the first image description, a predicted description is obtained using the text generation model (i.e., a pre-trained text generation model); based on the predicted description, a second generated image corresponding to the predicted description is obtained using the target image generation model.

[0049] Afterwards, the reward model is used to determine the image quality difference between the first generated image and the second generated image; and / or, based on the second generated image and the predicted description, the correlation between the second generated image and the predicted description is determined, wherein the image quality difference between the second generated image and the first generated image can characterize the quality improvement of the predicted description obtained by the text generation model compared to the first image description; the correlation between the second generated image and the predicted description can characterize the relevance and quality of the second generated image and the predicted description.

[0050] Next, based on the image quality difference, the loss of the text generation model is determined, and / or based on the correlation between the second generated image and the predicted description, the loss of the text generation model is determined, that is, the correlation between the second generated image and the predicted description and / or the image quality difference is used to determine the loss of the text generation model. Accordingly, the loss can characterize the quality improvement of the predicted description obtained by the text generation model compared to the first image description, and / or the relevance and quality of the second generated image and the predicted description; then, with the goal of minimizing the loss, the parameters of the text generation model are adjusted, that is, the parameters of the text generation model are adjusted in the direction of increasing the quality improvement of the predicted description obtained by the text generation model compared to the first image description, and / or the direction of improving the relevance and quality of the second generated image and the predicted description, thereby improving the image description generation capability of the text generation model, improving the quality of the image description generated by it, and thereby improving the quality of the image generated based on the image description.

[0051] In the above process, the reinforcement learning idea is adopted, and the first image description as the current environment state is input into the text generation model to obtain a predicted description as the execution action, and the first image description is processed by the target image generation model to obtain a first generated image, and the predicted description is processed by the target image generation model to obtain a second generated image. Then, based on the image quality difference between the second generated image and the first generated image, the current environment feedback, that is, the reward score, is determined, and / or, based on the correlation between the second generated image and the predicted description, the current environment feedback is determined, and then the loss of the text generation model is determined, and the loss is fed back to the text generation model, that is, with the goal of minimizing the loss, the parameters of the text generation model are adjusted to realize training of the text generation model in combination with the environment feedback, with the goal of improving the quality of the image generated using the predicted loss obtained by processing the text generation model, the parameters of the text generation model are adjusted to improve the image description generation ability of the text generation model, so as to obtain a better quality image description, and thus improve the quality of the image generated based on the image description.

[0052] Moreover, in the above process, there is no need to label the data in the training sample set, which can save labor costs while improving the ability of the text generation model to generate better quality image descriptions.

[0053] The following, in conjunction with exemplary embodiments, details the training methods, apparatuses, devices, and media for the text generation model, as well as the image generation methods, apparatuses, devices, and media provided by the present disclosure. For clarity, the pre-training processes for the target image generation model and the text generation model are first introduced.

[0054] In one possible implementation, the pre-training process of the target image generation model may include:

[0055] An image generation model to be trained is obtained, and a training dataset S for training the image generation model is obtained. The training dataset S may include multiple image-text pairs, wherein any image-text pair M includes a sample image T and its corresponding image description (referred to as a second image description) X. In one case, the training dataset S may be obtained from a public dataset that can provide a large number of image-text pairs, or an artificially pre-manufactured dataset. For example, the process of artificially pre-manufactured datasets may include: capturing an image, and annotating the image description to obtain an image-text pair, thereby obtaining an artificially pre-manufactured dataset.

[0056] In one case, the image description may also be referred to as a text prompt, that is, a text prompt used to generate an image.

[0057] Obtain any image-text pair M from a training dataset S, specifically, the second image description X and the corresponding sample image T in the image-text pair M. Based on the second image description X, the image generation model to be trained is used to obtain a generated image T' corresponding to the second image description X. The sample image T and the generated image T' are then used to determine the loss corresponding to the image generation model to be trained, which serves as the first prediction loss. For example, in one scenario, the first prediction loss can be determined using the image difference between the sample image T and the generated image T'.

[0058] Then, with the goal of minimizing the first prediction loss, that is, reducing the image difference between the sample image T and the second generated image T', the parameters of the image generation model to be trained are adjusted until the image generation model reaches a first convergence condition, thereby obtaining a target image generation model. For example, the first convergence condition may include, but is not limited to, the number of parameter adjustments being no less than a first adjustment threshold, or the calculated first prediction loss being less than the first loss threshold, or the training duration of the image generation model being no less than a preset training duration.

[0059] The above embodiment of the pre-training process of the target image generation model is described by taking an image-text pair (i.e., a pair consisting of a second image description and its corresponding sample image) as an example. In another embodiment, the above pre-training process can also be performed on a batch of samples, i.e., multiple image-text pairs, to obtain the generated images corresponding to the second image description in each image-text pair, and then determine the first prediction loss based on the generated image corresponding to the second image description in each image-text pair and the sample image in each image-text pair. With the goal of minimizing the first prediction loss, the parameters of the image generation model to be trained are adjusted. In this embodiment, the first prediction loss is determined for a batch of samples, and then the parameters of the image generation model are adjusted. This can reduce the number of parameter adjustments to the image generation model and make it easier to implement the pre-training process.

[0060] In a possible implementation, the target image generation model may be an image generation model based on a diffusion model in related technologies.

[0061] In one possible implementation, the pre-training process of the text generation model may include: obtaining the text generation model to be trained; constructing a sample description pair set D for pre-training the text generation model, where the sample description pair set D may include a plurality of sample description pairs, wherein the construction process of a sample description pair Di may include:

[0062] Get the third image description yi (for example, it can be expressed as {(a)(sunny)(windless)(morning)}, or it can also be expressed as {(a)(clear)(bright)(windless)(morning)(morning)}, where the content in each bracket represents the third image description y i , and the first representation is taken as an example below), and the third image description yi is included in the sample description pair Di.

[0063] Using a preset masking algorithm, part of the description (part of the content) of the third image description yi is masked to obtain the corresponding masked third image description as the fourth image description xi (for example, it can be expressed as {(a)(?)(?)(of)(morning)}, where the question mark "?" is the masked word), and the fourth image description xi is included in the sample description pair Di, that is: a sample description pair Di includes a third image description yi and its corresponding fourth image description xi. The preset masking algorithm can be any algorithm in the relevant technology that can achieve text masking. In a possible implementation, using the preset masking algorithm, any content (or called word) at any other position of the third image description yi except the first sentence (for example, the sentence before the first punctuation mark) can be randomly masked; words at any other position of the third image description yi except the first a words can also be randomly masked, and a can be any positive integer.

[0064] In one embodiment, the third image description yi can be an image description that can generate an image that meets preset conditions, where the preset conditions may include, but are not limited to: an image aesthetic score (which can represent the image's aesthetics) that is not less than a certain aesthetic score threshold, an image clarity score that is not less than a certain clarity threshold, and / or a correlation value between the image and the corresponding image description that is not less than a certain correlation threshold. In another embodiment, the third image description yi can be an image description of any quality obtained from the aforementioned public dataset that provides a large number of image-text pairs.

[0065] Next, based on the fourth image description xi in any sample description pair Di, the output of the text generation model to be trained is obtained through the text generation model to be trained; using the output of the text generation model to be trained and the third image description yi in the sample description pair Di, the loss corresponding to the text generation model to be trained is determined, which is called the second prediction loss.

[0066] In one implementation, after obtaining the output of the text generation model to be trained, the probability of the words at the corresponding positions of the third image description yi appearing at each occluded position in the fourth image description xi (or each position of the fourth image description xi) can be determined from the output of the text generation model to be trained according to the third image description yi (for example, in the above example, the fourth image description x i The first occluded position in the fourth image description x i The probability of the word "sunny" appearing at the position of the second word, the fourth image description x i The probability of the word “no wind” appearing at the second blocked position, i.e. the third word in the sentence).

[0067] Afterwards, the cross-entropy loss function is used to determine the aforementioned second prediction loss based on the probability of the word at the corresponding position of the third image description yi appearing at each occluded position in the fourth image description xi (or each position in the fourth image description xi). The greater the probability of the word at the corresponding position of the third image description yi appearing at each occluded position in the fourth image description xi (or each position in the fourth image description xi), the smaller the second prediction loss, indicating that the image description generation ability of the text generation model to be trained is better. Accordingly, in one case, the second prediction loss l can be expressed by the following formula:

[0068] In the above formula, p(yi|xi) represents the probability that the text generation model to be trained outputs the third image description yi for the fourth image description xi. This p(yi|xi) can be determined based on the probability that the word corresponding to the third image description yi appears at each occluded position in the fourth image description xi (or each position in the fourth image description xi) (for example, based on the average of the sum of the probabilities of the word corresponding to the third image description yi appearing at each occluded position in the fourth image description xi, or the product of the sum of the probabilities); D represents the aforementioned sample description pair set D to which the third image description yi and the fourth image description xi belong; It means calculating the expected value of logp(yi|xi).

[0069] In another implementation, the target program can determine the generated description corresponding to the fourth image description based on the output of the text generation model to be trained and the specified word determination rule, and then determine the second prediction loss based on the difference between the generated description corresponding to the fourth image description and the third image description (for example, the Euclidean distance), wherein the smaller the distance between the generated description and the third image description, the smaller the second prediction loss.

[0070] Afterwards, in order to minimize the second prediction loss, that is, to maximize the probability that the text generation model to be trained outputs the third image description yi for the fourth image description xi, that is, to reduce the difference between the generated description determined based on the output of the text generation model to be trained and the third image description, the parameters of the text generation model to be trained are adjusted until the text generation model reaches the second convergence condition, thereby obtaining a pre-trained text generation model. For example, the second convergence condition may include, but is not limited to, the number of parameter adjustments is not less than the second adjustment threshold, or the calculated second prediction loss is less than the second loss threshold. The second adjustment threshold may be the same as or different from the aforementioned first adjustment threshold, and the second loss threshold may be the same as or different from the first loss threshold of the signature.

[0071] The above embodiment of the pre-training process of the text generation model is explained by taking a sample description pair (i.e., a pair consisting of a third image description and its corresponding fourth image description) as an example. In another embodiment, the above pre-training process can also be performed on a batch of samples, i.e., multiple sample description pairs, to obtain the output of the text generation model to be trained corresponding to the fourth image description in each sample description pair, and then determine the second prediction loss based on the output of the text generation model to be trained corresponding to the fourth image description in each sample description pair and the third image description in each sample description pair. With the goal of minimizing the second prediction loss, adjust the parameters of the text generation model to be trained. In this embodiment, the second prediction loss is determined for a batch of samples, and then the parameters of the text generation model are adjusted. This can reduce the number of parameter adjustments to the text generation model and make it easier to implement the pre-training process.

[0072] In one possible implementation, the text generation model can be a transformer-based text generation model in related technologies. The text generation model can expand and supplement the input image description to generate a richer image description.

[0073] After obtaining the pre-trained target image generation model and text generation model, the pre-trained text generation model is then fine-tuned (i.e., retrained) to improve its ability to generate high-quality image descriptions. Figure 2 shows a flow chart of a method for training a text generation model in one embodiment of the present disclosure. The method can be implemented by a target program. The target program can be installed in an electronic device, and the electronic device can be implemented by any device, equipment, platform, device cluster, etc. with computing and processing capabilities. As shown in Figure 2, the method may include the following steps S210-S260:

[0074] In step S210, any first image description in a training sample set is obtained. The training sample set includes a plurality of first image descriptions. In one case, the first image description can be any type of image description used to generate an image.

[0075] Next, in step S220, based on the first image description, a first generated image is obtained through the target image generation model. In this step, the target program can input the first image description into the target image generation model, which processes the first image description and outputs an image corresponding to the first image description, i.e., the first generated image. Accordingly, the target program obtains the first generated image.

[0076] Then, in step S230, a predicted description is obtained based on the first image description through a text generation model. Then, in step S240, a second generated image is obtained based on the predicted description through a target image generation model.

[0077] In one implementation, the text generation model may include a text encoder and a text decoder. The target program inputs the first image description into the text generation model, encodes the first image description through the text encoder of the text generation model to obtain the encoding result corresponding to the first image description, and inputs the encoding result corresponding to the first image description into the text decoder of the text generation model to obtain the model output of the text generation model for the first image description.

[0078] For example, the model output may include a word probability set corresponding to each of the several positions determined based on the first image description, and the word probability set for each position may include the probability of each word in a preset token set appearing at the position, where the preset token set is a set of words that can be predicted by the text generation model. Based on the word probability set corresponding to each of the several positions included in the model output and the specified word determination rules, the word corresponding to each position can be determined (the word corresponding to the position may be empty), that is, the description after the text generation model is expanded, that is, the predicted description, and accordingly, the target program obtains the predicted description.

[0079] The specified word determination rule may be any rule in the related art that can determine the corresponding word based on the word probability set corresponding to each of the aforementioned multiple positions. The specific number of the multiple positions can be determined based on specified parameters set for the text generation model, and can refer to the maximum length of the model output, and the embodiments of the present disclosure are not limited thereto.

[0080] Next, the target program inputs the predicted description into the target image generation model, processes the predicted description through the target image generation model, and obtains a second generated image corresponding to the predicted description.

[0081] After the target program obtains the first generated image corresponding to the first image description, the predicted description and its corresponding second generated image, in step S250, the loss of the text generation model is determined based on the image quality difference between the second generated image and the first generated image, and / or the loss of the text generation model is determined based on the correlation between the second generated image and the predicted description.

[0082] In one possible implementation, the target program may determine the loss of the text generation model based on the image quality difference between the second generated image and the first generated image; or, determine the loss of the text generation model based on the correlation between the second generated image and the predicted description; or, jointly determine the loss of the text generation model based on the image quality difference between the second generated image and the first generated image, and the correlation between the second generated image and the predicted description.

[0083] It is understandable that the purpose of fine-tuning (training) the text generation model is to improve the ability of the text generation model to generate image descriptions with better quality. In theory, the better the image quality of the second generated image obtained based on the predicted description generated by the text generation model is than the image quality of the first generated image generated based on the first image description, that is, the greater the difference in image quality between the second generated image and the first generated image, the better the quality of the predicted description is than the quality of the first image description, and accordingly, the better the ability of the text generation model to generate high-quality image descriptions. The higher the correlation between the second generated image and the predicted description, the better the correlation and quality of the predicted description and its second generated image, and accordingly, the better the ability of the text generation model to generate high-quality image descriptions.

[0084] In view of this, combined with the learning mechanism taking into account reinforcement learning, the current environmental feedback, i.e., the reward score, can be determined based on the image quality difference between the second generated image and the first generated image, and / or the correlation between the second generated image and the predicted description, and the loss of the text generation model can be determined based on the image quality difference between the second generated image and the first generated image, and / or the loss of the text generation model can be determined based on the correlation between the second generated image and the predicted description, so that the loss of the text generation model is negatively correlated with the image quality difference between the second generated image and the first generated image, and / or the loss of the text generation model is negatively correlated with the correlation between the second generated image and the predicted description.

[0085] In one embodiment, the loss of the text generation model can also be determined by combining the correlation between the second generated image and the predicted description, and the difference in correlation between the first generated image and the first image description. For example, step S250 can include the following steps 11-13:

[0086] In step 11, a first sub-loss is determined based on an image quality difference between the second generated image and the first generated image.

[0087] Before determining the first sub-loss in step 11, the target program may also include a process of determining image quality differences. The process of determining image quality differences may include the following steps 21-22: In step 21, using a preset quality assessment model, determine a first quality assessment value corresponding to the first generated image and a second quality assessment value corresponding to the second generated image.

[0088] In one implementation, the quality (i.e., quality assessment value) of the second generated image and the first generated image can be determined from various perspectives. For example, these may include, but are not limited to: determining the second quality assessment value (first quality assessment value) based on the clarity of the second generated image (first generated image); determining the second quality assessment value (first quality assessment value) based on the aesthetics of the second generated image (first generated image); determining the second quality assessment value (first quality assessment value) based on a combination of the clarity of the second generated image (first generated image) and the aesthetics of the second generated image (first generated image), etc.

[0089] Accordingly, in one embodiment, the aforementioned preset quality assessment model may include an aesthetic prediction model F for determining the aesthetics of an image. B , and / or, an image gradient algorithm F for determining image sharpness T .

[0090] That is, the first quality assessment value may include: a first aesthetic score determined by the aesthetic prediction model based on the first generated image, and / or a first clarity score determined by the image gradient algorithm based on the first generated image; the second quality assessment value may include a second aesthetic score determined by the aesthetic prediction model based on the second generated image, and / or a second clarity score determined by the image gradient algorithm based on the second generated image.

[0091] The aesthetic score can represent the aesthetics of the corresponding image. A larger aesthetic score can represent a more beautiful image and a better image quality. In one case, the aesthetic score can take a value between [0, 1].

[0092] The clarity score may represent the clarity of the corresponding image. A larger clarity score may represent a clearer corresponding image and a better image quality. In one embodiment, the clarity score may take a value between [0, 1].

[0093] It is understandable that the aforementioned aesthetic prediction model F B The model can be any model in the related art that can evaluate the aesthetics of an image, i.e., its aesthetic score. In one possible embodiment, the aesthetic prediction model can be a model pre-trained based on images and their corresponding aesthetic label values. The aesthetic prediction model can output an aesthetic score for an input image. The images and their corresponding aesthetic label values ​​used to train the aesthetic prediction model can be selected from any dataset containing a large number of images and their corresponding aesthetic score labels representing the image's aesthetics, and the embodiments of the present disclosure are not limited in this regard.

[0094] The aforementioned image gradient algorithm F TThe Tenengrad gradient algorithm can be used. The following describes the process of determining an image clarity score using an image gradient algorithm based on a first generated image, taking the process of determining a first clarity score using the image gradient algorithm as an example. This process may include, for example, the following: the target program may use the Sobel operator to calculate the horizontal gradient (which may be referred to as the horizontal gradient) and the vertical gradient (which may be referred to as the vertical gradient) of each pixel point (referred to as the first pixel point) in the first generated image. The target program then uses the horizontal and vertical gradients of each first pixel point to determine the total gradient of each first pixel point. The target program then calculates the average of the sum of the total gradients of each first pixel point as the first gradient of the first generated image. The target program then uses the first gradient of the first generated image to determine the first clarity score of the first generated image. For example, the first gradient of the first generated image may be directly determined as the first clarity score of the first generated image; alternatively, the target program may use the first gradient of the first generated image and a preset correspondence between gradients and clarity scores to determine the first clarity score of the first generated image.

[0095] The larger the gradient of the first generated image (or the second generated image), the higher its clarity, and correspondingly the larger its clarity score, that is, the better the image quality.

[0096] In one case, the total gradient of each first pixel point (for example, the j-th first pixel point) can be determined by the following formula (1):

[0097] In the above formula (1), T j Represents the total gradient of the jth first pixel, t j1 Represents the horizontal gradient of the jth first pixel, t j2 Represents the vertical gradient of the j-th first pixel.

[0098] Using a preset quality assessment model, a first quality assessment value corresponding to the first generated image and a second quality assessment value corresponding to the second generated image are determined. In step 22, an image quality difference is determined based on the difference between the second quality assessment value and the first quality assessment value.

[0099] In one embodiment, the process of determining the image quality difference may include: determining the image quality difference according to a difference between the second aesthetic score and the first aesthetic score, and / or determining the image quality difference according to a difference between the second clarity score and the first clarity score.

[0100] For example, when the first quality assessment value includes a first aesthetic score and the second quality assessment value includes a second aesthetic score, the image quality difference can be determined based on the difference between the aesthetics of the second generated image and the first generated image, that is, the difference (e.g., the difference) between the second aesthetic score and the first aesthetic score.

[0101] For another example, when the first quality assessment value includes a first clarity score and the second quality assessment value includes a second clarity score, the image quality difference can be determined based on the difference between the clarity of the second generated image and the first generated image, that is, the difference (e.g., the difference) between the second clarity score and the first clarity score.

[0102] For another example, when the first quality evaluation value includes a first aesthetic score and a first clarity score, and the second quality evaluation value includes a second aesthetic score and a second clarity score, the image quality difference is determined based on the difference between the second aesthetic score and the first aesthetic score, and the difference between the second clarity score and the first clarity score.

[0103] The greater the difference between the second aesthetic score and the first aesthetic score, the greater the difference between the second clarity score and the first clarity score, and the greater the image quality difference between the second generated image and the first generated image.

[0104] In the case where the image quality difference is determined based on the difference between the second aesthetic score and the first aesthetic score and the difference between the second clarity score and the first clarity score, the image quality difference can be expressed by the following formula (2): E1 = (F B (I P )-F B (I o ))+(F T (I P )-F T (I o ); (2)

[0105] In the above formula (2), E1 represents the image quality difference, F B (I P ) represents the second aesthetic score of the second generated image, F B (I o ) represents the first aesthetic score of the first generated image, F B (I P )-F B (I o )) represents the difference between the second aesthetic score and the first aesthetic score, F T (I P ) represents the second clarity score of the second generated image, F T (Io ) represents the first clarity score of the first generated image, (F T (I P )-F T (I o ) represents the difference between the second clarity score and the first clarity score, I P represents the second generated image, I o represents the first generated image.

[0106] Afterwards, the target program determines a first sub-loss based on the image quality difference between the second generated image and the first generated image, wherein the greater the image quality difference between the second generated image and the first generated image, the better the image description generation and expansion capability representing the text generated image, and accordingly, the image quality difference is negatively correlated with the first sub-loss.

[0107] Next, in step 12, a second sub-loss is determined based on the difference between a first correlation value determined based on the second generated image and the predicted description and a second correlation value determined based on the first generated image and the first image description. The first correlation value may represent the correlation between the second generated image and the predicted description, with the greater the correlation between the second generated image and the predicted description, the greater the first correlation value. Similarly, the second correlation value may represent the correlation between the first generated image and the first image description, with the greater the correlation between the first generated image and the first image description, the greater the second correlation value. In one embodiment, the correlation value may take a value between [0, 1].

[0108] The greater the difference between the first correlation value and the second correlation value, the better the image description generation expansion capability representing the text-generated image. Correspondingly, the difference (e.g., differential) between the first correlation value and the second correlation value is negatively correlated with the second sub-loss.

[0109] In one implementation, the first relevance value is determined by using the second generated image and the predicted description through the target image-text association model; the second relevance value is determined by using the first generated image and the first image description through the target image-text association model. Accordingly, the aforementioned reward model may include the target image-text association model F G . The target image-text association model can be a model pre-trained based on a number of sample image-text pairs (each sample image-text pair includes an image and an image description) and their corresponding labels (including sample correlation values ​​that characterize the correlation of the corresponding sample image-text pairs). For example, the target image-text association model can be a CLIP (Contrastive Language–Image Pre-training) model. The target image-text association model can determine the correlation between the input image and image description.

[0110] The difference E2 between the first correlation value determined based on the second generated image and the predicted description and the second correlation value determined based on the first generated image and the first image description can be expressed by the following formula (3): E2 = F G (I P ,x2)-F G (I O ,x1); (3)

[0111] In the above formula (3), F G (I P , x2) represents the second generated image I P and the first correlation value determined by the prediction description x2, F G (I O ,x1) represents the first generated image I O and a second correlation value determined by the first image description x1.

[0112] After obtaining the first sub-loss and / or the second sub-loss, the loss of the text generation model is determined based on the first sub-loss and / or the second sub-loss in step 13. The loss of the text generation model may be positively correlated with the first sub-loss and the second sub-loss.

[0113] In one implementation, the aforementioned reward model includes an aesthetic prediction model F B , Image Gradient Algorithm F T And the target image-text association model F G In the case of , the loss of the text generation model can be determined based on the first sub-loss and the second sub-loss. The loss of the text generation model can be expressed by the following formula (4):

[0114] In the above formula (4), Indicates the calculation of expected value.

[0115] In another implementation, the aforementioned reward model includes an aesthetic prediction model F B and image gradient algorithm F T In the case of , the first sub-loss can be determined as the loss of the text generation model. G In the case of , the second sub-loss can be determined as the loss of the text generation model.

[0116] It is understandable that the loss of the text generation model can characterize the image description (i.e., text) generation expansion effect of the text generation model. The smaller the loss of the text generation model, the better the image description generation expansion effect of the text generation model. Accordingly, in step S260, the parameters of the text generation model are adjusted with the goal of minimizing the loss of the text generation model. This step may, for example, include: based on the loss of the text generation model, using a back propagation algorithm to determine the adjustment gradient of the parameters of the text generation model; using the determined adjustment gradient and the current value of the parameters of the text generation model to determine the updated value of the parameters of the text generation model; and then adjusting the parameters of the text generation model based on the updated value of the parameters.

[0117] In one possible embodiment, the adjustment gradient of the parameters of the text generation model is determined with the goal of minimizing the loss of the text generation model. Minimizing the loss of the text generation model is to improve the ability of the text generation model to generate high-quality image descriptions, that is, to improve (increase) the aesthetic score and clarity score corresponding to the image generated by the text generation model (i.e., increase the aforementioned image quality difference) and / or the correlation value between the second generated image and the predicted description (or increase the difference between a first correlation value determined based on the second generated image and the predicted description and a second correlation value determined based on the first generated image and the first image description).

[0118] The above embodiment of the execution process of the aforementioned steps S210-S260 is explained by taking a first image description as an example. In another embodiment, the above steps S210-S250 can also be performed for a batch of samples, that is, multiple first image descriptions, to obtain the first generated image and predicted description corresponding to each first image description and the second generated image corresponding to the predicted description, respectively, and then in step S260, based on the image quality difference between the second generated image corresponding to each first image description and the first generated image, the loss of the text generation model is determined, and / or, based on the correlation between the second generated image corresponding to each first image description and the predicted description, the loss of the text generation model is determined, and the parameters of the text generation model are adjusted with the goal of minimizing the loss of the text generation model. In this embodiment, the loss of the text generation model is determined for a batch of samples, and then the parameters of the text generation model are adjusted. This can reduce the number of parameter adjustments to the text generation model and make it easier to implement the pre-training process.

[0119] Adjusting parameters of the text generation model until the text generation model reaches a preset convergence condition, thereby obtaining a trained text generation model. For example, the preset convergence condition may include, but is not limited to: the number of parameter adjustments of the text generation model exceeds a preset threshold number, or the loss of the text generation model is lower than a preset generation loss threshold.

[0120] In this embodiment, the idea of ​​reinforcement learning is adopted, and the first image description as the current environmental state is input into the text generation model to obtain a predicted description as the execution action, and the first image description is processed by the target image generation model to obtain a first generated image, and the predicted description is processed by the target image generation model to obtain a second generated image. Then, the current environmental feedback, i.e., the reward score, is determined based on the image quality difference between the second generated image and the first generated image, and / or the current environmental feedback is determined based on the correlation between the second generated image and the predicted description, and then the loss of the text generation model is determined, and the loss is fed back to the text generation model, that is: with the goal of minimizing the loss, the parameters of the text generation model are adjusted to realize training of the text generation model in combination with the environmental feedback, and with the goal of improving the quality of the image generated using the predicted loss obtained by processing the text generation model, the parameters of the text generation model are adjusted to improve the image description generation ability of the text generation model, so as to obtain a better quality image description, and thus improve the quality of the image generated based on the image description.

[0121] Corresponding to the above method embodiment, the embodiment of the present disclosure further provides an image generation method, which can be implemented by a first program. The first program can be the same physical program as the aforementioned target program, or a different physical program from the aforementioned target program. As shown in Figure 3, the method can include the following steps S310-S330:

[0122] In step S310, a description of the image to be processed is obtained. In one possible embodiment, the description of the image to be processed can be any image description input by a user when the user needs to generate an image using the image description (for example, when generating corresponding illustrations for novels, news, etc.). Accordingly, the target program can obtain the image description input by the user as the description of the image to be processed.

[0123] Next, in step S320, based on the image description to be processed, a target image description is obtained by training the text generation model using the aforementioned text generation model training method. The process of obtaining the target image description in this embodiment can be referred to the process of obtaining the predicted description in the aforementioned embodiment, and will not be repeated here.

[0124] Then, in step S320, based on the target image description, a target generated image is obtained through the target image generation model. In this embodiment, the target image generation model can be the target image generation model described in the previous embodiment. It is understood that in step S320, based on the image description to be processed, a text generation model trained using the aforementioned text generation model training method can be used to obtain a higher quality target image description. Furthermore, based on the target image description, a higher quality target generated image can be obtained through the target image generation model, thereby improving the user experience.

[0125] The foregoing describes exemplary embodiments of the present disclosure. Other embodiments based on the foregoing embodiments are also within the scope of the appended claims without departing from the spirit of the present disclosure. In some cases, the actions or steps recited in the claims may be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily need to be performed in the specific order shown or in a sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0126] Corresponding to the above method embodiment, an embodiment of the present disclosure further provides a text generation model training device 400, a schematic block diagram of which is shown in FIG4 , for executing the method flow shown in FIG2 . The device 400 may include:

[0127] A first acquisition module 410 is configured to acquire any first image description in a training sample set;

[0128] A first generating module 420 is configured to generate a first generated image based on the first image description using a target image generation model;

[0129] A second generation module 430 is configured to obtain a predicted description based on the first image description through a text generation model;

[0130] A third generating module 440 is configured to generate a second generated image based on the prediction description using the target image generation model;

[0131] a first determination module 450 configured to determine a loss of the text generation model based on an image quality difference between the second generated image and the first generated image, and / or to determine a loss of the text generation model based on a correlation between the second generated image and the predicted description;

[0132] The first adjustment module 460 is configured to adjust the parameters of the text generation model with the goal of minimizing the loss.

[0133] In an optional implementation, the first determining module 450 includes:

[0134] a first determining unit (not shown in the figure), configured to determine a first sub-loss based on an image quality difference between the second generated image and the first generated image;

[0135] a second determining unit (not shown in the figure), configured to determine a second sub-loss based on a difference between a first correlation value determined based on the second generated image and the predicted description and a second correlation value determined based on the first generated image and the first image description;

[0136] A third determination unit (not shown in the figure) is configured to determine the loss of the text generation model based on the first sub-loss and / or the second sub-loss.

[0137] In an optional embodiment, the first correlation value is determined by using the second generated image and the predicted description through a target image-text association model;

[0138] The second correlation value is determined by using the first generated image and the first image description through the target image-text association model.

[0139] In an optional implementation, the first determining module 450 further includes:

[0140] a fourth determining unit (not shown in the figure), configured to, before determining the second sub-loss, use a preset quality assessment model to determine a first quality assessment value corresponding to the first generated image and a second quality assessment value corresponding to the second generated image;

[0141] A fifth determining unit (not shown in the figure) is configured to determine the image quality difference based on a difference between the second quality evaluation value and the first quality evaluation value.

[0142] In an optional embodiment, the preset quality assessment model includes an aesthetic prediction model and / or an image gradient algorithm;

[0143] The first quality assessment value includes: a first aesthetic score determined by the aesthetic prediction model based on the first generated image, and / or a first clarity score determined by the image gradient algorithm based on the first generated image;

[0144] The second quality assessment value comprises: a second aesthetic score determined by the aesthetic prediction model based on the second generated image, and / or a first clarity score determined by the image gradient algorithm based on the second generated image;

[0145] The fifth determining unit is configured to determine the image quality difference according to a difference between the second aesthetic score and the first aesthetic score, and / or determine the image quality difference according to a difference between the second clarity score and the first clarity score.

[0146] In an optional embodiment, the invention further comprises:

[0147] a third acquisition module (not shown in the figure), configured to acquire the second image description and its corresponding sample image;

[0148] a sixth generation module (not shown in the figure), configured to obtain a generated image corresponding to the second image description based on the second image description using an image generation model to be trained;

[0149] A second determination module (not shown in the figure) is configured to determine a first prediction loss corresponding to the image generation model to be trained using the sample image and the generated image corresponding to the second image description;

[0150] The second adjustment module (not shown in the figure) is configured to adjust the parameters of the image generation model to be trained with the goal of minimizing the first prediction loss to obtain the target image generation model that meets the first convergence condition.

[0151] In an optional embodiment, the apparatus 400 may further include:

[0152] a fourth acquisition module (not shown in the figure), configured to acquire a third image description before obtaining a predicted description based on the first image description through the text generation model;

[0153] an occlusion module (not shown in the figure), configured to occlude part of the description of the third image description using a preset occlusion algorithm to obtain a fourth image description;

[0154] a seventh generation module (not shown in the figure), configured to obtain an output of the to-be-trained text generation model based on the fourth image description through the to-be-trained text generation model;

[0155] a third determination module (not shown in the figure), configured to determine a second prediction loss corresponding to the text generation model to be trained using the output of the text generation model to be trained and the third image description;

[0156] The third adjustment module (not shown in the figure) is configured to adjust the parameters of the text generation model to be trained with the goal of minimizing the second prediction loss to obtain the text generation model that meets the second convergence condition.

[0157] Corresponding to the above method embodiment, an embodiment of the present disclosure further provides an image generating apparatus 500, a schematic block diagram of which is shown in FIG5 , and is configured to execute the method flow shown in FIG3 . The apparatus 500 may include:

[0158] A second acquisition module 510 is configured to acquire a description of the image to be processed;

[0159] The fourth generating module 520 is configured to obtain a target image description based on the image description to be processed by using a text generation model trained by the aforementioned text generation model training device;

[0160] The fifth generation module 530 is configured to obtain a target generated image based on the target image description through a target image generation model.

[0161] The above-mentioned device embodiments correspond to the method embodiments. For detailed descriptions, please refer to the description of the method embodiments, which will not be repeated here. The device embodiments are obtained based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. For detailed descriptions, please refer to the corresponding method embodiments.

[0162] The present disclosure also provides an electronic device including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method flow shown in FIG2 or FIG3 is implemented. Referring now to FIG6 , a schematic diagram of the structure of an electronic device suitable for implementing the present disclosure is shown. The electronic device shown in FIG6 is merely an example and should not limit the functionality and scope of use of the present disclosure.

[0163] As shown in Figure 6, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the electronic device are also stored in RAM 603. The processing device 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0164] Typically, the following devices can be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although FIG6 shows an electronic device with various devices, it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have alternatively. Each box shown in FIG6 may represent one device, or may represent multiple devices as needed.

[0165] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0166] An embodiment of the present disclosure also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed in a computer, the computer is caused to execute the text generation model training method shown in FIG. 2 or the image generation method shown in FIG. 3 provided in an embodiment of the present disclosure.

[0167] It should be noted that the computer-readable medium described in the embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or device. In the embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or convey a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code embodied on the computer-readable medium may be conveyed using any suitable medium, including but not limited to wires, optical cables, RF (Radio Frequency), or any suitable combination thereof.

[0168] The computer-readable medium may be included in the electronic device, or may exist separately and not incorporated into the electronic device. The computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: obtain a first image description from a training sample set; obtain a first generated image based on the first image description using a target image generation model; obtain a predicted description based on the first image description using a text generation model; obtain a second generated image based on the predicted description using the target image generation model; determine the loss of the text generation model based on the image quality difference between the second generated image and the first generated image, and / or determine the loss of the text generation model based on the correlation between the second generated image and the predicted description; and adjust the parameters of the text generation model with the goal of minimizing the loss. Alternatively, the electronic device may: obtain a description of an image to be processed; obtain a target image description based on the image to be processed using a text generation model trained using the aforementioned text generation model training method; and obtain a target generated image based on the target image description using the target image generation model.

[0169] Computer program code for performing the operations of embodiments of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0170] The various embodiments of this disclosure are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other, and each embodiment focuses on the differences between the other embodiments. In particular, the storage medium and computing device embodiments are described briefly because they are generally similar to the method embodiments. For relevant portions, reference can be made to the description of the method embodiments.

[0171] Those skilled in the art will appreciate that, in one or more of the above examples, the functions described in the embodiments of the present invention may be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0172] The specific implementation methods described above further illustrate the purpose, technical solutions, and beneficial effects of the embodiments of the present invention. It should be understood that the above description is only a specific implementation method of the embodiments of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A training method for a text generation model, comprising: Get any first image description in the training sample set; Based on the first image description, a first generated image is obtained by using a target image generation model; Based on the first image description, obtaining a predicted description through a text generation model; Based on the prediction description, a second generated image is obtained by generating a model of the target image; determining a loss of the text generation model based on a difference in image quality between the second generated image and the first generated image, and / or determining a loss of the text generation model based on a correlation between the second generated image and the predicted description; With the goal of minimizing the loss, the parameters of the text generation model are adjusted.

2. The method of claim 1, wherein: Determining the loss of the text generation model includes: determining a first sub-loss based on an image quality difference between the second generated image and the first generated image; determining a second sub-loss based on a difference between a first correlation value determined based on the second generated image and the predicted description and a second correlation value determined based on the first generated image and the first image description; Based on the first sub-loss and / or the second sub-loss, the loss of the text generation model is determined.

3. The method of claim 2, wherein: The first correlation value is determined by using the second generated image and the predicted description through a target image-text association model; The second correlation value is determined by using the first generated image and the first image description through the target image-text association model.

4. The method according to claim 2 or 3, before determining the second sub-loss, further comprising: Determine, by using a preset quality assessment model, a first quality assessment value corresponding to the first generated image and a second quality assessment value corresponding to the second generated image; The image quality difference is determined based on a difference between the second quality assessment value and the first quality assessment value.

5. The method of claim 4, wherein: The preset quality assessment model includes an aesthetic prediction model and / or an image gradient algorithm; The first quality assessment value comprises: a first aesthetic score determined by the aesthetic prediction model based on the first generated image, and / or a first clarity score determined by the image gradient algorithm based on the first generated image; The second quality assessment value comprises: a second aesthetic score determined by the aesthetic prediction model based on the second generated image, and / or a first clarity score determined by the image gradient algorithm based on the second generated image; The determining the image quality difference comprises: The image quality difference is determined based on a difference between the second aesthetic score and the first aesthetic score, and / or the image quality difference is determined based on a difference between the second clarity score and the first clarity score.

6. The method according to any one of claims 1 to 5, further comprising: Obtaining a second image description and a corresponding sample image; Based on the second image description, obtaining a generated image corresponding to the second image description through an image generation model to be trained; Determine a first prediction loss corresponding to the image generation model to be trained by using the sample image and a generated image corresponding to the second image description; With the goal of minimizing the first prediction loss, the parameters of the image generation model to be trained are adjusted to obtain the target image generation model that meets the first convergence condition.

7. The method according to any one of claims 1 to 6, before obtaining the predicted description based on the first image description by using a text generation model, further comprising: obtaining a third image description; Using a preset occlusion algorithm, occluding part of the description of the third image description to obtain a fourth image description; Based on the fourth image description, obtaining an output of the text generation model to be trained through the text generation model to be trained; Determine a second prediction loss corresponding to the text generation model to be trained by using the output of the text generation model to be trained and the third image description; With the goal of minimizing the second prediction loss, adjust the text generation model to be trained Parameters to obtain the text generation model that meets the second convergence condition.

8. A method for generating an image, comprising: Get the description of the image to be processed; Based on the to-be-processed image description, obtaining a target image description by using a text generation model trained by the training method for a text generation model according to any one of claims 1 to 7; Based on the target image description, a target generated image is obtained through a target image generation model.

9. A training device for a text generation model, comprising: A first acquisition module, configured to acquire any first image description in a training sample set; A first generating module is configured to obtain a first generated image through a target image generating model based on the first image description; A second generation module is configured to obtain a predicted description based on the first image description through a text generation model; A third generation module is configured to obtain a second generated image based on the prediction description through the target image generation model; a first determination module configured to determine a loss of the text generation model based on an image quality difference between the second generated image and the first generated image, and / or to determine a loss of the text generation model based on a correlation between the second generated image and the predicted description; The first adjustment module is configured to adjust the parameters of the text generation model with the goal of minimizing the loss.

10. An image generating device, comprising: A second acquisition module is configured to acquire a description of the image to be processed; a fourth generating module configured to obtain a target image description based on the to-be-processed image description by using a text generation model trained by the training device for a text generation model according to claim 9; The fifth generation module is configured to obtain a target generated image through a target image generation model based on the target image description.

11. An electronic device comprising a memory and a processor, wherein: The memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 8 is implemented.

12. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 8.

Images