Text to Image Model Training Method, Model, Device, and Electronic Device

Abstract

To provide a method, model, device and electronic apparatus for training Text-to-Image model.SOLUTION: The method comprises: obtaining a first Text-to-Image model for generating a corresponding generated image based on input text, and a pre-trained reward model for scoring based on a data pair consisting of the input text and the corresponding generated image; and adjusting parameters of the first Text-to-Image model based on the pre-trained reward model and on a reinforcement learning strategy to thereby obtain a second Text-to-Image model.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to reinforcement learning and computer vision technology fields, and in particular to a method for training a text-to-image model, a text-to-image model, a text-to-image model training apparatus, an electronic device, a computer-readable storage medium, and a computer program product. [Background technology]

[0002] Artificial intelligence is a field that studies how computers can mimic some of the human thought processes and intelligent behaviors (e.g., learning, reasoning, thinking, planning, etc.), and includes both hardware and software technologies. Artificial intelligence hardware technologies generally include sensors, dedicated chips for artificial intelligence, cloud computing, distributed storage, and big data processing, while artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, machine learning / deep learning, big data processing technology, knowledge graph technology, and several other major areas.

[0003] A Text-Image Model (TIM) is a model that generates a corresponding image based on input text. Recent research has focused on diffusion models, which can generate artistic and aesthetic images based on the user's ambiguous natural language description. In a Text-to-Image model, the model's output image is aligned with the semantics and details of the input text, resulting in as much artistic quality as possible, and is a research direction that has attracted a lot of attention.

[0004] The approaches described in this section are not necessarily approaches that have been previously conceived or adopted. Unless otherwise noted, any approach described in this section should not be considered prior art merely because it is included in this section. Likewise, unless otherwise noted, the subject matter addressed in this section should not be considered an admission of any prior art. Summary of the Invention

[0005] The present disclosure provides a method for training a Text to Image model, a Text to Image model, a Text to Image model training apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

[0006] According to one aspect of the present disclosure, there is provided a method for training a Text to Image model, the method including: obtaining a first Text to Image model for generating a corresponding generated image based on an input text; and a pre-trained reward model for scoring based on a data pair consisting of the input text and the corresponding generated image; the method for training a Text to Image model further includes: obtaining a second Text to Image model by adjusting parameters of the first Text to Image model based on the pre-trained reward model and a reinforcement learning strategy; wherein the second Text to Image model is configured such that a cumulative reward obtained in a generation trajectory for realizing the Text to Image satisfies a preset condition, and the cumulative reward is obtained based on the reward of each stage in the generation trajectory.

[0007] According to another aspect of the present disclosure, a Text to Image model is provided, which is obtained by training according to the training method for a Text to Image model provided above.

[0008] According to another aspect of the present disclosure, there is provided an apparatus for training a Text to Image model, the apparatus including: an acquisition module configured to acquire a first Text to Image model that generates a corresponding generated image based on an input text, and a pre-trained reward model that assigns a score based on a data pair consisting of the input text and the corresponding generated image; and an adjustment module configured to obtain a second Text to Image model by adjusting parameters of the first Text to Image model based on the pre-trained reward model and a reinforcement learning strategy, wherein the second Text to Image model is configured such that a cumulative reward obtained in a generation trajectory for realizing the Text to Image satisfies a preset condition, and the cumulative reward is obtained based on a sum of rewards of each term in the generation trajectory.

[0009] According to another aspect of the present disclosure, there is provided an electronic device including: a memory communicatively coupled to at least one processor; and instructions stored in the memory executable by the at least one processor, the instructions being executable by the at least one processor to cause the at least one processor to perform the target detection method or training method provided above in this disclosure.

[0010] According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to cause a computer to perform the target detection method or training method provided above in this disclosure.

[0011] According to another aspect of the present disclosure, there is provided a computer program product including a computer program which, when executed by a processor, implements the target detection method or training method provided above in the present disclosure.

[0012] According to one or more embodiments of the present disclosure, the generation process of a Text to Image model is optimized based on reinforcement learning, and the entire generation process is controlled by a reward signal, optimizing the model in the direction of a higher cumulative reward.

[0013] It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of the present disclosure, and is not intended to limit the scope of the present disclosure. Other features of the present disclosure will be readily apparent from the following specification. [Brief explanation of the drawings]

[0014] The drawings illustratively illustrate examples, constitute a part of the specification, and together with the written description serve to explain exemplary embodiments of the examples. The illustrated examples are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar, but not necessarily identical, elements. [Figure 1] 1 is a schematic diagram illustrating an example system 100 in which various methods and apparatus described herein may be implemented, according to embodiments of the present disclosure. [Figure 2] FIG. 2 is a schematic diagram illustrating the interaction of each Text to Image model described in several embodiments of the present disclosure. [Figure 3] 1 is a flowchart illustrating a method for training a Text to Image model according to an embodiment of the present disclosure. [Figure 4] FIG. 1 is a schematic diagram illustrating a reinforcement learning strategy in a method for training a Text to Image model according to an embodiment of the present disclosure. [Figure 5] 1 is a flowchart illustrating a method for training a Text to Image model according to an embodiment of the present disclosure. [Figure 6] 1 is a flowchart illustrating a method for training a Text to Image model according to an embodiment of the present disclosure. [Figure 7]FIG. 7 is a block diagram illustrating a training apparatus 700 for a Text to Image model according to an embodiment of the present disclosure. [Figure 8] FIG. 8 is a block diagram illustrating a training apparatus 800 for a Text to Image model according to an embodiment of the present disclosure. [Figure 9] FIG. 9 is a block diagram illustrating a training apparatus 900 for a Text to Image model according to an embodiment of the present disclosure. [Figure 10] FIG. 1 is a block diagram illustrating a configuration of a training apparatus 1000 for a Text to Image model according to an embodiment of the present disclosure. [Figure 11] FIG. 11 is a block diagram illustrating a training apparatus 1100 for a Text to Image model according to an embodiment of the present disclosure. [Figure 12] FIG. 1 is a block diagram illustrating an exemplary electronic device that can be used to implement embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0015]

[0023] The following description will be made in conjunction with the drawings to illustrate exemplary embodiments of the present disclosure. Various details of the embodiments of the present disclosure are included to facilitate understanding and should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for clarity and conciseness, the following description omits descriptions of known functions and structures.

[0016] In this application, unless otherwise specified, the terms "first," "second," and the like, used to describe various elements are not intended to limit the location, timing, or importance of these elements. Such terms are used only to distinguish one element from another. In some instances, a first element and a second element may refer to the same instance of the element, or in some cases, may refer to different instances based on the context.

[0017] The terms used in the description of various examples of the present disclosure are intended only to describe particular examples and are not intended to be limiting. Unless the context clearly indicates otherwise, an element may be one or more, unless the number of elements is specifically limited. Furthermore, the term "and / or" as used in this disclosure covers any and all possible combinations of the listed items.

[0018] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0019] 1 illustrates a schematic diagram of an exemplary system 100 in which various methods and apparatus described herein may be implemented, according to embodiments of the present disclosure. Referring to FIG. 1, the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 may be configured to run one or more applications.

[0020] In an embodiment of the present disclosure, the server 120 operates to run one or more services or software applications of the method for training a Text to Image model according to an embodiment of the present disclosure.

[0021] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtualized and virtualized environments. In some embodiments, these services may be provided as web-based or cloud services, for example, provided to users of client devices 101, 102, 103, 104, 105, and / or 106 in a Software as a Service (SaaS) model.

[0022] In the configuration shown in FIG. 1 , server 120 may include one or more assemblies that implement the functionality performed by server 120. These assemblies may include software assemblies, hardware assemblies, or a combination thereof, executable on one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 may, in turn, utilize one or more client applications to interact with server 120 to access services provided by these assemblies. It should be understood that a variety of different system configurations are possible and may differ from system 100. Thus, FIG. 1 is intended to be illustrative of an example system for implementing various methods described herein and is not intended to be limiting.

[0023] A client device may provide an interface through which a user of the client device interacts with the client device. The client device may also output information to the user through the interface. Although only six client devices are shown in Figure 1, one skilled in the art will understand that the present disclosure can support any number of client devices.

[0024] Client devices 101, 102, 103, 104, 105, and / or 106 may include various types of computing devices, such as portable handheld devices, general-purpose computers (e.g., personal computers or laptops), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices. These computing devices may run various types and versions of software applications and operating systems, such as Microsoft Windows, Apple iOS, UNIX-like operating systems, Linux or Linux-like operating systems (e.g., Google Chrome OS), and may include various mobile operating systems, such as Microsoft Windows Mobile OS, iOS, Windows Phone, and Android. Portable handheld devices may include mobile phones, intelligent phones, tablets, personal digital assistants (PDAs), and the like. Wearable devices may include head-mounted displays (e.g., smart glasses) and other devices. Gaming systems may include various handheld gaming devices, Internet-enabled gaming devices, and the like. The client device may run a variety of applications, such as Internet-related applications, communication applications (eg, email applications), and short message service (SMS) applications, and may use a variety of communication protocols.

[0025] Network 110 may be any type of network known to those skilled in the art, which may use any one of several available protocols (including, but not limited to, TCP / IP, SNA, IPX, etc.) to support data communications. By way of example, one or more networks 110 may be a local area network (LAN), an Ethernet-based network, a token loop, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0026] Server 120 may include one or more general-purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframes, server clusters, or any other suitable arrangement and / or combination. Server 120 may also include one or more virtual machines running virtual operating systems or other computing architectures involving virtualization (e.g., one or more flexible pools of virtualized logical storage devices to maintain virtual storage devices for the servers). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0027] The computing units in server 120 may run one or more operating systems, including any of the operating systems listed above and any commercial server operating system. Server 120 may also run any one of a variety of additional server and / or middle-tier applications, such as an HTTP server, an FTP server, a CGI server, a JAVA server, a database server, etc.

[0028] In some embodiments, server 120 may include one or more applications for analyzing and consolidating data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and / or 106. Server 120 may include one or more applications for displaying data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and / or 106.

[0029] In some embodiments, server 120 may be a server in a distributed system or a server incorporating blockchain. Server 120 may be a cloud server, or an intelligent cloud computing server or intelligent cloud host equipped with artificial intelligence technology. A cloud server is a host product in a cloud computing service system that solves the drawbacks of traditional physical hosts and virtual private server (VPS) services, such as high management difficulty and poor business scalability.

[0030] System 100 may further include one or more databases 130. In some embodiments, these databases may be used to store data or other information. For example, one or more of databases 130 may be used to store information such as audio files or video files. Databases 130 may be located in a variety of locations. For example, a database used by server 120 may be local to server 120 or may be remote from server 120 and in communication with server 120 over a network or dedicated connection. Databases 130 may be of a variety of types. In some embodiments, a database used by server 120 may be a relational database. One or more of these databases may store, update, and retrieve data from the databases in response to commands.

[0031] In some embodiments, one or more of the databases 130 may be used by an application to store data for the application. The databases used by the application may be various types of databases, such as a key-value repository, an object repository, or a general-purpose repository supported by a file system.

[0032] The system 100 of FIG. 1 can be configured and operated in a variety of ways to accommodate the various methods and apparatus described in accordance with this disclosure.

[0033] Figure 2 shows an interaction diagram of each Text-to-Image model described in multiple embodiments of the present disclosure. Referring to Figure 2, a Text-Image Model (TIM) 200 refers to a model that generates a corresponding image based on input text. Recent research generally focuses on diffusion models, which can generate artistic and aesthetic images based on a user's vague natural language prompt, i.e., input text 201, by generating a corresponding generated image 202 based on the input text 201. In the Text-to-Image model 200, the generated image 202 of the model is aligned with the semantics and details of the input text 201 to achieve maximum artistic quality, which is a research direction that has attracted many people's attention.

[0034] Taking the Text to Image model 200 shown in FIG. 2 as an example, the input text 201 includes "rainbow clouds surrounding a golden palace, a flock of birds, a Chinese celestial maiden, and ribbons on her clothes." That is, the input text 201 includes at least four entities: clouds, palace, birds, and celestial maiden. Here, the entity attributes of the clouds and palace are color attributes (rainbow clouds and a golden palace), the entity attribute of the bird is a quantity attribute (multiple bird entities form a flock of birds), and the entity attribute of the celestial maiden is a style attribute (Chinese-style celestial maiden, ribbons on her clothes). The generated image 202 of the Text to Image model 200 only includes "rainbow clouds surrounding a golden palace and a flock of birds." That is, the generated image 202 includes only the three entities: clouds, palace, and birds, and does not include the entity celestial maiden. Therefore, the quantities of the entities in the generated image 202 and the input text 201 are not aligned. Therefore, for a user who generates an image using the Text to Image model, in order to evaluate whether the generated image 202 of the Text to Image model 200 is aligned with the input text 201 based on a human perspective, the following details still need to be improved: 1) entity quantity, 2) entity attributes, 3) multi-entity combination, 4) drawing background, and 5) drawing style. Generating detailed and error-free images in each direction will enhance the technical capabilities of Text to Image products and improve user satisfaction.

[0035] Regarding the above technical problem, the present disclosure provides a method for training a Text to Image model.

[0036] 3 shows a flowchart of a method for training a Text to Image model according to an embodiment of the present disclosure. As shown in FIG. 3, the method for training a Text to Image model includes the following steps:

[0037] In step S301, a first Text to Image model for generating a corresponding generated image based on an input text and a pre-trained reward model for scoring based on a data pair consisting of the input text and the corresponding generated image are obtained.

[0038] That is, since the same input text generates multiple output results (i.e., generated images), the generated images need to be scored by a reward model (RM) to generate a reward signal, which sorts or evaluates the generated images from a human perspective.

[0039] A possible implementation could be to use a simple binary reward signal, e.g., using a "+" or "-" sign to represent a given reward or penalty, i.e., the score in the reward model is either 0 or 1.

[0040] Because a binary reward signal may not adequately reflect the differences in the output generated images in some cases, a possible embodiment is to represent the reward signal as an integer between 0 and 5, i.e., the score of the reward model is an integer between 0 and 5, where 5 represents the highest reward and 0 represents the lowest reward. Such a reward signal can help the model better understand whether the generated images are good or bad, which helps to improve the model's performance representation in subsequent training stages.

[0041] Furthermore, when the same generated image is scored from different evaluation angles, for example, if different evaluators score the same generated image, evaluator 1 may give it 5 points and evaluator 2 may give it 3 points, making it difficult for the model to determine whether the image is good or bad during training. Since it is difficult to unify the evaluation standards due to absolute scores, a possible embodiment is to sort the good and bad results using a relative sorting method. For example, for generated image A and generated image B, evaluator 1 considers A>B, meaning that evaluator 1 believes that generated image A is more consistent with expectations than generated image B, and evaluator 2 also considers A>B. Therefore, based on the relative sorting method, the model can better distinguish between high-quality and low-quality images among many generated images.

[0042] The reward model collects artificial feedback data, trains the reward model in a comparative learning format, and adapts the discrimination order of the reward model to people's common understanding.

[0043] In step S302, based on the pre-trained reward model and the reinforcement learning strategy, a second Text to Image model is obtained by adjusting the parameters of the first Text to Image model, where the cumulative reward obtained in the generation trajectory for realizing Text to Image satisfies a preset condition, and the cumulative reward is obtained based on the reward of each stage in the generation trajectory.

[0044] Reinforcement learning strategies are a product of machine learning behaviorism, and the basic idea is that an agent acquires intelligence through continuous interaction with its environment. Reinforcement learning strategies are based on the environment (State), the actor (Action), and the reward (Reward), where the environment is the current state, the actor is the object that interacts with the environment and performs an action, the action is the action the actor performs, and the reward is the feedback given to the actor's specific action.

[0045] Corresponding to the generation trajectory for realizing Text to Image, the actor is the Text to Image model at the current stage, the state can be the input text corresponding to the Text to Image model and the generated generated image, the action is the output noise corresponding to the Text to Image model, and the reward can be designed according to artificial demand; if focusing on user feedback of the product, the reward can be designed based on user feedback. In this process, the image generation step through noise removal of the Text to Image model is used as a reinforcement learning trajectory, and the entire generation process is controlled by the reward signal, optimizing the model in the direction of higher cumulative reward.

[0046] Using a single first text-to-image model as the training basis can effectively reduce errors that occur when the model is run sequentially, thereby improving the quality of the generated results. In this manner, each input and its corresponding output can be better understood by the initial model (first text-to-image model) and corresponding operations can be performed. In a possible embodiment, high-quality data pairs can be used to fine-tune and train the first text-to-image model, thereby improving the overall performance of the model. Here, the high-quality data pairs can be additional image-text pairs, for example, manually labeled image-text pairs.

[0047] Various aspects of the target detection method according to the embodiment of the present disclosure will be further described below.

[0048] According to some embodiments, the preset condition includes that the cumulative reward obtained by the second Text to Image model in the generation trajectory for realizing Text to Image is higher than the cumulative reward obtained by the first Text to Image model in the generation trajectory for realizing Text to Image.

[0049] Text-to-image models often produce a single final result after multiple cycles of environment (state)-action (action). This means that there may be multiple input texts and generated images, but there is one reward for each action, and each output noise is scored. However, the final result is the sum of all these rewards, i.e., the cumulative reward. Reinforcement learning strategies can have countless states, and even a single state can have many feasible solutions. Therefore, updating parameters for each state-action-reward cycle makes the model very "myopic" and difficult to converge. This model can only adapt to the "current situation" and may not be able to adapt to an infinite number of states. Therefore, the ultimate goal of reinforcement learning strategies is to optimize the trajectory, not the actions.

[0050] According to some embodiments, Figure 4 shows a schematic diagram of a reinforcement learning strategy in a method for training a Text to Image model according to embodiments of the present disclosure. As shown in Figure 4, the reinforcement learning strategy includes a Proximal Policy Optimization (PPO) algorithm.

[0051] The neighborhood policy optimization algorithm is an improved version of the policy gradient algorithm. Traditional policy gradient algorithms update policy weights based on the gradient and step size of the target function. In this update process, two common problems can occur: overshooting and undershooting. Overshooting means that the update misses the reward peak and falls into the suboptimal policy region, and undershooting means that an update step size that is too small in the gradient direction results in slow convergence.

[0052] In supervised learning problems, overshooting is not a big problem, and since the data is fixed, it can be re-corrected in the next epoch. However, in reinforcement learning problems, if overshooting leads to a bad policy region, future sample batches may not provide meaningful information, and updating the policy with bad data samples can lead to bad positive feedback that cannot be recovered from.

[0053] The neighborhood policy optimization (PPO) algorithm solves this problem by establishing a target divergence, hoping that each update will be located within a certain interval near the target divergence. Here, the target divergence must be large enough to significantly change the policy, but small enough to stabilize the update. After each update, the neighborhood policy optimization (PPO) algorithm checks the size of the update. If the final updated divergence exceeds 1.5 times the target divergence, the loss factor β is doubled in the next iteration, imposing a heavier penalty. Conversely, if the update is too small, the loss factor β is halved, effectively widening the trust region.

[0054] According to some embodiments, the neighborhood strategy optimization (PPO) algorithm uses a behavior sub-model 403 that is initialized based on a first Text to Image model 401 and an evaluation sub-model 404 that is initialized based on a pre-trained reward model 402.

[0055] The choice of initial point can determine to some extent whether the algorithm converges, and when it does, whether the initial point can converge to multiple blocks of learning convergence, high-cost points, or low-cost points. Initialization that is too large can lead to gradient explosion, while initialization that is too small can lead to gradient vanishing. Therefore, a policy can be trained with offline data (i.e., data collected by a human presenter, a scripted policy, or other reinforcement learning agent) and used to initialize a new reinforcement learning policy. This process makes the new reinforcement learning policy appear pre-trained. Next, this policy is used to initialize and fine-tune an agent (i.e., action submodel, Actor)-evaluation (i.e., evaluation submodel, Critic) network, where a pre-trained first Text-to-Image model 401 is used as the initial agent (Actor) and a pre-trained reward model 402 is used as the initial critic (Critic). Using a priori information avoids random exploration of the state space. Such a priori information helps the agent understand which state of the environment is good and should be further explored. At the same time, the reward model 402 and the first Text to Image model 401 are simultaneously fine-tuned, so that the fine-tuned second Text to Image model 405 takes into account the elements of the reward model, and the details problem can be avoided.

[0056] According to some embodiments, the generation trajectory includes at least one stage, where for each stage in the generation trajectory, the behavior sub-model 403 generates a corresponding output noisy image based on the provided input text, and the evaluation sub-model 404 outputs a reward for the current stage based on the input text and the output noisy image of the current stage.

[0057] For example, based on the same input text X, we generate two generated images Y1 and Y2, one generated from the first Text-to-Image model and the other generated from the current iteration Text-to-Image model via a reinforcement learning strategy. The reward, which compares the generated images of the two models and calculates the difference, can also be thought of as a penalty term, since this reward can be positive or negative. This term is used to reward or penalize the reinforcement learning strategy for the degree to which it deviates from the initial model (i.e., the first Text-to-Image model) in each training batch to ensure that the model outputs reasonable generated images. Removing this penalty term could result in the model generating a jumble of nonsense images during optimization, potentially fooling the reward model and providing a high reward value.

[0058] A reward is a function that generates a scalar that represents the "advantage" of an agent being in a particular state and performing a particular action.

[0059] According to some embodiments, the reward for the current stage comprises the relative entropy between the output of the behavioral sub-model 403 of the previous stage prior to the current stage and the output of the behavioral sub-model 403 of the current stage.

[0060] Within the generative trajectory, the reward for the noisy image is only the Kullback-Leible divergence (i.e., KL divergence) with the latter loss, which can be used to measure the degree of difference between the two distributions. If the difference between the two is small, the KL divergence will be small. If the two distributions match, the KL divergence will be zero.

[0061] Therefore, we adopt KL divergence as the penalty term in the reinforcement learning strategy, and fine-tune the reward model and the pre-trained model jointly, so that the generative model takes into account the elements of the reward model and avoid the details problem.

[0062] According to some embodiments, the reward for the current stage may include the difference between the rating value of the previous stage before the current stage and the rating value of the current stage, where the rating value is obtained by scoring the pre-trained reward model 402 based on the provided input text and the corresponding output noisy image.

[0063] Since the generated noisy image itself can be evaluated, it can also use the score from the reward model itself as the reward, and the reward for each step in the generated trajectory can be the score of the reward model.

[0064] As a possible embodiment, it is also possible to directly use a manually identified reward fraction instead of the reward model, it is understood that the reward model can be designed according to artificial demand, and when paying attention to user feedback of the product, the reward can be designed based on user feedback.

[0065] According to some embodiments, the cumulative reward obtained on the generation trajectory includes a total score obtained by the pre-trained reward model based on the initial inputs and final outputs of the generation trajectory, and a loss term that is the product of the reward and a loss coefficient at the final stage in the generation trajectory. The reward function is

[0066]

number

[0067] The first term in the formula is a positive value, and the purpose of score is to increase the cumulative total score so that it matches expectations. The second term is a penalty term, which can prevent the trained model from deviating from the previously adjusted model, otherwise the results may not match expectations.

[0068] According to some embodiments, the parameters of the second Text to Image model 405 are obtained by a backpropagation algorithm based on the accumulated rewards in the generated trajectory of the second Text to Image model 405.

[0069] The emergence of the backpropagation (BP) algorithm marked a major breakthrough in the development of neural networks and is the basis of many current deep learning training methods. This method calculates the gradient of each parameter in the loss function of a neural network, updates the parameters in accordance with the optimization method, and reduces the loss function. The reward function is considered a positive loss function, and the entire generation process is controlled by the reward signal, optimizing the model toward a higher cumulative reward.

[0070] According to some embodiments, Figure 5 shows a flowchart of a method for training a Text to Image model according to an embodiment of the present disclosure. As shown in Figure 5, the method for training a Text to Image model includes: A step S501 of obtaining a first Text to Image model for generating a corresponding generated image based on an input text and a pre-trained reward model for scoring based on a data pair consisting of the input text and the corresponding generated image; Step S502: obtaining a second Text to Image model by adjusting parameters of the first Text to Image model based on the pre-trained reward model and the reinforcement learning strategy, where the second Text to Image model satisfies a predetermined condition in which the cumulative reward obtained in the generation trajectory for realizing the Text to Image satisfies a predetermined condition, and the cumulative reward is obtained based on the reward of each stage in the generation trajectory; Before step S501, the method further includes step S503 of training a reward model based on the feedback dataset.

[0071] In a possible embodiment, the pre-trained reward model is obtained by training based on a feedback dataset including a plurality of feedback data, where the plurality of feedback data includes data pairs each consisting of an input text and a corresponding generated image, and a feedback state corresponding to the data pair, where the feedback state is used to indicate whether the corresponding generated image generated for the same input text belongs to positive feedback or negative feedback. That is, the feedback data in the feedback dataset is actually a triplet of "input-output-evaluation," where the feedback state is typically given based on human feedback.

[0072] Assume that there are four sorted generated images A, B, C, and D based on one input text x, and that they are sorted based on human feedback, with A>B>C>D. Here, for input text x, image A is of higher quality than image B in general human perception. When training a reward model with known sorting, the higher-order data will be biased toward positive feedback (high-quality images), and the lower-order data will be biased toward negative feedback (low-quality images).

[0073] According to some embodiments, training the reward model includes training the reward model in a comparative learning format based on the plurality of feedback data to output a first reward score for a data pair whose feedback state is positive feedback and a second reward score for a data pair whose feedback state is negative feedback, wherein a difference between the first reward score and the second reward score is used to represent a difference in quality of a corresponding generated image.

[0074] Based on one input text x, there are four sorted generated images A, B, C and D, and human feedback is sorted as follows: A>B>C>D. The reward model requires that the scores of the four generated images satisfy r(A)>r(B)>r(C)>r(D), so the loss function of the reward model is:

[0075]

number

[0076] Since the data in the feedback dataset is sorted by default from highest to lowest score, we simply traverse and sum the score differences between the previous and next terms.

[0077] In a possible embodiment, step 503 can be performed multiple times to achieve better optimization of the reward model. According to some embodiments, the feedback dataset includes multiple feedback data from at least two different sources. The feedback dataset may also include feedback data from multiple different sources. By introducing more data sources, data can be collected from various angles, such as user feedback and manual labeling. For feedback data from different sources, the optimized text-to-image model can take into account alignment factors such as quantity, attributes, and background, as well as multi-entity combinations and drawing styles.

[0078] According to some embodiments, the plurality of feedback data includes at least two of user-feedback data and manually-compared data with manually-labeled data, wherein the user-feedback data derives the feedback state based on user behavior, the manually-labeled data derives the feedback state based on manually-labeled results, and the manually-compared data derives the feedback state based on different versions of generated images.

[0079] D RM is a dataset used for reward models and includes three parts: user feedback, manual labeling, and manual comparison. User feedback is generally related to product formats, such as when users split, zoom, or rate data they may like. These actions can be used to consider drawing style based on the user's judgment. Manual labeling generally involves expert annotators who help distinguish between good and bad images by labeling them. It compares the same input text and generated image data pairs using a different version of the Text to Image model than manual comparison, which can improve entity matching.

[0080] According to some embodiments, Figure 6 shows a flowchart of a method for training a Text to Image model according to an embodiment of the present disclosure. As shown in Figure 6, the method for training a Text to Image model includes: A step S601 of obtaining a first Text to Image model for generating a corresponding generated image based on an input text and a pre-trained reward model for scoring based on a data pair consisting of the input text and the corresponding generated image; Step S602: adjusting parameters of the first Text to Image model based on the pre-trained reward model and the reinforcement learning strategy to obtain a second Text to Image model, where the second Text to Image model satisfies a predetermined condition in which the cumulative reward obtained in the generation trajectory for realizing the Text to Image satisfies a predetermined condition, and the cumulative reward is obtained based on the reward of each stage in the generation trajectory; Before step S601, the method includes step S603 of obtaining manually labeled image-text pairs as training samples of the first Text to Image model to be trained, and step S604 of obtaining a supervised trained first Text to Image model by updating parameters of the first Text to Image model to be trained based on the backpropagation algorithm.

[0081] A first text-to-image model can be obtained through supervised fine-tuning (SFT) using a pre-trained text-to-image model. Fine-tuning using a pre-trained text-to-image model STF can then be performed using standard supervised learning methods, i.e., manually labeled (input, output) text pairs are used as training samples, and the model parameters can be updated using the backpropagation algorithm. This approach allows the model to better understand each input and its corresponding output and perform corresponding operations. Furthermore, the supervised fine-tuning STF using a pre-trained text-to-image model can effectively reduce errors that occur during sequential model execution, thereby improving the quality of the generated results.

[0082] According to some embodiments, the present disclosure further provides a Text to Image model obtained by training according to the Text to Image model training method provided in the above embodiments.

[0083] 7 shows a block diagram of a Text to Image model training apparatus 700 according to an embodiment of the present disclosure. As shown in FIG. 7, the Text to Image model training apparatus 700 includes:

[0084] An acquisition module 701 configured to acquire a first Text to Image model that generates a corresponding generated image based on an input text, and a pre-trained reward model that assigns a score based on a data pair consisting of the input text and the corresponding generated image.

[0085] Since the same input text generates multiple output results (generated images), the generated images need to be scored by a reward model (RM) to generate a reward signal, which sorts or evaluates the generated images from a human perspective.

[0086] The reward model collects artificial feedback data, trains the reward model in a comparative learning format, and adapts the discrimination order of the reward model to people's common understanding.

[0087] an adjustment module 702, which obtains a second Text to Image model by adjusting parameters of the first Text to Image model based on the pre-trained reward model and a reinforcement learning strategy, wherein the second Text to Image model is configured such that a cumulative reward obtained in a generation trajectory for realizing Text to Image satisfies a preset condition, and the cumulative reward is obtained based on a sum of rewards of each term in the generation trajectory.

[0088] The image generation step of the Text to Image model through noise removal is used as a reinforcement learning trajectory, and the entire generation process is controlled by a reward signal, optimizing the model in the direction of higher cumulative rewards.

[0089] Using a single first text-to-image model as the training basis can effectively reduce errors that occur when the model is run sequentially, thereby improving the quality of the generated results. In this manner, each input and its corresponding output can be better understood by the initial model (first text-to-image model) and corresponding operations can be performed. In a possible embodiment, high-quality data pairs can be used to fine-tune and train the first text-to-image model, thereby improving the overall performance of the model. Here, the high-quality data pairs can be additional image-text pairs, for example, manually labeled image-text pairs.

[0090] According to some embodiments, the preset condition includes that the cumulative reward obtained by the second Text to Image model in the generation trajectory for realizing Text to Image is higher than the cumulative reward obtained by the first Text to Image model in the generation trajectory for realizing Text to Image.

[0091] Text-to-image models often produce a single final result after multiple cycles of environment (state)-action (action). This means that there may be multiple input texts and generated images, but there is one reward for each action, and each output noise is scored. However, the final result is the sum of all these rewards, i.e., the cumulative reward. Reinforcement learning strategies can have countless states, and even a single state can have many feasible solutions. Therefore, updating parameters for each state-action-reward cycle makes the model very "myopic" and difficult to converge. This model can only adapt to the "current situation" and may not be able to adapt to an infinite number of states. Therefore, the ultimate goal of reinforcement learning strategies is to optimize the trajectory, not the actions.

[0092] According to some embodiments, the reinforcement learning strategy includes a Proximal Policy Optimization (PPO) algorithm.

[0093] The neighborhood policy optimization algorithm is an improved algorithm of policy gradient. Traditional policy gradient algorithms update policy weights based on the gradient and step size of the target function. In this update process, two common problems can occur: overshooting and undershooting. Here, overshooting means that the update misses the reward peak and falls into the suboptimal policy region, and undershooting means that an update step size that is too small in the gradient direction results in slow convergence.

[0094] In supervised learning problems, overshooting is not a big problem, and since the data is fixed, it can be re-corrected in the next epoch. However, in reinforcement learning problems, if overshooting leads to a bad policy region, future sample batches may not provide meaningful information, and updating the policy with bad data samples can lead to bad positive feedback that cannot be recovered from.

[0095] The neighborhood policy optimization (PPO) algorithm solves this problem by establishing a target divergence, hoping that each update will be located within a certain interval near the target divergence. Here, the target divergence must be large enough to significantly change the policy, but small enough to stabilize the update. After each update, the neighborhood policy optimization (PPO) algorithm checks the size of the update. If the final updated divergence exceeds 1.5 times the target divergence, the loss factor β is doubled in the next iteration, imposing a heavier penalty. Conversely, if the update is too small, the loss factor β is halved, effectively widening the trust region.

[0096] According to some embodiments, the neighborhood policy optimization algorithm uses a behavior sub-model and an evaluation sub-model. Figure 8 shows a block diagram of a training apparatus 800 for a Text to Image model according to an embodiment of the present disclosure. As shown in Figure 8, the adjustment module 801: a behavior sub-module 8011 configured to initialize the behavior sub-model based on the first Text to Image model; and an evaluation sub-module 8012 configured to initialize the evaluation sub-model based on the pre-trained reward model.

[0097] The choice of initial point can determine to some extent whether the algorithm converges, and when it does, whether the initial point can converge to multiple blocks of learning convergence, high-cost points, or low-cost points. Initialization that is too large can lead to gradient explosion, while initialization that is too small can lead to gradient vanishing. Therefore, a policy can be trained with offline data (i.e., data collected by a human presenter, a scripted policy, or other RL agent) and used to initialize a new RL policy. This process makes the new RL policy appear pre-trained. Next, this policy is used to initialize and fine-tune an actor-critic network, where the initial actor is a pre-trained first text-to-image model and the initial critic is a pre-trained reward model. Using a priori information avoids random exploration of the state space. Such a priori information helps the agent understand which state of the environment is favorable and should be further explored. At the same time, the reward model and the first Text to Image model are simultaneously fine-tuned, so that the fine-tuned second Text to Image model takes into account the elements of the reward model, thereby avoiding the detail problem.

[0098] The adjustment module 802 is the same as the adjustment module in the previous embodiment, and therefore a detailed description thereof will be omitted here.

[0099] According to some embodiments, the generating trajectory comprises at least one stage, wherein for each stage in the generating trajectory: The behavior sub-module 8011 is further configured to generate a corresponding output noisy image based on the provided input text.

[0100] The evaluation submodule 8012 is further configured to output a reward for the current stage based on the input text and the output noisy image for the current stage.

[0101] For example, based on the same input text X, two generated images Y1 and Y2 are generated, one from the first Text-to-Image model and the other from the current iteration Text-to-Image model via a reinforcement learning strategy. The reward, which compares the generated images of the two models and calculates the difference, can also be thought of as a penalty term, since this reward can be positive or negative. This term is used to reward or penalize the reinforcement learning strategy for the degree to which the model deviates from the initial model (the first Text-to-Image model) in each training batch to ensure that the model outputs reasonable generated images. Removing this penalty term could result in the model generating a jumble of nonsense images during optimization, potentially fooling the reward model and providing a high reward value.

[0102] A reward is a function that generates a scalar that represents the "advantage" of an agent being in a particular state and performing a particular action.

[0103] According to some embodiments, Figure 9 illustrates a block diagram of a training device 900 for a Text to Image model according to an embodiment of the present disclosure. As shown in Figure 9, the training module 902: The reward submodule 9021 is configured to generate a cumulative reward for the generation trajectory based on a total score and a loss term, where the total score is obtained by a pre-trained reward model based on the initial input and final output of the generation trajectory, and the loss term is a product of the reward and loss coefficient of the final stage in the generation trajectory. The reward function is

[0104]

number

[0105] The first term in the formula is a positive value, and the purpose of score is to increase the cumulative total score so that it matches expectations. The second term is a penalty term, which can prevent the trained model from deviating from the previously adjusted model, otherwise the results may not match expectations.

[0106] The acquisition module 901 is the same as the acquisition module in the previous embodiment, and therefore a detailed description thereof will be omitted here.

[0107] According to some embodiments, Figure 10 illustrates a block diagram of a training apparatus 1000 for a Text to Image model according to an embodiment of the present disclosure. As shown in Figure 10, the training apparatus 1000 for a Text to Image model includes: The system further includes a first pre-training module 1003 configured to train the reward model based on a feedback dataset including a plurality of feedback data to obtain a pre-trained reward model, the plurality of feedback data including a data pair consisting of an input text and a corresponding generated image, and a feedback state corresponding to the data pair, where the feedback state is used to indicate that the corresponding generated image generated for the same input text belongs to positive feedback or negative feedback.

[0108] The pre-trained reward model is obtained by training based on a feedback dataset containing multiple feedback data, where the multiple feedback data includes data pairs each consisting of an input text and a corresponding generated image, and a feedback state corresponding to the data pair, where the feedback state is used to indicate whether the corresponding generated image generated for the same input text belongs to positive feedback or negative feedback. That is, the feedback data in the feedback dataset is actually a ternary set of "input-output-evaluation," in which the feedback state is usually given based on human feedback.

[0109] Assume that there are four sorted generated images A, B, C, and D based on one input text x, and they are sorted based on human feedback: A>B>C>D. Here, for input text x, image A is of higher quality than image B in general human perception. When training a reward model with known sorting, the higher-order data will be biased toward positive feedback (high-quality images), and the lower-order data will be biased toward negative feedback (low-quality images).

[0110] Based on one input text x, there are four sorted generated images A, B, C and D, and human feedback is sorted as follows: A>B>C>D. The reward model requires that the scores of the four generated images satisfy r(A)>r(B)>r(C)>r(D), so the loss function of the reward model is:

[0111]

number

[0112] Since the data in the feedback dataset is sorted by default from highest to lowest score, we simply traverse and sum the score differences between the previous and next terms.

[0113] The acquisition module 1001 and the adjustment module 1002 are the same as the acquisition module and adjustment module in the above-described embodiment, and therefore a detailed description thereof will be omitted here.

[0114] According to some embodiments, Figure 11 illustrates a block diagram of a training apparatus 1100 for a Text to Image model according to an embodiment of the present disclosure. As shown in Figure 11, the training apparatus 1100 for a Text to Image model includes: The method further includes a second pre-training module 1104 configured to train the first Text to Image model to be trained based on manually labeled image-text pairs to obtain a supervised trained first Text to Image model.

[0115] A first text-to-image model can be obtained through supervised fine-tuning (SFT) using a pre-trained text-to-image model. Fine-tuning using a pre-trained text-to-image model STF can then be performed using standard supervised learning methods, i.e., manually labeled (input, output) text pairs are used as training samples, and the model parameters can be updated using the backpropagation algorithm. This approach allows the model to better understand each input and its corresponding output and perform corresponding operations. Furthermore, the supervised fine-tuning STF using a pre-trained text-to-image model can effectively reduce errors that occur during sequential model execution, thereby improving the quality of the generated results.

[0116] The acquisition module 1101 and the adjustment module 1102 are the same as the acquisition module and adjustment module in the above-described embodiment, and therefore a detailed description thereof will be omitted here.

[0117] According to embodiments of the present disclosure, an electronic device, a readable storage medium, and a computer program product are further provided.

[0118] Referring to FIG. 12 , a block diagram of an electronic device 1000 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device applicable to various aspects of the present disclosure. The electronic device may represent various forms of digital electronic computing devices, such as laptop computers, desktop computers, stage computers, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing devices, mobile phones, intelligent phones, wearable devices, and other similar computing devices. The components, their connections, and their functions shown herein are merely exemplary and do not limit the scope of the present disclosure as described and / or claimed herein.

[0119] 12, electronic device 1200 includes a computing unit 1201, which can perform various appropriate operations and processes according to a computer program stored in a read-only memory (ROM) 1202 or loaded from a storage unit 1208 into a random access memory (RAM) 1203. RAM 1203 may further store various programs and data necessary for operating electronic device 1200. Computing unit 1201, ROM 1202, and RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to bus 1204.

[0120] The components of the electronic device 1200 are connected to an I / O interface 1205, and include an input unit 1206, an output unit 1207, a storage unit 1208, and a communication unit 1209. The input unit 1206 may be any type of device capable of inputting information into the electronic device 1200. The input unit 1206 can receive input numeric or character information and generate key signal input for user settings and / or function control of the electronic device, and may include, but is not limited to, a mouse, keyboard, touchscreen, trackboard, trackball, joystick, microphone, and / or remote control. The output unit 1207 may be any type of device capable of presenting information, and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. The storage unit 1208 may include, but is not limited to, a magnetic disk or an optical disk. The communication unit 1209 enables the electronic device 1200 to exchange information / data with other devices via a computer network, e.g., the Internet, and / or various telecommunications networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, e.g., a Bluetooth device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0121] The computing unit 1201 may be various general-purpose and / or special-purpose processing components having processing and computing capabilities. Some examples of the computing unit 1201 may include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that execute machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1201 executes each of the methods and processes described above, such as the Text-to-Image model training method provided in the above embodiments. For example, in some embodiments, the Text-to-Image model training method provided in the above embodiments may be implemented as a computer software program and tangibly included in a machine-readable medium, such as the storage unit 1208. In some embodiments, some or all of the computer program may be loaded and / or installed in the electronic device 1200 via the ROM 1202 and / or the communication unit 1209. When the computer program is loaded into the RAM 1203 and executed by the computing unit 1201, it can perform one or more steps of the training method for the Text to Image model provided in the above embodiment. Alternatively, in another embodiment, the computing unit 1201 may be configured to perform the training method for the Text to Image model provided in the above embodiment in any other suitable manner (e.g., by firmware).

[0122] Various embodiments of the systems and techniques described herein may be implemented in digital electronic circuitry systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include being embodied in one or more computer programs that may be executed and / or interpreted by a programmable system including at least one programmable processor, which may be a special-purpose or general-purpose programmable processor, and may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0123] Program code implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, so that when executed by the processor or controller, the program code performs the functions / operations specified in the flowcharts and / or block diagrams. The program code may be entirely executed on a machine, partially executed on a machine, partially executed on a machine and partially executed on a remote machine as a separate software package, or entirely executed on a remote machine or server.

[0124] In the context of this disclosure, a machine-readable medium may be a tangible medium, including or storing a program for use in or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include an electrical connection with one or more leads, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0125] To provide for user interaction, a computer may implement the systems and techniques described herein and include a display device (e.g., a CRT (Cathode Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to a user, and a keyboard and pointing device (e.g., a mouse or trackball) through which a user may provide input to the computer. Other types of devices may also be used to provide for user interaction, for example, providing feedback to a user in any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback) and receiving input from a user in any form (including sound input, speech input, or tactile input).

[0126] The systems and techniques described herein may be implemented in a computing system including backstage components (e.g., as a data server), middleware components (e.g., as an application server), front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with the system or technique implementation), or any combination of backstage components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0127] The computer system may include a client and a server. The client and the server are generally remote from each other and usually interact via a communication network. The client-server relationship is created by running computer programs on corresponding computers. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0128] It should be understood that the various forms of flow described above may be used to rearrange, add, or remove steps, and for example, the steps described in this disclosure may be performed in parallel, sequentially, or in a different order, as long as the technical solutions disclosed in this disclosure can achieve the desired results, and the present disclosure is not limited thereto.

[0129] Although embodiments or examples of the present disclosure have been described with reference to the drawings, it should be understood that the above-described methods, systems, and devices are merely exemplary embodiments or examples, and that the scope of the present invention is not limited by these embodiments or examples, but only by the appended claims and their equivalents. Various elements of the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, steps may be performed in a different order than described in this disclosure. Furthermore, various elements of the embodiments or examples may be combined in various ways. It is important to note that as technology evolves, many elements described herein may be replaced by equivalent elements that appear later in this disclosure.

Claims

1. A method for training a Text to Image model, comprising: Obtaining a first text-to-image model for generating a corresponding generated image based on an input text, and a pre-trained reward model for scoring based on a data pair consisting of the input text and the corresponding generated image; and adjusting parameters of the first text-to-image model based on the pre-trained reward model and a reinforcement learning strategy to obtain a second text-to-image model; The second Text to Image model is such that a cumulative reward obtained in a generation trajectory for realizing Text to Image satisfies a predetermined condition, and the cumulative reward is obtained based on a reward at each stage in the generation trajectory; The method for training a text-to-image model, wherein the cumulative reward obtained in the generated trajectory includes a total score obtained by the pre-trained reward model based on the initial input and final output of the generated trajectory, and a loss term which is the product of the reward of the final stage in the generated trajectory and a loss coefficient.

2. 2. The method of claim 1, wherein the preset condition includes that a cumulative reward obtained on a generation trajectory for realizing Text to Image of the second Text to Image model is higher than a cumulative reward obtained on a generation trajectory for realizing Text to Image of the first Text to Image model.

3. The method of claim 1 , wherein the reinforcement learning strategy comprises a neighborhood policy optimization algorithm.

4. 4. The method of claim 3, wherein the neighborhood policy optimization algorithm uses a behavioral sub-model initialized based on the first text-to-image model and an evaluation sub-model initialized based on the pre-trained reward model.

5. The generating trajectory includes at least one step, and for each step in the generating trajectory: the behavioral sub-model generates a corresponding output noisy image based on the provided input text; The method of claim 4 , wherein the evaluation sub-model outputs a reward for the current stage based on the input text and the output noisy image for the current stage.

6. The method of claim 5 , wherein the reward for the current stage comprises a relative entropy between an output of a previous stage of the behavioral sub-model prior to the current stage and an output of the current stage of the behavioral sub-model.

7. 6. The method of claim 5, wherein the reward for the current stage comprises a difference between an evaluation value for a previous stage before the current stage and an evaluation value for the current stage, and the evaluation value is obtained by scoring the pre-trained reward model based on a provided input text and a corresponding output noisy image.

8. The method of claim 1 , wherein parameters of the second text-to-image model are obtained by a backpropagation algorithm based on the accumulated rewards in a generation trajectory of the second text-to-image model.

9. 2. The method of claim 1 , wherein the pre-trained reward model is obtained by training based on a feedback dataset including a plurality of feedback data, the plurality of feedback data including a data pair consisting of the input text and the corresponding generated image and a feedback state corresponding to the data pair, and the feedback state is used to indicate whether the corresponding generated image generated for the same input text belongs to positive feedback or negative feedback.

10. Training the reward model comprises:

10. The method of claim 9, further comprising: training the reward model in a form of comparative learning to output a first reward score for the data pair whose feedback state is positive feedback and a second reward score for the data pair whose feedback state is negative feedback based on the plurality of feedback data; and wherein a difference between the first reward score and the second reward score is used to represent a difference in quality of the corresponding generated image.

11. The method of claim 9 , wherein the feedback data set includes the plurality of feedback data from at least two different sources.

12. the plurality of feedback data includes at least two of data fed back from a user and data manually compared with manually labeled data; The data fed back from the user is used to obtain the feedback state based on the user's behavior; The manually labeled data is used to obtain the feedback state based on the manually labeled results; The method of claim 11 , wherein the manually compared data derives the feedback state based on different versions of generated images.

13. The step of obtaining the first text to image model includes: Obtaining manually labeled image-text pairs as training samples for the first text-to-image model to be trained; and updating parameters of the first text-to-image model to be trained based on a backpropagation algorithm to obtain a supervised trained first text-to-image model.

14. 1. A text-based image generation method comprising: Training a Text to Image model according to the method of any one of claims 1 to 13; and generating an image based on input text using the Text to Image model.

15. A training device for a Text to Image model, comprising: an acquisition module configured to acquire a first text-to-image model that generates a corresponding generated image based on an input text, and a pre-trained reward model that assigns a score based on a data pair consisting of the input text and the corresponding generated image; an adjustment module configured to adjust parameters of the first text-to-image model based on the pre-trained reward model and a reinforcement learning strategy to obtain a second text-to-image model; The second Text to Image model is such that a cumulative reward obtained in a generation trajectory for realizing Text to Image satisfies a predetermined condition, and the cumulative reward is obtained based on the sum of rewards of each term in the generation trajectory; The adjustment module includes: a reward module configured to generate a cumulative reward for the generated trajectory based on a total score and a loss term; the total score obtained by the pre-trained reward model based on the initial inputs and final outputs of the generated trajectory; A training apparatus for a Text to Image model, wherein the loss term is a product of the reward of the final stage in the generation trajectory and a loss coefficient.

16. 16. The apparatus of claim 15, wherein the preset condition includes a cumulative reward obtained on a generation trajectory for realizing text to image of the second text to image model being higher than a cumulative reward obtained on a generation trajectory for realizing text to image of the first text to image model.

17. The apparatus of claim 15 or 16, wherein the reinforcement learning strategy comprises a neighborhood strategy optimization algorithm.

18. The neighborhood strategy optimization algorithm uses a behavioral sub-model and an evaluation sub-model; The adjustment module includes: a behavioral sub-module configured to initialize the behavioral sub-model based on the first Text to Image model; and an evaluation sub-module configured to initialize the evaluation sub-model based on the pre-trained reward model.

19. The generating trajectory includes at least one step, and for each step in the generating trajectory: the behavioral sub-module is further configured to generate a corresponding output noisy image based on the provided input text; The apparatus of claim 18 , wherein the evaluation sub-module is further configured to output a reward for a current stage based on the input text and the output noisy image for the current stage.

20. a first pre-training module configured to train a reward model based on a feedback dataset comprising a plurality of feedback data to obtain the pre-trained reward model; 16. The device of claim 15, wherein the plurality of feedback data includes a data pair consisting of the input text and the corresponding generated image and a feedback state corresponding to the data pair, the feedback state being used to indicate whether the corresponding generated image generated for the same input text belongs to positive feedback or negative feedback.

21. 16. The apparatus of claim 15, further comprising: a second pre-training module configured to train the first text-to-image model to be trained based on manually labeled image-text pairs, thereby obtaining a supervised trained first text-to-image model.

22. An electronic device, at least one processor; a memory communicatively coupled to the at least one processor; The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to cause the at least one processor to perform the method of any one of claims 1 to 13.

23. A non-transitory computer readable storage medium having stored thereon computer instructions, the computer instructions being used to cause a computer to perform the method of any one of claims 1 to 13.

24. A computer program product that causes a processor to carry out the method according to any one of claims 1 to 13.

Images