Text generation image optimization method and device based on AI feedback and medium

Abstract

The invention discloses a text generation image optimization method and device based on AI feedback and a medium, and relates to the technical field of text generation image optimization. According to the method, a large language model is introduced to serve as an alternative scheme of human preference annotation, AI feedback is used for approximating human preferences, so that high-cost manual dependence of a human feedback mode is eliminated, and the method generally comprises the two processes that 1, preference approximation is conducted, the large language model is used for myopia of the human preferences, and ranking pairs are generated; and 2) aligning: aligning a more exquisite image with a simple text, and generating a higher-quality image by an expected text generation image model.According to the text generation image optimization method and device based on AI feedback and the medium provided by the invention, the problems of high labor cost and fine adjustment overhead of a human feedback optimization method are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of text-generated image optimization, and in particular to a text-generated image optimization method, device and medium based on AI feedback. Background Art

[0002] In recent years, the field of artificial intelligence-generated content has made tremendous revolutionary progress. As an important branch of this field and also as an intersection of natural language processing and computer vision, text-to-image generation has broad application prospects and important research value. Given an appropriate text description, the text-to-image generation model aims to generate high-fidelity and semantically relevant images. Thanks to the excellent ability of the diffusion model in generative modeling of continuous data, it has become the most advanced text-to-image generation method by training on network-scale data. Although some progress has been made in text-to-image generation models, such as Dalle, Imagen, stable diffusion, etc., the generated images are still far from perfect. Specifically, the current text-to-image diffusion model faces two major challenges: the semantics of the generated image may not be completely consistent with the text description and the generated image is inconsistent with human aesthetic preferences.

[0003] The optimization method based on human feedback learning is a very effective technique that can align pre-trained models with human preferences and values, thereby guiding the text-generated image model to generate images that are more in line with human preferences. This method mainly involves manually annotating a large-scale human preference dataset. Specifically, the annotator needs to label the corresponding multiple images with the specified text, that is, use the image preference ranking as a label, and then train a reward model that represents human preferences. The reward model can serve as a reference for human preference consistency by scoring the images corresponding to the text. The higher the score, the more in line with human preferences the image is. In order to use the reward model to improve the alignment ability of the text-generated image model and human preferences, there are currently two main optimization methods: 1) Reward learning, using the score output by the reward model as gradient feedback to guide the text-generated image model to generate higher-scoring images. 2) Data screening, using the reward model to screen out high-scoring data to supervise the fine-tuning of the text-generated image model. These fine-tuning methods have achieved some success in aligning the text-generated image model with human preferences.

[0004] However, the fine-tuning method based on reward learning is to randomly sample a noisy image at a time step in the later stage of image denoising, and use the reward model to output a score as the gradient feedback at that time step to guide the model to generate samples with higher scores. Therefore, this method needs to introduce a reward model during the training process, which will bring additional memory overhead. The fine-tuning method based on data screening uses the reward model to screen the training data for high scores, but the improvement ability is very limited. These methods often require training a reward model as a judgment of human preference, such as Figure 1 As shown, the final improvement result also greatly depends on the fitting performance of the reward model to the face preference data.

[0005] Although incorporating human preferences into the text-to-image model can generate higher quality results that are more in line with human intentions, the biggest bottleneck of the existing optimization methods based on human feedback learning is the high and expensive labor costs. The collection process of preference data sets is time-consuming and labor-intensive, so how to make the model's generation results more in line with human preferences while not relying on manual annotation is a very practical and meaningful problem. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a method, device and medium for text-generated image optimization based on AI feedback, which uses AI feedback to approximate human preferences as an alternative to manual labeling, and proposes a simple and effective alignment strategy to improve the text-generated image model, thereby solving the problems of expensive labor costs and fine-tuning overhead of human feedback optimization methods.

[0007] In a first aspect, the present invention provides a method for optimizing text-generated images based on AI feedback, comprising: The process of generating the preference data set is as follows: a data set including real user text prompts and corresponding generated images is obtained, and then data with less than N text prompts are filtered out from the data set to obtain simple text prompts; a large language model is used to rewrite the simple text prompts and expand them into complex text prompts, and then a generative model is used to generate corresponding simple images and complex images according to the simple text prompts and complex text prompts, respectively, and simple text prompts, simple images, complex text prompts, and complex images are used as ranking pairs to form a preference data set as a replacement for human preference annotations; Alignment process: We use our preferred dataset to align simple text cues with complex images using supervised fine-tuning to optimize the text-to-image model.

[0008] Furthermore, in the process of generating the preference data set, after the simple text prompts are screened, the simple text prompts with a similarity of more than 0.7 are filtered out to ensure the diversity of data distribution.

[0009] Furthermore, during the generation of the preference dataset, complex text prompts are structured and represented as {original simple prompts}, {extended descriptions generated by a large language model}, and {terms specifying image quality}.

[0010] Furthermore, in the alignment process, the optimization objective of supervised fine-tuning is expressed as:

[0011] in, represents the mathematical expectation, Represents a variational autoencoder VAE, which is used to convert complex text prompts Generated Image Obtaining hidden variables from pixel space compression to low-dimensional latent space , represents the potential space The hidden variable of the step, Represents a frozen CLIP text encoder for encoding simple text prompts , Represents the standard Gaussian noise distribution The noise sampled in .

[0012] In a second aspect, the present invention provides a text generation image optimization device based on AI feedback, comprising: The preference data set generation module is used to obtain a data set including real user text prompts and corresponding generated images, and then filter out data with less than N text prompts from the data set to obtain simple text prompts; use the large language model to rewrite the simple text prompts and expand them into complex text prompts, and then use the generation model to generate corresponding simple images and complex images according to the simple text prompts and complex text prompts respectively, and use the simple text prompts, simple images, complex text prompts, and complex images as ranking pairs to form a preference data set as a replacement for human preference annotations; An alignment module is used to optimize the text-to-image model by aligning simple text cues and complex images using supervised fine-tuning using a preferred dataset.

[0013] Furthermore, in the preference data set generation module, after screening and obtaining simple text prompts, simple text prompts with a similarity of more than 0.7 are filtered out to ensure the diversity of data distribution.

[0014] Furthermore, in the preference dataset generation module, complex text prompts are structured as {original simple prompts}, {extended descriptions generated by large language models}, and {terms specifying image quality}.

[0015] Furthermore, in the alignment module, the optimization objective of supervised fine-tuning is expressed as:

[0016] in, represents the mathematical expectation, Represents a variational autoencoder VAE, which is used to convert complex text prompts Generated Image Obtaining hidden variables from pixel space compression to low-dimensional latent space , represents the potential space The hidden variable of the step, Represents a frozen CLIP text encoder for encoding simple text prompts , Represents the standard Gaussian noise distribution The noise sampled in .

[0017] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0018] The technical solution provided by the present invention has at least the following technical effects: Regarding the cost of human annotation, the solution of the present invention uses a large language model to construct text ranking pairs of uneven quality as an alternative to manual annotation, thereby getting rid of the high-cost manual dependence on human feedback. Regarding the cost of fine-tuning, the solution of the present invention directly fine-tunes on the preference data set generated based on AI feedback, aligning simple text prompts and complex images as the optimization direction, in the hope that the model will generate results that are more in line with human expectations. The training process does not require the participation of the reward model, thereby reducing overhead, and at the same time can have a more obvious improvement effect than the reward screening method. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.

[0020] Figure 1 It is a schematic diagram of the overall process of learning based on human feedback in the prior art; Figure 2 It is a schematic diagram of the overall process of an embodiment of the present invention; Figure 3 Schematic diagram of the process of the method in Embodiment 1 of the present invention; Figure 4 The evaluation results of the image generated by the rewritten complex text prompt in the first embodiment of the present invention in terms of aesthetic performance and user preference; Figure 5 This is a schematic diagram of experimental results of a reward model trained on a preference data set fed back by AI in Embodiment 1 of the present invention; Figure 6 A schematic diagram of the structure of the device in the second embodiment of the present invention DETAILED DESCRIPTION The embodiments of the present invention provide a method, device and medium for text-generated image optimization based on AI feedback, which utilizes AI feedback to approximate human preferences as an alternative to manual annotation, and proposes a simple and effective alignment strategy to improve the text-generated image model, thereby solving the problems of expensive labor costs and fine-tuning overhead of the human feedback optimization method.

[0021] The technical solution in the embodiment of the present invention has the following general idea: This paper aims to solve the problem of expensive labor costs and fine-tuning overhead of human feedback optimization methods by introducing a large language model as an alternative to human preference annotation. Using AI feedback to approximate human preferences, it can get rid of the high cost of manual dependence on human feedback. The overall process is as follows Figure 2 As shown in the figure, the overall process includes two processes: 1) approximate preference, using a large language model to approximate human preferences and generate ranking pairs; 2) alignment, aligning more exquisite images with simple texts, expecting the text-to-image model to generate higher quality images.

[0022] Regarding the cost of human annotation, the present invention explores the potential of large language models and human preference consistency, and uses large language models to construct text ranking pairs of uneven quality. This step is crucial to the quality of constructing an effective preference data set. The present invention uses a large language model to rewrite simple texts to generate more detailed prompts as an alternative to manual annotation, thereby getting rid of the expensive manual dependence on human feedback. At the same time, the method of this embodiment has better consistency between images and texts, and is more in line with human preference consistency.

[0023] As for the fine-tuning overhead, the present invention directly performs fine-tuning on the preference data set generated based on AI feedback. Since humans' usual language habits are close to brief descriptions, and a text-to-image model is usually used to generate the desired image, a more detailed description is often required. Therefore, in order to make up for the gap between humans' daily expression habits and the usage scenarios of text-to-image models, the present invention uses the alignment of simple texts and complex images as an optimization direction, in the expectation that the model will generate results that are more in line with human expectations. Compared with the reward-based learning method, the training process of the present invention does not require the participation of the reward model, thereby reducing overhead, and can also have a more obvious improvement effect than the reward screening method.

[0024] Embodiment 1 This embodiment provides a text generation image optimization method based on AI feedback, such as Figure 3 As shown, including: S1, preference data set generation process: obtain a data set including real user text prompts and corresponding generated images, and then filter out data with less than N text prompts from the data set (N can be 10) to obtain simple text prompts; the data of this embodiment comes from the public DiffusionDB data set, which includes 1.8 million unique real user prompts and 14 million images generated by the stable diffusion model. Sampling real user prompts can ensure that the model generation results are more in line with user expectations during reasoning. Filtering out simple text prompts with less than 10 words can ensure consistency with the usage scenarios of real users. After filtering out simple text prompts, filtering out simple text prompts with a similarity of more than 0.7 can ensure the diversity of data distribution. Finally, this embodiment uses a graph-based selective annotation method to select representative, thematically diverse text prompts as a simple text prompt data set.

[0025] Then, a large language model is used to rewrite the simple text prompts and expand them into complex text prompts. Then, a generative model is used to generate corresponding simple images and complex images based on the simple text prompts and complex text prompts respectively. The simple text prompts, simple images, complex text prompts, and complex images are used as ranked pairs to form a preference data set. The semantic alignment and aesthetic quality of complex images have been greatly improved. Therefore, a set of ranked pairs are naturally generated, which can be used as a substitute for human preference annotations.

[0026] S2. Alignment process: We use the preferred dataset to align simple text prompts and complex images using supervised fine-tuning to optimize the text-to-image model.

[0027] Assume use Represents a simple text prompt, which is then rewritten by a large language model to produce a more sophisticated complex prompt , and then use the text-generated image model to generate the corresponding image to obtain a simple text-generated image and images generated from complex text , the optimization objective of supervised fine-tuning can be expressed as:

[0028] in, represents the mathematical expectation, Represents a variational autoencoder VAE, which is used to convert complex text prompts Generated Image Obtaining hidden variables from pixel space compression to low-dimensional latent space , represents the potential space The hidden variable of the step, Represents a frozen CLIP text encoder for encoding simple text prompts , Represents the standard Gaussian noise distribution The noise sampled in .

[0029] In a preferred implementation, in order to make the method have better consistency between images and texts and be more consistent with human preferences, during the generation of the preference data set, by using prompt word engineering and carefully designed complex text prompt structure templates, the large language model is guided to rewrite simple texts and expand them into more detailed complex text prompts. The complex text prompts are structured as {original simple text prompts}, {extended descriptions generated by the large language model}, and {terms specifying image quality}.

[0030] Figure 4 The evaluation results of the rewritten complex text prompts in terms of aesthetic performance and user preference are shown. Complex is the winning ratio of complex text prompts, Simple is the winning ratio of simple text prompts, and Tie is the tie ratio. The evaluation is conducted from multiple angles. Aesthetic is used to evaluate aesthetic quality; ImageReward and HPSV2 are reward models trained on a large number of human-annotated preference datasets to evaluate the consistency of human preferences; Human Preference is the result of a user questionnaire.

[0031] The present invention uses AI feedback to approximate human preferences as an alternative to manual annotation. At the same time, a simple and effective alignment strategy is proposed to improve the text generation image model. The preferences of the comparison constructed in this embodiment are consistent with human preferences, and the performance is comparable to that of the method based on human feedback learning, which shows the effectiveness of the method of the present invention. In addition, the method of the present invention does not require the participation of the reward model, further reducing the training cost. The present invention provides new insights into AI feedback learning for text image generation, while paving the way for more efficient and cost-effective text image generation and other training processes.

[0032] In order to prove the effectiveness of the method of using AI feedback to approximate human preferences, the reward model was further trained on the obtained preference dataset. Two different fine-tuning methods for optimizing the text-based graph model were considered, namely ReFL and RAFT. These two methods are successful practices of two paradigms: reinforcement learning and supervised fine-tuning. ReFL uses the reward model as real-time gradient feedback during training to maximize the expectation of the reward value; RAFT uses the reward model to screen high-scoring text-based graph pairs offline as a dataset for supervised fine-tuning. Figure 5As shown, IR represents ImageReward, which uses BLIP as the backbone network and fine-tunes the reward model on a large amount of manually annotated preference data. It can score the corresponding image according to the text, and the higher the score, the more it meets human expectations; S2C represents S2CReward, which follows the training method of ImageReward, uses BLIP as the backbone network, and is a reward model trained on the preference data set constructed by AI feedback. The experimental results show that the reward model trained on the preference data set of AI feedback can also achieve performance comparable to that of the reward model trained on human preference data. In addition, joint optimization with the reward model trained on human preference data can bring further effect gains. This shows that the AI ​​feedback preference data set of the present invention can be used as an effective substitute for the human preference data set, thereby providing an efficient data annotation method for image-text generation tasks.

[0033] Based on the same inventive concept, the present application also provides a device corresponding to the method in Example 1, see Example 2 for details.

[0034] Embodiment 2 In this embodiment, a text generation image optimization device based on AI feedback is provided, such as Figure 6 As shown, including: The preference data set generation module is used to obtain a data set including real user text prompts and corresponding generated images, and then filter out data with less than N text prompts from the data set to obtain simple text prompts; use the large language model to rewrite the simple text prompts and expand them into complex text prompts, and then use the generation model to generate corresponding simple images and complex images according to the simple text prompts and complex text prompts respectively, and use the simple text prompts, simple images, complex text prompts, and complex images as ranking pairs to form a preference data set as a replacement for human preference annotations; An alignment module is used to optimize the text-to-image model by aligning simple text cues and complex images using supervised fine-tuning using a preferred dataset.

[0035] Furthermore, in the preference data set generation module, after screening and obtaining simple text prompts, simple text prompts with a similarity of more than 0.7 are filtered out to ensure the diversity of data distribution.

[0036] Furthermore, in the preference dataset generation module, complex text prompts are structured as {original simple prompts}, {extended descriptions generated by large language models}, and {terms specifying image quality}.

[0037] Furthermore, in the alignment module, the optimization objective of supervised fine-tuning is expressed as:

[0038] in, represents the mathematical expectation, Represents a variational autoencoder VAE, which is used to convert complex text prompts Generated Image Obtaining hidden variables from pixel space compression to low-dimensional latent space , represents the potential space The hidden variable of the step, Represents a frozen CLIP text encoder for encoding simple text prompts , Represents the standard Gaussian noise distribution The noise sampled in .

[0039] Since the device introduced in the second embodiment of the present invention is a device used to implement the method of the first embodiment of the present invention, those skilled in the art can understand the specific structure and deformation of the device based on the method introduced in the first embodiment of the present invention, so it is not described here in detail. All devices used in the method of the first embodiment of the present invention belong to the scope of protection of the present invention.

[0040] Based on the same inventive concept, the present application provides a storage medium corresponding to Example 1, see Example 3 for details.

[0041] Embodiment 3 This embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, any implementation method in the first embodiment can be implemented.

[0042] Regarding the cost of human annotation, the solution of the present invention uses a large language model to construct text ranking pairs of uneven quality as an alternative to manual annotation, thereby getting rid of the high-cost manual dependence on human feedback. Regarding the cost of fine-tuning, the solution of the present invention directly fine-tunes on the preference data set generated based on AI feedback, aligning simple text prompts and complex images as the optimization direction, in the hope that the model will generate results that are more in line with human expectations. The training process does not require the participation of the reward model, thereby reducing overhead, and at the same time can have a more obvious improvement effect than the reward screening method.

[0043] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0044] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0045] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0046] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0047] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A text generation image optimization method based on AI feedback, characterized in that: include: The process of generating the preference data set is as follows: a data set including real user text prompts and corresponding generated images is obtained, and then data with less than N text prompts are filtered out from the data set to obtain simple text prompts; a large language model is used to rewrite the simple text prompts and expand them into complex text prompts, and then a generative model is used to generate corresponding simple images and complex images according to the simple text prompts and complex text prompts, respectively, and simple text prompts, simple images, complex text prompts, and complex images are used as ranking pairs to form a preference data set as a replacement for human preference annotations; Alignment process: We use our preferred dataset to align simple text cues with complex images using supervised fine-tuning to optimize the text-to-image model.

2. The method according to claim 1, characterized in that: During the generation of the preference data set, after the simple text prompts are screened, the simple text prompts with a similarity of more than 0.7 are filtered out to ensure the diversity of data distribution.

3. The method according to claim 1 or 2, characterized in that: During the generation of the preference dataset, complex text prompts are structured as {original simple prompts}, {extended descriptions generated by large language models}, and {terms specifying image quality}.

4. The method according to claim 1, characterized in that: During the alignment process, the optimization objective of supervised fine-tuning is expressed as: in, represents the mathematical expectation, Represents a variational autoencoder VAE, which is used to convert complex text prompts Generated Image Obtaining hidden variables from pixel space compression to low-dimensional latent space , represents the potential space The hidden variable of the step, Represents a frozen CLIP text encoder for encoding simple text prompts , Represents the standard Gaussian noise distribution The noise sampled in .

5. A text generation image optimization device based on AI feedback, characterized in that: include: The preference data set generation module is used to obtain a data set including real user text prompts and corresponding generated images, and then filter out data with less than N text prompts from the data set to obtain simple text prompts; use the large language model to rewrite the simple text prompts and expand them into complex text prompts, and then use the generation model to generate corresponding simple images and complex images according to the simple text prompts and complex text prompts respectively, and use the simple text prompts, simple images, complex text prompts, and complex images as ranking pairs to form a preference data set as a replacement for human preference annotations; An alignment module is used to optimize the text-to-image model by aligning simple text cues and complex images using supervised fine-tuning using a preferred dataset.

6. The device according to claim 5, characterized in that: In the preference data set generation module, after screening and obtaining simple text prompts, simple text prompts with a similarity of more than 0.7 are filtered out to ensure the diversity of data distribution.

7. The device according to claim 5 or 6, characterized in that: In the preference dataset generation module, complex text prompts are structured as {original simple prompts}, {extended descriptions generated by large language models}, and {terms specifying image quality}.

8. The device according to claim 5, characterized in that: In the alignment module, the optimization objective of supervised fine-tuning is expressed as: in, represents the mathematical expectation, Represents a variational autoencoder VAE, which is used to convert complex text prompts Generated Image Obtaining hidden variables from pixel space compression to low-dimensional latent space , represents the potential space The hidden variable of the step, Represents a frozen CLIP text encoder for encoding simple text prompts , Represents the standard Gaussian noise distribution The noise sampled in .

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images