An AI feedback-based text-to-image optimization method and device, and a medium

Abstract

The application discloses an AI feedback-based text-to-image optimization method and device and medium, and relates to the technical field of text-to-image optimization. The method introduces a large language model as an alternative to human preference annotation, uses AI feedback to approximate human preference, thereby getting rid of the high labor dependence of human feedback mode, and generally includes two processes: 1) approximating preference, using a large language model to approximate human preference to generate a ranking pair; and 2) alignment, aligning a more exquisite image with simple text, and expecting a text-to-image model to generate higher-quality images. The AI feedback-based text-to-image optimization method and device and medium provided by the application solve the problems of high labor cost and fine-tuning overhead of the human feedback optimization method.

Description

Technical Field

[0001] This invention relates to the field of text-to-image optimization technology, and in particular to a text-to-image optimization method, apparatus and medium based on AI feedback. Background Technology

[0002] In recent years, the field of AI-generated content has made tremendous revolutionary progress. Text-to-image generation, as an important branch and an intersection of natural language processing and computer vision, has broad application prospects and significant research value. Given an appropriate text description, text-to-image generation models aim to generate high-fidelity and semantically relevant images. Thanks to the excellent generative modeling capabilities of diffusion models on continuous data, trained on network-scale data, it has become the state-of-the-art method for text-to-image generation. Although text-to-image generation models have made some progress, such as Dalle, Imagen, and stable diffusion, the generated images are still far from perfect. Specifically, current text-to-image diffusion models face two main challenges: the semantics of the generated images may not be entirely consistent with the text description, and the generated images may not align with human aesthetic preferences.

[0003] Human feedback-based learning optimization is a highly effective technique that aligns pre-trained models with human preferences and values, thereby guiding text-to-image (TGE) models to generate images that better reflect human preferences. This method primarily involves manually annotating a large-scale dataset of human preferences. Specifically, annotators assign preference labels to multiple images based on given text, using the image preference ranking as the label. A reward model representing human preferences is then trained. This reward model scores the images corresponding to the text as a reference for consistency with human preferences; higher scores indicate that the image better aligns with human preferences. To leverage the reward model to improve the alignment between TGE models and human preferences, two main optimization approaches exist: 1) Reward learning, using the reward model's output score as gradient feedback to guide the TGE model to generate higher-scoring images; 2) Data filtering, using the reward model to filter high-scoring data for supervised fine-tuning of the TGE model. These fine-tuning methods have achieved some success in aligning TGE models with human preferences.

[0004] However, reward-based fine-tuning methods randomly sample noisy images at a specific time step during the later stages of image denoising. The reward model outputs a score as gradient feedback at that time step, guiding the model to generate higher-scoring samples. Therefore, this method requires an incentive model during training, incurring additional memory overhead. Data-filtering-based fine-tuning methods utilize reward models to filter high-scoring training data, but their improvement capabilities are very limited. These methods often require training a reward model to represent human preferences, such as... Figure 1 As shown, the final improvement result also depends heavily on the performance of the reward model in fitting the face preference data.

[0005] While incorporating human preferences into text-to-image models can generate higher-quality results that better reflect human intent, the biggest bottleneck of existing optimization methods based on human feedback learning is the high cost of manual labor. Collecting preference datasets is time-consuming and labor-intensive; therefore, how to make the model's generated results more in line with human preferences without relying on manual annotation is a highly valuable and significant problem. Summary of the Invention

[0006] The technical problem to be solved by this invention is to provide a text-to-image optimization method, apparatus and medium based on AI feedback. It uses AI feedback to approximate human preferences as an alternative to manual annotation, and proposes a simple and effective alignment strategy to improve the text-to-image model, thereby solving the problems of high manual cost and fine-tuning overhead of human feedback optimization methods.

[0007] In a first aspect, the present invention provides a text-to-image optimization method based on AI feedback, comprising:

[0008] The preference dataset generation process involves: acquiring a dataset containing real user text prompts and corresponding generated images; selecting data with fewer than N text prompts from the dataset to obtain simple text prompts; rewriting and expanding the simple text prompts into complex text prompts using a large language model; and then using a generative model to generate corresponding simple and complex images based on the simple and complex text prompts respectively. The simple text prompts, simple images, complex text prompts, and complex images are then used as ranking pairs to form a preference dataset, serving as a substitute for annotations of human preferences.

[0009] Alignment process: The text-to-image model is optimized by aligning simple text prompts with complex images using a preference dataset in a supervised fine-tuning manner.

[0010] Furthermore, during the generation of the preference dataset, after obtaining simple text prompts, simple text prompts with a similarity of 0.7 or higher are filtered out to ensure the diversity of data distribution.

[0011] Furthermore, during the generation of the preference dataset, the complex text prompts are structured as follows: the original simple prompts, the extended descriptions generated by the large language model, and the terms specifying image quality.

[0012] Furthermore, in the alignment process, the optimization objective of the supervised fine-tuning is expressed as:

[0013]

[0014] in, Represents mathematical expectation, This refers to a variational autoencoder (VAE), used to encode complex text prompts. generated image Latent variables are obtained by compressing the pixel space into a low-dimensional latent space. , The first representing the potential space Hidden variables of the step, This indicates a frozen CLIP text encoder used for encoding simple text prompts. , Represents the standard Gaussian noise distribution Noise in the sampling.

[0015] Secondly, the present invention provides a text-to-image optimization device based on AI feedback, comprising:

[0016] The preference dataset generation module is used to acquire a dataset including real user text prompts and corresponding generated images. Then, it filters out data with fewer than N text prompts from the dataset to obtain simple text prompts. The simple text prompts are rewritten and expanded into complex text prompts using a large language model. Then, a generative model is used to generate corresponding simple and complex images based on the simple and complex text prompts, respectively. The simple text prompts, simple images, complex text prompts, and complex images are used as ranking pairs to form a preference dataset, which serves as a substitute for annotations of human preferences.

[0017] The alignment module is used to optimize the text-to-image model by aligning simple text prompts and complex images with a preference dataset using supervised fine-tuning.

[0018] Furthermore, in the preference dataset generation module, after obtaining simple text prompts, simple text prompts with a similarity of 0.7 or higher are filtered out to ensure the diversity of data distribution.

[0019] Furthermore, in the preference dataset generation module, the complex text prompts are structured as follows: the original simple prompts, the extended descriptions generated by the large language model, and the terms specifying image quality.

[0020] Furthermore, in the alignment module, the optimization objective of supervised fine-tuning is expressed as:

[0021]

[0022] in, Represents mathematical expectation, This refers to a variational autoencoder (VAE), used to encode complex text prompts. generated image Latent variables are obtained by compressing the pixel space into a low-dimensional latent space. , The first representing the potential space Hidden variables of the step, This indicates a frozen CLIP text encoder used for encoding simple text prompts. , Represents the standard Gaussian noise distribution Noise in the sampling.

[0023] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0024] The technical solution provided by this invention has at least the following technical effects:

[0025] To address the cost of human annotation, this invention utilizes a large language model to construct text ranking pairs of varying quality as an alternative to manual annotation, thus eliminating the high cost and reliance on human feedback. Regarding the overhead of fine-tuning, this invention directly fine-tunes on a preference dataset generated based on AI feedback, using the alignment of simple text prompts with complex images as optimization directions. The aim is to generate results that better align with human expectations. The training process does not require the participation of a reward model, thereby reducing overhead and achieving a more significant improvement over reward-based selection methods. Attached Figure Description

[0026] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0027] Figure 1 This is a schematic diagram of the overall process of learning based on human feedback in existing technologies.

[0028] Figure 2 This is a schematic diagram of the overall process of an embodiment of the present invention;

[0029] Figure 3 This is a flowchart illustrating the method in Embodiment 1 of the present invention;

[0030] Figure 4 The evaluation results of the image generated by the rewritten complex text prompt in Embodiment 1 of the present invention in terms of aesthetic performance and user preference;

[0031] Figure 5 This is a schematic diagram of the experimental results of the reward model trained on the AI ​​feedback preference dataset in Embodiment 1 of the present invention;

[0032] Figure 6 This is a schematic diagram of the device in Embodiment 2 of the present invention. Detailed Implementation

[0033] This invention provides a text-to-image optimization method, apparatus, and medium based on AI feedback. It utilizes AI feedback to approximate human preferences as an alternative to manual annotation. At the same time, it proposes a simple and effective alignment strategy to improve the text-to-image model, thereby solving the problems of high manual costs and fine-tuning overhead of human feedback optimization methods.

[0034] The overall concept of the technical solutions in the embodiments of the present invention is as follows:

[0035] This invention aims to address the high manual costs and fine-tuning overhead of human feedback optimization methods by introducing a large language model as an alternative to human preference annotation. It utilizes AI feedback to approximate human preferences, thereby eliminating the high cost and manual dependence on human feedback. The overall process is as follows: Figure 2 As shown, the overall process includes two steps: 1) approximation of preferences, using a large language model to approximate human preferences and generate ranked pairs; 2) alignment, aligning more refined images with simple text, hoping that the text-to-image model will generate higher quality images.

[0036] To address the cost of human annotation, this invention explores the potential of large language models and their alignment with human preferences, utilizing large language models to construct text ranking pairs of varying quality. This step is crucial for constructing a high-quality, effective preference dataset. This invention uses large language models to rewrite simple text, generating more detailed prompts as an alternative to manual annotation, thereby eliminating the high cost and manual dependence on human feedback. Furthermore, the method in this embodiment exhibits better consistency between text and images and is more aligned with human preferences.

[0037] To address the overhead of fine-tuning, this invention directly fine-tunes on a preference dataset generated based on AI feedback. Since human language habits tend towards concise descriptions, while generating desired images from text-to-image models often requires more detailed descriptions, this invention aims to bridge the gap between human daily expression habits and the usage scenarios of text-to-image models. This approach focuses on aligning simple text with complex images to optimize the model's output, ensuring it better matches human expectations. Compared to reward-based learning, this invention's training process does not require a reward model, thus reducing overhead and achieving a more significant improvement over reward-based selection methods.

[0038] Example 1

[0039] This embodiment provides a text-to-image optimization method based on AI feedback, such as... Figure 3 As shown, it includes:

[0040] S1. Preference Dataset Generation Process: A dataset including real user text prompts and corresponding generated images is obtained. Then, data with fewer than N text prompts (N can be 10) is filtered from the dataset to obtain simple text prompts. The data in this embodiment comes from the publicly available DiffusionDB dataset, which includes 1.8 million unique real user prompts and 14 million corresponding images generated by a stable diffusion model. Sampling real user prompts ensures that the model's generated results better match user expectations during inference. Filtering out simple text prompts with fewer than 10 words ensures consistency with real user scenarios. After obtaining the simple text prompts, those with a similarity greater than 0.7 are filtered out to ensure data diversity. Finally, this embodiment uses a graph-based selective annotation method to select representative, thematically diverse text prompts as the simple text prompt dataset.

[0041] Then, a large language model is used to rewrite the simple text prompts and expand them into complex text prompts. A generative model is then used to generate corresponding simple and complex images based on the simple and complex text prompts, respectively. The simple text prompts, simple images, complex text prompts, and complex images are used as ranking pairs to form a preference dataset. The semantic alignment and aesthetic quality of the complex images are greatly improved, thus naturally generating a set of ranking pairs that can be used as annotations for human preferences.

[0042] S2. Alignment process: Using a preference dataset, simple text prompts and complex images are aligned to optimize the text-to-image model.

[0043] Assuming to use We first present simple text prompts, then rewrite them using a large language model to obtain more sophisticated and complex prompts. Then, a text-to-image model is used to generate the corresponding image to obtain a simple text-generated image. Images generated from complex text The optimization objective of supervised fine-tuning can be expressed as:

[0044]

[0045] in, Represents mathematical expectation, This refers to a variational autoencoder (VAE), used to encode complex text prompts. generated image Latent variables are obtained by compressing the pixel space into a low-dimensional latent space. , The first representing the potential space Hidden variables of the step, This indicates a frozen CLIP text encoder used for encoding simple text prompts. , Represents the standard Gaussian noise distribution Noise in the sampling.

[0046] In a preferred implementation, to achieve better consistency between the image and text and to better align with human preferences, the preference dataset generation process employs cue word engineering. A carefully designed complex text cue structure template guides a large language model to rewrite simple text, expanding it into more detailed complex text cuees. The complex text cue is structured as follows: the original simple cue, the expanded description generated by the large language model, and terms specifying image quality.

[0047] Figure 4 This section presents the evaluation results of images generated from rewritten complex text prompts in terms of aesthetic performance and user preference. "Complex" represents the winning percentage for images generated from complex text prompts, "Simple" represents the winning percentage for images generated from simple text prompts, and "Tie" represents the tie rate. Evaluation is conducted from multiple perspectives: Aesthetic assesses aesthetic quality; ImageReward and HPSV2 are reward models trained on large datasets of human-annotated preferences to assess the consistency of human preferences; and Human Preference represents the results of a user survey.

[0048] This invention utilizes AI feedback to approximate human preferences as an alternative to manual annotation. It also proposes a simple and effective alignment strategy to improve text-to-image generation models. The comparison pairs constructed in this embodiment exhibit consistency with human preferences and achieve performance comparable to methods based on human feedback learning, demonstrating the effectiveness of the proposed method. Furthermore, the method eliminates the need for a reward model, further reducing training costs. This invention provides new insights into AI feedback learning for text-to-image generation, paving the way for more efficient and cost-effective text-to-image generation and other training processes.

[0049] To demonstrate the effectiveness of this invention's method of using AI feedback to approximate human preferences, a reward model was further trained on the obtained preference dataset. Two different fine-tuning methods for optimizing the text-to-image model were considered: ReFL and RAFT. These two methods represent successful practices of two paradigms: reinforcement learning and supervised fine-tuning. ReFL uses the reward model as real-time gradient feedback during training to maximize the expected reward value; RAFT uses the reward model to select high-scoring text-to-image pairs offline for supervised fine-tuning on the dataset. Figure 5 As shown, IR stands for ImageReward, which uses BLIP as the backbone network and is fine-tuned on a large amount of manually labeled preference data to obtain a reward model that can score corresponding images based on text; higher scores indicate better alignment with human expectations. S2C stands for S2CReward, which follows the training method of ImageReward, using BLIP as the backbone network and is trained on a preference dataset constructed from AI feedback. Experimental results show that the reward model trained on the AI ​​feedback preference dataset can achieve performance comparable to that trained on human preference data. Furthermore, joint optimization with the reward model trained on human preference data can bring further performance gains. This indicates that the AI ​​feedback preference dataset of this invention can serve as an effective alternative to the human preference dataset, thus providing an efficient data annotation method for image-text generation tasks.

[0050] Based on the same inventive concept, this application also provides an apparatus corresponding to the method in Embodiment 1, as detailed in Embodiment 2.

[0051] Example 2

[0052] This embodiment provides a text-to-image optimization device based on AI feedback, such as... Figure 6 As shown, it includes:

[0053] The preference dataset generation module is used to acquire a dataset including real user text prompts and corresponding generated images. Then, it filters out data with fewer than N text prompts from the dataset to obtain simple text prompts. The simple text prompts are rewritten and expanded into complex text prompts using a large language model. Then, a generative model is used to generate corresponding simple and complex images based on the simple and complex text prompts, respectively. The simple text prompts, simple images, complex text prompts, and complex images are used as ranking pairs to form a preference dataset, which serves as a substitute for annotations of human preferences.

[0054] The alignment module is used to optimize the text-to-image model by aligning simple text prompts and complex images with a preference dataset using supervised fine-tuning.

[0055] Furthermore, in the preference dataset generation module, after obtaining simple text prompts, simple text prompts with a similarity of 0.7 or higher are filtered out to ensure the diversity of data distribution.

[0056] Furthermore, in the preference dataset generation module, the complex text prompts are structured as follows: the original simple prompts, the extended descriptions generated by the large language model, and the terms specifying image quality.

[0057] Furthermore, in the alignment module, the optimization objective of supervised fine-tuning is expressed as:

[0058]

[0059] in, Represents mathematical expectation, This refers to a variational autoencoder (VAE), used to encode complex text prompts. generated image Latent variables are obtained by compressing the pixel space into a low-dimensional latent space. , The first representing the potential space Hidden variables of the step, This indicates a frozen CLIP text encoder used for encoding simple text prompts. , Represents the standard Gaussian noise distribution Noise in the sampling.

[0060] Since the apparatus described in Embodiment 2 of the present invention is an apparatus used to implement the method of Embodiment 1 of the present invention, those skilled in the art can understand the specific structure and variations of the apparatus based on the method described in Embodiment 1 of the present invention, and therefore will not be described again here. All apparatuses used in the method of Embodiment 1 of the present invention fall within the scope of protection of the present invention.

[0061] Based on the same inventive concept, this application provides a storage medium corresponding to Embodiment 1, as detailed in Embodiment 3.

[0062] Example 3

[0063] This embodiment provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it can implement any of the implementation methods in Embodiment 1.

[0064] To address the cost of human annotation, this invention utilizes a large language model to construct text ranking pairs of varying quality as an alternative to manual annotation, thus eliminating the high cost and reliance on human feedback. Regarding the overhead of fine-tuning, this invention directly fine-tunes on a preference dataset generated based on AI feedback, using the alignment of simple text prompts with complex images as optimization directions. The aim is to generate results that better align with human expectations. The training process does not require the participation of a reward model, thereby reducing overhead and achieving a more significant improvement over reward-based selection methods.

[0065] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0066] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0067] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The function specified in one or more boxes.

[0068] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0069] While specific embodiments of the present invention have been described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and not intended to limit the scope of the present invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A text-to-image optimization method based on AI feedback, characterized in that, include: The preference dataset generation process involves: acquiring a dataset containing real user text prompts and corresponding generated images; selecting data with fewer than N text prompts from the dataset to obtain simple text prompts; rewriting and expanding the simple text prompts into complex text prompts using a large language model; and then using a generative model to generate corresponding simple and complex images based on the simple and complex text prompts respectively. The simple text prompts, simple images, complex text prompts, and complex images are then used as ranking pairs to form a preference dataset, serving as a substitute for annotations of human preferences. Alignment process: The text-to-image model is optimized by aligning simple text prompts with complex images using a preference dataset in a supervised fine-tuning manner; During the generation of the preference dataset, the complex text prompts are structured as follows: the original simple prompts, the extended descriptions generated by the large language model, and the terms specifying image quality; During the alignment process, the optimization objective of the supervised fine-tuning is expressed as: in, Represents mathematical expectation, This refers to a variational autoencoder (VAE), used to encode complex text prompts. generated image Latent variables are obtained by compressing the pixel space into a low-dimensional latent space. , The first representing the potential space Hidden variables of the step, This indicates a frozen CLIP text encoder used for encoding simple text prompts. , Represents the standard Gaussian noise distribution Noise in the sampling.

2. The method according to claim 1, characterized in that: During the generation of the preference dataset, after obtaining simple text prompts, those with a similarity of 0.7 or higher are filtered out to ensure the diversity of data distribution.

3. A text generation and image optimization device based on AI feedback, characterized in that, include: The preference dataset generation module is used to acquire a dataset including real user text prompts and corresponding generated images. Then, it filters out data with fewer than N text prompts from the dataset to obtain simple text prompts. The simple text prompts are rewritten and expanded into complex text prompts using a large language model. Then, a generative model is used to generate corresponding simple and complex images based on the simple and complex text prompts, respectively. The simple text prompts, simple images, complex text prompts, and complex images are used as ranking pairs to form a preference dataset, which serves as a substitute for annotations of human preferences. The alignment module is used to optimize the text-to-image model by aligning simple text prompts and complex images with a preference dataset using supervised fine-tuning. In the preference dataset generation module, the complex text prompts are structured as follows: the original simple prompts, the extended descriptions generated by the large language model, and the terms specifying image quality; In the alignment module, the optimization objective of supervised fine-tuning is expressed as: in, Represents mathematical expectation, This refers to a variational autoencoder (VAE), used to encode complex text prompts. generated image Latent variables are obtained by compressing the pixel space into a low-dimensional latent space. , The first representing the potential space Hidden variables of the step, This indicates a frozen CLIP text encoder used for encoding simple text prompts. , Represents the standard Gaussian noise distribution Noise in the sampling.

4. The apparatus according to claim 3, characterized in that: In the preference dataset generation module, after obtaining simple text prompts, simple text prompts with a similarity of 0.7 or higher are filtered out to ensure the diversity of data distribution.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 2.

Images