Text generation image diffusion model enhancement method based on multi-target preference optimization

Abstract

The invention discloses a text generation image diffusion model enhancement method based on multi-target preference optimization. The method comprises the following steps: firstly, determining a plurality of reward models, and constructing a sample pair training set comprising positive and negative samples; and then, generating a loss weight of each sample pair in the sample pair training set, performing fine tuning training on the text map diffusion model by using the sample pair training set, and in the fine tuning training process, calculating a loss function value of each sample pair in combination with the loss weight of each sample pair until the training is completed, thereby obtaining an aligned text map diffusion model. According to the method provided by the invention, manual data annotation is not needed, the problems of preference inconsistency and over-optimization in a multi-reward scene are effectively solved, and the image quality, the text alignment capability and the multi-target optimization performance of the text-to-image generation model are remarkably improved. The method is superior to an existing optimization method under single-reward and multi-reward setting, shows higher generation quality and robustness, and can be seamlessly applied to various picture generation models.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence and computer vision and relates to a method for enhancing a text-generated image diffusion model, in particular to a method for enhancing a text-generated image diffusion model based on multi-objective preference optimization. Background Art

[0002] In recent years, significant progress has been made in text-to-image generation technology based on generative models. However, aligning the generated results with human preferences still faces many challenges. Existing technologies mostly rely on reinforcement learning from human feedback (RLHF) or optimization methods based on a single reward signal, which have the following major limitations:

[0003] 1) Strong data dependence: RLHF methods rely on a large amount of manually labeled preference data, which is not only costly but also difficult to scale to large-scale and diverse application scenarios.

[0004] 2) Insufficient modeling capabilities for fine-grained and diverse preferences: In practical applications, human preferences are often highly fine-grained and diverse. For example, artists may prefer the aesthetic quality of an image, while ordinary users may prioritize the details of an image. However, current mainstream methods focus on improving the overall quality of images and lack the ability to accurately model and respond to diverse user preferences. Furthermore, there is no effective technology that can effectively optimize diffusion models under multiple different types of preference objectives.

[0005] 3) Inconsistencies and conflicts between multiple reward signals: In multi-objective optimization scenarios, the various reward dimensions are not independent of each other and often influence or even conflict with each other. Simply weighting and combining multiple reward signals can easily lead to over-optimization of certain objectives, thereby causing overall performance imbalance. For example, the existing DPO (Direct Preference Optimization) method is based only on pairwise preference data and has difficulty in effectively utilizing multi-dimensional reward signals. While the IPO (Identity Preference Optimization) method introduces a dynamic target mechanism, it still has shortcomings in handling multi-reward conflicts.

[0006] Therefore, there is an urgent need for a diffusion model optimization method that does not require manual labeling, can effectively coordinate conflicts among multiple reward signals, and adapt to the needs of diversity preferences, so as to improve the quality of generated images and personalized alignment capabilities. Summary of the Invention

[0007] This paper proposes a method for enhancing the text-to-image diffusion model based on multi-objective preference optimization, aiming to address existing issues such as strong data dependency, severe multi-reward conflicts, and over-optimization. By introducing a multidimensional reward evaluation mechanism, a Pareto frontier sample screening strategy, and a weighted loss function design, this method achieves efficient optimization of the text-to-image diffusion model and alignment with human preferences. This method not only significantly improves the overall quality of generated images and text semantic alignment capabilities, but also exhibits excellent multi-objective optimization performance, making it suitable for a variety of text-to-image generation tasks and possessing broad application prospects and practical value.

[0008] The technical solution adopted by the present invention includes the following steps:

[0009] 1. A Text Generation Image Diffusion Model Enhancement Method Based on Multi-Objective Preference Optimization

[0010] 1) Determine multiple reward models and construct a sample pair training set including positive and negative samples based on the determined reward models;

[0011] 2) Generate loss weights for each sample pair in the sample pair training set, and use the sample pair training set to fine-tune the Vincent graph diffusion model. During the fine-tuning training process, calculate the loss function value of each sample pair based on the loss weights of each sample pair until the training is completed, and obtain the aligned Vincent graph diffusion model.

[0012] The specific embodiment of 1) is:

[0013] 1.1) Obtain a text prompt and use a pre-trained text-image diffusion model to generate multiple images corresponding to the current text prompt, thereby obtaining multiple text-image samples corresponding to the current text prompt;

[0014] 1.2) Determine multiple reward models and calculate multiple reward values ​​corresponding to each text-image sample in the current text prompt;

[0015] 1.3) Based on the multiple reward values ​​corresponding to each text-image sample, the multiple text-image samples corresponding to the current text prompt are divided into a positive sample set and a negative sample set corresponding to the current text prompt, thereby generating a set of positive and negative sample pairs for the current text prompt;

[0016] 1.4) Obtain other text prompts and repeat 1.1)-1.3) to generate positive and negative sample pairs corresponding to different text prompts. The positive and negative sample pairs corresponding to all text prompts constitute a sample pair training set including positive and negative samples.

[0017] In 1.3), a non-dominated sorting method is used to sort multiple text-image samples corresponding to the current text prompt according to multiple reward values ​​corresponding to each text-image sample to obtain a sorting result; images that are not dominated by any other samples in the sorting result are classified as positive samples, otherwise they are negative samples, thereby obtaining a positive sample set and a negative sample set corresponding to the current text prompt.

[0018] The reward model includes a model for evaluating the aesthetic quality, clarity, and artifact level of an image.

[0019] In 2), the loss weight of each sample pair in the sample pair training set is generated, including:

[0020] The loss weight of each sample pair is calculated based on the reward difference between the positive and negative samples in the sample pair. The calculation formula is as follows:

[0021] w=σ(γ(r(x + )-r(x - )))

[0022] Among them, σ(.) is the sigmoid function; x + and x - Represent the positive samples and negative samples in each sample pair respectively; r(.) represents the average reward value; γ() represents the scaling value used to adjust the reward difference scale; ω is the loss weight coefficient of the sample pair.

[0023] In 2), the calculation formula of the loss function value of each sample pair is as follows:

[0024]

[0025] in, is the loss function, is the mathematical expectation of the loss function for all samples, ω ij is the loss weight of the sample pair consisting of the i-th positive sample and the j-th negative sample, σ() is the sigmoid function; β is a hyperparameter, T represents the number of iterations of the diffusion model, π θ (|) is the currently optimized diffusion model strategy, π ref (|) is the pre-training diffusion model strategy, and Respectively represent the positive samples of the optimization process at time step t-1 and time step t, and They represent the negative samples corresponding to the optimization process at time step t-1 and time step t respectively.

[0026] 2. A computer device,

[0027] The computer device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the steps of the text generation image diffusion model enhancement method based on multi-objective preference optimization are implemented.

[0028] 3. A computer-readable storage medium

[0029] The medium stores a computer program, which, when executed by a processor, implements the steps of the method for enhancing a text-generated image diffusion model based on multi-objective preference optimization.

[0030] 4. A computer program product

[0031] The product includes a computer program / instruction, which, when executed by a processor, implements the steps of the text generation image diffusion model enhancement method based on multi-objective preference optimization.

[0032] The present invention has the following beneficial effects:

[0033] 1) This method eliminates the need for manual labeling of preference data. Instead, it uses a pre-trained text graph model to automatically generate samples and utilizes a reward model to perform multi-dimensional preference evaluation on the samples, significantly reducing the cost of data acquisition and labeling and demonstrating good scalability and practicality.

[0034] 2) This paper introduces a Pareto front sample screening strategy, which can effectively balance different optimization objectives (such as image quality, aesthetic effect, consistency between text and image, etc.) in multi-objective evaluation scenarios, avoiding the performance imbalance caused by excessive bias towards a single objective in traditional methods;

[0035] 3) This method combines Direct Preference Optimization (DPO) loss with dynamically calculated loss weights during training, achieving faster convergence than traditional methods and generating higher-quality image samples with better preference alignment.

[0036] 4) The present invention has good versatility and compatibility, and can be flexibly adapted to a variety of mainstream text-to-image models (such as Stable Diffusion XL and Stable Diffusion 3) and various types of reward models (such as CLIP-based and BLIP-based models, or rule-based customized reward functions), making it suitable for diverse application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Flow chart of the method of the present invention.

[0038] Figure 2Comparison of images generated by the stablediffusion-1.5 model before and after fine-tuning training Figure 1 , where (a) is the image generated before training, and (b) is the image generated after fine-tuning training.

[0039] Figure 3 Comparison of images generated by the stablediffusion-1.5 model before and after fine-tuning training Figure 2 , where (a) is the image generated before training, and (b) is the image generated after fine-tuning training.

[0040] Figure 4 Comparison of images generated by the stablediffusion-1.5 model before and after fine-tuning training Figure 3 , where (a) is the image generated before training, and (b) is the image generated after fine-tuning training.

[0041] Figure 5 Comparison of images generated by the stablediffusion-xl model before and after fine-tuning training Figure 1 , where (a) is the image generated before training, and (b) is the image generated after fine-tuning training.

[0042] Figure 6 Comparison of images generated by the stablediffusion-xl model before and after fine-tuning training Figure 2 , where (a) is the image generated before training, and (b) is the image generated after fine-tuning training.

[0043] Figure 7 Comparison of images generated by the stablediffusion-xl model before and after fine-tuning training Figure 3 , where (a) is the image generated before training, and (b) is the image generated after fine-tuning training. DETAILED DESCRIPTION

[0044] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0045] like Figure 1 As shown, the method proposed by the present invention comprises the following steps:

[0046] 1) Determine multiple reward models and construct a sample pair training set including positive and negative samples based on the determined reward models;

[0047] 1) Specifically:

[0048] 1.1) Prepare a text dataset for training. The text prompts should be text describing the target image content and should cover as much content diversity as possible. After obtaining the text prompt, use the pre-trained text-image diffusion model to generate multiple images corresponding to the current text prompt. This yields multiple text-image samples corresponding to the current text prompt. Each text prompt and its corresponding image form a text-image sample. The text-image diffusion model is both a diffusion model and a text-image model. Typically, eight images are generated for each text prompt.

[0049] 1.2) Determine multiple reward models and calculate multiple reward values ​​corresponding to each text-image sample in the current text prompt; the reward model includes models for evaluating the aesthetic quality, clarity, degree of artifacts, etc. of the image, which can be set according to needs. For example, a model for evaluating the aesthetic quality of an image, or a model for evaluating the consistency between the generated image and the input text prompt (Clip model). Specifically, each image sample will obtain a reward vector output by each reward model on different evaluation dimensions, with a dimension of m, where m represents the number of reward models used. Ultimately, each text prompt corresponds to a reward matrix of size 8×m, which is used to comprehensively characterize the performance of the candidate image set generated by the text on each evaluation dimension.

[0050] 1.3) Based on the multiple reward values ​​corresponding to each text-image sample, the Pareto front selection method is used to divide the multiple text-image samples corresponding to the current text prompt into a positive sample set and a negative sample set corresponding to the current text prompt, thereby generating a positive and negative sample pair set for the current text prompt; based on the multiple reward values ​​corresponding to each text-image sample, the non-dominated sorting method is used to sort the multiple text-image samples corresponding to the current text prompt to obtain a sorting result; the images in the sorting result that are not dominated by any other samples are classified as positive samples, and vice versa, they are negative samples; the images that do not dominate any other samples are classified as negative samples, thereby obtaining a positive sample set and a negative sample set corresponding to the current text prompt. Specifically, there are M reward models, let R i Represents the reward value measured by the i-th reward model, defined as: if and only if for all reward models i = 1, ..., M, there is R i (x1)≥R i (x2), image x1 is said to dominate image x2. Under this definition, the non-dominated set of samples (i.e., samples that are not simultaneously surpassed by other samples in all dimensions) constitutes the upper Pareto frontier, while the set of samples dominated by other samples constitutes the lower Pareto frontier. The filtered non-dominated set is considered the positive sample set, and the dominated set is considered the negative sample set.

[0051] 1.4) Obtain other text prompts and repeat 1.1)-1.3) to generate positive and negative sample pairs corresponding to different text prompts. The positive and negative sample pairs corresponding to all text prompts constitute a sample pair training set including positive and negative samples.

[0052] 2) Generate the loss weight of each sample pair in the sample pair training set including positive and negative samples, and use the sample pair training set including positive and negative samples to fine-tune the current text-to-image diffusion model. During the fine-tuning training process, combine the loss weight of each sample pair and the direct preference optimization loss to calculate the loss function value of each sample pair until the training is completed, and obtain the aligned text-to-image diffusion model.

[0053] Generate the loss weight of each sample pair in the sample pair training set including positive and negative samples, including:

[0054] The loss weight of each sample pair is calculated based on the reward difference between the positive and negative samples in the sample pair. The reward difference is the sum and average of all the reward differences of the current positive and negative samples. The calculation formula is as follows:

[0055] w=σ(γ(r(x + )-r(x - )))

[0056] Among them, σ(.) is the sigmoid function; x + and x - where represents the positive and negative samples in each sample pair; r(.) represents the average reward; γ() represents the scaling value used to adjust the reward difference; and ω is the loss weight coefficient for the sample pair. This method dynamically adjusts the loss weight based on the reward difference between positive and negative samples. The weight is proportional to the sample reward value, which allows the model to focus more on positive and negative samples with large differences, accelerating model convergence.

[0057] The calculation formula of the loss function value for each sample pair is as follows:

[0058]

[0059] in, is the loss function, is the mathematical expectation of the loss function for all samples, ω ij is the loss weight of the sample pair consisting of the i-th positive sample and the j-th negative sample, σ() is the sigmoid function; β is a hyperparameter, T represents the number of iterations of the diffusion model, π θ (|) is the currently optimized diffusion model strategy, π ref (|) is the pre-training diffusion model strategy, and Respectively represent the positive samples of the optimization process at time step t-1 and time step t, and They represent the negative samples corresponding to the optimization process at time step t-1 and time step t respectively. Specific embodiment:

[0061] The present invention conducted experiments based on two models, Stable Diffusion 1.5 and Stable Diffusion XL. The text dataset used consists of 12,000 high-quality texts after screening. During the training and evaluation process, three reward models were introduced to evaluate the aesthetic quality of the image, the degree of alignment between the text and the image, and the plausibility of the image. These reward models are used together to construct a multi-dimensional reward vector for the training samples, guiding the construction of positive and negative sample pairs and the optimization process of the model. The original model image and the trained image are shown in Figure 2. Figures 2 to 7 shown. Figures 2 to 4 Comparison of images generated by the stablediffusion-1.5 model before and after fine-tuning training. Figures 5 to 7 Comparison of images generated by the stablediffusion-xl model before and after fine-tuning training.

[0062] Comparing the images generated by the original models of the above embodiments with the images generated after training, it can be seen that the method of the present invention can improve the quality of images in many aspects. The method of the present invention is simple, easy to implement, and can be widely applied to various generative models.

[0063] The above specific embodiments are used to illustrate the present invention rather than to limit the present invention. Any modifications and changes made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A text generation image diffusion model enhancement method based on multi-objective preference optimization, characterized in that: The following steps are involved: 1) Determine multiple reward models and construct a sample pair training set including positive and negative samples based on the determined reward models; 2) Generate loss weights for each sample pair in the sample pair training set, and use the sample pair training set to fine-tune the Vincent graph diffusion model. During the fine-tuning training process, calculate the loss function value of each sample pair based on the loss weights of each sample pair until the training is completed, and obtain the aligned Vincent graph diffusion model.

2. The text generation image diffusion model enhancement method based on multi-objective preference optimization according to claim 1 is characterized in that: The specific embodiment of 1) is: 1) Obtain a text prompt and use the pre-trained text-image diffusion model to generate multiple images corresponding to the current text prompt, thereby obtaining multiple text-image samples corresponding to the current text prompt; 1).2 Determine multiple reward models and calculate multiple reward values ​​corresponding to each text-image sample in the current text prompt; 1).3 Based on the multiple reward values ​​corresponding to each text-image sample, the multiple text-image samples corresponding to the current text prompt are divided into a positive sample set and a negative sample set corresponding to the current text prompt, thereby generating a positive and negative sample pair set for the current text prompt; 1).4 Obtain other text prompts and repeat 1.1)-1.3) to generate positive and negative sample pairs corresponding to different text prompts. The positive and negative sample pairs corresponding to all text prompts constitute a sample pair training set including positive and negative samples.

3. The text generation image diffusion model enhancement method based on multi-objective preference optimization according to claim 2 is characterized in that: In 1.3), a non-dominated sorting method is used to sort multiple text-image samples corresponding to the current text prompt according to multiple reward values ​​corresponding to each text-image sample to obtain a sorting result; images that are not dominated by any other samples in the sorting result are classified as positive samples, otherwise they are negative samples, thereby obtaining a positive sample set and a negative sample set corresponding to the current text prompt.

4. The text generation image diffusion model enhancement method based on multi-objective preference optimization according to claim 1 is characterized in that: The reward model includes a model for evaluating the aesthetic quality, clarity, and artifact level of an image.

5. The text generation image diffusion model enhancement method based on multi-objective preference optimization according to claim 1 is characterized in that: In 2), the loss weight of each sample pair in the sample pair training set is generated, including: The loss weight of each sample pair is calculated based on the reward difference between the positive and negative samples in the sample pair. The calculation formula is as follows: w=σ(γ(r(x + )-r(x - ))) Among them, σ(.) is the sigmoid function; x + and x - Represent the positive samples and negative samples in each sample pair respectively; r(.) represents the average reward value; γ() represents the scaling value used to adjust the reward difference scale; ω is the loss weight coefficient of the sample pair.

6. The text generation image diffusion model enhancement method based on multi-objective preference optimization according to claim 1 is characterized in that: In 2), the calculation formula of the loss function value of each sample pair is as follows: in, is the loss function, is the mathematical expectation of the loss function for all samples, ω ij is the loss weight of the sample pair consisting of the i-th positive sample and the j-th negative sample, σ() is the sigmoid function; β is a hyperparameter, T represents the number of iterations of the diffusion model, π θ (|) is the currently optimized diffusion model strategy, π ref (|) is the pre-training diffusion model strategy, and Respectively represent the positive samples of the optimization process at time step t-1 and time step t, and They represent the negative samples corresponding to the optimization process at time step t-1 and time step t respectively.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method for enhancing a text-generated image diffusion model based on multi-objective preference optimization as described in any one of claims 1 to 6 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for enhancing a text-generated image diffusion model based on multi-objective preference optimization as described in any one of claims 1 to 6 are implemented.

9. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the method for enhancing a text generation image diffusion model based on multi-objective preference optimization as described in any one of claims 1 to 6 are implemented.

Images