A diffusion model preference alignment method and device based on self-evolution agent and dynamic rule evolution

Abstract

The application discloses a diffusion model preference alignment method and device based on self-evolution intelligent agents and dynamic rule evolution, the method is judged by the performer according to the rules of the basic rule book or the current rule book to the sample image pair and the judgment basis is obtained, the evaluator provided by the application carries out blind evaluation, and through comparison, the content not covered in the rules cited by the performer is obtained to form the review opinion, the judge makes the final judgment based on the judgment of the performer and the review opinion of the evaluator, if the conclusions of the performer and the evaluator conflict, a new rule is formed to continue optimizing the judgment of the performer, and the judge generates an enhanced text prompt and labels the winning image and the losing image in the sample image pair in each iteration process, which is used for training of the subsequent diffusion model, so that the static evaluation system is broken through, and continuous improvement of the diffusion model preference alignment is realized.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a method and apparatus for aligning preferences based on a diffusion model of self-evolving intelligent agents and dynamic rule evolution. Background Technology

[0002] In recent years, text-to-image diffusion models have made significant progress in image generation. However, their pre-training method, which involves fitting data distributions, inherently creates a gap between the generated results and human preferences. To bridge this gap, drawing on the success of reinforcement learning based on human feedback in language models, current mainstream methods train reward models as proxies of human preferences and use the output of these reward models to guide the post-training optimization of the diffusion model. However, these methods are generally limited by the fixed nature of their evaluation criteria.

[0003] In offline methods, preference labels are generated by human annotators or pre-trained reward models, and the granularity of the annotation directly determines the upper limit of the training data quality. In online methods, although reward models can provide real-time feedback, their evaluation criteria remain fixed after pre-training, and their discriminative ability directly limits the size of the optimization space. Some recent methods have attempted to use visual language models for preference annotation, but they still evaluate each sample independently, failing to accumulate evaluation knowledge across different samples. This cross-sample knowledge accumulation is precisely necessary because only by abstracting recurring patterns from diverse judgments can the evaluation criteria gradually evolve from coarse-grained basic constraints to fine-grained attributes that match the model's capabilities. Regardless of the form of feedback signal used, what is ultimately used for training is still implicit scalar scores or win / loss labels. The model can only infer the direction of optimization from these, but cannot know which specific visual attributes should be improved.

[0004] The paper “Wallace B, Dang M, Rafailov R, et al. Diffusion model alignment using direct preference optimization[C] / / Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition. 2024: 8228-8238.” discloses a preference alignment method that applies direct preference optimization to a diffusion model. Diffusion-DPO extends the objective function of the original DPO from the language model to the denoising process of the diffusion model, enabling the diffusion model to learn directly from the preference contrast data. The specific steps are as follows: (1) Collect a dataset of preference image pairs, each containing a text cue. A winning image And a failed image The winning image is the one judged as better by human annotators; (2) the optimization objective of DPO is elevated to the full reverse path of the diffusion model. The above is simplified into a single-time-step training formula using the lower bound of evidence; (3) In each training step, the winning and losing images are noise-added to random time steps respectively. , calculate the difference in denoising error between the current model and the reference model on the two images, and construct the gradient signal for preference optimization; (4) train the diffusion model through multiple rounds of iteration to learn to generate images that are more in line with human preferences. Diffusion-DPO does not require explicit training of the reward model, which simplifies the training process and achieves better results than the baseline model in human evaluation. However, the preference data of this method comes from the pre-labeled win and lose labels in a fixed dataset. The labeling granularity of these labels is determined before training. The model adapts to the same fixed evaluation criteria throughout the training cycle and lacks a mechanism for accumulating and evolving evaluation knowledge.

[0005] Therefore, it is necessary to design a method that can dynamically evolve the evaluation criteria during training and inject the criteria into the training process with explicit semantic information, thereby breaking through the inherent upper bound of the static evaluation system and achieving continuous improvement of the preference alignment of the diffusion model. Summary of the Invention

[0006] This invention provides a method for aligning preferences in a diffusion model based on a self-evolving intelligent agent and dynamic rule evolution. This method can break through static evaluation systems and achieve continuous improvement in the preference alignment of the diffusion model.

[0007] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention involves an executor judging sampled image pairs based on rules in a basic or current rulebook and obtaining the judgment criteria. An evaluator provided by this invention performs blind evaluation and, through comparison, obtains content not covered by the rules cited by the executor, thus forming a review opinion. A referee makes a final judgment based on the executor's judgment and the evaluator's review opinion. If the conclusions of the executor and the evaluator conflict, new rules are formed to further optimize the executor's judgment. In each iteration, the referee generates enhanced text prompts and labels the winning and losing images in the sampled image pairs for subsequent training of the diffusion model, thereby breaking through the static evaluation system and achieving continuous improvement of the diffusion model's preference alignment. Attached Figure Description

[0008] Figure 1 A flowchart of a preference alignment method and apparatus based on a diffusion model of self-evolving intelligent agents and dynamic rule evolution, provided for a specific embodiment of the present invention; Figure 2 The images show the effect diagrams of various models provided for specific embodiments of the present invention. Detailed Implementation

[0009] This invention provides a diffusion model alignment method based on self-evolving preference optimization, called SEPO (Self-Evolving Preference Optimization). This method is a novel framework for achieving diffusion model preference alignment through dynamic rule refinement. Existing preference optimization methods all rely on fixed evaluation criteria before training. Whether using human-annotated fixed datasets or pre-trained reward models, their evaluation capabilities do not improve during training, resulting in an inherent upper bound on the model's alignment performance. This invention constructs a multi-role intelligent agent architecture comprising four roles: executor, evaluator, referee, and rule compressor, maintaining a dynamic rule manual stored in natural language. This allows the evaluation criteria to be continuously refined and expanded as the model's capabilities improve during training. Furthermore, this invention proposes a semantic injection mechanism, transforming the evolved rules into enhanced conditional text describing the visual quality of the winning image and injecting it into DPO training. This enables the model to directly learn the specific visual attributes to be optimized, rather than inferring the optimization direction solely from implicit win / loss labels. This invention can achieve continuous improvement in preference alignment of text-to-image diffusion models, surpassing existing methods on multiple evaluation metrics.

[0010] The method proposed in this invention achieves continuous improvement in the preference alignment of the diffusion model by constructing a complete training framework that includes a dynamic rule manual, a multi-role intelligent agent architecture, and a semantic injection mechanism. Specifically, SEPO maintains a dynamic rule manual stored in natural language. Through the collaboration of four roles—executor, evaluator, referee, and rule compressor—it continuously discovers evaluation blind spots in the rule manual during training and refines the visual differences in these blind spots into new rules, thereby continuously expanding the coverage and granularity of the evaluation criteria. Simultaneously, this invention proposes a semantic injection mechanism that transforms the evolved rules into enhanced conditional text describing the visual quality of the winning image. This text is concatenated with the original text prompts and injected into the DPO training, enabling the model to directly learn the specific visual attributes that should be optimized, rather than inferring the optimization direction solely from implicit win / loss labels. The design of each component is described below.

[0011] A specific embodiment of the present invention provides a preference alignment method based on a diffusion model of self-evolving intelligent agents and dynamic rule evolution, comprising: (1) Construct a multi-role intelligent agent, which includes an executor, an evaluator and a referee. Based on the set basic rule manual or the current rule manual, sampled image pairs and original text prompts, the executor obtains the first preference conclusion and the first judgment basis.

[0012] In one specific embodiment, the present invention provides a current rule manual. , For the nth natural language rule, each rule describes a criterion for evaluating image quality. During the initialization phase, the basic rule manual... It contains only six natural language rules, covering basic dimensions such as object existence (e.g., "all objects explicitly mentioned in the prompt must appear in the image"), quantity consistency (e.g., "when a specific quantity is mentioned, the count must be accurate"), attribute accuracy (e.g., "explicitly described attributes such as color and size must match the prompt"), anatomical correctness, physical plausibility, and safety. These seed rules only cover the most basic text-image alignment requirements, leaving more granular aesthetic dimensions (e.g., composition, color harmony, lighting and atmosphere) to be automatically discovered and refined by the system during training.

[0013] In one specific embodiment, each natural language rule includes the following information items: rule number (index of the rule under the current category), category, rule content, source information, iteration steps at creation, total number of times it is cited by the executor, number of times it is judged to be adopted by the system after citation, and iteration steps of the last citation, rule status, and validity rate. The rule status indicates whether the natural language rule is in a citationable state; when the natural language rule is not deleted, deactivated, or merged in the rule compression process, the rule status is valid; when the natural language rule is no longer available for executor citation due to redundancy, inefficiency, expiration, or merging, the rule status is invalid.

[0014] The specific embodiments of the present invention provide the efficiency ,in This represents the total number of times the rule is referenced by those enforcing it. This represents the number of times a rule is adopted by the system after being cited (confirmed by the referee after arbitration, and ultimately adopted). Efficiency reflects the quality of the rule's judgment. High efficiency indicates that the rule accurately captures preference features, and its guided judgments are frequently adopted by the system; low efficiency indicates that the rule may have biases or be too broad in its expression, requiring pruning in subsequent compression. This efficiency is used for subsequent rule compression: when a rule's citation count reaches a preset usage threshold and its efficiency is lower than a preset efficiency threshold, the rule is pruned as an inefficient rule; rules that have not been cited for a long time and whose age exceeds a preset age threshold can also be pruned. In this example, the preset usage threshold is 5 times, the preset efficiency threshold is 0.4, the preset age threshold is 200 iterations, and protection is set for seed rules and recently generated rules.

[0015] The multi-role intelligent agent provided in this specific embodiment of the invention also includes a rule compressor S, wherein the executor, evaluator, and referee collaboratively complete preference labeling and rule evolution, and the rule compressor is responsible for optimizing the rule manual during the compression phase. All four roles are implemented using a visual language model; in one specific embodiment, this visual language model is Gemini 3.5 Flash.

[0016] The executor provided in the specific embodiments of the present invention Preference decisions are made strictly according to the rulebook. In each iteration step, the current diffusion model is based on the original text prompts. Generate a pair of images That is, sampled image pairs, which the executor determines according to the current rule manual. Output preference judgment and judgment criteria: , among which, among which For the preference judgment result, express Better express Better This indicates that the existing rules cannot distinguish quality differences, which is the first preference conclusion. This is a subset of rules referenced by the executor. When making judgments, the executor must explicitly refer to the rule number on which they are based, and when multiple rules apply, the newer rule should be prioritized, as it typically supplements the blind spots of earlier rules and represents a more refined judgment standard. When the executor cannot distinguish between two images based on existing rules, the judgment result is "indistinguishable," which precisely exposes the blind spots in the rule manual and provides a signal for subsequent rule evolution.

[0017] In a specific embodiment of the present invention, the executor prompts enable the executing agent to perform preference judgments on sampled image pairs, obtaining the winning and losing images provided by the executing agent, i.e., the first preference conclusion, and the first judgment basis, which includes citation rules and judgment reasons. The executor prompt provided in a specific embodiment of this invention is as follows: "You are the rule executor and can only evaluate images according to the given rules. The text prompt to be evaluated is [Text Prompt], and the current rule manual is [Rule List]. Please check image A and image B, and apply the above rules to make a preference judgment; when multiple rules are applicable, the rules that were recently formed or have a higher iteration number are given priority, because these rules are derived from experience and may capture details missed by older rules; only when the recent rules are not applicable will you fall back to the initial R of the seed rule; if the rules cannot distinguish between the two images, a conclusion that they cannot be distinguished is given. When making a judgment, you must cite the specific rule number and explain how each cited rule affects the conclusion." Here, [Text Prompt] refers to the original text generation prompt corresponding to the current sample, [Rule List] refers to the set of natural language rules provided to the executor in the current rule manual, and image A and image B refer to the pair of images to be compared in this round.

[0018] (2) Based on the obtained sampled image pairs and original text prompts, the second preference conclusion and the second judgment basis are obtained through blind evaluation by the evaluator. The second preference conclusion and the second judgment basis are compared with the first preference conclusion and the first judgment basis. The review opinion is constructed through the conclusion conflict between the first and second preference conclusions and the basis not covered by the first judgment basis.

[0019] Specifically, the evaluator provided in the specific embodiments of the present invention This approach provides independent assessments across a broader range of visual quality dimensions to uncover visual quality differences that the executor might have overlooked. The evaluator's work is divided into two phases: the first phase is a blind evaluation, where the evaluator independently assesses the visual quality differences between image pairs without knowing the executor's judgments, analyzing aspects such as composition, lighting, texture detail, color harmony, and overall aesthetics; the second phase is a review phase, where the evaluator compares their independent findings with the executor's judgments and reasoning to determine whether the rules cited by the executor cover key visual quality differences. This two-phase design avoids the evaluator being anchored to the executor's judgments. The evaluator's output consists of: independent findings, which are blind observations formed by the evaluator without referring to the executor's conclusions (the evaluator does not look at the R, but evaluates based on agent capabilities); the executor's judgments and reasoning, which refer to their preferred conclusions, the rules they rely on, and the reasons they give based on those rules; and a review opinion, which checks each key difference found in the blind evaluation to see if it is covered by the rules and reasons cited by the executor, and if not, or if the conclusions conflict. Key visual differences include variations in composition, lighting, texture, color, aesthetics, artifacts, or distortion that can significantly influence preference decisions.

[0020] In one specific embodiment, this embodiment enables the evaluator agent to make preference judgments on sampled image pairs through evaluator prompts, namely, a second preference conclusion, a second judgment basis, and a review opinion. The evaluator prompts include requiring the evaluator to find visual quality differences in the sampled image pairs based on multi-faceted analysis, including composition layout, lighting atmosphere, detail texture, color coordination, overall aesthetics, artifacts, or distortion. Based on the visual quality differences, a second preference conclusion is formed. Then, a first preference conclusion and a first judgment basis are obtained. The first judgment basis includes citation rules and judgment reasons. The evaluator is required to review whether the citation rules of the first judgment basis can capture visual quality differences and find omissions or errors. The found omissions or errors and the conflicting conclusions are used to form a review opinion.

[0021] Specifically, the evaluator prompt provided in this embodiment is: "You are an independent visual evaluation expert, responsible for reviewing the executor's judgment. The text prompt to be evaluated is [Text Prompt]. In the first stage, please do not read the executor's judgment, but only observe Image A and Image B, and point out the most significant visual quality differences. You can analyze them from aspects such as composition, lighting atmosphere, detail texture, color coordination, overall aesthetics, artifacts or distortion, and form an independent preference. In the second stage, read the executor's preference conclusion, cited rules, and judgment reasons, and review whether their rules capture the key differences you found in the first stage, whether important visual aspects have been omitted, and whether the conclusion is fully supported by the rules; if you disagree, please explain the omissions or errors and their impact." Here, [Text Prompt] refers to the original text generation prompt corresponding to the current sample; Image A and Image B refer to the pair of images to be compared in this round.

[0022] The reason why this invention provides an independent evaluator role instead of allowing the executor to self-censor is that the executor is confined to reasoning within the framework of the rulebook, while the evaluator examines the samples from an open perspective, thus complementing each other. Reviewing one's own judgment using the same model can easily lead to confirmation bias, while an independent role can provide a more objective evaluation. The review opinion κ is used to explain what visual differences the executor has overlooked, why this overlooking affects preference determination, which judgment dimensions are lacking in the current rulebook, whether the executor's judgment of sample preferences is correct, and if so, what has been overlooked.

[0023] (3) In a specific embodiment of the present invention, based on the first preference conclusion, the first judgment basis and the review opinion, the referee provides the final preference judgment and the judgment reason, generates new rules based on the final preference judgment and adds them to the basic rule manual, as well as enhanced text prompts describing the winning image, and marks the sampled image as the winning image and the losing image.

[0024] Specifically, the referee provided in the specific embodiments of the present invention Integrating the opinions of implementers and evaluators, it is responsible for generating enhancement conditions and, when disagreements arise between the two, for arbitrating and refining the rules. ,in For the final preference determination, For new rules refined from disagreements (when) hour ), To enhance text suggestions. When That is, when the evaluator agrees with the executor's judgment, the adjudicator directly adopts the executor's judgment. ), and generate enhancement conditions based on the judgment criteria. Skip the arbitration and rule refinement steps. When When the implementer and the evaluator disagree, the referee intervenes and performs three tasks: (1) Make a final preference judgment by combining the analyses of both parties. The referee does not simply adopt the conclusion of either party, but makes an independent judgment based on a comprehensive analysis of the two images; (2) Evaluate each potential blind spot proposed by the evaluator, selectively adopt suggestions that meet the requirements of universality and non-redundancy, and abstract them into new rules. Update the rule manual At the same time, it rejects suggestions that are too specific, already covered by existing rules, or lack sufficient basis; (3) it generates enhanced text prompts. The new rules must meet the requirement of generality, meaning that the wording of the rules should be independent of specific samples and applicable to similar situations encountered in the future. For example, it should not generate a specific description such as "Image A has better lighting and shadow effects," but should abstract it to a general principle such as "Light and shadow should have a sense of layering and volume, with natural interaction between light sources and shadows." The enhanced text prompt c' describes the specific visual qualities of the winning image; its content should be a concrete visual description rather than an abstract quality evaluation. For example, it should not generate "High quality, good lighting and shadows," but should generate "Warm evening sunlight shines through the window, soft volumetric lighting is accompanied by natural shadows, and delicate hair texture." In specific implementation, the referee receives the executor's preference conclusion, cited rules, and reasoning basis (review opinion k), as well as the evaluator's blind review results, whether they agree with the executor's conclusion and review opinion; based on this, the referee gives a final preference judgment, whether to confirm the executor's original judgment, the reason for the final judgment, the rule refinement result, and the enhanced conditions. The referee generates new rules only when the evaluator disagrees; the new rules must be general, enforceable and not duplicate existing rules, potential blind spots come from rule gaps pointed out in the review comments, and enhancement conditions are generated from the specific visual advantages of the winning image.

[0025] In one specific embodiment, the referee's prompts enable the referee agent to provide a final preference determination and the reasoning behind it. The referee's prompts include a request for the referee to review the sampled image pairs, the first preference conclusion, the first judgment basis, the second preference conclusion, and the second judgment basis, and to make a final preference determination and the reasoning behind it. When the second preference conclusion of the evaluator and the executor conflicts with the first preference conclusion, a new rule is generated. The requirements for the new rule include that it does not repeat the existing rules, is applicable to multiple types of images, and an enhanced text prompt is generated for the winning image based on the final preference determination.

[0026] Specifically, the referee prompt provided in this embodiment is: "You are the final referee, responsible for resolving disagreements between the executor and the evaluator and refining the rules. The text to be evaluated is prompted as '[Text Prompt]'. Please review Image A, Image B, the executor's rule-based judgment, and the evaluator's independent observation simultaneously. First, synthesize the opinions of both parties to make a final preference determination, and explain the specific reasons why the winning image is superior to the other image, while also determining whether the executor's original conclusion can be confirmed. Second, only propose new rules when the evaluator disagrees with the executor; if both parties agree, the current rules are considered sufficient, and no new rules are generated; if both parties disagree, it indicates that there are blind spots in the rule manual." Then, abstract this blind spot into rules that can be used for future samples. New rules must be applicable to multiple image types, clear enough to guide subsequent judgments, and must not duplicate existing rules. Avoid specific examples like "Image A has better lighting," and instead use general principles such as "Lighting should have a sense of depth and volume, and exhibit natural light source and shadow interaction." Third, based on the final judgment criteria, generate enhancement conditions for the winning image. Enhancement conditions should use specific visual descriptions, not abstract evaluations like "higher quality" or "better looking"; for example, write "Warm evening sunlight streams in through the window, soft volumetric light accompanies natural shadows, and the hair texture is delicate." Among them, [Text Prompt] refers to the original text prompt corresponding to the current sample; Image A and Image B refer to a pair of images that the judge needs to ultimately determine the superiority or inferiority of.

[0027] Specific embodiments of the present invention also provide rule compression. As training progresses, the rule manual will continuously accumulate new rules, which may lead to problems such as redundant rules and inefficient rules. Specific embodiments of the present invention compress rules every... Rule compression is performed once per iteration step, and the compression process is divided into two stages.

[0028] The first stage is automatic cleanup based on statistics. This includes cleaning up citation counts. But effective The rules were pruned because, although frequently referenced, their guiding judgments were often rejected by the system, indicating a bias in the rules themselves. For rules created more than [time period missing], [further details needed]. Rules with fewer than one iteration step and a reference count below the threshold are also pruned, as these rules have not been referenced for a long time and may have been replaced by more precise new rules. Simultaneously, quotas are dynamically allocated based on the cumulative usage frequency of rules in each category to prevent the overexpansion of one category of rules from crowding out other categories. The total capacity limit of the rule manual is set to [value missing]. Seed rules and recently created rules are protected during the compression process.

[0029] The second stage is semantic compression. After statistical cleaning, the compression agent, based on the prompts, utilizes its semantic recognition capabilities to perform semantic analysis on the remaining rules by category. This identifies semantic redundancy (rules with different expressions but similar meanings), merging opportunities (fine-grained rules that can be abstracted into more general principles), and ambiguous rules that need clarification or removal. Executable operations include removing redundant rules, modifying rule expressions, and merging multiple rules into a single, more abstract rule.

[0030] In one specific embodiment, the present invention utilizes a rule compressor agent to compress rules through a rule compressor prompt. The rule compressor prompt provided in this specific embodiment is: "You are a rule compressor, responsible for analyzing rules by category to optimize the rule manual. The current category being processed is [Rule Category], the current iteration step is [Current Iteration Step], the number of rules in this category is [Number of Rules], and the number of inefficient rules removed is [Number of Removed Rules]. Input also includes statistical information for each rule in this category, specifically including each rule's number, creation iteration step, age, number of citations, number of adoptions, effectiveness rate, and rule content. Your task is only for the current category: First, identify rules with the same or highly similar meanings; second, discover opportunities to merge multiple narrow rules into more abstract principles; third, clarify ambiguous rules or remove completely useless, redundant, or inefficient rules. Three types of operations can be performed: removal, modification, and merging. Removal is used to delete redundant or invalid rules; modification is used to improve the clarity and executability of rule expressions; merging is used to rewrite multiple fine-grained rules into a more general new rule while retaining its category." Among them, [Rule Category] refers to the topic category to which the current rule to be compressed belongs, [Current Iteration Step] refers to the training or annotation iteration round when rule compression occurs, [Number of Rules] refers to the number of rules to be analyzed under this category, and [Number of Removed Rules] refers to the number of inefficient rules that have been deleted during the statistical cleanup phase.

[0031] (4) Repeat steps (1)-(3) until the set number of iterations is reached, stop the iteration, train the diffusion model based on the obtained multiple labeled sampled image pairs and the corresponding original text prompts and enhanced text prompts, and obtain the trained diffusion model, or perform step (4) after step (3). Step (4) is to train the diffusion model based on labeled sampled image pairs, original text prompts and enhanced text prompts, repeat steps (1)-(4) until the set number of iterations is reached, stop the iteration, and obtain the trained diffusion model.

[0032] In a specific embodiment of the present invention, after the above-described intelligent agent annotation, for the image pairs generated by the current model... and text prompts Winning samples can be obtained. Failed samples (Based on final preference judgment) (Confirmed) and enhanced conditions The specific embodiments of the present invention will use the original conditions. and enhanced conditions Concatenate into the final text conditions This is used for Diffusion-DPO training. The specific form of the SEPO training objective is derived below. Here, c retains the original user cue, and c' explicitly describes the specific visual advantage of the winning image; the two are concatenated to obtain the enhanced textual condition used in the training phase, which transforms the implicit win / loss labels into learnable visual attribute supervision. This enhanced textual condition is only used for DPO training; the original cue is still used in the inference phase.

[0033] In a specific embodiment of the present invention, the training conditions are changed from the original... Replace with spliced This makes the visual preferences expressed by the rules explicitly included in the training signal, thus obtaining the training objective of SEPO. for: ,in, Indicates the SEPO training objective; This represents the training dataset consisting of labeled samples, each of which includes the winning image. Failure image and enhanced text conditions ; Indicates a prompt from the original text. and enhanced text prompts The training conditions obtained by splicing together; The winning image is shown. Indicates a failure image; and These represent the winning and losing images at diffusion time steps 1 to 2000. diffusion path on, and These represent the complete diffusion trajectories of the winning and losing images, including the original image and its diffusion path, respectively. This represents the total number of time steps in the diffusion process; This represents the parameters of the diffusion model currently being trained. This represents the path probability corresponding to the current diffusion model to be trained; This represents the path probability corresponding to the reference diffusion model, whose parameters are frozen during training. Represents the Sigmoid function; Indicates the preference optimization intensity coefficient; This represents the expectation of samples in the training dataset and their corresponding diffusion paths; It represents logarithmic operations.

[0034] Compared with standard Diffusion-DPO only under original conditions Due to different training conditions, SEPO uses The condition is defined by the explicit inclusion of visual preferences expressed by the rules. In the process of learning preferences, the model is actually learning to generate visual preferences that conform to... The image describing the attributes transforms preference learning from fitting implicit win / loss signals to following explicit conditional descriptions. It is important to emphasize that... Use only during the training phase; use only the original hints during the inference phase. This is done because fine-grained visual criteria have been internalized into the model parameters during training.

[0035] When applied, the text prompt is input into the trained diffusion model to obtain the optimized image.

[0036] A specific embodiment of the present invention also provides a preference alignment device based on a diffusion model of self-evolving intelligent agents and dynamic rule evolution, characterized in that it includes a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the preference alignment method based on a diffusion model of self-evolving intelligent agents and dynamic rule evolution as described in any one of the present invention.

[0037] The technical solution provided in the specific embodiments of this invention is applicable to both offline and online training settings. The core process is the same for both, the difference lying only in the source of the image pairs: in the offline setting, the image pairs... and its corresponding text prompts From a fixed preference dataset; in the online setting, a prompt is sampled from the text prompt dataset. According to the current diffusion model Generate two images in real time and The overall process is as follows: Step 1: Initialize the rule manual Six seed rules are set, covering basic dimensions such as object correctness, visual fidelity, and security. The diffusion model is then initialized. and reference model .

[0038] Step 2: Obtain image pairs and text prompts. In offline settings, sample a pair of images from the preference dataset. and its corresponding text prompts In the online settings, sample a text prompt from the text prompt dataset. Using the current diffusion model Generate two images as prompted. and .

[0039] Step 3: Pair the images Text prompts and the current rulebook Input executor The executor makes preference judgments based on the rules in the rule manual and outputs the preference results. and a subset of the rules referenced .

[0040] Step 4: Pair the images Text prompts The executor's judgment result and citation rules Input evaluator The evaluators first conduct a blind review, independently analyzing the visual quality differences between image pairs, and then compare their results with the executor's judgment, outputting whether they agree or disagree. and review opinions .

[0041] Step 5: Input the image pair, text prompts, and all information about the executor and evaluator into the referee's input. .like The adjudicator directly adopts the executor's judgment and generates enhancing conditions. ;like The referee will arbitrate and make a final preference decision. Refining new rules from disagreements And update the rulebook. Simultaneously generate enhancement conditions .

[0042] Step Six: Determine based on final preference Determine the winning sample and failed samples , the original conditions and enhanced conditions spliced ​​as The training loss of SEPO is calculated based on the obtained SEPO training objective formula, and the diffusion model is updated accordingly. The parameters.

[0043] Step 7: Every Each iteration performs compression operations on the rule manual, including statistical-based automatic cleanup and semantic compression, pruning inefficient rules, merging redundant rules, and controlling the size of the rule manual. Within.

[0044] Step 8: Repeat steps 2 through 7 until training is complete. During training, the rulebook continues to evolve as the model's capabilities improve, gradually expanding from basic constraints to fine-grained aesthetic attributes.

[0045] Step 9: For a pre-trained diffusion model, you can directly use the original text prompts. Image generation can be performed without enhancing text prompts. Because during training The encoded fine-grained visual criteria have been internalized into the model parameters.

[0046] This invention provides Example 1: Offline preference alignment based on SDXL, including: S1 initializes the model, rulebook, and agent.

[0047] S2 performs agent annotation on the preference dataset and simultaneously evolves the rule manual, saving the annotation results.

[0048] S3 uses labeled data to train a diffusion model.

[0049] S4 uses a trained model to generate images.

[0050] Specific steps: S1 reads the weights of the pre-trained SDXL model and uses it as the diffusion model. The initial parameters. Copy the parameters as a reference model The reference model's parameters are frozen and not updated throughout the entire training process. Initialization rule manual. It contains 6 seed rules, covering the following dimensions: (1) Object existence: All objects explicitly mentioned in the prompt must appear in the image; (2) Quantity consistency: When a specific quantity is mentioned, the count must be accurate; (3) Attribute accuracy: Attributes such as color and size must match the prompt; (4) Human anatomy correctness: The limbs, facial features, etc. of a person should conform to the normal human body structure; (5) Physical rationality: The spatial relationship and physical laws between objects should be reasonable; (6) Security: Images containing harmful content such as violence and pornography should not be generated. Each rule is stored in a structured record form, including information such as rule number, category, content, creation iteration steps, number of references, and number of adoptions. During initialization, the number of references and adoptions of the rule are both set to 0. Gemini 2.0 Flash is selected as the executor. evaluator , referee The visual language model backbone consists of four intelligent agents: S (rule compressor), S (rule compactor), and S (rule manual). The upper limit of the rule manual's capacity is set. Compression interval We used the Pick-a-Pic v2 dataset, which contains 851,293 pairs of image preferences and their corresponding text prompts.

[0051] S2 performs agent annotation on image pairs in the dataset, annotating a total of 5000 samples, with the rule manual evolving synchronously during the annotation process: S2-1 Take a sample sequentially from the Pick-a-Pic v2 dataset to obtain a pair of images. and its corresponding text prompts .

[0052] S2-2 will pair the images Text prompts and the current complete rulebook Enter the executor together After receiving two images and text prompts, the executor reviews each rule in the rule manual, assessing the differences in performance between the two images across various rule dimensions. When multiple rules apply, the newer rule (creating a rule with a larger iteration count) is prioritized, as newer rules typically correspond to finer-grained judgment criteria. The executor provides the preference judgment result, a list of referenced rule numbers, the primary rule number used, and an explanation of the judgment reasoning. Simultaneously, the citation count of the referenced rule is updated. This step calls the aforementioned executor prompt template.

[0053] S2-3 will pair the images Text prompts The executor's judgment result and citation rules Input evaluator The evaluator's work is carried out in two phases: In the first phase, without knowing the executor's judgment, the evaluator independently analyzes the visual quality differences between the two images from open dimensions such as composition, lighting, texture details, color harmony, and overall aesthetics, forming their own preference judgment; in the second phase, the evaluator compares their independent findings with the executor's judgment and reasoning to determine whether the rules cited by the executor cover the key visual differences. The evaluator's output is whether they agree or disagree. and review opinions The specific method is as follows: the evaluator first forms a blind review conclusion, then reads the executor's preference conclusion, cited rules, and reasoning basis, and generates an agreement and review opinion based on three questions: "whether key differences were captured, whether important visual aspects were omitted, and whether the conclusion is fully supported by the rules." This step uses the aforementioned evaluator prompt template.

[0054] S2-4 Input the image pair, text prompts, and all information about the enforcer and evaluator into the referee. .like That is, the evaluator agrees with the executor's judgment, and the adjudicator directly adopts the executor's judgment as the final preference determination. Skip arbitration and rule refinement, and generate enhanced conditions solely based on the judgment criteria. Enhanced conditions This is a piece of natural language text describing the specific visual qualities of the winning image, such as "warm evening sunlight streams in through the window, soft volumetric lighting is accompanied by natural shadows, and delicate hair textures." It also updates the number of times the cited rule has been adopted. When the executor and the evaluator disagree, the referee intervenes to perform the following three tasks: (1) synthesize the analyses of both parties and make a final preference judgment based on an independent review of the two images. (2) The evaluator in For each potential blind spot identified, suggestions that meet the requirements of generality and non-redundancy are selectively adopted and abstracted into new rules. The number of references and adoptions of the new rule are initialized to 0, the iteration step record is created as the current iteration step, and then the rule manual is updated; (3) Generate enhancement conditions. If the final judgment of the referee is consistent with that of the executor, the adoption number of the corresponding referenced rule is increased; if they are inconsistent, the adoption number is not increased. The specific method is as follows: the referee first generates the final preference judgment and its reasons based on the output of both parties, and then decides whether to generate a new rule based on whether the evaluator agrees with the executor's conclusion; if a new rule is generated, its source is the potential blind spot pointed out in the review opinion, and it must meet the requirements of being general, executable, and non-repetitive, and at the same time, enhancement conditions describing the visual quality of the winning image are generated. This step calls the aforementioned referee prompt word template.

[0055] S2-5 Save the annotation results of this sample as a quadruple. Among them, the winning sample and failed samples Based on the final preference determination Confirmed: If but ,like but .

[0056] S2-6 After processing every 100 samples, the rulebook is compressed once. The first stage is statistical cleanup: calculating the effectiveness of each rule. ,prune and The rules; pruning creation time exceeding 200 iterations and Rules below a threshold are dynamically allocated quotas based on the cumulative usage frequency of each rule category. Seed rules and new rules created within the last 100 steps are protected. The second stage is semantic compression: semantic analysis is performed on the remaining rules by category, identifying and merging semantically similar rules, and removing redundant rules. The size of the compressed rule manual is controlled within [a certain range]. Within one category. This step calls the aforementioned rule compressor prompt word template to perform semantic deduplication, merging, modification, and removal on a category-by-category basis.

[0057] S2-7 Repeat S2-1 to S2-6 until all 5000 samples are labeled. During the labeling process, the rule manual evolved from 6 seed rules to 150 rules, covering multiple dimensions from basic correctness to fine-grained aesthetic attributes such as composition, lighting, and color harmony.

[0058] S3 trained the SDXL model using labeled data in the labeled order, following the Diffusion-DPO training configuration, for a total of 5000 steps. The experiment was performed on a single NVIDIA H800 GPU. In each training step, a quadruple was extracted in the labeled order. The training process is as follows: S3-1 will display the original text prompt. and enhanced conditions Concatenation to enhance text conditions .

[0059] S3-2 Random Sampling Time Step and noise For the winning sample and failed samples Noise is added separately to obtain the noisy samples. and ,in This is the cumulative product of the noise scheduling parameters.

[0060] S3-3 Adding noise to the sample and Input the current model respectively and reference model Calculate the denoising error for each: S3-4 Calculate the SEPO training loss: in The regularization coefficient is . For signal-to-noise ratio, This is the weighting function. The gradient is calculated based on this loss, and the model parameters are updated. Reference Model Keep frozen throughout the training process.

[0061] S3-5 Repeat S3-1 to S3-4, training for a total of 5000 steps. Training in the labeled order preserves the coarse-to-fine course effect: the early samples... Generated from basic rules, the description is rather rough; later samples... It is generated from evolved fine-grained rules and contains more refined descriptions of visual attributes.

[0062] After S4 training is complete, the trained SDXL model is used to generate images. (A text prompt is provided.) Initial noise is sampled from a standard Gaussian distribution. Using the trained Gradual noise reduction. At each time step Given the current noisy sample Perform one step of noise reduction to obtain : in It is random noise. This represents the noise figure for the corresponding time step. Note that only the original text hints are used during inference. As a condition, there is no need to make enhanced text prompts. ,because The encoded fine-grained visual criteria have been internalized into the model parameters during training.

[0063] This invention provides Embodiment 2: Online Preference Alignment Based on FLUX: The online alignment method based on the diffusion model of self-evolutionary preference optimization includes the following steps: S1 initializes the model, rulebook, and agent.

[0064] S2 alternately performs sample generation, agent labeling, and model training, while simultaneously evolving the rule manual.

[0065] S3 uses a trained model to generate images.

[0066] Specific steps: S1 reads the weights of the pre-trained FLUX.1-dev model. Due to the large number of parameters in the FLUX model, the LoRA method is used for efficient parameter fine-tuning. A low-rank adaptation matrix is ​​inserted into the model's attention layer. The LoRA rank is set to 4, and the learning rate is set to... The LoRA scaling factor is set to 0.1. The original FLUX.1-dev model (with the LoRA parameter initialized to zero) is used as the reference model. The reference model's parameters are frozen and not updated during training. (Initialization rule manual) It contains the same six seed rules as in Example 1, and the metadata initialization method for each rule is the same as in Example 1. Gemini 2.0 Flash is selected as the backbone for the four intelligent agent roles: executor E, evaluator V, referee J, and rule compressor S. An upper limit for the rulebook capacity is set. Compression interval Text prompts from the Pick-a-Pic v2 dataset were used as the training prompt source.

[0067] During S2 training, sample generation, agent labeling, and model updates are performed synchronously in each iteration step, as follows: S2-1 Sample a text prompt from the text prompt dataset. Using the current model (That is, FLUX.1-dev plus the current LoRA parameter) in this prompt Two images are generated below. and The generation process is as follows: two initial noise samples are sampled from a standard Gaussian distribution. Use the current model For each piece of noise under the condition The complete denoising process is then executed, and the following results are obtained: and Because the two initial noise samples are different, the two generated images have different visual appearances.

[0068] S2-2 The image generated by the current model is compared... Text prompts and the current complete rulebook Input executor The executor reviews each rule, providing preference judgments, a list of referenced rule numbers, and the reasons for the judgments. Simultaneously, the reference count of the referenced rules is updated. The process is the same as S2-2 in Example 1.

[0069] S2-3 Input the image pair, the executor's judgment result, and the reference rules into the evaluator. The evaluators followed a two-stage process of blind review and assessment, and their output was whether they agreed or disagreed. and review opinions The process is the same as S2-3 in Example 1.

[0070] S2-4 Input all information into the referee The adjudicator performs different actions based on whether the evaluator agrees with the executor's judgment: if agreeing, the adjudicator adopts the executor's judgment and generates enhanced conditions, updating the adoption count of the corresponding rule; if disagreeing, arbitration, rule refinement, and enhanced condition generation are conducted. The process is the same as S2-4 in Example 1. The final preference determination is then made. Determine the winning sample and failed samples .

[0071] S2-5 calculates the SEPO training loss and updates the model parameters in real time. (The original conditions are then set aside.) and enhanced conditions spliced ​​as Random sampling time step and noise ,right and Noise was added separately to obtain and Calculate the difference in denoising error between the current model and the reference model on two samples. and Calculate the SEPO training loss according to the obtained SEPO training objective formula. Calculate the gradient and update only the LoRA parameters, FLUX backbone parameters, and reference model. All remain frozen.

[0072] S2-6 performs rule compression every 100 steps. The compression process is the same as S2-6 in Example 1: first, statistical cleaning (pruning) is performed. and The rules are modified (expired rules are pruned, quotas are allocated by category), and then semantic compression is performed (semantic similar rules are merged and redundant rules are removed) to keep the rule manual size within 150 rules.

[0073] S2-7 Repeat S2-1 to S2-6 until training is complete. In the online setting, since image pairs are generated in real time by the current model, the evolution of the rulebook can adaptively adjust to the model's current generation capabilities: when the model has improved in certain visual dimensions, the agents' disagreements will focus on new fine-grained differences, thereby driving the rules to evolve in a more refined direction.

[0074] After S3 training is complete, the trained FLUX model (original weights plus trained LoRA parameters) is used to generate images. A text prompt is then provided. Initial noise is sampled from a standard Gaussian distribution. The trained model is used to progressively denoise and generate the final image. Only the original text prompts are used during inference. As a condition, there is no need to make enhanced text prompts. .

[0075] First, compared with existing diffusion model preference optimization methods, the SEPO method proposed in this invention achieves superior preference alignment quality in both offline and online training settings. It achieves optimal results on multiple mainstream human preference evaluation metrics such as ImageReward, HPSv2, and LAION-Aesthetic, and its performance is balanced across metrics, avoiding the reward intrusion phenomenon that occurs when using fixed reward models, where only a single metric scores the highest while other metrics degenerate.

[0076] Secondly, the self-evolutionary evaluation criteria and semantic injection mechanism proposed in the specific embodiments of this invention are universal and can be applied to diffusion models with different architectures (such as SDXL, FLUX, etc.), providing an effective post-training optimization scheme for improving the quality of text-to-image generation systems, and have good application prospects.

[0077] Table 1. Comparison results of the method of the specific embodiment of the present invention and the baseline method on the Pick-a-Pic v2 dataset. Table 1 shows the comparison results between the method of the specific embodiment of the present invention and existing methods. The evaluation metrics include CLIPScore (text-image alignment), PickScore (human preference prediction), HPSv2 (human preference score), ImageReward (image reward score), and LAION-Aesthetic (aesthetic score). Higher values ​​for these metrics indicate better quality generated images. Online-DPO (IR) and Online-DPO (Aes) use ImageReward and LAION-Aesthetic as fixed reward models, respectively. The results show that, under the three experimental settings, the SEPO method proposed in the specific embodiment of the present invention achieves the best results on most metrics. The Online-DPO method using the fixed reward model only scores highest on the single metric it directly optimizes, while other metrics degenerate, whereas SEPO performs evenly across all metrics.

[0078] Figure 2 This paper presents a comparison of the text prompt generation results of the method of this invention and the baseline method on the SDXL architecture. The text prompt is: "Pink-haired woman looking straight ahead, with full lips, wearing a white military uniform with a few red details, against a background of blue sky and blurry clouds, with color difference effects and geometric shapes." It can be seen that, compared to existing methods, the SEPO method proposed in this invention generates images that are superior in terms of prompt compliance, natural lighting, and fine-grained visual quality.

Claims

1. A preference alignment method based on a diffusion model of self-evolving intelligent agents and dynamic rule evolution, characterized in that, include: (1) Construct a multi-role intelligent agent, which includes an executor, an evaluator and a referee. Based on the set basic rule manual or the current rule manual, sampled image pairs and original text prompts, the executor obtains the first preference conclusion and the first judgment basis. (2) Based on the obtained sampled image pairs and original text prompts, the second preference conclusion and the second judgment basis are obtained through blind evaluation by the evaluator. The second preference conclusion and the second judgment basis are compared with the first preference conclusion and the first judgment basis. The review opinion is constructed through the conclusion conflict between the first and second preference conclusions and the basis not covered by the first judgment basis. (3) Based on the first preference conclusion, the first judgment basis and the review opinion, the referee provides the final preference judgment and the reason for the judgment, and generates new rules based on the final preference judgment to be added to the basic rule manual, as well as enhanced text prompts describing the winning image, and at the same time, the sampled image is labeled as the winning image and the losing image; (4) Repeat steps (1)-(3) until the set number of iterations is reached, stop the iteration, train the diffusion model based on the obtained multiple labeled sampled image pairs and the corresponding original text prompts and enhanced text prompts, and obtain the trained diffusion model, or perform step (4) after step (3). Step (4) is to train the diffusion model based on labeled sampled image pairs, original text prompts and enhanced text prompts, repeat steps (1)-(4) until the set number of iterations is reached, stop the iteration, and obtain the trained diffusion model. When applied, the text prompt is input into the trained diffusion model to obtain the optimized image.

2. The preference alignment method based on a diffusion model of self-evolving intelligent agents and dynamic rule evolution according to claim 1, characterized in that, The executor prompts enable the executing agent to make preference judgments on the sampled image pairs, obtain the winning and losing images provided by the executing agent, i.e., the first preference conclusion, and the first judgment basis, which includes the reference rules and the judgment reasons; The executor prompt includes instructing the executor agent to first search for an evolutionary rule applicable to the sampled image pair in the current rule manual; when multiple applicable evolutionary rules exist, the evolutionary rule with a larger creation iteration step number is preferred, which is a rule generated or updated in a later iteration round; if none of the evolutionary rules in the current rule manual are applicable to the current sampled image pair, then the rules in the basic rule manual are referenced, and it is explained how the referenced rules affect the preference conclusion.

3. The preference alignment method based on a diffusion model of self-evolving intelligent agents and dynamic rule evolution according to claim 1, characterized in that, The evaluator's prompts enable the evaluator agent to make preference judgments on the sampled image pairs, namely, the second preference conclusion, the second judgment basis, and the provision of review opinions. The evaluator's prompts include requiring the evaluator to identify visual quality differences between sampled images based on multi-faceted analysis. These multi-faceted analyses include composition, lighting, detail, texture, color coordination, overall aesthetics, and artifacts or distortion. Based on these visual quality differences, a second preferred conclusion is formed. Then, a first preferred conclusion and a first judgment criterion are obtained. The first judgment criterion includes citation rules and judgment reasons. The evaluator is required to review whether the citation rules of the first judgment criterion can capture visual quality differences and identify omissions or errors. The identified omissions or errors, as well as conflicting conclusions, are used to form a review opinion.

4. The preference alignment method based on a diffusion model of self-evolving intelligent agents and dynamic rule evolution according to claim 1, characterized in that, The referee's prompts enable the referee agent to provide the final preference judgment and the reasoning behind it. The referee's prompts include a request for the referee to review the sampled image pairs, the first preference conclusion, the first judgment basis, the second preference conclusion and the second judgment basis, and to make a final preference determination and reasoning. When the second preference conclusion of the evaluator and the executor conflicts with the first preference conclusion, a new rule is generated. The requirements for the new rule include that it does not repeat the existing rules, is applicable to multiple types of images, and also generates an enhanced text prompt for the winning image based on the final preference determination.

5. The preference alignment method based on a diffusion model of self-evolving intelligent agents and dynamic rule evolution according to claim 4, characterized in that, The requirements for the new rules also include: the new rules are general natural language evaluation rules, which are rules that are not limited to the current sampled image pair and can be used to judge the visual quality differences of the same type in subsequent sampled image pairs; the enhanced text prompts are specific visual attribute descriptions for the winning image.

6. The preference alignment method based on a diffusion model of self-evolving intelligent agents and dynamic rule evolution according to claim 1, characterized in that, The basic rule manual is a set of natural language rules, which includes basic natural language rules for existence, quantity consistency, attribute accuracy, human anatomy correctness, physical rationality, and security.

7. The preference alignment method based on a diffusion model of self-evolving intelligent agents and dynamic rule evolution according to claim 1, characterized in that, The basic rule manual or current rule manual includes multiple natural language rules and basic information about each natural language rule. The basic information about each natural language rule includes its category, creation time, index within the current category, last usage time, number of citations, number of adoptions, rule status, and validity rate. The rule status indicates whether the natural language rule is citationable. A rule is valid when it has not been deleted, deactivated, or merged during the rule compression process; it is invalid when it is no longer citationable by executors due to redundancy, inefficiency, expiration, or merging. The validity rate is the ratio of the number of times an executor cites the current natural language rule and it is adopted by the adjudicator to the total number of times the executor cites the current natural language rule.

8. The preference alignment method based on a diffusion model of self-evolving intelligent agents and dynamic rule evolution according to claim 1, characterized in that, At every set iteration step, the current rule manual is compressed, including: The rules of the same category in the current rule manual are compressed. The compression criteria include: removing natural language rules whose citation count reaches a preset citation count threshold and whose effectiveness is lower than a preset effectiveness threshold; and removing natural language rules whose interval between the creation iteration step and the current iteration step reaches a preset age threshold and whose citation count is lower than the preset citation count threshold. The preset citation count threshold is used to determine whether the natural language rule has sufficient usage statistics, and it is the same threshold in the above two compression criteria. Then, the remaining rules are semantically compressed to obtain the compressed current rule manual. The validity rate is the ratio of the number of times the executor references the current natural language rule and the number of times the referee adopts it to the number of times the executor references the current natural language rule.

9. The preference alignment method based on a diffusion model of self-evolving intelligent agents and dynamic rule evolution according to claim 1, characterized in that, The diffusion model is trained using the SEPO training objective to obtain the trained diffusion model; The training objective of SEPO for: ,in, Indicates the SEPO training objective; This represents the training dataset consisting of labeled samples, each of which includes the winning image. Failure image and enhanced text conditions ; Indicates a prompt from the original text. and enhanced text prompts The training conditions obtained by splicing together; The winning image is shown. Indicates a failure image; and These represent the winning and losing images at diffusion time steps 1 to 2000. diffusion path on, and These represent the complete diffusion trajectories of the winning and losing images, respectively. This represents the total number of time steps in the diffusion process; This represents the parameters of the diffusion model currently being trained. This represents the path probability corresponding to the current diffusion model to be trained; This represents the path probability corresponding to the reference diffusion model, whose parameters are frozen during training. Represents the Sigmoid function; Indicates the preference optimization intensity coefficient; This represents the expectation of samples in the training dataset and their corresponding diffusion paths; It represents logarithmic operations.

10. A preference alignment device based on a diffusion model of self-evolving intelligent agents and dynamic rule evolution, characterized in that, The device includes a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the preference alignment method based on the diffusion model of self-evolving intelligent agents and dynamic rule evolution as described in any one of claims 1-9.

Images