Deterministic trajectory pair preference optimization method and device for flow-oriented matching model

Abstract

The application relates to a deterministic trajectory pair preference optimization method and device for a flow matching model. The generation quality of each deterministic trajectory in a trajectory group is evaluated based on input prompt text to obtain an evaluation result. The trajectory group is generated based on deterministic ODE sampling by a current flow matching model. Main trajectories and reference trajectories are divided in the deterministic trajectories based on the evaluation result. The main trajectories are trajectories participating in gradient updating. The reference trajectories serve as comparison targets and do not participate in gradient updating. Preference optimization is performed on the current flow matching model according to the main trajectories and the reference trajectories to obtain a preference alignment model. The main trajectories and the reference trajectories are used for preference optimization directly on original ODE trajectories, the inconsistency problem caused by additional random noise in training and reasoning is avoided, and the performance of the final preference alignment model is improved.

Description

Technical Field

[0001] This application relates to the field of machine learning technology, and in particular to a deterministic trajectory pair preference optimization method and apparatus for flow matching models. Background Technology

[0002] In recent years, significant progress has been made in the generation capabilities of large-scale pre-trained models. However, the content output by these models still struggles to consistently align with the complex value orientations and preferences of humans. To bridge this gap, introducing human feedback mechanisms to guide models in generating higher-quality content that better meets expectations has become an important research direction in the field of artificial intelligence.

[0003] Among existing preference alignment methods, the GRPO (Group Relative Policy Optimization) method, based on the reconstruction of stochastic differential equations, has attracted widespread attention. This method reconstructs the original deterministic ordinary differential equation sampling process of the flow matching model into a stochastic sampling process, introducing additional random noise at each denoising time step, so that one sampling step can be modeled as a stochastic exploration process. Subsequently, the transition probability is calculated based on the distance between the sampling result and the model's predicted mean, and the GRPO optimization objective is constructed using the probability ratio between the current model and the old model.

[0004] However, this method requires the introduction of an additional stochastic process, which changes the original sampling mechanism of the flow matching model. Since random sampling is used in the training phase while deterministic sampling is still relied upon in the inference phase, the training and inference processes are inconsistent, resulting in poor model stability and unsatisfactory preference alignment. Summary of the Invention

[0005] This embodiment provides a deterministic trajectory pair preference optimization method and apparatus for flow matching models to solve the problem of insufficient model performance caused by the introduction of additional random noise during training in related technologies.

[0006] Firstly, this embodiment provides a deterministic trajectory pair preference optimization method for a flow matching model, the method comprising:

[0007] Based on the input prompt text, the generation quality of each deterministic trajectory in the trajectory group is evaluated to obtain the evaluation result; the trajectory group is generated by the current flow matching model based on deterministic ODE sampling.

[0008] Based on the evaluation results, the deterministic trajectory is divided into a main trajectory and a reference trajectory; the main trajectory is the trajectory that participates in gradient update; the reference trajectory serves as a comparison target and does not participate in gradient update.

[0009] Based on the main trajectory and the reference trajectory, the current flow matching model is optimized to obtain a preference alignment model.

[0010] In some embodiments, the generation quality of each deterministic trajectory in the trajectory group is evaluated based on the input prompt text, and the evaluation results are obtained, including:

[0011] The input prompt text and each deterministic trajectory in the trajectory group are input into a preset reward model to calculate a reward score that describes the generation quality of each deterministic trajectory.

[0012] The reward score is normalized within the trajectory group to obtain the relative advantage of each deterministic trajectory within the group;

[0013] The evaluation result is obtained based on the reward score and the relative advantage within the group.

[0014] In some embodiments, the trajectory group acquisition process includes:

[0015] Obtain a shared fundamental noise and multiple independent perturbation noises;

[0016] Based on a preset noise disturbance coefficient, the shared basic noise and multiple independent disturbance noises are combined to obtain a set of initial noise;

[0017] The initial noise is input into the current flow matching module, and a trajectory group containing multiple deterministic trajectories is generated based on deterministic ODE sampling.

[0018] In some embodiments, the primary trajectory and reference trajectory are divided in the deterministic trajectory based on the evaluation results, including:

[0019] Based on the reward score or the relative advantage within the group, the deterministic trajectories are sorted within the trajectory group;

[0020] In the sorting results, the deterministic trajectory located at the middle end is marked as the main trajectory, and the remaining deterministic trajectories are marked as reference trajectories.

[0021] In some embodiments, the current flow matching model is optimized for preference based on the main trajectory and the reference trajectory to obtain a preference alignment model, including:

[0022] Take one of the main trajectories and one of the reference trajectories as a trajectory pair;

[0023] For each pair of trajectories, the current distance score is obtained based on the distance between the trajectories, and the corresponding preference direction is obtained based on the difference in relative advantages within the group.

[0024] Based on the current distance score and the preference direction, an optimization objective is constructed;

[0025] Based on the optimization objective, the current flow matching model is iteratively trained to obtain a preference alignment model.

[0026] In some embodiments, an optimization objective is constructed based on the current distance score and the preference direction, including:

[0027] Obtain the old distance score corresponding to the old model; the old model is the previous model version that is immediately adjacent to the current model during iterative training;

[0028] Based on the difference between the current distance score function and the old distance score, a deterministic comparison value is obtained;

[0029] An optimization objective is constructed based on the deterministic ratio and the preference direction.

[0030] In some embodiments, an optimization objective is constructed based on the deterministic ratio and the preference direction, including:

[0031] The first product is obtained by multiplying the preference direction by the deterministic comparison value of the unpruned determinism;

[0032] A second product is formed by multiplying the preference direction with the comparative value of the determinedness after clipping;

[0033] The minimum value between the first product and the second product is used as the optimization objective.

[0034] In some embodiments, the current flow matching model is iteratively trained based on the optimization objective to obtain a preference alignment model, including:

[0035] Based on the optimization objective, the model parameters of the current flow matching model are updated;

[0036] After the model parameters are updated, determine whether the preset number of training rounds has been reached;

[0037] If the training rounds are reached, the preference alignment model is output based on the current model parameters; if the training rounds are not reached, iterative training continues.

[0038] Secondly, this embodiment provides a deterministic trajectory pair preference optimization device for a flow matching model, the device comprising:

[0039] The quality evaluation module is used to evaluate the generation quality of each deterministic trajectory in the trajectory group based on the input prompt text, and obtain the evaluation result; the trajectory group is generated by the current flow matching model based on deterministic ODE sampling;

[0040] The trajectory segmentation module is used to segment the deterministic trajectory into a main trajectory and a reference trajectory based on the evaluation results; the main trajectory is the trajectory that participates in gradient update; the reference trajectory serves as a comparison target and does not participate in gradient update.

[0041] The model optimization module is used to perform preference optimization on the current flow matching model based on the main trajectory and the reference trajectory to obtain a preference alignment model.

[0042] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the deterministic trajectory pair preference optimization method for a flow-oriented matching model as described in the first aspect.

[0043] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the deterministic trajectory pair preference optimization method for a flow-oriented matching model as described in the first aspect.

[0044] Compared with related technologies, the deterministic trajectory pair preference optimization method and apparatus for flow matching models provided in this embodiment evaluates the generation quality of each deterministic trajectory in the trajectory group based on the input prompt text, and obtains the evaluation result. The trajectory group is generated by the current flow matching model based on deterministic ODE sampling. Based on the evaluation result, the deterministic trajectory is divided into a main trajectory and a reference trajectory. The main trajectory is the trajectory that participates in gradient update. The reference trajectory serves as a comparison target and does not participate in gradient update. Based on the main trajectory and the reference trajectory, the current flow matching model is optimized to obtain a preference alignment model, which solves the problems of poor model stability and unsatisfactory preference alignment effect. It does not require reconstructing the deterministic ODE sampling into random SDE sampling, but directly performs preference optimization on the original ODE trajectory by dividing it into a main trajectory and a reference trajectory. This avoids the training and inference inconsistency problem caused by additional random noise, maintains the stability of the flow matching model under low-step sampling and high-order ODE solvers, and improves the model stability and preference alignment effect.

[0045] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0046] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0047] Figure 1 This is a hardware structure block diagram of the terminal for the deterministic trajectory pair preference optimization method for the flow matching model in this application embodiment;

[0048] Figure 2 This is a flowchart illustrating the deterministic trajectory pair preference optimization method for a flow matching model in this application embodiment;

[0049] Figure 3 This is a flowchart illustrating the method for optimizing a model based on a main trajectory and a reference trajectory in an embodiment of this application.

[0050] Figure 4 This is a flowchart illustrating the deterministic trajectory pair preference optimization method for a flow matching model in a preferred embodiment of this application.

[0051] Figure 5 This is a block diagram of the deterministic trajectory pair preference optimization device for the flow matching model in this application embodiment.

[0052] Reference numerals: 102, processor; 104, memory; 106, transmission device; 108, input / output device; 51, quality evaluation module; 52, trajectory division module; 53, model optimization module. Detailed Implementation

[0053] To better understand the purpose, technical solution, and advantages of this application, the application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0054] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these” used in this application do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to these processes, methods, products, or devices. Words such as “connected,” “linked,” and “coupled” used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. Normally, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," "third," etc., used in this application are merely to distinguish similar objects and do not represent a specific order of objects.

[0055] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. For example, it can run on a terminal. Figure 1 This is a hardware structure block diagram of the terminal for the deterministic trajectory pair preference optimization method for the flow matching model in this embodiment. Figure 1 As shown, a terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 and a memory 104 for storing data are also included. The processor 102 may be, but is not limited to, a microprocessor (MCU) or a programmable logic device (FPGA). The terminal may also include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that… Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown are illustrated.

[0056] Memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the deterministic trajectory pair preference optimization method for the flow matching model in this embodiment. Processor 102 executes various functional applications and data processing by running the computer programs stored in memory 104, thereby implementing the above-described method. Memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, memory 104 may further include memory remotely located relative to processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0057] The transmission device 106 is used to receive or send data via a network. This network includes a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 can be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0058] This embodiment provides a deterministic trajectory pair preference optimization method for flow matching models. Figure 2 This is a flowchart of the deterministic trajectory pair preference optimization method for the flow matching model in this embodiment, as shown below. Figure 2 As shown, this process can be executed by the processor and specifically includes the following steps:

[0059] Step S210: Based on the input prompt text, evaluate the generation quality of each deterministic trajectory in the trajectory group and obtain the evaluation result; the trajectory group is generated by the current flow matching model based on deterministic ODE sampling.

[0060] Specifically, the user provides a text c, which describes the user's expectations for the generated result. These expectations include the semantic content, style, or structural requirements of the generated result. The processor inputs the text c into a text encoder to obtain the corresponding text conditional embedding. Based on this text conditional embedding, the processor uses it as a cue condition to evaluate the quality of each deterministic trajectory. The flow matching model is a continuous generation model, a variant or simplified form of the diffusion model. It directly learns a velocity field (vector field) and smoothly maps a simple prior distribution (such as a standard Gaussian) to the target data distribution using an Ordinary Differential Equation (ODE). Since the ODE sampling process is deterministic, the entire trajectory is determined once the model parameters and initial noise are determined. This embodiment utilizes this property to transform preference optimization from "single-step stochastic exploration optimization" to "deterministic trajectory pair optimization."

[0061] Step S220: Based on the evaluation results, divide the deterministic trajectory into a main trajectory and a reference trajectory; the main trajectory is the trajectory that participates in gradient update; the reference trajectory serves as a comparison target and does not participate in gradient update.

[0062] Specifically, this embodiment does not employ a pairwise optimization approach for all trajectories to avoid each trajectory serving as both the optimized object and a reference object for other trajectories, which could lead to overlapping optimization roles and gradient coupling. Therefore, the processor divides the same trajectory group into two sets: a main trajectory set and a reference trajectory set, allowing a main trajectory to be compared pairwise with a reference trajectory. When dividing the sets, the processor can sort or threshold-filter the evaluation results of each deterministic trajectory, selecting the best and / or worst deterministic trajectories as reference trajectories to provide comparison targets, and selecting multiple deterministic trajectories with moderate quality performance as main trajectories.

[0063] Step S230: Based on the main trajectory and the reference trajectory, perform preference optimization on the current flow matching model to obtain a preference alignment model.

[0064] Specifically, the processor calculates pairwise optimization terms based on the state of each main trajectory and each reference trajectory at each selected time step. In some embodiments, the reference trajectory is used as a frozen fixed preference target, and only the main trajectory (generated by the current model) is allowed to unidirectionally move towards or away from the latent space features of the reference trajectory through gradient descent to achieve preference alignment. In other embodiments, the trajectory generated by the old model is also used as a benchmark, and the improvement ratio of the preference matching degree of the current flow matching model relative to the old model is calculated. Only the flow matching model parameters corresponding to the current main trajectory participate in gradient updates, and the current reference trajectory and the old model are frozen throughout. The processor automatically prioritizes the optimization of trajectory pairs with significant improvement in preference matching degree, ultimately obtaining a preference alignment model that better fits the prompt text. The specific process of preference optimization is not limited in this embodiment.

[0065] When the maximum number of training rounds is reached, training stops and an updated flow matching model is output as the preference alignment model. This preference alignment model still uses the native deterministic ODE sampling process during the inference phase, without introducing SDE (stochastic differential equations) random noise, and can generate preference-aligned images based on the input text prompts.

[0066] In this embodiment, the generation quality of each deterministic trajectory in the trajectory group is evaluated based on the input prompt text, and an evaluation result is obtained. The trajectory group is generated by the current flow matching model based on deterministic ODE sampling. Based on the evaluation result, the deterministic trajectory is divided into a main trajectory and a reference trajectory. The main trajectory is the trajectory that participates in gradient update; the reference trajectory serves as a comparison target and does not participate in gradient update. Based on the main trajectory and the reference trajectory, the current flow matching model is optimized for preferences to obtain a preference alignment model, which solves the problems of poor model stability and unsatisfactory preference alignment effect. Directly optimizing preferences on the original ODE trajectory by dividing it into a main trajectory and a reference trajectory can avoid the inconsistency between training and inference caused by additional random noise, maintain the stability of the flow matching model under low-step sampling and high-order ODE solvers, and improve the performance of the final preference alignment model.

[0067] In some embodiments, for step S210, based on the input prompt text, the generation quality of each deterministic trajectory in the trajectory group is evaluated to obtain evaluation results, including:

[0068] Step S211: Input the input prompt text and each deterministic trajectory in the trajectory group into the preset reward model to calculate the reward score used to describe the generation quality of each deterministic trajectory;

[0069] Specifically, the reward model can be a human preference reward model, an image-text alignment scoring model, an aesthetic scoring model, or other models capable of scoring the quality of the generated image. For each trajectory, the final generated image... The preference score is calculated using a reward model:

[0070] ;

[0071] Among them, R i Let represent the reward score for the generated image corresponding to the i-th trajectory under the text prompt c.

[0072] Step S212: Normalize the reward score within the trajectory group to obtain the relative advantage of each deterministic trajectory within the group;

[0073] Specifically, the formula for normalizing the reward of all trajectories within the same trajectory group is as follows:

[0074] ;

[0075] Among them, A i This represents the degree of advantage of the i-th trajectory relative to other trajectories in the same group, i.e., the relative advantage within the group.

[0076] Step S213: Obtain the evaluation result based on the reward score and the relative advantage within the group.

[0077] In this embodiment, the relative advantage within the group is calculated based on the reward score, which can weaken the absolute score deviation and accurately reflect the differences in quality within the trajectory group.

[0078] In some embodiments, the trajectory group acquisition process includes:

[0079] Step S310: Obtain a shared fundamental noise. and multiple independent disturbance noises ;

[0080] Step S320: Based on the preset noise disturbance coefficient α, combine and share the basic noise. and multiple independent disturbance noises We obtain a set of initial noise; the initial noise of the i-th trajectory can be expressed as:

[0081] ;

[0082] The noise disturbance coefficient α is used to control the degree of difference between different trajectories. When α is small, the trajectories are more similar; when α is large, the differences between the trajectories are greater.

[0083] Step S330: Input the initial noise into the current flow matching module, and generate a trajectory group containing multiple deterministic trajectories based on deterministic ODE sampling. For the i-th initial noise, its generated trajectory can be denoted as:

[0084] ;

[0085] in: This represents the initial noise state of the i-th trajectory. This represents the final generated image, an intermediate state. This represents the latent space state at time step t during the deterministic ODE sampling process. Since the ODE sampling process is deterministic, the entire trajectory is determined once the model parameters and initial noise are determined.

[0086] In this embodiment, a shared base noise is introduced to ensure that the trajectories have the same origin characteristic, and the intensity of independent disturbances is flexibly adjusted using a coefficient α. The similarity and difference between trajectories can be controlled as needed, which can generate diverse trajectory samples and stably control the distribution range to adapt to different evaluation and optimization needs.

[0087] In some embodiments, for step S220, dividing the deterministic trajectory into a main trajectory and a reference trajectory based on the evaluation results includes:

[0088] Step S221: Sort the deterministic trajectories within the trajectory group based on the reward score or relative advantage within the group;

[0089] Step S222: In the sorting results, the deterministic trajectory located at the middle end is marked as the main trajectory, and the remaining deterministic trajectories are marked as reference trajectories.

[0090] For example, when there are G=12 trajectories in the trajectory group, the four trajectories in the middle after sorting can be selected as the main trajectories, and the remaining eight trajectories can be used as reference trajectories.

[0091] In this preferred embodiment, the trajectory roles are divided based on the ranking results. The rules are clear and controllable, and can provide balanced and effective supervision samples for subsequent preference optimization.

[0092] In some embodiments, for step S230, the current flow matching model is optimized based on the main trajectory and the reference trajectory to obtain a preference alignment model, see [link to relevant documentation]. Figure 3 This step specifically includes:

[0093] Step S231: Take a main trajectory and a reference trajectory as a trajectory pair;

[0094] Step S232: For each pair of trajectories, obtain the current distance score based on the distance between the trajectories, and obtain the corresponding preference direction based on the difference in relative advantage within the group;

[0095] Specifically, for a main trajectory i and a reference trajectory j, at a certain time step t, the distance between the latent space states of the two trajectories is calculated, defined as follows:

[0096] ;

[0097] in, This represents the hidden space state of the main trajectory at time step t; This represents the hidden space state of the reference trajectory at time step t; This indicates that gradient operations are stopped, meaning the reference trajectory does not participate in gradient updates; The larger the value, the closer the two trajectories are at that time step. The technical implication of this score is that it preserves the "negative distance" structure of Gaussian transition probabilities in GRPO-type methods, but eliminates the need to construct random SDE transition probabilities, estimate noise variance, or normalize the entire group. In other words, it replaces the original "probability calculation between the sampled state and the model prediction mean" with "deterministic trajectory pair distance calculation between the main trajectory and the reference trajectory."

[0098] This embodiment also utilizes the relative advantage between the two trajectories to construct the preference direction. For the main trajectory i and the reference trajectory j, a symbolic pairwise preference difference is defined:

[0099] ;

[0100] Among them, A i A represents the relative advantage of the group representing the main trajectory. j This represents the relative advantage of the group representing the reference trajectory. When S ij When the value is greater than 0, it indicates that the reward of the reference trajectory is higher than that of the main trajectory. In this case, the reference trajectory should attract the main trajectory, causing it to move closer to it. Conversely, the reference trajectory should repel the main trajectory, causing it to move away from the less desirable generation direction. This is different from directly using A. j Compared to the optimized signal, S ij It can explicitly express the relative quality difference between two trajectories. If the two trajectories have similar quality, the optimization signal is weak; if the two trajectories have a large quality difference, the optimization signal is strong.

[0101] Step S233: Construct an optimization objective based on the current distance score and preference direction;

[0102] Specifically, in order to maintain the probability ratio form in GRPO, the distance score under the current flow matching model can be compared with the distance score under the old model to construct a deterministic comparison value, and an optimization objective can be constructed based on the deterministic comparison value and the preference direction.

[0103] Step S234: Iteratively train the current flow matching model based on the optimization objective to obtain the preference alignment model.

[0104] Specifically, based on the optimization objective, the model parameters of the current flow matching model are updated; after the model parameters are updated, it is determined whether the preset number of training rounds has been reached; if the number of training rounds has been reached, the preference alignment model is output based on the current model parameters; if the number of training rounds has not been reached, the training continues iteratively.

[0105] In this embodiment, the optimization signal comes from a deterministic trajectory pair comparison between the main trajectory and the reference trajectory, rather than random perturbations in a single denoising time step. Therefore, it can more fully utilize preference information at the complete trajectory level and propagate the final reward signal to the intermediate ODE state, improving preference optimization efficiency.

[0106] In some embodiments, for step S233, an optimization objective is constructed based on the current distance score and preference direction, including:

[0107] Step S410: Obtain the old distance score corresponding to the old model; the old model is the previous model version that is immediately adjacent to the current model in the iterative training.

[0108] Specifically, at the beginning of each training iteration, the parameters of the current model are copied and frozen as the parameters of the old model. The old model is used to calculate the old distance scores of the trajectory pairs before the update and does not participate in gradient updates; the current flow matching model is used to calculate the current distance scores of the updated trajectory pairs and participates in gradient updates.

[0109] Step S420: Based on the difference between the current distance score function and the old distance score, obtain the deterministic comparison value;

[0110] Step S430: Construct an optimization objective based on deterministic comparison values ​​and preference directions.

[0111] Specifically, the formula for the deterministic ratio is:

[0112] ;

[0113] in: This represents the distance score between the main trajectory and the reference trajectory in the current model. This represents the distance score under the old model.

[0114] This ratio is consistent with the strategy probability ratio in traditional GRPO, both using exponential difference. The difference is that the deterministic ratio does not depend on random transition probabilities, but is constructed entirely based on the distance changes between deterministic ODE trajectory pairs.

[0115] In some embodiments, for step S430, an optimization objective is constructed based on deterministic comparison values ​​and preference directions, including:

[0116] Step S431: Obtain the first product based on the product of the preference direction and the deterministic contrast value of the unpruned direction;

[0117] Step S432: Based on the product of the preference direction and the deterministic comparison value after clipping, a second product is formed;

[0118] Step S433: The minimum value between the first product and the second product is taken as the optimization objective.

[0119] Specifically, construct the GRPO style optimization objective:

[0120] ;

[0121] Where P represents the main trajectory set, T represents the time step set, and R represents the reference trajectory set. The clipping process involves limiting the ratio term to... Within the range. For each main trajectory, each selected time step, and each reference trajectory, the corresponding pairwise optimization term is calculated. This optimization term considers both the product of the unpruned contrast ratio and the preference difference, and the product of the contrast ratio and the preference difference after clipping, taking the smaller of the two as the final optimization term. This can limit the magnitude of a single model update and maintain the stability of the training process.

[0122] The present embodiment will now be described and illustrated through preferred embodiments.

[0123] Existing reinforcement learning preference optimization methods, such as GRPO and PPO, typically rely on the policy ratio for model updates. However, for flow matching models, the native sampling process is deterministic probabilistic flow ordinary differential equation (ODE) sampling, lacking natural random transition probabilities. Therefore, traditional GRPO methods are difficult to apply directly. Existing methods usually require introducing additional stochastic processes or constructing alternative probability distributions to approximate the policy ratio, which can easily lead to problems such as inconsistency between training and inference processes, optimization signal coupling, training instability, and high computational costs. Especially in the sampling scenario of short-step flow models, these problems further affect preference alignment efficiency and generation quality.

[0124] To address the above problems, there are two common solutions: one is the GRPO method based on stochastic differential equation (SDE) reconstruction, and the other is the GRPO method based on deterministic ODE neighborhood probability modeling.

[0125] The first category of methods is represented by DanceGRPO and Flow-GRPO. These methods reconstruct the original deterministic ODE sampling process of the flow matching model into a stochastic SDE sampling process, introducing additional random noise at each denoising time step, so that each sampling step can be modeled as a stochastic exploration process. Subsequently, existing methods calculate the transition probability based on the distance between the sampling results and the model's predicted mean, and construct the GRPO optimization objective using the probability ratio between the current model and the old model.

[0126] The second type of method is represented by NeighborGRPO, etc. This type of method maintains the deterministic ODE sampling process, generating a set of deterministic sampling trajectories (i.e., a trajectory group) using different initial noise levels, and constructing a neighborhood probability distribution at each time step based on the distance between samples within the group. This probability distribution requires normalization of candidate samples in the entire trajectory group, and then constructing a replacement probability ratio using the ratio of the neighborhood probabilities under the current model to those under the old model.

[0127] However, the above solutions still have shortcomings, such as: (1) SDE reconstruction leads to inconsistency between training and inference: SDE reconstruction-based methods need to introduce random noise in the deterministic ODE sampling process, which changes the original sampling method of the flow matching model. Random SDE sampling is used in the training stage, while deterministic ODE sampling is usually used in the inference stage, which easily causes inconsistency between training and inference processes and affects the stability under low-step sampling and high-order ODE solvers; (2) The optimization signal is difficult to fully utilize the preference information at the complete trajectory level: SDE reconstruction-based methods mainly rely on random perturbations in a single denoised time step to construct the optimization signal, and the preference reward is difficult to effectively propagate to the complete generated trajectory; Although ODE neighborhood probability modeling methods retain deterministic sampling, they still mainly model on the sample neighborhood at the same time step and fail to directly utilize the overall preference relationship between two deterministic trajectories; (3) Group-level probability modeling methods are prone to signal coupling and optimization instability: ODE neighborhood probability modeling methods need to normalize the model of the entire trajectory group, so that the optimization signal of a single sample pair is affected by other samples, group size and group structure. Meanwhile, samples within a group are usually processed symmetrically, and the same trajectory may be used as both the optimization target and the reference target at the same time, which can easily lead to overlapping of optimization roles, target offset, and training instability.

[0128] Based on this, this preferred embodiment proposes a deterministic trajectory pair preference optimization method for flow matching models. Figure 4 This is a flowchart of the deterministic trajectory pair preference optimization method for a flow-oriented matching model according to a preferred embodiment. Figure 4 As shown, the method includes the following steps:

[0129] Step S1: Input text prompt. Given a text prompt c, which describes the semantic content, style, or structural requirements of the image to be generated. Input the text prompt c into the text encoder to obtain the corresponding text conditional embedding, which is used in the subsequent conditional generation process of the stream matching model.

[0130] Step S2: Construct relevant initial noise and generate trajectory groups through deterministic ODE sampling.

[0131] A set of relevant initial noise is constructed to generate a set of candidate trajectories under the same cue conditions. Specifically, a shared base noise is first sampled. Then, sample independent perturbation noise for each trajectory. . No. The initial noise of the trajectory can be expressed as:

[0132] ;

[0133] Here, α is the noise disturbance coefficient, used to control the degree of difference between different trajectories. When α is small, the trajectories are more similar; when α is large, the differences between the trajectories are greater.

[0134] Each initial noise input stream is matched to the model, and a complete trajectory is generated through a deterministic ODE sampling process. For the i-th initial noise, its generated trajectory can be denoted as:

[0135] ;

[0136] in: This represents the initial noise state of the i-th trajectory. This represents the final generated image, an intermediate state. Let represent the latent space state at time step t during the deterministic ODE sampling process. Since the ODE sampling process is deterministic, the entire trajectory is determined once the model parameters and initial noise are determined. This preferred embodiment utilizes this property to transform preference optimization from "single-step stochastic exploration optimization" to "deterministic trajectory pair optimization".

[0137] Step S3: Calculate the reward and relative advantage for each trajectory.

[0138] The final image generated for each trajectory The preference score is calculated using a reward model:

[0139] ;

[0140] Among them, R iThis represents the reward score for the generated image corresponding to the i-th trajectory under text prompt c. The reward model can be a human preference reward model, an image-text alignment scoring model, an aesthetic scoring model, or other models capable of scoring the quality of the generated image. Subsequently, the rewards of all trajectories within the same trajectory group are normalized to obtain the group's relative advantage:

[0141] ;

[0142] Among them, A i This indicates the degree of advantage of the i-th trajectory relative to other trajectories in the same group.

[0143] Step S4: Sort by reward and divide into main trajectory and reference trajectory.

[0144] This preferred embodiment does not employ a pairwise optimization approach for all trajectories, as this would cause each trajectory to act as both the optimized object and a reference object for other trajectories, easily leading to overlapping optimization roles and gradient coupling. Therefore, this preferred embodiment divides the same trajectory group into two sets: main trajectories that participate in gradient updates; and reference trajectories that do not participate in gradient updates but only serve as comparison targets. Specifically, this preferred embodiment sorts the trajectories based on the reward score or advantage obtained in step S3, and selects the middle-ranked trajectory as the main trajectory, with the remaining trajectories serving as reference trajectories. For example, when there are G=12 trajectories in the trajectory group, the four trajectories in the middle after sorting can be selected as the main trajectories, and the remaining eight trajectories as reference trajectories.

[0145] Step S5: Calculate the distance between the main trajectory and the reference trajectory at each selected time step.

[0146] For a master trajectory i and a reference trajectory j, at a certain time step t, calculate the distance between the latent space states of the two trajectories. Specifically, define:

[0147] ;

[0148] in: This represents the hidden space state of the main trajectory at time step t; This represents the hidden space state of the reference trajectory at time step t; This indicates that gradient operations are stopped, meaning the reference trajectory does not participate in gradient updates; The larger the value, the closer the two trajectories are at that time step.

[0149] The technical implication of this score is that it retains the "negative distance" structure of Gaussian transition probabilities in GRPO-type methods, but no longer requires constructing random SDE transition probabilities, nor does it require estimating noise variance or normalizing the entire group. In other words, it replaces the original "probability calculation between the sampled state and the model prediction mean" with "deterministic trajectory pair distance calculation between the main trajectory and the reference trajectory".

[0150] Step S6: Construct symbolic pairwise preference differences as preference directions.

[0151] This preferred embodiment further utilizes the relative advantage between the two trajectories to construct the preference direction. For the main trajectory Given a reference trajectory j, define symbolic pairwise preference differences:

[0152] ;

[0153] Where A i A represents the relative advantage of the group representing the main trajectory. j This represents the relative advantage of the group representing the reference trajectory. When S ij When the value is greater than 0, it indicates that the reward of the reference trajectory is higher than that of the main trajectory. In this case, the reference trajectory should attract the main trajectory, causing it to move closer to it. Conversely, the reference trajectory should repel the main trajectory, causing it to move away from the less desirable generation direction. This is different from directly using A. j Compared to the optimized signal, S ij It can explicitly express the relative quality difference between two trajectories. If the two trajectories have similar quality, the optimization signal is weak; if the two trajectories have a large quality difference, the optimization signal is strong.

[0154] Step S7: Construct deterministic comparison values.

[0155] To preserve the probability ratio form in GRPO, the distance scores under the current model are compared with the distance scores under the old model to construct a deterministic comparison value:

[0156] ;

[0157] in: This represents the distance score between the main trajectory and the reference trajectory in the current model. This represents the distance score corresponding to the old model. This ratio is consistent with the policy probability ratio in traditional GRPO, both using exponential difference. However, unlike GRPO, the ratio in this preferred embodiment does not rely on random transition probabilities but is constructed entirely based on the distance changes between deterministic ODE trajectory pairs.

[0158] Step S8: Construct the optimization objective and update the model parameters.

[0159] In this preferred embodiment, the symbolic pairwise preference difference obtained in step S6 is combined with the deterministic contrast value obtained in step S7 to construct the GRPO style optimization objective:

[0160] ;

[0161] Where P represents the set of main trajectories, T represents the set of time steps, and R represents the set of reference trajectories. For each main trajectory, each selected time step, and each reference trajectory, a corresponding pairwise optimization term is calculated. This optimization term considers both the product of the unpruned contrast ratio and the preference difference, and the product of the contrast ratio and the preference difference after clipping, taking the smaller of the two as the final optimization term. This limits the magnitude of a single model update and maintains stability during training.

[0162] Step S9: Output the trained preference alignment model and use it to generate images.

[0163] When the maximum number of training epochs is reached, training stops and the updated flow matching model is output. During the inference phase, this model still uses the native deterministic ODE sampling process, generating preference-aligned images based on input text prompts without introducing SDE random noise or constructing trajectory sets.

[0164] To verify the effectiveness of this preferred embodiment, preference optimization experiments were conducted on the FLUX.1-dev visual generation model for flow matching. Experimental comparisons are shown in Table 1. All methods used the same base model, training data, optimizer, number of training steps, batch size, and number of sampling steps. Training and evaluation were performed using the HPDv2 dataset, with both training and evaluation employing 8-step ODE sampling. The reward model used HPS-v2.1 as the optimization objective, and various metrics were used to evaluate the generation quality and preference alignment effect, including HPS-v2.1, PickScore, ImageReward, AestheticScore, and UniReward.

[0165] Table 1

[0166]

[0167] As shown in Table 1, this preferred embodiment achieves superior results in metrics such as HPS-v2.1, ImageReward, Aesthetic, and UniReward, while maintaining a single iteration time of 80 seconds. This indicates that the method of this preferred embodiment can significantly reduce training overhead while maintaining high generation quality and preference alignment. Compared with DanceGRPO based on SDE reconstruction, this preferred embodiment does not require introducing random noise at each denoising time step, nor does it require calculating random transition probabilities, thus making the training process more efficient. Compared with NeighborGRPO based on ODE neighborhood probability modeling, this preferred embodiment does not require constructing normalized neighborhood probabilities for the entire trajectory group, but directly optimizes the preferences of trajectory pairs between the main trajectory and the reference trajectory, thus optimizing the signal more directly.

[0168] In this preferred embodiment, while maintaining the original deterministic ODE sampling of the flow matching model, a GRPO-style preference optimization signal is directly constructed between deterministic trajectory pairs. Specifically, this preferred embodiment generates multiple deterministic trajectories under the same text prompt through relevant initial noise, and divides the trajectory group into the main trajectory participating in gradient update and the reference trajectory stopping the gradient; then, a distance score is constructed based on the distance between the main trajectory and the reference trajectory in the intermediate ODE state, and a deterministic comparison value is formed by the difference between the distance scores under the current model and the old model; simultaneously, a symbolic pair preference difference is constructed using the advantage difference between the two trajectories to generate an attraction or repulsion signal according to the relative preference relationship. This technical solution avoids the coupling of SDE random reconstruction and group-level probability normalization, achieves more stable and efficient flow model preference optimization, and ensures a high alignment effect of the model formation.

[0169] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0170] This embodiment also provides a deterministic trajectory pair preference optimization device for a flow matching model. This device is used to implement the above embodiments and preferred embodiments, and will not be repeated as already described. The terms "module," "unit," "subunit," etc., used below can refer to combinations of software and / or hardware that implement a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0171] Figure 5 This is a block diagram of the deterministic trajectory pair preference optimization device for the flow-oriented matching model in this embodiment, as shown below. Figure 5As shown, the device includes: a quality evaluation module 51, a trajectory division module 52, and a model optimization module 53.

[0172] The quality evaluation module 51 is used to evaluate the generation quality of each deterministic trajectory in the trajectory group based on the input prompt text, and obtain the evaluation result; the trajectory group is generated by the current flow matching model based on deterministic ODE sampling.

[0173] The trajectory segmentation module 52 is used to segment the deterministic trajectory into a main trajectory and a reference trajectory based on the evaluation result; the main trajectory is the trajectory that participates in gradient update; the reference trajectory serves as a comparison target and does not participate in gradient update.

[0174] The model optimization module 53 is used to perform preference optimization on the current flow matching model based on the main trajectory and the reference trajectory to obtain a preference alignment model.

[0175] In some embodiments, the quality evaluation module 51 is further configured to input the input prompt text and each deterministic trajectory in the trajectory group into a preset reward model, calculate a reward score to describe the generation quality of each deterministic trajectory; normalize the reward score within the trajectory group to obtain the relative advantage of each deterministic trajectory within the group; and obtain an evaluation result based on the reward score and the relative advantage within the group.

[0176] In some embodiments, the quality evaluation module 51 is further configured to acquire a shared base noise and multiple independent disturbance noises; based on a preset noise disturbance coefficient, combine the shared base noise and multiple independent disturbance noises to obtain a set of initial noise; input the initial noise into the current flow matching module, and generate a trajectory group containing multiple deterministic trajectories based on deterministic ODE sampling.

[0177] In some embodiments, the trajectory segmentation module 52 is further configured to sort the deterministic trajectories within the trajectory group based on the reward score or the relative advantage within the group; in the sorting result, the deterministic trajectory located at the middle end is marked as the main trajectory, and the remaining deterministic trajectories are marked as reference trajectories.

[0178] In some embodiments, the model optimization module 53 is further configured to take one of the main trajectories and one of the reference trajectories as a pair of trajectories; for each pair of trajectories, obtain the corresponding current distance score based on the distance between the trajectories, obtain the corresponding preference direction based on the difference in relative advantages within the group; construct an optimization objective based on the current distance score and the preference direction; and iteratively train the current flow matching model based on the optimization objective to obtain a preference alignment model.

[0179] In some embodiments, the model optimization module 53 is further configured to obtain the old distance score corresponding to the old model; the old model is the previous model version that is adjacent to the current model in iterative training; a deterministic comparison value is obtained based on the difference between the current distance score function and the old distance score; and an optimization objective is constructed based on the deterministic comparison value and the preference direction.

[0180] In some embodiments, the model optimization module 53 is further configured to obtain a first product based on the product of the preference direction and the deterministic comparison value without pruning; form a second product based on the product of the preference direction and the deterministic comparison value after pruning; and use the minimum value of the first product and the second product as the optimization target.

[0181] In some embodiments, the model optimization module 53 is further configured to update the model parameters of the current flow matching model based on the optimization objective; after the model parameters are updated, determine whether a preset training round has been reached; if the training round has been reached, output a preference alignment model based on the current model parameters; if the training round has not been reached, continue iterative training.

[0182] This embodiment also provides a computer device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0183] Optionally, the computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0184] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0185] Furthermore, in conjunction with the deterministic trajectory pair preference optimization method for the flow matching model provided in the above embodiments, this embodiment can also provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, this computer program implements any of the deterministic trajectory pair preference optimization methods for the flow matching model in the above embodiments.

[0186] It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. All other embodiments derived by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0187] Obviously, the accompanying drawings are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar situations based on these drawings without any creative effort. Furthermore, it is understood that although the work done in this development process may be complex and lengthy, for those skilled in the art, certain design, manufacturing, or production modifications made based on the technical content disclosed in this application are merely conventional technical means and should not be considered as insufficient disclosure of this application.

[0188] The term "embodiment" in this application refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily imply the same embodiment, nor does it imply that it is mutually exclusive with or independent of other embodiments. It will be clearly or implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0189] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims

1. A deterministic trajectory pair preference optimization method for flow matching models, characterized in that, The method includes: Based on the input prompt text, the generation quality of each deterministic trajectory in the trajectory group is evaluated to obtain the evaluation result; the trajectory group is generated by the current flow matching model based on deterministic ODE sampling. Based on the evaluation results, the deterministic trajectory is divided into a main trajectory and a reference trajectory; the main trajectory is the trajectory that participates in gradient update; the reference trajectory serves as a comparison target and does not participate in gradient update. Based on the main trajectory and the reference trajectory, the current flow matching model is optimized to obtain a preference alignment model.

2. The deterministic trajectory pair preference optimization method for flow matching models according to claim 1, characterized in that, Based on the input prompt text, the generation quality of each deterministic trajectory in the trajectory group is evaluated, and the evaluation results are obtained, including: The input prompt text and each deterministic trajectory in the trajectory group are input into a preset reward model to calculate a reward score that describes the generation quality of each deterministic trajectory. The reward score is normalized within the trajectory group to obtain the relative advantage of each deterministic trajectory within the group; The evaluation result is obtained based on the reward score and the relative advantage within the group.

3. The deterministic trajectory pair preference optimization method for flow matching models according to claim 2, characterized in that, The trajectory group acquisition process includes: Obtain a shared fundamental noise and multiple independent perturbation noises; Based on a preset noise disturbance coefficient, the shared basic noise and multiple independent disturbance noises are combined to obtain a set of initial noise; The initial noise is input into the current flow matching module, and a trajectory group containing multiple deterministic trajectories is generated based on deterministic ODE sampling.

4. The deterministic trajectory pair preference optimization method for flow matching models according to claim 2, characterized in that, Based on the evaluation results, the deterministic trajectory is divided into a main trajectory and a reference trajectory, including: Based on the reward score or the relative advantage within the group, the deterministic trajectories are sorted within the trajectory group; In the sorting results, the deterministic trajectory located at the middle end is marked as the main trajectory, and the remaining deterministic trajectories are marked as reference trajectories.

5. The deterministic trajectory pair preference optimization method for flow matching models according to claim 2, characterized in that, Based on the main trajectory and the reference trajectory, the current flow matching model is optimized to obtain a preference alignment model, including: Take one of the main trajectories and one of the reference trajectories as a trajectory pair; For each pair of trajectories, the current distance score is obtained based on the distance between the trajectories, and the corresponding preference direction is obtained based on the difference in relative advantage within the group. Based on the current distance score and the preference direction, an optimization objective is constructed; Based on the optimization objective, the current flow matching model is iteratively trained to obtain a preference alignment model.

6. The deterministic trajectory pair preference optimization method for flow matching models according to claim 5, characterized in that, Based on the current distance score and the preference direction, an optimization objective is constructed, including: Obtain the old distance score corresponding to the old model; the old model is the previous model version that is immediately adjacent to the current model during iterative training; Based on the difference between the current distance score function and the old distance score, a deterministic comparison value is obtained; An optimization objective is constructed based on the deterministic ratio and the preference direction.

7. The deterministic trajectory pair preference optimization method for flow matching models according to claim 6, characterized in that, Based on the deterministic comparison value and the preference direction, an optimization objective is constructed, including: The first product is obtained by multiplying the preference direction by the deterministic comparison value of the unpruned determinism; A second product is formed by multiplying the preference direction with the comparative value of the determinedness after clipping; The minimum value between the first product and the second product is used as the optimization objective.

8. The deterministic trajectory pair preference optimization method for flow matching models according to claim 5, characterized in that, Based on the optimization objective, the current flow matching model is iteratively trained to obtain a preference alignment model, including: Based on the optimization objective, the model parameters of the current flow matching model are updated; After the model parameters are updated, determine whether the preset number of training rounds has been reached; If the training rounds are reached, the preference alignment model is output based on the current model parameters; if the training rounds are not reached, iterative training continues.

9. A deterministic trajectory pair preference optimization device for flow matching models, characterized in that, The device includes: The quality evaluation module is used to evaluate the generation quality of each deterministic trajectory in the trajectory group based on the input prompt text, and obtain the evaluation result; the trajectory group is generated by the current flow matching model based on deterministic ODE sampling; The trajectory segmentation module is used to segment the deterministic trajectory into a main trajectory and a reference trajectory based on the evaluation results; the main trajectory is the trajectory that participates in gradient update; the reference trajectory serves as a comparison target and does not participate in gradient update. The model optimization module is used to perform preference optimization on the current flow matching model based on the main trajectory and the reference trajectory to obtain a preference alignment model.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

Images