A post-training method for models that integrates slow reward ranking calibration and intra-group policy optimization

Abstract

This invention discloses a post-training method for models that integrates slow reward ranking calibration and intra-group policy optimization, belonging to the fields of artificial intelligence and computer vision technology. Addressing the industry pain points of high-precision physics simulation engines being non-differentiable and slow in computation, preventing their direct participation in online model training backpropagation, and the disconnect between fast and slow reward evaluation standards, this invention proposes: constructing a candidate video library, calculating high-precision slow rewards based on physics simulation and fast rewards based on surrogate metrics respectively; constructing a ranking calibrator, training fast rewards based on intra-group ranking of slow rewards, so that the lightweight fast reward intra-group ranking approximates the ranking preference of high-precision slow rewards, obtaining calibrated fast rewards; and using the calibrated fast rewards as differentiable training signals for online fine-tuning. This invention also includes supervised fine-tuning warm-up and low-risk offline stream matching adaptation stages, and employs double KL divergence constraint intra-group policy optimization with the removal of traditional SFT anchors in the online fine-tuning stage.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and computer vision technology, specifically relating to a post-training method for a model that integrates slow reward ranking calibration and intra-group policy optimization. Background Technology

[0002] Physics-aware video generation is an interdisciplinary research direction combining physical simulation and generative models. Its core goal is to ensure that the generated video content conforms to real-world physical laws. Currently, in business scenarios such as virtual human live streaming, film and animation, and virtual try-on, there is an urgent need to generate videos where the character's identity is stable and the fabric movement conforms to real-world physical laws (such as drooping, fluttering, and kinetic energy decay). Existing methods often employ diffusion models combined with supervised learning to achieve this.

[0003] However, existing technologies have the following main drawbacks:

[0004] 1. There is an inherent contradiction between physical simulation and online training: high-precision physical simulation engines (such as ClothSim) have slow calculation speed (single frame simulation exceeds 100ms), and the engine itself is not differentiable, so it cannot be directly embedded into the backpropagation of training iteration. Forcing it in will cause the time consumption of a single round of training to increase exponentially, making it impossible to scale up and deploy.

[0005] 2. The single reward mechanism has a "score-boosting trap": using only lightweight proxy rewards (fast rewards) will ignore the real physical logic, resulting in physical anomalies such as cloth shaking and flying in the generated video; if high-precision physical rewards (slow rewards) are used directly, the computational cost is too high and cannot meet the training efficiency requirements.

[0006] 3. Direct training of reinforcement learning is prone to model collapse: Conventional GRPO / PPO algorithms lack effective policy drift constraint mechanisms in video generation scenarios. The model is prone to deviating from its basic capabilities in order to "boost scores", which will eventually lead to training failure problems such as identity distortion, video flickering, and parameter divergence.

[0007] 4. Disconnect between fast and slow reward standards: There is a lack of ranking calibration between fast agent rewards and slow physical rewards. The ranking of the two is incompatible, which means that high-precision physical evaluation signals cannot be converted into gradient signals that can be used for online training of the model, and physical realism cannot be quantitatively improved. Summary of the Invention

[0008] To address the industry pain points of existing technologies where high-precision physical simulation evaluation (slow reward) suffers from high computational cost, non-differentiability, inability to directly participate in model backpropagation, and disconnection from fast surrogate reward evaluation standards, resulting in poor physical realism, low training efficiency, and susceptibility to model crashes, this invention provides a model post-training method that integrates slow reward ranking calibration and intra-group policy optimization. The core technical problem this invention aims to solve is: how to transform non-differentiable and computationally expensive physical simulation evaluation signals into efficiently usable differentiable online training signals, while ensuring stable model training without sacrificing fundamental generative capabilities.

[0009] The specific technical solution is as follows:

[0010] A post-training method for models that integrates slow reward ranking calibration and intra-group policy optimization includes the following steps:

[0011] The diffusion model in the physics-driven video generation task is used as the baseline model. ;

[0012] Construct a candidate video library, which contains candidate videos generated by the baseline model under multiple input conditions;

[0013] For the candidate videos, a physics simulation engine is used to calculate high-precision and non-differentiable slow rewards. And use agent evaluation metrics to calculate low-precision and differentiable fast rewards. ;

[0014] Construct a ranking calibrator, determine the fast and slow reward preference pairs within the group based on the ranking of the slow reward within the input condition group, and train the ranking calibrator using the preference pairs;

[0015] The fast reward is calibrated using the trained sorting calibrator to obtain the calibrated fast reward. This makes the intra-group ranking of the fast reward approximate the intra-group ranking of the slow reward after calibration.

[0016] Utilizing the aforementioned calibration fast reward The diffusion model is trained online as a training signal, and the ranking calibrator includes learnable weight parameters, which are obtained through training based on a pairwise ranking optimization objective of the preference pairs.

[0017] In some embodiments, prior to constructing the candidate video library, a step of supervised fine-tuning SFT pre-training of the baseline model is further included:

[0018] Using the full training set at the business level, the baseline model was tested. Optimize the objective function. for:

[0019] ,

[0020] in, To reconstruct the difference weights, To account for the L1 reconstruction loss, the similarity between the video footage and the real labels is constrained; For identity deviation weight, For identity consistency loss, cosine similarity is calculated. As the weight for time series deviation, The temporal consistency loss is calculated based on optical flow stability.

[0021] In some embodiments, the fast reward The calculation formula is:

[0022] ,

[0023] in, The total number of dimensions of the reward agent features. Let m be the weight of the business feature in dimension m. The m-th surrogate feature value includes identity preservation, temporal stability, and structural matching features; Let KL divergence penalty coefficient be denoted as . This is a KL divergence penalty term;

[0024] The slow reward The calculation formula is:

[0025] ,

[0026] in, The physical safety dynamics reward is used to filter invalid videos such as simulation divergence and NaN pixels; The slow reward is a consistent reward for the decay of kinetic energy after the fabric is released, used to measure the realism of physical motion; the slow reward is only used for sorting calibration and offline evaluation, and does not participate in the backpropagation of the online training.

[0027] In some embodiments, in the step of training the ranking calibrator using the preference pair, a pairwise ranking optimization objective is employed. :

[0028] ,

[0029] in, , The first The first in the group The initial fast reward score of k candidate videos, For the first The set of preference sample pairs within each input condition group The L2 regularization coefficient is... The weight parameters can be learned for the sorting calibrator network. For calibrator parameters, use the L2 regularization term;

[0030] When the Spearman correlation coefficient is used to evaluate the calibration effect of the ranking calibrator and the preset threshold is reached, the training of the ranking calibrator is determined to be complete.

[0031] In some embodiments, an offline adaptation phase is included before online training of the diffusion model using the calibrated fast reward:

[0032] An offline adaptation candidate library is constructed, wherein the samples in the offline adaptation candidate library are generated by new input conditions and do not reuse the samples in the candidate library constructed in the calibration stage;

[0033] The candidate videos in the offline adaptation candidate library are normalized and weighted using the slow reward, and the weighting formula is as follows:

[0034] ,

[0035] Weighted Flow Matching Loss The baseline model is fine-tuned, and the objective function is:

[0036] ,

[0037] in, For the first The slow reward normalized weight of the j-th candidate video in the group, For the basic loss of Flow Matching, For the model to select candidate videos The predicted output, To ensure accurate annotation of the reference video, For the first The slow reward score for the j-th candidate video in the group.

[0038] In some embodiments, the online training is performed based on the model from the offline adaptation phase, using an intra-group policy optimization algorithm, which includes:

[0039] For each set of input conditions, sample K candidate videos and calculate the calibrated fast reward value for the K candidate videos. ;

[0040] Based on within-group mean and standard deviation Calculate the within-group advantage standardized function :

[0041] ,

[0042] in, For the first The average of all candidate rewards within the group. The number of candidate videos sampled in each group. To ensure numerical stability and prevent the denominator from being zero, the calculation results of the intra-group advantage are standardized. As weights, they are applied synchronously to each time step of the diffusion model's denoising process to address the temporal credit allocation problem in multi-step iterations.

[0043] In some embodiments, the within-group policy optimization algorithm employs double KL divergence regularization constraints, and its total reward optimization objective is... for:

[0044] ,

[0045] in, These are the weighting coefficients for the primary objective of RL. The GRPO strategy within the group is used as a proxy reward to widen the generation probability difference between high-scoring and low-scoring samples within the group; The KL constraint coefficients are used as a reference model. For the current strategy and the frozen baseline model KL divergence; The KL constraint coefficients of the old strategy control the single-step update magnitude. The KL divergence between the current strategy and the old strategy from the previous iteration;

[0046] The dual KL divergence regularization constraint replaces the supervised fine-tuning SFT loss anchor in online reinforcement learning to prevent policy drift and catastrophic forgetting during parameter updates in the diffusion model.

[0047] In some embodiments, during the online training process, the dynamic refresh step of the sorting calibrator is repeated once every preset training round:

[0048] The validation set candidate videos are resampled using the latest version of the diffusion model, the slow and fast rewards are recalculated, and the learnable weight parameters of the ranking calibrator are retrained and updated to ensure the fast reward is correct after calibration. Always adhere to the slow reward Physical evaluation criteria.

[0049] In some embodiments, the online training stops when a preset termination condition is met, which includes any one or more of the following: the physical accuracy of the generated video meets the standard, the consistency of the person's identity meets the standard, the temporal stability meets the standard, and the loss function of the diffusion model converges to a set threshold; after training stops, the final physical-driven video generation commercial model is output.

[0050] In some embodiments, the physics simulation engine is the ClothSim cloth physics simulation engine, and the slow reward... The engine performs physical security checks and calculates the consistency of kinetic energy decay after release on the candidate videos.

[0051] Both the candidate video library and the offline adaptation candidate library are generated based on the physical condition constraint package (including reference identity image, normal vector video, velocity video and acceleration video) of virtual human live broadcast, film and television cloth animation or virtual try-on.

[0052] Compared with the prior art, the present invention has at least the following technical effects:

[0053] 1. Resolves the contradiction between physical simulation and online training: By using Slow-to-Fast sorting calibration, the non-differentiable and high-cost slow reward is transformed into a differentiable fast reward after calibration. The entire process does not require the slow reward to participate in backpropagation, which significantly reduces the training computing power and time cost while ensuring the accuracy of physical evaluation.

[0054] 2. Strong training stability and anti-collapse resistance: It adopts a phased (warm-up-calibration-offline adaptation-online fine-tuning) progressive training mode, and removes the traditional SFT anchor point that limits the optimization space in the online stage. Instead, it uses a double KL regularization constraint strategy with a frozen reference model to prevent drift, effectively preventing the degradation of the model's basic capabilities, identity distortion and collapse.

[0055] 3. Alignment of physical evaluation with online optimization: By constructing an intra-group preference pair training ranking calibrator, the fast reward completely replicates the ranking relationship of the slow reward, avoiding the "score-boosting trap" of a single agent reward, and ensuring that the generated video is significantly better than the traditional baseline in terms of physical dynamic rationality (such as the consistency of kinetic energy decay). Attached Figure Description

[0056] Figure 1 The following is a flowchart illustrating the four-stage implementation of this invention.

[0057] Figure 2 This is a visual comparison of the generation effects of Embodiment 1 of the present invention and five mainstream existing technical methods. The figures include a comparison of the effects under the tasks of pulling down the hem of clothing and blowing the skirt in the wind. Detailed Implementation

[0058] Various exemplary embodiments, features, and aspects of this application will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0059] Example 1: Four-stage progressive training process

[0060] This embodiment combines Figure 1 The flowchart shown illustrates a complete embodiment of the present invention in detail.

[0061] 1. Task Input and Output:

[0062] This invention addresses the task of generating human portrait videos driven by physical conditions such as virtual humans and film / television fabrics. It clarifies the input and output specifications to adapt to the data formats and requirements in actual business scenarios. The specific definitions are as follows:

[0063] Input conditional package Each component is designed to meet actual business needs:

[0064] Reference Identity Image Single high-resolution portrait image (resolution) or (RGB format) is used to constrain the consistency of the identity of people in the generated video (such as the anchor's identity remaining unchanged in virtual human live broadcasts, and the facial features of people remaining stable in virtual try-on), so as to avoid identity drift affecting the business experience.

[0065] Normal vector rendering video Timing length is Video sequence (frame rate) Single frame resolution and (Consistent), each frame is a normal vector map, used to constrain the general deformation structure of the character's body and the fabric in the generated video (such as the folds of film and television fabrics, and the pattern of clothing in virtual try-on).

[0066] Speed ​​rendering video With acceleration rendering video All are time series lengths of Dynamic signal video (frame rate and (Consistent), respectively used to constrain the movement direction, speed and acceleration changes of the character and the cloth (such as the limb speed of the virtual person walking, the acceleration decay of the cloth when it is blown), to ensure that the movement conforms to the laws of nature.

[0067] Model output: Time series length is ( RGB character cloth movement video (seconds, tailored to the needs of short videos and virtual live streaming segments in business) , denoted as:

[0068] ,

[0069] in The parameter is The diffusion video generation model (adopting a variant architecture of DDPM to adapt to the temporal characteristics of video generation and meet the high-definition and high-coherence requirements in business).

[0070] Core constraint: Identity preservation constraint (character identity and...) Constraints include: high consistency, conditional alignment (move matches input conditions), temporal stability (no flickering or jumps), physical reliability (cloth movement conforms to physical laws), and training stability (no collapse or policy drift).

[0071] 2. Stage 0: Supervised Fine-tuning Warm Start (SFT Warm Start):

[0072] This stage is the foundation of the entire training process. Its core purpose is to enable the diffusion model to first master the basic ability of physically driven video generation, thus avoiding problems such as model collapse, identity distortion, and temporal discontinuity in the subsequent reinforcement learning stage due to weak basic capabilities. In line with the industry's implementation logic of "first achieve stable generation, then optimize physical accuracy," this stage prioritizes ensuring three basic business indicators: consistency of virtual human identity, video image clarity, and inter-frame temporal continuity. This provides a stable benchmark model for subsequent calibration and optimization stages, avoiding ineffective training.

[0073] Training configuration:

[0074] Dataset selection: A full business-grade training set is used, covering mainstream virtual human images, common fabric materials such as cotton, linen, and silk, as well as conventional limb movements and fabric deformation trajectories. The input condition package and the real physical driving video are strictly paired and labeled, with no label noise, which meets the accuracy requirements of supervised learning and avoids model learning bias caused by poor data.

[0075] Optimizer and training strategy: The AdamW optimizer, which is commonly used for training AI generative models, is selected. The initial learning rate is set to 1e-5 and the weight decay is 1e-4 to avoid gradient oscillation. A 10-epoch scheme of "2 warm-up rounds + 8 stable training rounds" is adopted. Small steps are taken in the early stage to avoid parameter mutations, and stable learning is carried out in the later stage to converge basic features. It is suitable for small and medium computing power devices and can complete the training without high-end GPU clusters.

[0076] Key objective function:

[0077] ,

[0078] The weighting parameters are aligned with business priority settings: Reconstruct differential weights. (Ensuring video image fidelity), and weighting of identity deviations. (Strictly prevent virtual identity drift), time-series deviation weighting (Ensuring smooth, flicker-free video frames) The three major indicators work together to constrain the model, enabling it to accurately learn the basic mapping relationship between "input physical conditions and output compliant video".

[0079] Phase transition and termination criteria: When the joint objective value of the validation set reaches stability for three consecutive rounds, and the identity consistency score is ≥0.85, the temporal stability score is ≥0.8, and the structure matching score is ≥0.7, training for this phase is stopped, and the baseline model is saved. It directly connects to the next stage of sorting and calibration, without the need for additional model debugging.

[0080] 3. Stage 1: Candidate Library Construction and Slow-to-Fast Ranking Calibration (Bridging the Gap Between Physical Truth and Online Training):

[0081] This stage is the core of the entire invention, primarily addressing the industry challenges of high-precision physical simulation (slow reward) being non-differentiable, computationally slow, and unable to directly participate in online backpropagation. The core function of this stage is "translation + calibration": reducing the slow reward to a "physical truth calibration standard," without participating in any online gradient calculations. Only through a ranking calibration algorithm, the lightweight fast reward accurately replicates the intra-group ranking preferences of the slow reward, bridging the technical barriers between "high-precision physical evaluation" and "efficient online training." This achieves "using physical truth to define the standard and using fast rewards for training," transforming "unusable physical truth standards" into "signals directly usable for model training." This stage retains the physical realism standards required for film-level and high-end virtual fitting scenarios while minimizing the computational overhead of physical simulation. It solves the pain point of small and medium-sized enterprises being unable to perform high-precision physical optimization due to insufficient computing power, while also preventing the model from falling into the training trap of "farming shallow reward scores and ignoring physical logic."

[0082] Implementation steps:

[0083] Calibration candidate library construction: 100 typical business conditions were selected from the business-level verification set, covering different virtual humans, different fabric materials, and different motion amplitudes. Four candidate videos were sampled for each condition using the Stage 0 benchmark model. The two videos with the best quality were selected to calculate the slow reward, which ensured the representativeness of the samples and avoided invalid samples from interfering with the calibration accuracy.

[0084] Fast and slow reward division of labor:

[0085] (1) Fast Reward: Designed for high-throughput requirements of online training, it is lightweight and can be batch-computed. It extracts three key business features: identity preservation, temporal stability, and structure matching. After calibration, a policy drift penalty is added. The core formula is:

[0086] ,

[0087] Its function is to provide stable and efficient gradient signals for subsequent online optimization, participating in backpropagation throughout the entire process. Specifically:

[0088] : Fast reward after calibration, lightweight and differentiable, used for online training backpropagation;

[0089] : Total number of dimensions for fast reward agent features;

[0090] Quick reward number Dimensional business feature weights;

[0091] : No. Dimensional proxy features include identity preservation, temporal stability, and structural matching;

[0092] KL divergence penalty coefficient, the strength of drift constraint for the control strategy;

[0093] : KL divergence penalty term, penalizing the distribution offset between the penalty model and the baseline policy;

[0094] (2) Slow Reward: Based on ClothSim's professional physics simulation engine, it serves as the physical truth standard. Its core includes two main indicators: safety gating (filtering out simulation divergence, NaN pixels, and other invalid videos) and cloth kinetic energy decay consistency (measuring physical realism). The core formula is: Its function is to determine the physical quality of candidate videos; it does not participate in any gradient calculations during online training. Specifically:

[0095] Physical simulation has a slow reward, high accuracy, and is non-differentiable; it is only used for sorting, calibration, and evaluation.

[0096] Physical safety dynamics reward, filtering simulation divergence, NaN, and invalid anomaly videos;

[0097] The consistent reward for kinetic energy decay after the cloth is released measures the realism of physical motion.

[0098] (3) Core process of ranking calibration: Based on the slow reward score, construct within-group preference pairs, standardize the fast reward features, and learn the calibration weights through pairwise ranking optimization to make the within-group ranking of fast rewards completely match that of slow rewards. The core calibration target formula is: The Spearman correlation coefficient is used to measure the calibration effect, with a target of ≥0.7 to ensure that the calibration accuracy meets the standard. Simultaneously, a dynamic refresh mechanism for the calibrator is reserved to pave the way for accuracy iterations in subsequent online optimization phases. Specifically:

[0099] : Fast / slow reward ranking calibration loss, achieving ranking alignment based on preference pair learning;

[0100] : No. The first in the group The initial fast reward score for each candidate video;

[0101] : No. A set of preference sample pairs within each input condition group;

[0102] L2 regularization coefficient, used to prevent overfitting of the calibration network;

[0103] The sorting calibrator network can learn weight parameters;

[0104] : Calibrator parameter L2 regularization term.

[0105] Stage transition and termination criteria: After the calibrator target converges and the ranking correlation coefficient meets the standard, the calibration weight is saved, the standard unification of "physical truth value - fast reward" is completed, and the next stage of offline adaptation is smoothly connected to realize the transition from basic generation to physical preference learning.

[0106] 4. Stage 2: Offline Adaptation (Low-Risk Transition to Avoid Reinforcement Learning Collapse Risk):

[0107] The core objective of this stage is to enable the model to initially learn physical preferences based on the calibrated fast reward and candidate library through low-risk Reward-weighted Flow Matching, thereby achieving a smooth transition from SFT to online RL training and avoiding policy mutations and breakdowns caused by directly entering RL training.

[0108] Implementation steps:

[0109] 1. Offline candidate library construction: 150 new business conditions are resampled without reusing the calibration phase samples to ensure sample diversity. Four candidate videos are sampled for each condition. A weighted training set is constructed by slow reward scoring to allow the model to learn samples with higher physical authenticity first.

[0110] 2. Core optimization logic:

[0111] Slow reward normalized weights Weighted training allows the model to gradually optimize towards physical reality, while strictly constraining identity consistency and temporal stability metrics to ensure that basic business capabilities do not degrade.

[0112] The objective function is:

[0113] ,

[0114] in:

[0115] Slow reward-weighted flow matching offline adaptation loss;

[0116] : No. Group 1 Slow reward normalized weights for each candidate video;

[0117] Flow Matching is the basic loss function that fits the continuous generation process of diffusion.

[0118] Model for candidate videos The predicted output;

[0119] : Real-world reference video annotations;

[0120] : No. Group 1 Each candidate video receives a slow reward score.

[0121] Stage transition and termination criteria: The model objective converges and the physical accuracy improves by more than 15% compared to the base model; the offline adaptation model is then saved. As the initial strategy for final online optimization, it perfectly connects with the core reinforcement learning process.

[0122] 5. Stage 3: Online Pure GRPO Intra-group Policy Optimization (Improving Physical Accuracy and Training Efficiency): This stage is the final performance enhancement phase. Its core function is to utilize the pre-calibrated fast reward signal for online fine-tuning using an improved intra-group GRPO algorithm. Each time, only a few candidate videos under the same input conditions are compared, accurately selecting the physically more realistic ones for optimization. This reduces computational waste, accelerates training speed, and constrains the model to avoid deviation, ensuring training stability. This stage abandons the traditional SFT constraint anchor point of reinforcement learning, instead using a KL regularization constraint strategy with a frozen reference model for drift correction. Combined with intra-group sample comparison optimization, this significantly reduces memory overhead and improves convergence speed.

[0123] Implementation steps:

[0124] 1. Intra-group sampling strategy: In each round, 50 sets of input conditions covering all business scenarios are selected, and K candidate videos are sampled from each set. Only the ranking comparison within the same condition group is performed to avoid reward bias caused by the difference in difficulty across conditions. This allows the model to accurately learn the differences in physical quality and makes the optimization direction clearer.

[0125] 2. Standardization of intra-group strengths:

[0126] ,

[0127] Eliminating the inherent differences between samples with similar conditions makes the advantage estimation more reasonable, further improving training stability and avoiding gradient oscillations. Specifically:

[0128] Within-group standardized dominance function to eliminate inherent differences among samples under the same conditions; : No. Group 1 The fast reward value after calibration of each candidate video; : No. The average reward of all candidate rewards within the group; Standard deviation of group rewards; Number of candidate videos sampled in each group; Numerical stability is maintained at a small value to prevent the denominator from being zero. The calculated results of the intra-group advantage are then standardized. As weights, they are applied synchronously to each time step of the diffusion model's denoising process to address the temporal credit allocation problem in multi-step iterations.

[0129] 3. Intra-group strategy constraints and overall optimization objective: in, Optimize the overall reward objective (with maximizing this objective as the optimization direction). It is a proxy reward objective function for the within-group policy constructed based on the calibrated fast reward; maximizing this proxy reward objective is equivalent to increasing the generation probability of high-reward samples within the group and decreasing the generation probability of low-reward samples. The traditional SFT anchor point is removed, and double KL regularization is used to constrain policy drift, ensuring that the model does not lose its basic capabilities while fully releasing the optimization space of reinforcement learning; at the same time, within-group policy ratio normalization is added to solve the policy drift problem in the denoising step of the diffusion model, further improving training stability. The overall reward objective for online reinforcement learning (RL) is to maximize this objective to achieve strategy optimization. : RL primary objective weight coefficient; The GRPO strategy within the group provides agent rewards, widening the probability difference between generating high-scoring and low-scoring samples within the group. : Reference model KL constraint coefficients to prevent deviation from the SFT baseline model; Current strategy and frozen baseline model KL divergence; : KL constraint coefficient of the old strategy, which controls the single-step update range; : KL divergence between the current strategy and the old strategy in the previous iteration.

[0130] The dual KL divergence regularization constraint replaces the supervised fine-tuning (SFT) loss anchor in online reinforcement learning to prevent policy drift and catastrophic forgetting in the diffusion model during parameter updates.

[0131] 4. Dynamic calibration and termination: The calibrator is refreshed every 5 rounds to ensure that the fast reward always matches the slow reward standard; when physical accuracy, identity consistency and temporal stability all meet the standards, training is stopped and the final commercial model is saved.

[0132] 6. Experimental Results:

[0133] To verify the advancement of the proposed "model post-training method integrating slow reward ranking calibration and intra-group policy optimization" in the physics-driven video generation task, this experiment selected five current mainstream generation models or physics control frameworks as comparative examples.

[0134] Comparative Example 1: ForcePrompting: Based on the traditional text-to-video (T2V) diffusion model architecture, training mainly relies on simple supervised fine-tuning (SFT) using conventional mean squared error (MSE) or stream matching loss. Experimental Results: In the "pull down the hem" and "wind blowing the skirt" tasks, the fabric hardly produced any obvious physical deformation, resulting in stiff images that failed to realistically respond to external force feedback.

[0135] Comparative Example 2: PhysCtrl: Employs a diffusion model that introduces additional control branches (such as ControlNet variants). Experimental Results: Severe generation crashes occur during complex physical deformations (huge black artifacts appear in the image), even compromising basic video generation capabilities.

[0136] Comparative Example 3: InterDyn: Focuses on interactive dynamic generation, with network structures largely relying on multi-frame autoregression or Transformer-based temporal prediction networks. Experimental Results: Under physical interaction, facial features are easily lost, resulting in completely blurred and distorted faces, and unnaturally deformed skirts.

[0137] Comparative Example 4: PhysGen3D: A generative framework based on 3D perception. Experimental Results: When faced with external physical forces, it exhibits catastrophic structural collapse, with extremely bloated character torsos and severe artifacts such as "decapitation" or bizarrely flying off body parts.

[0138] Comparative Example 5: MPMAvatar: A 3D digital human reconstruction architecture combining the Material Point Method (MPM) or a mass-spring system. Experimental Results: Although it can produce some physical deformation, it faces a severe identity loss problem, with significant abrupt changes in the facial features of the person.

[0139] Experimental results of this invention: such as Figure 2 As shown, in the tasks of "pulling down the hem of the clothes" and "wind blowing the skirt", the fabric deforms naturally, the character's identity remains clear, and there are no abnormalities such as black artifacts, facial distortion, or body parts flying out inexplicably. In terms of physical realism and identity stability, it is significantly better than all the above-mentioned comparative technologies, proving the advanced nature of the present invention in real-world scenarios.

[0140] This invention addresses the core pain point in physics-driven video generation (such as virtual human live streaming and film cloth animation): the desire to ensure physical realism in the video (natural cloth movement, no clipping) while simultaneously enabling fast and stable model training. This is achieved through two crucial, complementary operations. The first half, sorting and calibration, acts as a "translation and calibration" mechanism, connecting the non-differentiable slow reward with the differentiable training signal. The second half, intragroup GRPO, improves training efficiency and stability. The core logic of this invention's technical solution is as follows: First, the basic generation capability of the model is warmed up through supervised fine-tuning. Then, slow rewards are used to rank candidate videos within a group. A ranking calibration algorithm is used to make the fast rewards approximate the ranking preferences of the slow rewards within the group—this step is the core embodiment of "translation + calibration," transforming the unusable physical reality standard (slow rewards) into a signal that can be directly used for model training (calibrated fast rewards). Finally, based on the calibrated fast rewards, the model parameters are optimized using an in-group GRPO strategy, while introducing policy drift constraints. That is, only a few candidate videos under the same input conditions are compared each time, accurately selecting the more physically realistic ones for optimization, forming a complete closed loop of "warm-up-calibration-optimization." The entire process does not require slow rewards to directly participate in online backpropagation, saving time and computing power while ensuring the physical authenticity of the generated videos, and achieving fast and stable training.

[0141] The above embodiments illustrate in detail the specific implementation of the technical solution of the present invention, the logical and connection relationships of each component, and the complete working process. Those skilled in the art will understand that various changes and modifications can be made to the above embodiments without departing from the principles and spirit of the present invention, and all such changes and modifications should fall within the protection scope of the appended claims.

Claims

1. A model post-training method that integrates slow reward ranking calibration and within-group policy optimization, characterized in that, Includes the following steps: The diffusion model in the physics-driven video generation task is used as the baseline model. ; Construct a candidate video library, which contains candidate videos generated by the baseline model under multiple input conditions; For the candidate videos, a physics simulation engine is used to calculate high-precision and non-differentiable slow rewards. And use agent evaluation metrics to calculate low-precision and differentiable fast rewards. ; Construct a ranking calibrator, determine the fast and slow reward preference pairs within the group based on the ranking of the slow reward within the input condition group, and train the ranking calibrator using the preference pairs; The fast reward is calibrated using the trained sorting calibrator to obtain the calibrated fast reward. This makes the intra-group ranking of the fast reward approximate the intra-group ranking of the slow reward after calibration. Utilizing the aforementioned calibration fast reward The diffusion model is trained online as a training signal, and the ranking calibrator includes learnable weight parameters, which are obtained through training based on a pairwise ranking optimization objective of the preference pairs.

2. The post-training method for the model according to claim 1, characterized in that, Before constructing the candidate video library, the method also includes a supervised fine-tuning SFT warm-up training step for the baseline model: Using the full training set at the business level, the baseline model was tested. Optimize the objective function. for: ， in, To reconstruct the difference weights, To account for the L1 reconstruction loss, the similarity between the video footage and the real labels is constrained; For identity deviation weight, For identity consistency loss, cosine similarity is calculated. As the weight for time series deviation, The temporal consistency loss is calculated based on optical flow stability.

3. The post-training method for the model according to claim 1, characterized in that, The fast reward The calculation formula is: ， in, The total number of dimensions of the reward agent features. Let m be the weight of the business feature in dimension m. The m-th surrogate feature value includes identity preservation, temporal stability, and structural matching features; Let KL divergence penalty coefficient be denoted as . This is a KL divergence penalty term; The slow reward The calculation formula is: ， in, The physical safety dynamics reward is used to filter invalid videos such as simulation divergence and NaN pixels; The slow reward is a consistent reward for the decay of kinetic energy after the fabric is released, used to measure the realism of physical motion; the slow reward is only used for sorting calibration and offline evaluation, and does not participate in the backpropagation of the online training.

4. The post-training method for the model according to claim 1, characterized in that, In the step of training the ranking calibrator using the preference pair, a pairwise ranking optimization objective is adopted. : ， in, , The first The first in the group The initial fast reward score of k candidate videos, For the first The set of preference sample pairs within each input condition group The L2 regularization coefficient is... The sorting calibrator network can learn weight parameters. For calibrator parameters, use the L2 regularization term; When the Spearman correlation coefficient is used to evaluate the calibration effect of the ranking calibrator and the preset threshold is reached, the training of the ranking calibrator is determined to be complete.

5. The post-training method for the model according to claim 1, characterized in that, Before using the calibrated fast reward to train the diffusion model online, an offline adaptation phase is also included: An offline adaptation candidate library is constructed, wherein the samples in the offline adaptation candidate library are generated by new input conditions and do not reuse the samples in the candidate library constructed in the calibration stage; The candidate videos in the offline adaptation candidate library are normalized and weighted using the slow reward, and the weighting formula is as follows: ， Weighted Flow Matching Loss The baseline model is fine-tuned, and the objective function is: ， in, For the first The slow reward normalized weight of the j-th candidate video in the group, For the basic loss of Flow Matching, For the model to select candidate videos The predicted output, For accurate annotation of reference videos, For the first The slow reward score for the j-th candidate video in the group.

6. The post-model training method according to claim 5, characterized in that, The online training is based on the model from the offline adaptation phase and is performed using an intra-group policy optimization algorithm, which includes: For each set of input conditions, sample K candidate videos and calculate the calibrated fast reward value for the K candidate videos. ; Based on within-group mean and standard deviation Calculate the within-group advantage standardized function : ， in, For the first The average of all candidate rewards within the group. The number of candidate videos sampled in each group. To ensure numerical stability and prevent the denominator from being zero, the calculation results are standardized by considering the group's dominance. As weights, they are applied synchronously to each time step of the diffusion model's denoising process to address the temporal credit allocation problem in multi-step iterations.

7. The post-model training method according to claim 6, characterized in that, The intra-group policy optimization algorithm employs double KL divergence regularization constraints, and its overall reward optimization objective is... for: ， in, These are the weighting coefficients for the primary objective of RL. The GRPO strategy within the group is used as a proxy reward to widen the generation probability difference between high-scoring and low-scoring samples within the group; The KL constraint coefficients are used as a reference model. For the current strategy and the frozen baseline model KL divergence; The KL constraint coefficients of the old strategy control the single-step update magnitude. The KL divergence between the current strategy and the old strategy from the previous iteration; The dual KL divergence regularization constraint replaces the supervised fine-tuning SFT loss anchor in online reinforcement learning to prevent policy drift and catastrophic forgetting during parameter updates in the diffusion model.

8. The post-training method for the model according to claim 1, characterized in that, During the online training process, the dynamic refresh step of the sorting calibrator is repeated once every preset training round: The validation set candidate videos are resampled using the latest version of the diffusion model, the slow and fast rewards are recalculated, and the learnable weight parameters of the ranking calibrator are retrained and updated to ensure the fast reward is correct after calibration. Always adhere to the slow reward Physical evaluation criteria.

9. The post-training method for the model according to claim 1, characterized in that, The online training stops when a preset termination condition is met. The preset termination condition includes any one or more of the following combinations: the physical accuracy of the generated video meets the standard, the consistency of the person's identity meets the standard, the temporal stability meets the standard, and the loss function of the diffusion model converges to a set threshold. After training stops, the final physical-driven video generation commercial model is output.

10. The post-training method for the model according to claim 1, characterized in that, The physics simulation engine is the ClothSim cloth physics simulation engine, and the slow reward... The engine performs physical security checks and calculates the consistency of kinetic energy decay after release on the candidate videos. Both the candidate video library and the offline adaptation candidate library are generated based on the physical condition constraint package of virtual human live streaming, film and television cloth animation, or virtual try-on.

Images