Identity preserving video generation method and system based on reinforcement evolution

Abstract

The application discloses an identity-preserving video generation method and system based on reinforcement evolution, comprising the following steps: step S1, input condition preparation; step S2, candidate video sampling; step S3, double-path reward evaluation; step S4, reward aggregation; step S5, constructing a noisy sample; step S6, multi-model parallel speed prediction; step S7, double-loss calculation; step S8, evolving exploration model updating; step S9, stable anchoring model EMA synchronization; and step S10, iterative convergence. The identity-preserving video generation method and system based on reinforcement evolution provided by the application is converted from a static imitation learning paradigm to a self-evolution optimization paradigm that can actively balance multiple objectives: by introducing a double-strategy architecture (Stable+Evolving) and the concept of an 'evolution direction', the generation model can autonomously and dynamically balance multiple-dimensional perception quality objectives such as identity fidelity, action naturalness, and time sequence coherence, thereby avoiding the 'conservative balance' caused by training based on a fixed reconstruction loss.

Description

Technical Field

[0001] This invention relates to the field of video generation technology, and in particular to a method and system for generating identity-preserving videos based on reinforcement evolution. Background Technology

[0002] Identity-Preserving Video Generation (IP Video Generation) aims to synthesize a dynamic video that retains the identity features of the reference subject and performs corresponding actions according to the text description, based on one or more reference identity images (such as facial images) and a text description. It is an important supporting technology for personalized media, virtual digital humans, visual effects and other fields.

[0003] Existing IP video generation solutions mostly use diffusion models as their backbone, and can be roughly divided into two categories based on their training paradigms: Fine-tuning the testing approach: Train a separate adapter or dedicated embedding for each new reference identity. This approach typically requires performing the training process separately for each identity, resulting in high computational and storage costs, making it difficult to meet the needs of large-scale deployment.

[0004] The no-fine-tuning approach: By using adapters, cross-attention, or independent reference branches, reference images are encoded as identity features and injected into the pre-trained video diffusion model backbone, thus eliminating the need for additional training on identity during the inference phase. This approach has become mainstream due to its good generalization ability.

[0005] In recent years, mainstream fine-tuning-free schemes have generally adopted the Diffusion Transformer (DiT) backbone structure and imitation learning as the basic training paradigm. On large-scale identity-video-text triplet data, the model minimizes the reconstruction loss (such as L1, L2, perceptual loss LPIPS, etc.) between itself and the training samples, thereby learning a static mapping from "reference identity image + text description" to "video sequence". Under this paradigm, the model imitates a fixed conditional distribution from the training data and then synthesizes the video through an iterative denoising process. Some previous works have further introduced alignment methods based on human preferences or reinforcement learning (RL) on top of imitation learning, such as PPO, DPO, reward-weighted regression, group relative policy optimization (GRPO), and schemes for stream matching models; other schemes further optimize the generative model by specifically training an identity preference reward model.

[0006] II. Disadvantages of existing technologies and the technical problems to be solved by this application. The inventors discovered in their research that existing IP video generation solutions still have the following significant drawbacks when generating videos that combine high identity fidelity, high motion naturalness, and high temporal coherence: Static training objectives are insufficient to accommodate the dynamic trade-offs of multidimensional perceived quality.

[0007] IP video quality is a composite indicator determined by multiple factors, including identity fidelity, natural movement, temporal coherence, and the appropriateness of lighting and camera angles. Reconstruction losses such as L1 / L2 / LPIPS are essentially "static" targets, calculated pixel-by-pixel or feature-by-feature, and have only a weak correlation with the aforementioned multidimensional perceptual quality. This paradigm cannot dynamically adjust the focus on quality across different input conditions, causing the model to easily fall into a "conservative equilibrium."

[0008] A typical "copy-paste" artifact problem.

[0009] Under conservative equilibrium, in order to pursue high identity similarity, the model often rigidly "attaches" the reference face to the generated image, resulting in unnatural phenomena such as stiff movements, incoordination between face and body movements, and distorted expressions, which seriously damages the visual realism and human subjective preferences.

[0010] A single objective indicator cannot characterize the overall perceived quality.

[0011] Single metrics such as ArcFace cosine similarity, optical flow amplitude, and aesthetic score often only reflect one aspect of quality and are competitive with each other (for example, increasing the amplitude of movements may reduce facial similarity). Directly weighting and summing these metrics as rewards, with fixed weight settings, cannot adapt to the actual improvement needs of different training stages and different samples, and is prone to "reward hacking".

[0012] Applying reinforcement learning directly to diffusion video generation can lead to quality drift and catastrophic forgetting. If the diffusion model is trained solely on reward without introducing appropriate regularization, the model may sacrifice basic generation capabilities in pursuit of specific rewards, such as losing scene diversity, introducing artifacts, or distorting human body structures, resulting in a deterioration in overall video quality during RL optimization.

[0013] There is a lack of holistic modeling methods for human preferences. Even with the introduction of rewards based on preference models or human feedback, most existing solutions still require additional training of dedicated preference models, rely on large-scale preference-labeled data, and suffer from poor reusability and high construction costs. Summary of the Invention

[0014] The purpose of this invention is to provide a method and system for generating identity-preserving videos based on enhanced evolution, which solves the problems mentioned in the background art.

[0015] This invention is implemented as follows: a video generation method based on reinforced evolution that preserves identity, the method comprising: Obtain the reference identity image and text description; A stable anchoring model and an evolutionary exploration model are provided, both of which are cloned from the same pre-trained identity-preserving diffusion model; The stable anchoring model generates multiple candidate videos based on the reference identity image and the text description; A reward value is calculated for each candidate video through a dual-path reward mechanism that includes an objective indicator reward path and a preference reward path based on a multimodal large language model. For each candidate video, the stable anchoring model and the evolutionary exploration model predict the velocity vector of the noisy candidate video at the same time step, and the difference between the two is used as the evolution direction; The policy loss is constructed based on the reward value and the evolution direction, and the total loss is constructed by combining the teacher regularization term. The teacher regularization term includes at least an alignment loss that aligns the velocity prediction of the evolutionary exploration model to the velocity prediction of the frozen textual video teacher model. The parameters of the evolutionary exploration model are updated by gradient backpropagation based on the total loss, and the parameters of the stable anchoring model are updated by exponential moving average using the evolutionary exploration model. Using the updated evolutionary exploration model or the stable anchoring model, an identity-preserving video is generated based on the target reference identity image and target text description input by the user.

[0016] A further technical solution of the present invention is: the generation of multiple candidate videos by the stable anchoring model based on the reference identity image and the text description includes: for a reference identity image and text description pair, the stable anchoring model generates multiple candidate videos under different random noise seeds. The candidate videos, The value is an integer greater than or equal to 2; the construction of the policy loss based on the reward value and the evolution direction includes: Noise is added to the candidate video to obtain a noisy sample; The first velocity vector is predicted from the noisy sample by the stable anchoring model. ; The second velocity vector is predicted from the noisy sample by the evolutionary exploration model. ; Will As the direction of evolution; According to the preset step size coefficient Constructing positive correction prediction and reverse correction prediction ; The policy loss is constructed as follows:

[0017] in The normalized reward value for the candidate video. The vector is the actual velocity vector corresponding to the candidate video.

[0018] A further technical solution of the present invention is: the total loss is a weighted sum of the policy loss and at least one regularization term, wherein the at least one regularization term includes: T2V Teacher Regularization : Used to align the velocity prediction of the evolutionary exploration model under the conditions of the reference identity image and the text description with the velocity prediction of the frozen text-based video teacher model under the condition of only the text description; and / or Self-constrained regularization terms The method is used to align the velocity prediction of the evolutionary exploration model with the velocity prediction of the pre-trained identity-preserving diffusion model under the same conditions using L2 distance; the step of updating the parameters of the stable anchoring model using the evolutionary exploration model through exponential moving average includes: in each iteration, according to... Update the parameters of the stable anchoring model, where This is the preset attenuation coefficient.

[0019] A further technical solution of the present invention is: the dual-path reward mechanism includes: An objective reward path is used to output an objective reward score for each candidate video from at least one dimension among identity fidelity, text fit, and visual quality through several visual evaluation models; and A preference reward path based on a multimodal large language model is used to provide each candidate video to the multimodal large language model and require it to output a comprehensive preference score according to a preset multidimensional evaluation guideline.

[0020] A further technical solution of the present invention is that the visual evaluation model included in the objective indicator reward path includes at least one of the following: A facial feature model is used to calculate the similarity between facial features in the candidate video and the reference identity image; A cross-modal matching model is used to evaluate the semantic consistency between the candidate video and the text description; An aesthetic scoring model is used to evaluate the visual aesthetic quality of the candidate videos; A temporal coherence scoring model is used to evaluate the stability of the candidate videos in the time dimension; A naturalness scoring model is used to evaluate the overall realism of the candidate videos; An optical flow estimation model is used to evaluate the inter-frame motion amplitude of the candidate video. The preference reward path based on the multimodal large language model includes: Provide the multimodal large language model with evaluation guidelines including at least two of the following: visual realism, text fit, facial identity consistency, motion naturalness, and camera movement rationality; The request is to use the multimodal large language model to output a single overall score after integrating the internal fusion of at least two dimensions, as the overall visual quality score; The overall visual quality score is used as the output reward signal of the preference reward path.

[0021] A further technical solution of the present invention is that the step of calculating the reward value for each candidate video further includes MLLM-guided dynamic weight adjustment, wherein the dynamic weight adjustment includes: Set of rewards Divided into Each group is assigned a prior weight, and a prior weight is set for each reward. ; The prior group proportions are obtained by summing the prior weights by group and normalizing them. ; The multiple candidate videos generated in this iteration are provided to a multimodal large language model, and its output is requested. dimensional focus vector ; Apply a smoothing factor to the focal vector Normalization yields the proportion of MLLM groups ; According to the mixing coefficient The prior group proportion and the MLLM group proportion are linearly mixed to obtain the mixed group proportion. ; The mixed group proportions are distributed to each reward according to the prior weight ratio within the group to obtain the final weights. The aggregate reward is obtained by weighting and summing the scores of each reward, and this aggregate reward is used as the reward value; the reward set is divided into... The awards are divided into three groups: identity, text, and quality. The identity group reflects the fidelity of the identity, the text group reflects the relevance of the text, and the quality group reflects the visual quality.

[0022] A further technical solution of the present invention is: the method further includes calculating a comprehensive evaluation score EvoScore for the candidate video or the identity-preserving video, wherein the calculation of the EvoScore includes: Video quality is broken down into three dimensions: identity consistency, text fit, and technical quality, with each dimension having several sub-items. By making several independent requests to the multimodal large language model, scores are given for all sub-items under each dimension; For each dimension, the minimum score of all sub-items under that dimension is taken as the score of that dimension according to the weakest link principle; The harmonic mean of the scores in the three dimensions is calculated to obtain the EvoScore; The method further incorporates the EvoScore as an additional reward signal into the dual-path reward mechanism, enabling the evolutionary exploration model to actively improve the weaker dimensions among the three dimensions during training.

[0023] A further technical solution of the present invention is: the stable anchoring model and / or the evolutionary exploration model are implemented by applying low-rank adaptation to several transformer blocks of the pre-trained identity-preserving diffusion model; the pre-trained identity-preserving diffusion model is trained based on flow matching, and its velocity vector objective is defined as The noise addition process satisfies ; The generated identity-preserving video includes: The target reference identity image is encoded into the latent space by a variational autoencoder to obtain the latent identity representation; Initial noise latent representation of sampling; Using the evolutionary exploration model or the stable anchoring model as conditions, the initial noisy latent representation is subjected to a preset number of denoising iterations. The identity-preserving video is obtained by decoding the denoised latent representation using a variational autoencoder decoder. A reinforcement evolution-based identity-preserving video generation system is characterized by comprising: The model management module is used to maintain a stable anchoring model, an evolutionary exploration model, a frozen Wensheng video teacher model, and an initial identity preservation diffusion model, wherein the stable anchoring model and the evolutionary exploration model are cloned from the initial identity preservation diffusion model; The candidate sampling module is used to generate multiple candidate videos based on the reference identity image and text description by the stable anchoring model; The reward evaluation module includes an objective indicator reward path and a preference reward path based on a multimodal large language model, which is used to output a reward value for each candidate video. The weight scheduling module is used to output a focus vector for multiple candidate videos in the current iteration through a multimodal large language model, and to calculate the final weight of each reward based on the prior weight ratio and the focus vector. The strategy optimization module is used to predict the velocity vector of the noisy sample and construct the evolution direction by the stable anchoring model and the evolution exploration model, respectively, and construct the strategy loss based on the reward value and the evolution direction, and construct the total loss by combining the T2V teacher regularization term and / or self-constraint regularization term. The parameter update module is used to update the parameters of the evolutionary exploration model based on the total loss, and to update the parameters of the stable anchoring model by exponential moving average. The video generation module is used to generate identity-preserving videos based on the target reference identity image and target text description input by the user after training is completed.

[0024] The beneficial effects of this invention: The identity-preserving video generation method and system based on reinforcement evolution provided in this application have the following significant technical advantages compared with the prior art: The paradigm shifts from static imitation learning to a self-evolving optimization paradigm that can actively weigh multiple objectives: By introducing a dual-policy architecture (Stable + Evolving) and the concept of "evolutionary direction," the reward signal is directly transformed into a directional constraint on the velocity field of the diffusion model. This allows the generative model to autonomously and dynamically weigh multiple perceptual quality objectives such as identity fidelity, action naturalness, and temporal coherence, avoiding the "conservative equilibrium" caused by training based on fixed reconstruction loss.

[0025] Mitigating "copy-paste" artifacts: Thanks to the synergistic effect of the MLLM-preferenced reward path and the MLLM-guided dynamic weight adjustment module, while pursuing facial similarity, the model is continuously guided to focus on the naturalness of actions and the overall perceptual rationality, thereby significantly suppressing the stiff movements and facial distortions caused by rigidly attaching reference faces to the generated images.

[0026] Stable reinforcement learning optimization process: Through dual regularization of T2V teacher regularization and initial IP model self-constraint, the evolutionary exploration model can fully explore in the reward direction and be anchored to the vicinity of a distribution with robust world priors, effectively alleviating the problems of "quality drift" and "catastrophic forgetting"; further, in conjunction with the EMA update mechanism of the stable anchoring model, the sampling distribution and optimization strategy form two sets of strategies, "slow-fast", to avoid training divergence.

[0027] Low-cost human preference alignment without relying on specialized preference models: It directly uses the VQScore output by a general MLLM as the overall preference reward, without the need to train a special preference model or large-scale preference annotation data. It has a low deployment threshold and strong reusability. At the same time, by decomposing quality into multiple evaluation dimensions and implicitly fusing them internally by MLLM, it is closer to human overall perception.

[0028] Adaptive weight adjustment for multi-reward aggregation: Through the dynamic weight mechanism guided by MLLM, the reward aggregation weights automatically adapt to the weak dimensions of the current model in each iteration, avoiding the problems of "reward hacking" and over-optimization of a certain dimension that are easily caused by fixed weight settings; the "weakest link effect" metric emphasized by EvoScore and others can be further introduced into the training as an additional reward to improve the model's performance on relatively weak dimensions.

[0029] Architecture-independent, customizable training paradigm: The training paradigm of this application does not depend on a specific IP model architecture or a specific VAE / text encoder. It can achieve targeted control of generation preferences (e.g., focusing more on identity similarity or focusing more on action amplitude) by changing, adding, removing, or reweighting reward signals without modifying the backbone model. This provides a flexible and low-cost tool for subsequent customized deployment of models in different business scenarios.

[0030] Both objective metrics and overall performance are significantly improved: Comparative experiments on the publicly available OpenS2V-Eval Human-Domain benchmark show that this application achieves a total score of 0.704 on the official weighted average score and a score of 0.682 on the proposed EvoScore. With the implementation method that uses EvoScore as an additional reward, the EvoScore is further improved to 0.718, which is comprehensively better than existing open source and commercial systems, verifying the overall advantage of the technical solution of this application in multi-dimensional perception quality. Attached Figure Description

[0031] Figure 1 This is a flowchart of an identity-preserving video generation method based on reinforcement evolution provided by the present invention; Figure 2 This is a schematic block diagram of an identity-preserving video generation system based on enhanced evolution provided by the present invention. Detailed Implementation

[0032] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention.

[0033] It should be noted that the structures, proportions, sizes, etc., illustrated in the accompanying drawings are merely for illustrative purposes to aid those skilled in the art and to facilitate understanding and reading. They are not intended to limit the scope of the invention and therefore have no substantial technical significance. Any modifications to the structure, changes in proportions, or adjustments to size, without affecting the effectiveness and purpose of the invention, should still fall within the scope of the technical content disclosed herein. Furthermore, the terms "upper," "lower," "left," "right," "middle," and "one" used in this specification are merely for clarity and not intended to limit the scope of the invention. Changes or adjustments to their relative relationships, without substantially altering the technical content, should also be considered within the scope of the invention.

[0034] Example 1: Figure 1 This paper presents an identity-preserving video generation method based on reinforcement evolution; the overall training framework of this application operates according to the following process in each training iteration: Step S1: Input Condition Preparation. Read a set of input conditions from the training set. The input conditions include a reference identity image and a text description, corresponding to the "Input Conditions" block on the far left of Figure 1.

[0035] Step S2: Candidate Video Sampling. The input conditions are provided to the stable anchoring model. The stable anchoring model is a clone of the pre-trained identity-preserving diffusion model, and its parameters are frozen during this iteration. The stable anchoring model performs denoising inference under multiple different random noise seeds, outputting several candidate videos, forming a set of "several videos generated by the stable anchoring model".

[0036] Step S3: Dual-path reward evaluation. Input all candidate videos obtained in Step S2 into two parallel reward evaluation paths simultaneously: (a) Subjective Preference Reward Path: The candidate videos are provided to the multimodal big language model, which outputs a comprehensive subjective preference reward for each candidate video according to the preset multi-dimensional evaluation guidelines. At the same time, the multimodal big language model observes all candidate videos generated in this iteration as a whole, analyzes the relative weaknesses of the current model in dimensions such as identity fidelity, text fit, and visual quality, and outputs a focus vector / weight that reflects the degree of attention in each dimension. (b) Objective reward signal path: The candidate videos are provided to several objective reward models (in one embodiment, including but not limited to face similarity model, optical flow model, art scoring model, text alignment model, temporal coherence model and naturalness model). Each objective reward model outputs an objective reward score from its corresponding dimension, and the whole constitutes a multi-dimensional objective reward signal.

[0037] Step S4: Reward Aggregation. The focus vector / weights obtained in Step S3 are weighted and fused with multi-dimensional objective reward signals and subjective preference rewards (referred to as "..." in the figure). The symbol is used to represent the unified reward scalar for each candidate video in this iteration; the weight allocation is dynamically calculated based on the focus vector and the preset prior weights, so that the reward signal of the weak dimension gets a higher weight in this iteration.

[0038] Step S5: Construct noisy samples. For each candidate video, sample one time step according to the preset time step distribution and apply corresponding noise (in Figure 1, the "⊕" symbol represents noise superposition) to obtain noisy samples.

[0039] Step S6: Multi-model parallel velocity prediction. The noisy samples obtained in Step S5, conditioned on the reference identity image and text description, are fed in parallel to three diffusion transformers for velocity prediction: Stable anchoring model: Output the first velocity vector , serving as the "pre-evolutionary" strategy baseline; Evolutionary exploration model: Like the stable anchoring model, it is cloned from the same pre-trained identity-preserving diffusion model, but its parameters are actually updated in each iteration, and it outputs a second velocity vector. As a "post-evolutionary" strategy; The Wensheng video teacher model is a frozen pre-trained Wensheng video model that is conditional only by textual descriptions and does not use reference identity images. Its output reflects the velocity predictions of the general world prior.

[0040] Step S7: Dual Loss Calculation. Based on the velocity prediction in Step S6, construct two loss paths respectively: Strategy loss: The difference between the velocity predictions of the stable anchoring model and the evolutionary exploration model is used as the "evolutionary direction". The strategy loss is constructed by combining the reward scalar obtained in step S4, so that high reward samples drive the evolutionary exploration model to continue moving forward along the evolutionary direction, while low reward samples drive it to shrink along the anti-evolutionary direction. Regularization loss: The velocity prediction of the evolutionary exploration model is anchored by L2 distance using the velocity prediction of the Wensheng video teacher model, and the evolutionary exploration model is anchored by self-constraint L2 using the initial identity-maintaining diffusion model. Both together suppress the quality drift and catastrophic forgetting that may occur in the evolutionary exploration model during the pursuit of rewards.

[0041] Step S8: Update the evolutionary exploration model. The total loss is obtained by weighted summation of the policy loss and regularization loss obtained in step S7, and the parameters of the evolutionary exploration model are updated using gradient descent via backpropagation.

[0042] Step S9: Synchronize the Stable Anchor Model (EMA). Using the updated evolutionary exploration model parameters, the parameters of the stable anchor model are progressively updated using an exponential moving average. After this step, the stable anchor model is used for candidate video sampling in the next iteration, thus forming a self-evolutionary closed loop of "sampling-evaluation-update-synchronization".

[0043] Step S10: Iterative convergence. Repeat steps S1 to S9 until the evolutionary exploration model converges or reaches the preset number of iterations; the final evolutionary exploration model (or its stable anchoring form) is used as the deployment model for identity-preserving video generation.

[0044] Example 2: Figure 2 This paper presents an identity-preserving video generation system based on reinforcement evolution; the video generation system constructed in this application includes the following core components: Pre-trained variational autoencoders (VAEs): used for transformations between pixel space and latent space, including VAE encoders and VAE decoders; Pre-trained text encoder: used to encode text descriptions into text features; Pre-trained Identity Preserving Diffusion Transformer (IP-DiT): As the base model of this application, it is capable of synthesizing videos based on reference identity images and text descriptions. express; Stable Transformer: a model with stable anchoring. Obtained by cloning, the parameters are denoted as , recorded as During training, it is responsible for sampling candidate videos and providing "pre-evolutionary" velocity predictions, and its parameters are updated asymptotically via EMA. Evolutionary exploration model: Evolving Transformer: also by Obtained by cloning, the parameters are denoted as , recorded as During training, it is responsible for updating parameters along the reward direction, and is the truly optimized "explorer". Frozen T2V Teacher Model (for online learning): Recorded as It shares the same VAE latent space with the IP model to provide a universal video prior without relying on identity information, serving as an "anchor point for the world prior"; The reward evaluation subsystem includes several models for objective metric evaluation (such as face recognition model, text-video alignment model, aesthetic model, temporal coherence model, naturalness model, optical flow model, etc.) and MLLM for preference evaluation. MLLM Weight Scheduling Subsystem: Used in each training iteration to output a focus vector from MLLM that reflects the current model's weakness, and adjust the aggregate weights of multiple rewards accordingly.

[0045] In one possible implementation, and Through the Low-rank adaptation (such as LoRA) is applied to carry out the tasks separately, thereby significantly reducing memory usage and training costs. Those skilled in the art will understand that LoRA can also be applied directly with full-parameter cloning, and this application is not limited thereto.

[0046] Task modeling and denoising process This application employs a diffusion modeling approach based on the idea of ​​rectified flow matching. The clean video latent representation is denoted as... Let Gaussian noise be denoted as Defined at time step The noisy samples are:

[0047] Model Reference identity image Text description and noisy samples Time step As input, predict the velocity vector. In one embodiment, After being encoded by VAE, the data is fed into the identity branch with the same structure as the main DiT, where multi-layer identity features are extracted at multiple depths and then injected into the intermediate features of the main DiT. This allows for precise control over identity while retaining general text-based video capabilities.

[0048] In the inference stage, this application uses Euler discretization as the ordinary differential equation solver, starts from pure noise, and sequentially removes noise to obtain the latent representation of the generated video, and then decodes it through a VAE decoder to obtain the final video sequence.

[0049] Stabilized reinforcement learning optimization In order to make This application introduces reinforcement learning as the core optimization mechanism to proactively optimize multidimensional perception quality, and stabilizes the optimization process through a dual-strategy architecture and multiple regularizations.

[0050] Dual-strategy architecture and sampling In each training iteration: Several pairs of "reference identity image + text description" are sampled from the training set to form the condition. ; For each Using a stable anchoring model Generating under different random noise seeds The candidate videos are recorded as follows: ; The dual-path reward mechanism (see Section 3.4) and the MLLM-guided dynamic weight adjustment module (see Section 3.5) of this application are used to calculate a normalized reward scalar for each candidate video. .

[0051] In one embodiment, each training step samples 4 identity-text pairs, and each pair generates 8 candidate videos, for a total of 32 candidate samples. However, those skilled in the art will understand that... The logarithm for each step can be adjusted based on computing resources.

[0052] Strategy loss based on "evolutionary direction" For those from A candidate video Add noise to it to get The velocities are predicted by two separate models:

[0053] Define the evolution direction of the current iteration as:

[0054] Further define the stable model predictions with corrections along the evolutionary direction and with corrections in the reverse evolutionary direction:

[0055] in This is a preset step size coefficient. The strategy loss requirement of this application is: when... When the sample is a high-reward sample, the prediction adjusted along the evolution direction It should be closer to the real speed Conversely, when When the sample has a low reward, the prediction adjusted along the inverse evolution direction It should be closer Formalistically:

[0056] In this form, the reward signal is directly transformed into a directional constraint on the velocity field of the diffusion model: high-reward samples will "pull" the evolutionary exploration model to continue moving along the direction of difference between it and the stable anchoring model; low-reward samples will suppress the update in this direction, thereby achieving reward-driven improvement of the generated policy without explicitly estimating the policy gradient.

[0057] Dual regularization based on T2V teacher and self-restraint Relying solely on Optimization carries the risk that the evolutionary exploration model sacrifices fundamental generative capabilities (e.g., loss of scene diversity, artifacts, abnormal human anatomy) in pursuit of high rewards, i.e., the reward hacking and catastrophic forgetting problem. To address this, this application introduces two regularization terms: (a) T2V Teacher Regularization Select a frozen one, and... Pre-trained textual video models sharing the same VAE latent space As a teacher, the requirement is that the velocity prediction of the evolutionary exploration model, without using identity information, should be close to the velocity prediction of the teacher model under the same text conditions:

[0058] The L2 alignment loss can be viewed as a simplified proxy for the KL divergence, used to anchor the evolutionary exploration model to a teacher distribution with a robust “world prior”.

[0059] (b) Self-constrained regularization terms The IP model will be trained at the initial time step. As an additional teacher, the deviation of the evolutionary exploration model relative to its original strategy is limited:

[0060] Overall Training Objectives and EMA Updates The overall training objective of the evolutionary exploration model is a weighted sum of the policy loss and the two regularization terms mentioned above:

[0061] in , The weighting coefficients are adjustable, and in one embodiment they are taken as 0.01 and 0.2 respectively, but this application does not limit the specific values. In each training step, the evolutionary exploration model... Through the Perform gradient backpropagation update; stabilize anchored model Then through the The Exponential Moving Average (EMA) is updated incrementally:

[0062] in EMA is the decay coefficient. This EMA mechanism enables the sampling model and the optimization model to form two sets of strategies, "slow" and "fast," avoiding a drastic mismatch between the sampling distribution and the optimization objective, thereby significantly stabilizing the RL training process.

[0063] Dual-path reward mechanism The design of the reward signal directly determines the direction of evolution. This application proposes a dual-path reward mechanism of "objective metric + MLLM preference" to simultaneously characterize fine-grained objective quality and overall subjective preference during training.

[0064] Objective Indicator Reward Path The objective metric path uses several existing visual evaluation models as reward sources to score the generated videos from multiple dimensions. In one embodiment, the reward model is configured according to the following three main categories: Identity authentication: For example, calculating the cosine similarity between facial features in a generated video and a reference identity image based on facial feature models (such as ArcFace, CurricularFace); Text-fitting type: For example, calculating the semantic consistency between video content and text description based on cross-modal matching models; Visual quality categories include, but are not limited to, aesthetic scoring models, temporal coherence scoring models, naturalness scoring models, and motion amplitude scoring based on dense optical flow estimation.

[0065] The output values ​​of each objective indicator were normalized to... The interval is then divided, and multiple candidate videos within the image-text pair are relatively normalized to obtain... Those skilled in the art will understand that the specific models listed are merely examples, and this application does not limit the implementation of specific reward models. Any model that can perform differentiable or non-differentiable scoring on any dimension such as identity, text fit, or visual quality can be used as a source of reward in the objective reward path of this application.

[0066] MLLM Preference Reward Path (VQScore) To compensate for the shortcomings of a single objective indicator in reflecting overall human preferences, and especially to alleviate the "copy-paste" phenomenon where objective facial similarity is high but visual performance is stiff, this application introduces a second MLLM preference reward path in addition to the objective reward path to generate a comprehensive preference score (Visual Quality Score, or VQScore for short).

[0067] Specifically, in one embodiment, the MLLM is provided with generated videos and evaluation guidelines, requiring the MLLM to simultaneously consider several evaluation dimensions (including visual realism, text fit, facial identity consistency, motion naturalness, and camera movement rationality in one embodiment), and explicitly requiring the MLLM to output a single overall score based on the evaluation results of all dimensions. This overall score is the VQScore, reflecting overall human perceptual preferences.

[0068] The key features of the MLLM preference reward path are: preference evaluation does not rely on a specially trained preference model, but directly reuses a general multimodal large language model; and the multi-dimensional evaluation is explicitly listed through carefully designed evaluation rubrics, enabling MLLM to achieve implicit fusion of multi-dimensionality internally.

[0069] EvoScore as an evaluation or additional reward signal To more accurately characterize the "weakest link effect" (i.e., the weak performance of a video in a certain key dimension often dominates the user's subjective evaluation), this application further designs an evaluation metric based on MLLM, EvoScore, which can be used as an evaluation metric or, in an optional implementation, as an additional reward signal to feed back into the training.

[0070] In one embodiment, the EvoScore calculation process is as follows: Video quality is broken down into three dimensions: identity consistency, text fit, and technical quality. Each dimension has several sub-items (for example, the identity dimension includes sub-items such as facial recognition accuracy, identity fidelity, and identity stability; the text fit dimension includes sub-items such as character appearance / action fit, background environment / object fit, image style and text compliance, camera movement mode compliance, and lighting atmosphere; and the technical quality dimension includes sub-items such as image fidelity, motion smoothness and continuity, character vividness, camera movement skills, aesthetic style, and content richness). For each dimension, use separate prompt words to request MLLM to score its sub-items; For each dimension, the minimum score of all sub-items within that dimension is taken as the score of that dimension according to the "weakest link principle". To improve stability, the scores for each dimension are averaged over several independent runs; The final EvoScore is the harmonic mean of the scores in the three dimensions.

[0071] Because the harmonic mean is significantly lowered when any dimension is too low, EvoScore emphasizes improving the weakest dimension. In one implementation, EvoScore is used only as an evaluation metric; in another implementation, EvoScore can also be incorporated into a dual-path reward mechanism as an additional preference reward signal to drive further optimization of the model.

[0072] MLLM-guided dynamic reward weighting Because there is an inherent competition among the various objective rewards mentioned above (e.g., pursuing large movements may impair facial similarity), simply adding the rewards with fixed weights cannot adapt to the specific shortcomings of the model during training. Therefore, this application proposes an MLLM-guided dynamic reward weight adjustment method, which allows the aggregated reward weights to adaptively change as the model training progresses.

[0073] Specifically, let there be a total A reward model In one embodiment, it is divided into groups according to the "identity / text / quality" grouping method. Groups (typically taken) ),Right now And for each reward Set prior weights .

[0074] Step a: Prior group proportion calculation. Calculate the proportion of each group. The prior proportion is:

[0075] Step b: MLLM Focus Vector Acquisition. In each training iteration, all videos generated in that iteration are provided to the MLLM, and a preset prompt is used to request the MLLM to output a focus vector. dimensional focal fraction vector Each component represents the degree to which the current model needs to focus on that set of dimensions.

[0076] Step c: MLLM scaling normalization. Perform with smoothing factor Normalization yields the MLLM ratio:

[0077] Step d: Mixing of prior and MLLM. Introducing mixing coefficients. The proportions of the mixed group are obtained by linearly mixing the prior proportions with the MLLM proportions:

[0078] Step e: Intra-group weight redistribution. While preserving the relative proportions of prior weights within the group, the weights are redistributed... Redistribute to the group Each reward The final individual weights are obtained as follows:

[0079] Step f: Calculate the aggregated reward. The final aggregated reward is:

[0080] in for Relative normalization results across multiple candidate videos for the same image-text pair.

[0081] Through the above process, the dimensions in which the model is currently performing weakly in each training iteration will be automatically identified by MLLM and given higher weights, thereby guiding the model to actively improve its weak aspects and ultimately achieve a balanced improvement in multidimensional quality.

[0082] Training process and inference process Training process In a typical implementation, the overall training process of this application includes the following steps: Initialization: Diffusion model from the same pre-trained IP Starting from this point, construct stable anchoring models respectively. and evolutionary exploration model (Both can be implemented using LoRA); Freeze the T2V teacher model With the initial IP teacher .

[0083] Sampling candidate videos: Sample several pairs of "reference identity image + text description" from the training set, and use the following for each pair: Generated under different random seeds 10 candidate videos.

[0084] Multi-reward scoring: For each candidate video, the objective metric reward path and the MLLM preference reward path are run sequentially to obtain multiple reward scores; optionally, the EvoScore is further calculated as an additional reward.

[0085] MLLM Dynamic Weight Output: For all videos generated in this iteration, request the MLLM output focus vector, and calculate the final weight of each reward according to the process in Section 3.5 to obtain the aggregated reward. .

[0086] Noise Addition and Velocity Prediction: Noise is added to each candidate video at random time steps, and velocities are predicted by both the stable anchoring model and the evolutionary exploration model. The evolutionary direction is then calculated. , and then construct and .

[0087] Loss calculation: based on , and Calculate total loss .

[0088] Parameter update: Updated via gradient descent Update via EMA .

[0089] Repeated iterations: Repeat steps 2 through 7 for the set number of iterations until convergence.

[0090] Reasoning process After training, the video generation inference process of this application includes: Receive reference identity image With text description ; The VAE encoder encodes the reference identity image into the latent space to obtain the identity latent representation, and samples the initial noise latent representation; In evolutionary exploration models Perform multi-step denoising (e.g., 20 steps) on (or its equivalent deployment model), with each step simultaneously using the latent identity representation and text features as conditional prediction speed and updating the latent representation according to the solver; The final latent representation is decoded into a video frame sequence by the VAE decoder and output.

[0091] In one alternative implementation, a stable anchoring model can also be used during the inference phase. (It is smoother than the evolutionary model and provides more robust generation results), or one of the two can be deployed, and this application is not limited to this.

[0092] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A video generation method for maintaining identity based on reinforcement evolution, characterized in that, The method includes: Obtain the reference identity image and text description; A stable anchoring model and an evolutionary exploration model are provided, both of which are cloned from the same pre-trained identity-preserving diffusion model; The stable anchoring model generates multiple candidate videos based on the reference identity image and the text description; A reward value is calculated for each candidate video through a dual-path reward mechanism that includes an objective indicator reward path and a preference reward path based on a multimodal large language model. For each candidate video, the stable anchoring model and the evolutionary exploration model predict the velocity vector of the noisy candidate video at the same time step, and the difference between the two is used as the evolution direction; The policy loss is constructed based on the reward value and the evolution direction, and the total loss is constructed by combining the teacher regularization term. The teacher regularization term includes at least an alignment loss that aligns the velocity prediction of the evolutionary exploration model to the velocity prediction of the frozen textual video teacher model. The parameters of the evolutionary exploration model are updated by gradient backpropagation based on the total loss, and the parameters of the stable anchoring model are updated by exponential moving average using the evolutionary exploration model. Using the updated evolutionary exploration model or the stable anchoring model, an identity-preserving video is generated based on the target reference identity image and target text description input by the user.

2. The identity-preserving video generation method based on reinforcement evolution according to claim 1, characterized in that, The generation of multiple candidate videos by the stable anchoring model based on the reference identity image and the text description includes: for a reference identity image and text description pair, the stable anchoring model generates multiple candidate videos under different random noise seeds. The candidate videos, The value is an integer greater than or equal to 2; the construction of the policy loss based on the reward value and the evolution direction includes: Noise is added to the candidate video to obtain a noisy sample; The first velocity vector is predicted from the noisy sample by the stable anchoring model. ; The second velocity vector is predicted from the noisy sample by the evolutionary exploration model. ; Will As the direction of evolution; According to the preset step size coefficient Constructing positive correction prediction and reverse correction prediction ; The policy loss is constructed as follows: in The normalized reward value for the candidate video. The vector is the actual velocity vector corresponding to the candidate video.

3. The identity-preserving video generation method based on reinforcement evolution according to claim 2, characterized in that, The total loss is a weighted sum of the policy loss and at least one regularization term, wherein the at least one regularization term includes: T2V Teacher Regularization : Used to align the velocity prediction of the evolutionary exploration model under the conditions of the reference identity image and the text description with the velocity prediction of the frozen text-based video teacher model under the condition of only the text description; and / or Self-constrained regularization terms The method is used to align the velocity prediction of the evolutionary exploration model with the velocity prediction of the pre-trained identity-preserving diffusion model under the same conditions using L2 distance; the step of updating the parameters of the stable anchoring model using the evolutionary exploration model through exponential moving average includes: in each iteration, according to... Update the parameters of the stable anchoring model, where This is the preset attenuation coefficient.

4. The identity-preserving video generation method based on reinforcement evolution according to claim 1, characterized in that, The dual-path reward mechanism includes: An objective reward path is used to output an objective reward score for each candidate video from at least one dimension among identity fidelity, text fit, and visual quality through several visual evaluation models; and A preference reward path based on a multimodal large language model is used to provide each candidate video to the multimodal large language model and require it to output a comprehensive preference score according to a preset multidimensional evaluation guideline.

5. The identity-preserving video generation method based on reinforcement evolution according to claim 4, characterized in that, The visual evaluation model included in the objective indicator reward path includes at least one of the following: A facial feature model is used to calculate the similarity between facial features in the candidate video and the reference identity image; A cross-modal matching model is used to evaluate the semantic consistency between the candidate video and the text description; An aesthetic scoring model is used to evaluate the visual aesthetic quality of the candidate videos; A temporal coherence scoring model is used to evaluate the stability of the candidate videos in the time dimension; A naturalness scoring model is used to evaluate the overall realism of the candidate videos; An optical flow estimation model is used to evaluate the inter-frame motion amplitude of the candidate video. The preference reward path based on the multimodal large language model includes: Provide the multimodal large language model with evaluation guidelines including at least two of the following: visual realism, text fit, facial identity consistency, motion naturalness, and camera movement rationality; The request is to use the multimodal large language model to output a single overall score after integrating the internal fusion of at least two dimensions, as the overall visual quality score; The overall visual quality score is used as the output reward signal of the preference reward path.

6. The identity-preserving video generation method based on reinforcement evolution according to claim 1, characterized in that, The step of calculating a reward value for each candidate video further includes MLLM-guided dynamic weight adjustment, the dynamic weight adjustment including: Set of rewards Divided into Each group is assigned a prior weight, and a prior weight is set for each reward. ; The prior group proportions are obtained by summing the prior weights by group and normalizing them. ; The multiple candidate videos generated in this iteration are provided to a multimodal large language model, and its output is requested. dimensional focus vector ; Apply a smoothing factor to the focal vector Normalization yields the proportion of MLLM groups ; According to the mixing coefficient The prior group proportion and the MLLM group proportion are linearly mixed to obtain the mixed group proportion. ; The mixed group proportions are distributed to each reward according to the prior weight ratio within the group to obtain the final weights. The aggregate reward is obtained by weighting and summing the scores of each reward, and this aggregate reward is used as the reward value; the reward set is divided into... The awards are divided into three groups: identity, text, and quality. The identity group reflects the fidelity of the identity, the text group reflects the relevance of the text, and the quality group reflects the visual quality.

7. The identity-preserving video generation method based on reinforcement evolution according to claim 1, characterized in that, The method further includes calculating a comprehensive evaluation score, EvoScore, for the candidate video or the identity-preserving video, wherein the calculation of EvoScore includes: Video quality is broken down into three dimensions: identity consistency, text fit, and technical quality, with each dimension having several sub-items. By making several independent requests to the multimodal large language model, scores are given for all sub-items under each dimension; For each dimension, the minimum score of all sub-items under that dimension is taken as the score of that dimension according to the weakest link principle; The harmonic mean of the scores in the three dimensions is calculated to obtain the EvoScore; The method further incorporates the EvoScore as an additional reward signal into the dual-path reward mechanism, enabling the evolutionary exploration model to actively improve the weaker dimensions among the three dimensions during training.

8. The identity-preserving video generation method based on reinforcement evolution according to claim 1, characterized in that, The stable anchoring model and / or the evolutionary exploration model are implemented by applying low-rank adaptations to several transformer blocks of the pre-trained identity-preserving diffusion model; the pre-trained identity-preserving diffusion model is trained based on flow matching, and its velocity vector objective is defined as The noise addition process satisfies ; The generated identity-preserving video includes: The target reference identity image is encoded into the latent space by a variational autoencoder to obtain the latent identity representation; Initial noise latent representation of sampling; Using the evolutionary exploration model or the stable anchoring model as conditions, the initial noisy latent representation is subjected to a preset number of denoising iterations. The identity-preserving video is obtained by decoding the denoised latent representation using a variational autoencoder decoder.

9. A video generation system for maintaining identity based on reinforcement evolution, characterized in that, include: The model management module is used to maintain a stable anchoring model, an evolutionary exploration model, a frozen Wensheng video teacher model, and an initial identity preservation diffusion model, wherein the stable anchoring model and the evolutionary exploration model are cloned from the initial identity preservation diffusion model; The candidate sampling module is used to generate multiple candidate videos based on the reference identity image and text description by the stable anchoring model; The reward evaluation module includes an objective indicator reward path and a preference reward path based on a multimodal large language model, which is used to output a reward value for each candidate video. The weight scheduling module is used to output a focus vector for multiple candidate videos in the current iteration through a multimodal large language model, and to calculate the final weight of each reward based on the prior weight ratio and the focus vector. The strategy optimization module is used to predict the velocity vector of the noisy sample and construct the evolution direction by the stable anchoring model and the evolution exploration model, respectively, and construct the strategy loss based on the reward value and the evolution direction, and construct the total loss by combining the T2V teacher regularization term and / or self-constrained regularization term. The parameter update module is used to update the parameters of the evolutionary exploration model based on the total loss, and to update the parameters of the stable anchoring model by exponential moving average. The video generation module is used to generate identity-preserving videos based on the target reference identity image and target text description input by the user after training is completed.

Images