Video generation model training method and device using physical law constraint

Abstract

The embodiment of the invention provides a video generation model training method and device utilizing physical law constraint. In the method, a first text prompt and a front-segment video frame of a real video are input into a video generation model, and a rear-segment video frame of the real video is generated through the video generation model to obtain a plurality of groups of generated rear-segment video frames. Then, extracting a first motion trail of the target object from the rear-segment video frames, and respectively extracting second motion trails corresponding to the target object from the plurality of groups of generated rear-segment video frames; and for a second movement track of any group of generated rear-segment video frames, calculating a track deviation according to the first movement track and the second movement track, and determining a reward score of the group of generated rear-segment video frames according to the track deviation. And when the plurality of groups of reward scores for generating the rear-segment video frames are obtained, executing model updating based on a reinforcement learning algorithm by using the plurality of reward scores so as to update the video generation model. In the model training process, privacy protection needs to be carried out on privacy data possibly involved in the model training process.

Description

TECHNICAL FIELD

[0001] One or more embodiments of the present specification relate to the technical field of video generation, and in particular to a video generation model training method and device using physical law constraints. BACKGROUND

[0002] With the emergence of video generation models such as Sora, Kling, etc., the visual quality of generated videos has made great breakthroughs. Currently, the training paradigm of video generation models mainly relies on large-scale data pixel-level reconstruction, and the model tends to statistically align the vision. At the same time, when private data is involved, the training process also needs to protect the private data. However, although the video generation model can generate realistic textures, it generally lacks understanding of physical laws. In scenes involving rigid body motion (such as object collision, free fall, rolling, pendulum motion), the generated content often violates physical common sense, such as object interpenetration, unstable trajectory, abnormal gravitational acceleration, or time discontinuity, etc. This "physical hallucination" phenomenon seriously limits the application of video generation models in simulation, engineering visualization, and high-quality film production.

[0003] Therefore, it is desirable to have an improved solution to improve the performance of the video generation model so that it can generate videos that are more consistent with physical laws in terms of motion. SUMMARY

[0004] One or more embodiments of the present specification describe a video generation model training method and device using physical law constraints to improve the performance of the video generation model so that it can generate videos that are more consistent with physical laws in terms of motion. The specific technical solutions are as follows.

[0005] In a first aspect, the embodiments provide a video generation model training method using physical law constraints, comprising:

[0006] inputting the first segment of video frames of a real video into a video generation model, generating the second segment of video frames of the real video through the video generation model, and obtaining a plurality of groups of generated second segment of video frames;

[0007] extracting a first motion trajectory of a target object from the second segment of video frames, and extracting a second motion trajectory corresponding to the target object from the plurality of groups of generated second segment of video frames;

[0008] for the second motion trajectory of any one group of generated second segment of video frames, calculating a trajectory deviation according to the first motion trajectory and the second motion trajectory, and determining a reward score of the group of generated second segment of video frames according to the trajectory deviation;

[0009] A model update based on a reinforcement learning algorithm is performed to update the video generation model using the reward scores of the generated post-episode video frames of the plurality of groups.

[0010] In an implementation manner, the step of extracting the first motion trajectory of the target object from the post-episode video frames comprises: inputting the post-episode video frames into a visual segmentation model to obtain an object motion mask of the post-episode video frames; and calculating the first motion trajectory of the target object according to the object motion mask.

[0011] In an implementation manner, the post-episode video frames and the generated post-episode video frames of any group comprise a plurality of video frames; and the step of calculating the trajectory deviation between the first motion trajectory and the second motion trajectory comprises:

[0012] calculating distance deviations between a plurality of coordinate values in the first motion trajectory and corresponding coordinate values in the second motion trajectory to obtain a plurality of distance deviations; and any coordinate value is a coordinate of the target object in a corresponding video frame;

[0013] calculating the trajectory deviation between the first motion trajectory and the second motion trajectory according to the plurality of distance deviations.

[0014] In an implementation manner, the step of calculating the trajectory deviation between the first motion trajectory and the second motion trajectory comprises:

[0015] determining the weight of the plurality of distance deviations by: when the acceleration of the target object in the first video frame included in the post-episode video frames suddenly changes, assigning a high weight to the distance deviation corresponding to the first video frame;

[0016] performing weighted summation on the plurality of distance deviations by using the weight;

[0017] determining the trajectory deviation according to the result of the weighted summation.

[0018] In an implementation manner, the step of determining the weight of the plurality of distance deviations further comprises:

[0019] calculating an acceleration sequence according to the first motion trajectory; wherein the acceleration sequence comprises accelerations of a plurality of time stamps, and the plurality of time stamps are time stamps included in the post-episode video frames;

[0020] detecting a time stamp at which the acceleration suddenly changes from the acceleration sequence, and taking a video frame corresponding to the time stamp in the post-episode video frames as the first video frame.

[0021] In an implementation manner, the step of determining the weight of the plurality of distance deviations further comprises:

[0022] The distance deviation corresponding to the adjacent video frame of the first video frame is given a medium weight, and the distance deviation corresponding to the other video frames except the first video frame and the adjacent video frame is given a low weight.

[0023] In an implementation manner, the step of determining the reward score of the generated post-episode video frame according to the trajectory deviation includes: performing a predetermined negative correlation transformation on the trajectory deviation, and taking a result of the transformation as the reward score of the generated post-episode video frame; wherein a numerical value of the transformation result is negatively correlated with a numerical value of the trajectory deviation.

[0024] In an implementation manner, when the video generation model is updated, the method further includes:

[0025] calculating an average trajectory deviation according to the trajectory deviations corresponding to the generated post-episode video frames;

[0026] when the average trajectory deviation is greater than a preset threshold, determining a flow matching loss according to pixel differences between the post-episode video frame and the generated post-episode video frames;

[0027] updating the video generation model according to the flow matching loss.

[0028] In an implementation manner, after the video generation model is trained, the method further includes:

[0029] inputting a second text prompt and a plurality of pre-episode video frames of a new real video into the video generation model, generating a post-episode video frame of the new real video according to the second text prompt by using the video generation model, and obtaining the generated post-episode video frame;

[0030] extracting a first motion trajectory of a target object from the post-episode video frame, and extracting a second motion trajectory corresponding to the target object from the generated post-episode video frame;

[0031] calculating a trajectory deviation according to the first motion trajectory and the second motion trajectory;

[0032] determining an evaluation score for evaluating the video generation model based on the trajectory deviation.

[0033] In a second aspect, embodiments provide a video generation model training device constrained by physical laws, including:

[0034] a video generation module configured to input pre-episode video frames of a real video into a video generation model, generate post-episode video frames of the real video by using the video generation model, and obtain a plurality of generated post-episode video frames;

[0035] The trajectory extraction module is configured to extract the first motion trajectory of the target object from the subsequent video frames, and to extract the second motion trajectory corresponding to the target object from the several sets of generated subsequent video frames respectively.

[0036] The deviation determination module is configured to calculate the trajectory deviation based on the first motion trajectory and the second motion trajectory for any set of generated subsequent video frames;

[0037] The reward determination module is configured to determine the reward score for the generated subsequent video frames based on the trajectory deviation.

[0038] The model update module is configured to use the reward scores from the several groups of generated subsequent video frames to perform a model update based on a reinforcement learning algorithm, so as to update the video generation model.

[0039] Thirdly, the embodiments provide a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method described in any one of the first aspects.

[0040] Fourthly, an embodiment provides a computing device including a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the method described in any one of the first aspects.

[0041] The method and apparatus provided in the embodiments of this specification compare the motion trajectory of the target object in the later video frame of the real video with the generated later video frame to obtain the trajectory deviation, and obtain a reward score based on the trajectory deviation. This reward score and reinforcement learning algorithm can then be used to update the video generation model. The motion trajectory in the later video frame conforms to the laws of motion. Determining the reward score based on the trajectory deviation between the two motion trajectories allows the motion trajectory generated by the video generation model in the generated later video frames to gradually move closer to conforming to the laws of motion and satisfying the constraints of physical laws. Therefore, the embodiments improve the performance of the video generation model through reinforcement learning, enabling it to generate videos that are more physically consistent in terms of motion. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0043] Figure 1 This is a schematic diagram illustrating an implementation scenario of one embodiment disclosed in this application;

[0044] Figure 2 A flowchart illustrating a video generation model training method constrained by physical laws, provided as an example;

[0045] Figure 3 This is a schematic diagram illustrating the principle of the coordinate values ​​of the target object in each video frame provided in the embodiment.

[0046] Figure 4 A schematic diagram of a process for performing supervised training on a video generation model M, provided as an example;

[0047] Figure 5 This is a schematic block diagram of a video generation model training device that utilizes physical laws as constraints, provided as an embodiment. Detailed Implementation

[0048] The solution provided in this specification will now be described with reference to the accompanying drawings.

[0049] Figure 1 This is a schematic diagram illustrating an implementation scenario of one embodiment disclosed in this application. It includes a dataset, a computing device, and a video generation model. The dataset stores a large number of real videos. The computing device can read real videos from the dataset and generate text prompts for those videos. The text prompts and the preceding video frames of the real videos are input into the video generation model, causing the model to generate the following video frames based on the text prompts and the preceding video frames, resulting in the generated following video frames. The computing device compares the motion trajectories between the generated following video frames and the preceding video frames to calculate the motion trajectory deviation, detects the positions of video frames with abrupt acceleration changes, weights the trajectory deviation using these acceleration changes, and calculates a reward score accordingly. The computing device uses the reward score to update the video generation model using a reinforcement learning algorithm.

[0050] Figure 1 This is merely one application scenario of the video generation method provided in this application. In practical applications, the video generation model can be implemented in a computing device or in other devices. The dataset can be stored in a computing device or in other devices. In this specification, the computing device can be any device, equipment, platform, device cluster, etc., with computing and processing capabilities.

[0051] The dataset contains multiple real videos, which are used as samples to train the video generation model. These real videos are actual videos captured by camera equipment, not videos generated by video generation models or similar tools. The real videos contain moving objects whose trajectories conform to the laws of physics, such as Newton's laws of motion, and do not contain anything that violates common sense. The motion scenarios include rigid body motion scenarios such as collisions, free fall, rolling, and pendulum motion. Rigid body motion is a branch of classical mechanics that studies the motion of rigid bodies under the action of forces, including collisions, free fall, and rolling; its motion is observable and deterministic.

[0052] Video Generative Models (VGMs) are artificial intelligence models used to automatically generate new video content based on input data such as text prompts, images, and / or videos. These models typically have billions to hundreds of billions of parameters, possessing powerful understanding and generation capabilities. Utilizing deep learning algorithms, they can process multimodal information (such as visual, audio, and text) and generate video clips with specific scenes, visual effects, and audio tracks. The generated video clips are formed from multiple video frames. Technically, video generative models predict and synthesize dynamic content by learning spatiotemporal patterns in data. They primarily rely on architectures such as Generative Adversarial Networks (GANs), autoregressive models, Diffusion Transformer (DiT) based models, and flow matching models. For example, diffusion models generate videos through a progressive denoising process, while GANs enhance the realism of videos through adversarial training between the generator and discriminator.

[0053] Video generation models are pre-trained on large-scale video-text pairs to learn video content, temporal dynamics, and semantic relationships. For example, a video generation model can be trained with 30 billion parameters using hundreds of thousands of video-text pairs, and can achieve coherent generation of long videos through pre-training on native video continuation tasks. The pre-trained video generation model can then be fine-tuned or subjected to other training for specific tasks to improve its practicality or performance.

[0054] Unlike image generation, video generation requires simultaneously modeling both the appearance of objects and their physical motion to avoid unnatural motion artifacts. However, current training paradigms for video generation models primarily rely on pixel-level reconstruction from large-scale data, leading to a preference for statistical visual alignment rather than intrinsic physical consistency. To address the issue of generated content frequently violating physical realities, reinforcement learning can be used to further train video generation models in practical applications.

[0055] Reinforcement learning (RL) is a machine learning method based on the Markov decision process. It allows an agent to learn better policies through trial and error in its interactions with the environment. The agent performs actions in the environment and receives feedback, or rewards, based on the results of those actions. These rewards guide the agent to adjust its policy to maximize long-term cumulative rewards.

[0056] Using a video generation model as an agent, when determining rewards, the generated videos can be scored using a Multi-modal Large Language Model (MLLM) or human feedback. This score can then be used as a reward to optimize the video generation model through reinforcement learning. This approach can also ensure that the generated content conforms to physical common sense to some extent. However, this feedback signal is subjective and coarse-grained, making it impossible to accurately verify whether the generated content adheres to physical laws (such as Newton's second law). This results in the model struggling to capture subtle dynamic differences (such as sudden acceleration changes during collisions).

[0057] To improve the quality of generated videos and enable video generation models to produce videos that better conform to physical laws, this application provides a video generation model training method that utilizes physical constraints. By comparing the motion trajectories of real and generated videos, trajectory deviations are obtained. The estimated deviations are used to determine reward scores, penalizing behaviors that violate physical laws as determined by the video generation model. This embodiment introduces a physically-aware reward function, so that the video generation model no longer merely optimizes pixel similarity but is directly penalized for behaviors that violate physical laws. This results in generated videos with a considerably high fidelity in rigid body dynamics. The following describes... Figure 2 The embodiments are described in detail below.

[0058] Figure 2 This is a flowchart illustrating a video generation model training method constrained by physical laws, provided as an embodiment. The method is executed via a computing device and includes the following steps.

[0059] Step S210: Input the first video frame R1 of the real video into the video generation model (hereinafter denoted by M). The video generation model M generates the second video frame R2 of the real video, resulting in several sets of generated second video frames V (V1~Vn). "Several" may include one or more. Each set of generated second video frames includes multiple video frames. The multiple sets of generated second video frames V are obtained by the video generation model M through multiple generation processes.

[0060] The aforementioned real video can be any real video obtained from the dataset. This real video contains moving objects, and the trajectory of these moving objects conforms to the laws of physics (e.g., the laws of motion). This real video is filmed in the real world using camera equipment, not a model-generated video. The moving objects include the target object, which is the object to be detected. The first video frame R1 consists of a few frames, for example, the first 5 frames of the real video, while the second video frame R2 consists of a large number of frames from the real video. Both the first video frame R1 and the second video frame R2 contain the trajectory of the target object, and this trajectory is continuous between them.

[0061] The video generation model M can be any pre-trained video generation model, used to generate the subsequent video frame R2 of the real video based on the preceding video segment R1, resulting in the generated subsequent video frame V. In practice, the preceding video segment R1 of the real video can be input into the video generation model M multiple times to obtain multiple sets of generated subsequent video frames V.

[0062] In practical applications, a prompt word can be constructed from the first text prompt and the preceding video frame R1 of the actual video. This prompt word is then input into the video generation model M. The first text prompt contains a description of the video frame to be generated, i.e., the following video frame R2, which instructs the video generation model M to generate the video frame according to the aforementioned description. The video generation model M can then generate a video based on the first text prompt and the preceding video frame R1, resulting in a set of generated following video frames.

[0063] The aforementioned description can be an accurate description of the motion trajectory of the target object in a real video. An accurate description allows the generated subsequent video frame V to more closely approximate the motion trajectory in the real video, thus making training more effective. For example, if a real video contains the motion trajectory of a kitten jumping across a stream, the first cue text could be an accurate description of that motion trajectory.

[0064] Step S220: Extract the first motion trajectory of the target object from the subsequent video frame R2, and extract the second motion trajectory corresponding to the same target object from several groups of generated subsequent video frames V.

[0065] For example, the second motion trajectory x1 of the target object in the generated video frame V1, the second motion trajectory x2 of the target object in the generated video frame V2, and the second motion trajectory xn of the target object in the generated video frame Vn were extracted respectively.

[0066] Any motion trajectory includes the coordinates of the target object in the corresponding video frame and the timestamp of that video frame.

[0067] When extracting motion trajectories from video frames, visual segmentation models can be used. These models can be, but are not limited to, SAM2 (Segment Anything Model 2) or similar models. The SAM2 model is an advanced visual segmentation model capable of continuously tracking and segmenting objects in a video, generating spatiotemporal masks.

[0068] In this step, when determining the first motion trajectory, the subsequent video frame R2 can be input into the visual segmentation model. The object motion mask of the subsequent video frame R2 can be obtained through the visual segmentation model. Based on the object motion mask, the first motion trajectory of the target object can be calculated.

[0069] When determining the second motion trajectory of any set of generated post-video frames V, the set of generated post-video frames V can be input into the visual segmentation model. The object motion mask of the set of generated post-video frames V can be obtained through the visual segmentation model. Based on the object motion mask, the second motion trajectory of the target object can be calculated.

[0070] Specifically, the coordinates (x, y) of the target object can be marked in the first frame of the subsequent video frame R2 and several sets of generated subsequent video frames V. The marked video frames are then input into the video segmentation model so that the video segmentation model can extract the motion trajectory of the target object from the subsequent video frame R2 and the set of generated subsequent video frames V based on the marked object coordinates.

[0071] A motion mask is a binary mask that corresponds one-to-one with a video frame, where white pixels represent object regions and black pixels represent non-object regions. Motion masks can distinguish moving pixels from the static background in each video frame.

[0072] The coordinates of the target object in each video frame can be extracted based on the object motion mask. The sequence of multiple coordinates formed in order constitutes the motion trajectory of the target object. That is, for any one of the first and second motion trajectories, the motion trajectory contains multiple coordinate values. Each coordinate value is the center coordinate of the target object in the corresponding video frame. The coordinate values ​​involved in this application can be, but are not limited to, coordinates in the image coordinate system, or coordinates in the camera coordinate system or the world coordinate system. Extracting coordinates from the video frame in the image coordinate system is a more easily calculated implementation. Regardless of which coordinate system is used, the first and second motion trajectories are represented using coordinates in the same coordinate system.

[0073] Figure 3 This is a schematic diagram illustrating the principle of the coordinate values ​​of a target object in various video frames provided in this embodiment. It includes the coordinate values ​​of each frame in the subsequent video frame R2 of the actual video and the coordinate values ​​of each frame in a generated subsequent video frame. Both the subsequent video frame R2 and the generated subsequent video frame contain 6 frames with timestamps t1 to t6. The timestamps of the 6 frames in the subsequent video frame R2 and the generated subsequent video frame are mutually corresponding and identical. Circles represent the position of the target object in the video frame, and dashed arrows represent the direction of movement of the target object in multiple video frames. The first motion trajectory of the target object in the subsequent video frame R2 includes the following coordinate values: (x1, y1), (x2, y2), ..., (x6, y6). The second motion trajectory of the target object in the generated subsequent video frame includes the following coordinate values: (x1′, y1′), (x2′, y2′), ..., (x6′, y6′).

[0074] Step S230: For any group of generated post-video frames (e.g., V1), calculate the trajectory deviation between the first and second motion trajectories, and determine the reward score reward1 for that group of generated post-video frames V1 based on the trajectory deviation. In this way, calculate the trajectory deviation between the first motion trajectory and the second motion trajectories of all groups of generated post-video frames V, and obtain the reward scores for several groups of generated post-video frames (V1, V2, ..., Vn), including reward1, reward2, ..., rewardn. The following explanation uses a group of generated post-video frames V1 as an example.

[0075] The first motion trajectory is the actual physical trajectory of the target object in the real video (Ground Truth). The trajectory offset (TO) is the deviation of the generated video frame V1 from the actual physical trajectory.

[0076] When calculating the trajectory deviation, the following steps 1 and 2 can be followed:

[0077] Step 1: Calculate the distance deviation between several coordinate values ​​in the first motion trajectory and the corresponding coordinate values ​​in the second motion trajectory to obtain several distance deviations.

[0078] It is understandable that the subsequent video frame R2 and the generated subsequent video frame V1 contain several corresponding video frames, and any coordinate value is the coordinate of the target object in the image coordinate system in the corresponding video frame.

[0079] by Figure 3 For example, the first motion trajectory includes the following coordinate values: (x1, y1), (x2, y2), ..., (x6, y6); the second motion trajectory includes the following coordinate values: (x1′, y1′), (x2′, y2′), ..., (x6′, y6′). The distance deviation d1 between (x1, y1) and (x1′, y1′), the distance deviation d2 between (x2, y2) and (x2′, y2′), ..., the distance deviation d6 between (x6, y6) and (x6′, y6′) can be calculated.

[0080] The distance deviation between any two coordinate values ​​can be calculated using the Euclidean distance formula to obtain the Euclidean distance value, which is then used as the distance deviation between the two coordinate values. Each distance deviation has a corresponding timestamp, which is the timestamp of the corresponding video frame.

[0081] Step 2: Based on several distance deviations, calculate the trajectory deviation between the first motion trajectory and the second motion trajectory, which is used as the trajectory deviation corresponding to the generated post-video frame V1, representing the trajectory deviation of the generated post-video frame V1 relative to the real video.

[0082] Specifically, the sum or average of several distance deviations can be used as the trajectory deviation. Using the average value as the trajectory deviation better reflects the degree of trajectory deviation of the generated video frame V1 relative to the real video.

[0083] To ensure that the trajectory deviation more accurately reflects its degree, this embodiment can also calculate a weighted average for several distance deviations, and use this weighted average as the trajectory deviation. The weight of each distance deviation is a crucial parameter.

[0084] The time point in a video frame where acceleration changes abruptly is more indicative of whether the motion of the target object conforms to the laws of physics and should be given higher weight. Therefore, in this embodiment, the first motion trajectory can be used as the standard trajectory, from which the time point of acceleration change abruptly is extracted. The trajectory deviation at this time point is assigned a higher weight, while the trajectory deviation at other non-acceleration time points is assigned a lower weight.

[0085] In one implementation, when performing step 2, the weights of several distance deviations can be determined, and the weights are used to perform a weighted summation of the several distance deviations. The trajectory deviation is then determined based on the result of the weighted summation.

[0086] When determining the weights of each distance deviation, an acceleration sequence can be calculated based on the first motion trajectory. The timestamps of abrupt acceleration changes are then detected from this sequence, and the video frame corresponding to these timestamps in the subsequent video frame R1 is taken as the first video frame. The first video frame is the video frame where the target object experiences a sudden acceleration change, and its timestamp is the timestamp of this acceleration change. The first video frame can be one or more video frames, meaning there can be one or more locations in the subsequent video frame R1 where the acceleration change occurs.

[0087] When determining the first video frame, a high weight can be assigned to the distance deviation corresponding to the first video frame, a medium weight to the distance deviation corresponding to the adjacent video frames, and a low weight to the distance deviations corresponding to other video frames besides the first and adjacent video frames. Alternatively, only two weight values ​​can be assigned, for example, assigning a high weight to the distance deviation corresponding to the first video frame and a low weight to the distance deviations corresponding to other video frames besides the first video frame.

[0088] The weight values ​​for high, medium, and low weights can be preset values. For example, high weight can be 3, medium weight can be 2, and low weight can be 1.

[0089] The acceleration sequence described above contains several accelerations, and the timestamp of each acceleration is the timestamp of the corresponding first video frame. When calculating the acceleration sequence based on the first motion trajectory, the acceleration can be calculated using multiple coordinate values ​​contained in the first motion trajectory and the timestamps corresponding to those coordinate values. For example, using three sets of coordinate values ​​of a vector and their timestamps, the corresponding acceleration can be calculated using the acceleration formula.

[0090] by Figure 3 Let's take an example to illustrate the assignment of weights. The first motion trajectory includes the following coordinate values: (x1, y1), (x2, y2), ..., (x6, y6) and the corresponding timestamps t1 to t6. Based on the coordinate values ​​(x1, y1), (x2, y2), and (x3, y3) and their corresponding timestamps, the acceleration a1 at timet3 can be calculated. Based on the coordinate values ​​(x2, y2), (x3, y3), and (x4, y4) and their corresponding timestamps, the acceleration a2 at timet4 can be calculated. And so on, the acceleration a3 at timet5 and the acceleration a4 at timet6 are calculated.

[0091] When detecting abrupt acceleration changes in an acceleration sequence, any acceleration can be compared with a vector acceleration. An acceleration change is considered to have occurred when the difference in accelerations exceeds a threshold. Assume an acceleration change occurs at acceleration a2, where a2 represents a significant change relative to a1 and / or a3. When assigning weights, the timestamp corresponding to acceleration a2 can be assigned a high weight (value 3), the timestamps of adjacent accelerations a1 and a3 can be assigned a medium weight (value 2), and the timestamps of other accelerations a4 can be assigned a low weight (value 1). The weight of each timestamp can be used as the weight of the distance deviation. The weighted sum or average of multiple distance deviations is used as the trajectory deviation.

[0092] When determining the reward score reward1 of the generated video frame V1 based on the trajectory deviation, a predetermined negative correlation transformation can be performed on the trajectory deviation value, and the transformation result can be used as the reward score reward1 of the generated video frame V1.

[0093] The value of the transformation result is negatively correlated with the value of the trajectory deviation. Negative correlation transformation includes operations such as taking a negative value or inverting the value. Negative correlation means that the two variables change in opposite directions; if one increases, the other decreases. A larger reward score indicates that the motion trajectory of the target object in the generated video frame approximately conforms to the laws of physics, meaning it's better. Conversely, a larger trajectory deviation indicates a greater degree of deviation between the generated video frame and the actual video trajectory. Therefore, the trajectory deviation can be transformed so that the transformed result can be used as a reward score.

[0094] As can be seen from the above description, steps S220 to S230 are the processing steps of the reward function. The input to the reward function is the subsequent video frame R2 and several sets of generated subsequent video frames V, and the output is the reward score corresponding to each of the several sets of generated subsequent video frames V. Acceleration abrupt changes occur because the target object is subjected to a force, which may be due to rigid body motion such as collision, free fall, rolling, or pendulum motion. Using the acceleration abrupt changes obtained from collision detection to weight the distance deviation, giving higher weight to the distance deviation corresponding to the acceleration abrupt changes, makes the obtained trajectory deviation more accurate. Using collision detection to weight the distance deviation not only measures the accuracy of the motion but also forces the model to focus on the moment of most intense physical interaction (i.e., the instant of collision).

[0095] In step S240, the reward scores of several sets of generated subsequent video frames V are used to perform a model update based on a reinforcement learning algorithm to update the video generation model M.

[0096] Here, reinforcement learning algorithms can employ GRPO, PPO, or other similar algorithms. GRPO (Group Relative Policy Optimization) is a reinforcement learning optimization algorithm that updates the model policy by evaluating the relative advantages of a set of generated samples. It is generally more efficient than the traditional PPO (Proximal Policy Optimization) algorithm because it does not require training a separate value model.

[0097] In this embodiment, the video generation model M acts as an agent, generating several sets of videos V as samples. The relative advantage assessment evaluates which of these samples has a greater advantage. In other words, using a set of samples generated by the current policy, the relative performance between them guides the policy to adjust in a better direction.

[0098] In this step, an average reward score can be calculated based on the reward scores of several groups of generated post-video frames V. Then, the relative reward of each group of generated post-video frames relative to this average reward score is calculated, thus obtaining several relative rewards. These relative rewards are then used to update the video generation model M. The specific parameter update process can be found in existing technologies and will not be detailed here.

[0099] Taking the DiT model as an example, all Transformer blocks participate in training. When using the GRPO algorithm for model updates, the denoising process is modeled as a Markov Decision Process (MDP). To improve exploration efficiency, a hybrid ODE-SDE sampling strategy is adopted, that is, random noise is introduced in the first few time steps of generation (SDE stage), and subsequent steps use deterministic ODE to solve, thereby increasing the diversity of the action space while maintaining stability.

[0100] In other words, the denoising process of the video generation model includes two different stages based on two techniques: a stage based on Differential Stochastic Processes (SDE) and a stage based on Ordinary Differential Stochastic Processes (ODE). The model uses the SDE strategy in the initial stage of denoising and the ODE strategy in the later stage. The SDE strategy has randomness, which in reinforcement learning manifests as exploring different directions. During the ODE execution stage, the image content is already determined, and the parameters for this part do not need to be updated in reinforcement learning.

[0101] In this embodiment, a physics-aware reward function based on trajectory deviation is introduced. Instead of simply optimizing pixel similarity as in the pre-training phase, the model is directly penalized for behaviors that violate physical laws. This improves the performance of the video generation model, making the motion trajectories of the generated videos more consistent with physical laws.

[0102] When the implementation incorporates a physics-aware reward function that includes collision detection, the model can be more specifically penalized for behaviors that violate the laws of physics, such as failing to bounce after a collision and unreasonable velocity abrupt changes—key behaviors related to abrupt acceleration changes. This results in videos with very high fidelity in rigid body dynamics.

[0103] In another embodiment of this application, in Figure 2 Based on the illustrated embodiment, when updating the video generation model M, it may further include... Figure 4 The following steps are shown. Figure 4 This is a schematic diagram illustrating a process for performing supervised training on a video generation model M, as provided in an embodiment.

[0104] Step S250: Calculate the average trajectory deviation based on the trajectory deviations corresponding to several groups of generated post-video frames V.

[0105] For example, the trajectory deviations corresponding to multiple sets of generated post-video frames V1, V2, ..., Vn are dev1, dev2, ..., devn, respectively. Summing these trajectory deviations and dividing the sum by n yields the average trajectory deviation.

[0106] Step S260: Compare the average trajectory deviation with a preset threshold. When the average trajectory deviation is greater than the preset threshold, determine the stream matching loss based on the pixel differences between the subsequent video frame R2 and several groups of generated subsequent video frames V.

[0107] Step S270: Update the video generation model M based on the stream matching loss. The parameter update of the video generation model M in this step is part of the supervised learning process during the pre-training phase. The specific update process can be found in existing techniques and will not be elaborated here.

[0108] When the average trajectory deviation is not greater than a preset threshold, only the model update based on the reinforcement learning algorithm in step S240 can be performed.

[0109] When the average trajectory deviation exceeds a preset threshold, it indicates that the video generation model M has a very poor grasp of the physical scene, and the generated video cannot even be effectively evaluated. At this point, the computing device activates a supervised learning branch, using flow matching loss to force the video generation model M to "imitate" the pixel distribution of the real data to quickly correct visual quality. Simultaneously, the aforementioned reward points are used to perform model updates based on reinforcement learning algorithms, forcing the video generation model M to "explore" and "understand" physical laws. Flow matching is a method for training generative models of continuous normalized flows, mapping a simple noise distribution to a complex data distribution by regressing a vector field.

[0110] When the average trajectory deviation is less than or equal to the preset threshold, it indicates that the video generation model M has basic generation capabilities and does not need to be executed. Figure 4 The supervised learning shown is performed only. Figure 2 The reinforcement learning update is shown in step S240. Regardless of the situation, the computing device always activates the reinforcement learning branch, using the GRPO algorithm to optimize the policy and maximize the physical trajectory deviation reward.

[0111] The model training process described above, including the model update processes shown in steps S210–S240 and S210–S270, can be performed iteratively. In the actual training process, imitation and exploration alternate. Initially, imitation and exploration are combined; as the model's capabilities improve, it gradually transitions to being primarily exploration-based, ultimately achieving the internalization of physical knowledge.

[0112] The aforementioned alternating training mechanism cleverly combines the stability of supervised learning (imitation) with the generalization of reinforcement learning (exploration). When the video learning model M "doesn't understand" physics (when the average trajectory deviation is greater than a preset threshold), it is first taught to "see" the data (imitation stage); when the model "understands" (when the average trajectory deviation is not greater than the preset threshold), it is then allowed to "understand" the rules through reinforcement learning (RL) (exploration stage).

[0113] The "imitation-exploration" dynamic switching training mechanism proposed in this embodiment adaptively switches between pixel-level supervised loss and physical-level reinforcement learning (RL) loss based on the model's current physical consistency performance. This solves the problems of difficulty in convergence and cold start in reinforcement learning during video generation, and also avoids the problem of failure due to sparse rewards.

[0114] Experiments have shown that the video generation model M trained using this embodiment significantly outperforms state-of-the-art models (such as CogVideoX and Kling) in terms of trajectory overlap (IoU) and trajectory deviation (TO), and can generate high-quality videos with accurate collisions and realistic gravity.

[0115] After training the video generation model M, it can be deployed online. In this embodiment, rewards are calculated using physical annotations only during the training phase. During the online inference phase, the model has internalized physical knowledge and can directly generate videos that conform to physical laws end-to-end without any external solver assistance.

[0116] In another embodiment of this application, after training the video generation model M, the video generation model M can also be evaluated based on physical law constraints. This can be achieved through steps 3 to 6.

[0117] Step 3: Input the second text prompt and the first segment video frame of the new real video into the video generation model M. Based on the second text prompt and the first segment video frame, the video generation model M generates the second segment video frame of the new real video, thus obtaining the generated second segment video frame.

[0118] The new real video can be any real video from the evaluation dataset. The second cue text is a relevance description used to explain the subsequent video frames.

[0119] Step 4: Extract the first motion trajectory of the target object from the later video frames, and extract the second motion trajectory of the target object from the generated later video frames.

[0120] Step 5: Calculate the trajectory deviation based on the first and second motion trajectories.

[0121] Step 6: Determine the evaluation score for evaluating the video generation model M based on the trajectory deviation.

[0122] For example, trajectory deviation can be directly used as the evaluation score. Alternatively, the trajectory deviation can be negatively transformed, such as by taking a negative value or its inverse, and the transformed result can be used as the evaluation score. In this case, the higher the evaluation score, the better the model performance, meaning that the video generation model M has a more thorough understanding of the physical laws.

[0123] The execution process of step 3 can be referred to the relevant description of step S210, the execution process of step 4 can be referred to the relevant description of step S220, and the execution process of step 5 can be referred to the relevant description of step S230. They will not be repeated here.

[0124] In this embodiment, the new real video can be selected from videos containing basic motion scenarios such as collision, pendulum, free fall and rolling, so as to effectively evaluate the physical properties of the video generation model M.

[0125] In this specification, the terms "first" in the first motion trajectory, the first video frame, etc., and the corresponding "second" (if present) in the text, are merely for the convenience of distinction and description, and do not have any limiting meaning.

[0126] The foregoing description describes specific embodiments of this specification; other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than those shown in the embodiments, and the desired result may still be achieved. Furthermore, the processes depicted in the drawings do not necessarily need to follow the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0127] Figure 5This is a schematic block diagram of a video generation model training device constrained by physical laws, provided as an embodiment. This device embodiment is related to... Figure 2 The method embodiment shown corresponds to this. The device 500 includes: a video generation module 510, a trajectory extraction module 520, a deviation determination module 530, a reward determination module 540, and a model update module 550. The video generation module 510 is configured to input several first-stage video frames of a real video into a video generation model, and generate several sets of generated second-stage video frames from the real video through the video generation model. The trajectory extraction module 520 is configured to extract a first motion trajectory of a target object from the second-stage video frames, and extract a second motion trajectory corresponding to the target object from each of the several sets of generated second-stage video frames. The deviation determination module 530 is configured to calculate a trajectory deviation between the first motion trajectory and the second motion trajectory of any set of generated second-stage video frames. The reward determination module 540 is configured to determine a reward score for the set of generated second-stage video frames based on the trajectory deviation. The model update module 550 is configured to use the reward scores of the several sets of generated second-stage video frames to perform a model update based on a reinforcement learning algorithm to update the video generation model.

[0128] In one implementation, the trajectory extraction module 520, when extracting the first motion trajectory of the target object from the subsequent video frame, includes: inputting the subsequent video frame into a visual segmentation model to obtain the object motion mask of the subsequent video frame, and calculating the first motion trajectory of the target object based on the object motion mask.

[0129] In one implementation, the subsequent video frames and any set of generated subsequent video frames contain several video frames. The deviation determination module 530 includes a distance calculation submodule 31 and a trajectory calculation submodule 32. The distance calculation submodule 31 is configured to calculate the distance deviations between several coordinate values ​​in the first motion trajectory and corresponding coordinate values ​​in the second motion trajectory, obtaining several distance deviations. Any coordinate value is the coordinate of the target object in the corresponding video frame. The trajectory calculation submodule 32 is configured to calculate the trajectory deviation between the first motion trajectory and the second motion trajectory based on the several distance deviations.

[0130] In one implementation, the trajectory calculation submodule 32 includes a weighting unit 32_1, a summation unit 32_2, and a determination unit 32_3. The weighting unit 32_1 is configured to determine the weights of several distance deviations by assigning a high weight to the distance deviation corresponding to the first video frame when the acceleration of the target object in the first video frame contained in the subsequent video frame changes abruptly. The summation unit 32_2 is configured to perform a weighted summation of the several distance deviations using the weights. The determination unit 32_3 is configured to determine the trajectory deviation based on the result of the weighted summation.

[0131] In one implementation, the weighting unit 32_1 is further configured to: calculate an acceleration sequence based on the first motion trajectory; detect the timestamp when the acceleration changes abruptly from the acceleration sequence; and use the video frame corresponding to the timestamp in the subsequent video frame as the first video frame. The acceleration sequence contains accelerations with several timestamps, and these timestamps are those included in the subsequent video frame.

[0132] In one implementation, the weighting unit 32_1 is further configured to: assign a medium weight to the distance deviation corresponding to the adjacent video frames of the first video frame, and assign a low weight to the distance deviation corresponding to the first video frame and other video frames other than the adjacent video frames.

[0133] In one implementation, the reward determination module 540 is specifically configured to: perform a predetermined negative correlation transformation on the trajectory deviation, and use the transformed result as the reward score for the generated subsequent video frames. The value of the transformation result is negatively correlated with the value of the trajectory deviation.

[0134] In one implementation, the device 500 further includes a supervised update module 560, configured to update the video generation model using supervised training. The supervised update module 560 includes a gating calculation submodule, a loss determination submodule, and a model update submodule (not shown in the figure). The gating calculation submodule is configured to calculate the average trajectory deviation based on the trajectory deviations corresponding to several sets of generated subsequent video frames when updating the video generation model. The loss determination submodule is configured to determine the stream matching loss based on the pixel differences between the subsequent video frames and several sets of generated subsequent video frames when the average trajectory deviation exceeds a preset threshold. The model update submodule is configured to update the video generation model based on the stream matching loss.

[0135] In one implementation, the device 500 further includes a model evaluation module (not shown in the figure), configured to evaluate the video generation model based on physical law constraints after training the video generation model. The model evaluation module includes a video generation submodule, a trajectory extraction submodule, a deviation calculation submodule, and a score determination submodule (not shown in the figure). The video generation submodule is configured to input a second text prompt and several first-segment video frames of a new real video into the video generation model, and generate second-segment video frames of the new real video based on the second text prompt. The trajectory extraction submodule is configured to extract a first motion trajectory of the target object from the second-segment video frames and extract a second motion trajectory corresponding to the target object from the generated second-segment video frames. The deviation calculation submodule is configured to calculate the trajectory deviation based on the first and second motion trajectories. The score determination submodule is configured to determine an evaluation score for evaluating the video generation model based on the trajectory deviation.

[0136] The above-described apparatus embodiments correspond to the method embodiments, and detailed descriptions can be found in the description of the method embodiments section, which will not be repeated here. The apparatus embodiments are derived based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments; detailed descriptions can be found in the corresponding method embodiments.

[0137] This specification also provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform... Figures 1 to 4 Any one of the methods described.

[0138] This specification also provides a computing device, including a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement... Figures 1 to 4 Any one of the methods described.

[0139] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for storage media and computing devices are basically similar to the method embodiments, so they are described more simply; relevant parts can be referred to the descriptions of the method embodiments.

[0140] Those skilled in the art will recognize that the functions described in the embodiments of the present invention in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0141] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, or improvements made based on the technical solutions of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for training a video generation model constrained by physical laws, comprising: The first part of the video frame of the real video is input into the video generation model, and the second part of the video frame of the real video is generated by the video generation model to obtain several sets of generated second part video frames. The first motion trajectory of the target object is extracted from the subsequent video frames, and the second motion trajectory corresponding to the target object is extracted from the several sets of generated subsequent video frames respectively. For any group of generated video frames, the second motion trajectory is used to calculate the trajectory deviation between the first motion trajectory and the second motion trajectory, and the reward score for the group of generated video frames is determined based on the trajectory deviation. Using the reward scores from the generated subsequent video frames from the aforementioned groups, a model update based on a reinforcement learning algorithm is performed to update the video generation model.

2. The method according to claim 1, wherein the step of extracting the first motion trajectory of the target object from the subsequent video frame comprises: The latter video frame is input into the visual segmentation model to obtain the object motion mask of the latter video frame; Calculate the first motion trajectory of the target object based on the object motion mask.

3. The method according to claim 1, wherein the subsequent video frames and any set of generated subsequent video frames include a plurality of video frames; the step of calculating the trajectory deviation based on the first motion trajectory and the second motion trajectory includes: Calculate the distance deviations between several coordinate values ​​in the first motion trajectory and the corresponding coordinate values ​​in the second motion trajectory to obtain several distance deviations; Any coordinate value is the coordinate of the target object in the corresponding video frame; Based on the aforementioned distance deviations, the trajectory deviation between the first motion trajectory and the second motion trajectory is calculated.

4. The method according to claim 2, wherein the step of calculating the trajectory deviation between the first motion trajectory and the second motion trajectory includes: The weights of the distance deviations are determined as follows: when the acceleration of the target object in the first video frame contained in the subsequent video frame changes abruptly, a high weight is assigned to the distance deviation corresponding to the first video frame. The weights are used to perform a weighted summation of the distance deviations; The trajectory deviation is determined based on the weighted summation result.

5. The method according to claim 4, wherein the step of determining the weights of the plurality of distance deviations by means of: An acceleration sequence is calculated based on the first motion trajectory; wherein, the acceleration sequence includes accelerations with several timestamps, and the several timestamps are timestamps contained in the subsequent video frames; Detect the timestamp when the acceleration changes abruptly from the acceleration sequence, and use the video frame corresponding to the timestamp in the subsequent video frame as the first video frame.

6. The method according to claim 4, wherein the step of determining the weights of the plurality of distance deviations by means of: A medium weight is assigned to the distance deviation corresponding to the adjacent video frames of the first video frame, and a low weight is assigned to the distance deviation corresponding to the first video frame and other video frames besides the adjacent video frames.

7. The method according to claim 1, wherein the step of determining the reward score for the generated subsequent video frames based on the trajectory deviation comprises: A predetermined negative correlation transformation is applied to the trajectory deviation, and the transformation result is used as the reward score for generating subsequent video frames in that group; wherein, the value of the transformation result is negatively correlated with the value of the trajectory deviation.

8. The method according to claim 1, further comprising, when updating the video generation model: The average trajectory deviation is calculated based on the trajectory deviations corresponding to the several groups of generated post-video frames. When the average trajectory deviation is greater than a preset threshold, the stream matching loss is determined based on the pixel difference between the subsequent video frame and the several groups of generated subsequent video frames. The video generation model is updated based on the stream matching loss.

9. The method according to claim 1, further comprising, after training the video generation model: The second text prompt and several first-segment video frames of the new real video are input into the video generation model. The video generation model generates the second-segment video frames of the new real video based on the second text prompt, thus obtaining the generated second-segment video frames. The first motion trajectory of the target object is extracted from the latter video frame, and the second motion trajectory corresponding to the target object is extracted from the generated latter video frame; Calculate the trajectory deviation based on the first motion trajectory and the second motion trajectory; An evaluation score is determined based on the trajectory deviation to evaluate the video generation model.

10. A training device for a video generation model constrained by physical laws, comprising: The video generation module is configured to input the first segment video frames of a real video into a video generation model, and generate the second segment video frames of the real video through the video generation model to obtain several sets of generated second segment video frames. The trajectory extraction module is configured to extract the first motion trajectory of the target object from the subsequent video frames, and to extract the second motion trajectory corresponding to the target object from the several sets of generated subsequent video frames respectively. The deviation determination module is configured to calculate the trajectory deviation based on the first motion trajectory and the second motion trajectory for any set of generated subsequent video frames; The reward determination module is configured to determine the reward score for the generated subsequent video frames based on the trajectory deviation. The model update module is configured to use the reward scores from the several groups of generated subsequent video frames to perform a model update based on a reinforcement learning algorithm, so as to update the video generation model.

11. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1-9.

12. A computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of any one of claims 1-9.

Images