Video generation method based on mask with body

Abstract

The invention discloses a video generation method based on a body mask, belongs to the technical field of artificial intelligence, and can solve the problems that an existing body world model is inconsistent in an action space and a pixel space, is sensitive to the change of a visual angle of a camera, and is not uniform in architecture among different body structures. The method comprises the following steps: S1, determining a mask sequence with a body according to a target video; s2, encoding the body mask sequence and the initial frame image of the target video respectively to correspondingly obtain body mask features and image encoding features; s3, inputting the body mask features into a control network module to obtain injection features, and inputting the image coding features into a backbone network of a video generation model to obtain backbone layer features; and S4, fusing the injection feature and the trunk layer feature to obtain a fused feature, and generating a prediction video according to the fused feature. The method is used for generating the predictive video with the body.

Description

TECHNICAL FIELD

[0001] The present application relates to a body mask-based video generation method, and belongs to the technical field of artificial intelligence. BACKGROUND

[0002] With the deepening of research on embodied intelligence and world model, embodied world model has shown significant potential in depicting the dynamic interaction between robots and the environment, and is widely used in robot operation, strategy evaluation and target planning tasks. Research shows that this kind of model can predict the future state of the environment under different actions, so as to realize strategy evaluation in a virtual simulation environment and target image-based planning. However, the existing methods still have limitations in design and generalization, mainly in the following three points: (1) Action-video space mismatch: current action-conditioned models usually represent the robot end effector pose as a low-dimensional coordinate vector, while video generation models run in high-dimensional pixel space. The two are inconsistent in representation dimension and semantics, making it difficult for action signals to effectively guide the generation process, limiting the full use of visual and motion priors in pre-trained video models. (2) Viewpoint sensitivity: actions defined based on coordinate space depend on a specific camera viewpoint. Once the viewpoint changes, even if the same action is performed, the model may still generate unreasonable or physically inconsistent prediction results. This sensitivity to the viewpoint limits the applicability of the model in multi-view and real scenes. (3) Cross-embodied structure inconsistency: most existing methods are designed for different embodied structures (such as single-arm and dual-arm systems) with independent action encoders and network structures, making it difficult to share knowledge and extend the model, hindering the construction of a universal embodied world model.

[0003] To solve the above problems, current research mainly follows two paths: one is to train an action-conditioned world model from scratch, relying only on robot data from a specific scene. This method performs well in a single scene, but has limited generalization ability across tasks and across embodied structures. The other class uses large-scale Internet videos or pre-trained video generation models, introducing action conditioning mechanisms through fine-tuning, thereby improving generation quality with visual and motion priors. However, these methods are still limited by the fundamental problems of action-video space misalignment, strong viewpoint dependence, and inconsistent model structure, making it difficult to adapt to real complex robot application scenarios. SUMMARY

[0004] The present application provides a body mask-based video generation method, which can solve the problems of inconsistency between action space and pixel space, sensitivity to camera viewpoint changes, and inconsistent architecture between different embodied structures in existing embodied world models.

[0005] The present application provides a body mask-based video generation method, which comprises:

[0006] S1, determining a body mask sequence according to the target video;

[0007] S2, respectively encoding the body mask sequence and an initial frame image of the target video to obtain body mask features and image encoding features;

[0008] S3, inputting the body mask features into a control net module to obtain injection features, and inputting the image encoding features into a backbone network of a video generation model to obtain backbone layer features;

[0009] S4, fusing the injection features and the backbone layer features to obtain fusion features, and generating a predicted video according to the fusion features.

[0010] Optionally, the S1 specifically comprises:

[0011] determining a robot action sequence according to the target video;

[0012] determining body three-dimensional pose information according to a robot structure description file, camera parameters and the robot action sequence, and projecting the body three-dimensional pose information to an image plane to obtain the body mask sequence.

[0013] Optionally, the S1 specifically comprises:

[0014] extracting the body mask sequence from the target video by using a segmentation model.

[0015] Optionally, before the S3, the method further comprises:

[0016] constructing a loss function; the loss function comprises a diffusion loss;

[0017] determining parameters of the control net module according to the loss function.

[0018] Optionally, the loss function further comprises a dynamic consistency loss.

[0019] constructing the dynamic consistency loss, specifically comprising:

[0020] constructing the dynamic consistency loss according to motion difference values of the target video and the predicted video at multiple time offsets in a latent space.

[0021] Optionally, the loss function further comprises a compound optical flow loss.

[0022] constructing the compound optical flow loss, specifically comprising:

[0023] constructing the compound optical flow loss according to optical flow fields of the target video and the predicted video.

[0024] Optionally, a composite optical flow loss is constructed according to the optical flow field of the target video and the optical flow field of the predicted video, and specifically includes:

[0025] A direction consistency term of the optical flow field of the target video and the optical flow field of the predicted video is calculated, and an amplitude term of the optical flow field of the target video and the optical flow field of the predicted video is calculated.

[0026] The composite optical flow loss is constructed according to the direction consistency term and the amplitude term.

[0027] Optionally, a weight coefficient of the composite optical flow loss is 0 when a training round number is less than an activation round number threshold, and is a fixed weight value when the training round number is greater than or equal to the activation round number threshold.

[0028] Optionally, the step S4 of fusing the injection feature and the backbone layer feature specifically includes:

[0029] The injection feature and the backbone layer feature are additively fused layer by layer.

[0030] The beneficial effects that can be produced by the present application include:

[0031] The video generation method based on the embodiment mask provided by the present application maps the coordinate space action into the embodiment mask aligned with the video pixel space, takes the embodiment mask as the control condition, injects into the backbone network of the video generation model through the conditional injection path of the control network module, aligns the action control signal and the generated video content in the time and space dimensions, and thus realizes the action-driven video generation. The present application projects the action sequence into the embodiment mask aligned with the pixel space, realizes the unified representation of the action signal and the video generation process, effectively solves the action-video mismatching problem, improves the robustness under the multi-view condition, and realizes the unified architecture among the single-arm and double-arm and other multi-embodiment structures.

[0032] The application provides a body mask-based video generation method, which comprises the following steps: firstly, generating a body mask sequence according to a robot structure description file, camera parameters and a robot action sequence, so as to convert the coordinate space action of a robot end effector into a body control signal aligned with a video pixel space; secondly, inputting the body mask sequence as a control condition into a pre-trained video generation model, and through the condition injection path of a control network module, the action control signal is aligned with the generated video content in the time and space dimensions, so that the action-driven video generation is realized; meanwhile, a motion loss function based on an optical flow is introduced in the module training, and the dynamic area of the body and object interaction is focused on, so as to inhibit the overfitting of the model to the static background. The method can realize a unified world modeling and video generation framework on different body platforms such as single-arm and double-arm. Experimental results show that the application can significantly improve the video generation quality and action semantic consistency in multi-view and complex scenes, and can be further applied to downstream tasks, including strategy evaluation and action planning based on target images. The application has strong cross-body generalization ability, high generation precision and good practical application value, and is suitable for the fields of intelligent manufacturing, service robots and human-machine cooperation. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 A body mask-based video generation method flow chart is provided for the embodiment of the application.

[0034] Figure 2 A body mask-based video generation method main body structure schematic diagram is provided for the embodiment of the application.

[0035] Figure 3 A multi-view video generation effect schematic diagram is provided for the embodiment of the application.

[0036] Figure 4 A video generation effect schematic diagram in an unseen scene is provided for the embodiment of the application.

[0037] Figure 5 A cross-body structure video generation effect schematic diagram is provided for the embodiment of the application. DETAILED DESCRIPTION

[0038] The application will be described in detail below with reference to the embodiments, but the application is not limited to these embodiments.

[0039] The embodiment of the application provides a body mask-based video generation method, as shown in Figure 1 and Figure 2 , the method comprises the following steps:

[0040] S1, determining a body mask sequence according to a target video.

[0041] The embodied world model is defined based on the kinematic structure of the robot embodiment. Its input includes initial image frames and motion sequences, and the output is an RGB video corresponding to the motion sequence. The robot's motion space is embodiment-independent and suitable for multi-arm robotic systems. Each motion consists of position, orientation, and gripper control variables, or is represented by joint angle parameters, which are used to describe the temporal motion state of the embodiment during the task.

[0042] Specifically, according to the operational definition of the embodied world model, the input includes the initial frame image. With action sequence The output is the corresponding predicted video sequence. ,in This refers to the video duration.

[0043] Action space Regardless of the physical structure, allowing for multi-arm structures, defined as:

[0044] ;

[0045] Each action Can be defined in Cartesian space:

[0046] ;

[0047] in, For displacement, For rotation amount, The robotic arm gripper is in gripper mode;

[0048] Or defined in joint space:

[0049] ;

[0050] in, For joint angle, The total number of joints;

[0051] To achieve a unified representation of actions across different bodies.

[0052] After defining the inputs and outputs of the embodied world model, the next step is to generate the embodied mask and map actions.

[0053] In one embodiment of the present invention, S1 specifically includes:

[0054] First, the robot's motion sequence is determined based on the target video. Then, the embodied 3D pose information is determined based on the robot's structural description file, camera parameters, and robot motion sequence. The embodied 3D pose information is then projected onto the image plane to obtain the embodied mask sequence.

[0055] Using the robot motion sequence determined from the target video, combined with the robot structural description file (URDF description file) and camera parameters (i.e., camera intrinsic and extrinsic parameters), the 3D pose information of the avatar is calculated through forward kinematics (to represent the structural changes of the avatar in 3D space). Then, the 3D pose information of the avatar is projected onto the image plane through a projection model to generate the avatar mask at each moment. This leads to the formation of an embodied mask sequence. This embodied mask sequence is used to align the coordinate space action with the video pixel space, ensuring the spatiotemporal consistency between the motion control signal and the image generation result.

[0056] This embodied mask sequence can guarantee every action With mask One-to-one correspondence, constructing sample triples based on this embodied mask sequence. Used for training embodied mask-driven video generation models; among which, The target video sequence.

[0057] In another embodiment of the present invention, S1 specifically includes:

[0058] The embodied mask sequence is extracted from the target video using a segmentation model.

[0059] In the absence of action annotations or camera calibration, segmentation models (such as GroundedSAM) can be used to directly extract embodied mask sequences from the target video. This enables the construction of embodied signals from unlabeled datasets.

[0060] S2. Encode the embodied mask sequence and the initial frame image of the target video respectively to obtain the embodied mask features and image coding features.

[0061] S3. Input the embodied mask features into the control network module to obtain the injected features, and input the image encoding features into the backbone network of the video generation model to obtain the backbone layer features.

[0062] S4. The injected features and the backbone layer features are fused to obtain fused features, and a predicted video is generated based on the fused features.

[0063] Specifically, the fusion of injected features and backbone features involves additive fusion of injected features and backbone features layer by layer.

[0064] A pre-trained video generation model is used as the backbone network. An embodied mask control branch (i.e., a control network module) similar to ControlNet is introduced. The embodied mask sequence is encoded by a 3D variational autoencoder (3D VAE) and then injected into the backbone network of the video generation model (such as a DiT (Diffusion Transformer) network) through this control network module. This control network module fuses the injected features with the backbone layer features layer by layer through zero-initialized convolutional layers, guiding the generation process to follow the action semantics corresponding to the embodied mask, thus achieving action-driven video generation.

[0065] Specifically, the DiT network of the pre-trained video generation model CogVideoX-5B-I2V is used as the backbone network, and the initial frame image of the input is processed by a three-dimensional variational autoencoder (3D-VAE). With target video Encoding yields image encoded features Latent variables of the target video , .

[0066] Based on this, embodied mask sequences are introduced. The embodied mask features are obtained through the same VAE encoding. It is then injected into each layer of the backbone DiT network through the control network module.

[0067] Each DiT network contains zero convolutional layers to control the stable injection of features. The fusion process employs layer-by-layer additive fusion, as shown in the following expression:

[0068] ;

[0069] in, Injection features; Characteristics of the main trunk layer; This is a feature of fusion.

[0070] The generated fusion features can maintain action-video consistency during the diffusion generation process.

[0071] Prior to S3, the method further includes:

[0072] Construct a loss function; the loss function includes diffusion loss;

[0073] The parameters of the control network module are determined based on the loss function.

[0074] Furthermore, the loss function also includes dynamic consistency loss;

[0075] The dynamic consistency loss is constructed, specifically including:

[0076] A dynamic consistency loss is constructed based on the motion differences between the target video and the predicted video at multiple time offsets in the latent space.

[0077] During video diffusion training, an inter-frame dynamic consistency loss is constructed to monitor motion changes between adjacent time points in the video latent space. By calculating the differences in video latent variables across multiple time offsets, the generated predicted video maintains true physical continuity in the temporal dimension, thereby improving the stability and coherence of temporal modeling.

[0078] Specifically, to improve the temporal coherence of the predicted video, a dynamic consistency loss is introduced. This is achieved by calculating the motion difference across multiple time offsets in the video latent space, and its expression is shown below:

[0079] ;

[0080] in, Maximum time offset; The latent space length of the target video; To predict the latent variables of the video at time t+j; To predict the latent variables of the video at time t; Let be the latent variable of the target video at time t+j; Let be the latent variable of the target video at time t.

[0081] This dynamic consistency loss This ensures that the generated video maintains physical continuity and the plausibility of actions in the temporal domain.

[0082] By combining diffusion loss and dynamic consistency loss, an overall training objective is constructed:

[0083] ;

[0084] in, The loss function for the control network module; For diffusion loss; This represents the loss due to dynamic consistency. These are custom hyperparameters.

[0085] Furthermore, the loss function also includes composite optical flow loss;

[0086] The composite optical flow loss is constructed by constructing a composite optical flow loss based on the optical flow field of the target video and the optical flow field of the predicted video.

[0087] Specifically, this invention includes:

[0088] Calculate the direction consistency term of the optical flow field of the target video and the optical flow field of the predicted video, and calculate the amplitude term of the optical flow field of the target video and the optical flow field of the predicted video;

[0089] The composite optical flow loss is constructed based on the direction consistency term and the magnitude term.

[0090] In practical applications, the weighting coefficient of the composite optical flow loss is 0 when the number of training rounds is less than the activation rounds threshold, and is a fixed weight value when the number of training rounds is greater than or equal to the activation rounds threshold.

[0091] A pre-trained RAFT optical flow estimation model is used to calculate the motion field of the predicted and target videos. A composite optical flow loss, incorporating both orientation and amplitude consistency, is designed to emphasize the dynamic regions of embodied and interactive objects and suppress overfitting to static backgrounds. During training, a delayed activation mechanism for the composite optical flow loss is implemented. It is disabled in the early stages of model training to stabilize the model and enabled with fixed weights after a set number of training epochs.

[0092] Specifically, the optical flow field of the target video is calculated using a pre-trained RAFT optical flow estimation model between the predicted video and the target video. Optical flow field of the predicted video And construct a composite optical flow loss Its expression is as follows:

[0093] ;

[0094] in, It also includes a direction consistency term (based on cosine distance) and an amplitude term (based on Huber distance) to constrain the motion field differences between the predicted video and the target video at both the direction and velocity levels. This composite optical flow loss can highlight the dynamic region learning of embodied and interactive objects, suppress ineffective reconstruction of static background pixels, thereby guiding the module to focus more on task-related motion changes and improving the dynamic accuracy and temporal consistency of video generation.

[0095] To prevent unstable optical flow estimation due to poor video quality during the initial training phase, this invention employs a delayed activation strategy to introduce optical flow loss in stages, specifically defined as follows:

[0096] ;

[0097] in, The activation round threshold for optical flow supervision. This is the current training round number. These are custom hyperparameters.

[0098] Optical flow supervision is not enabled when video generation is unstable in the early stages of training; after the module has basic video generation capabilities, composite optical flow loss with fixed weights is enabled to ensure smooth convergence of the training process and improve overall stability.

[0099] At this point, the loss function of the control network module includes: frame-level diffusion loss, dynamic consistency loss, and composite optical flow loss. The loss weight parameters are adjusted to balance video quality and dynamic consistency.

[0100] Finally, the total loss function of the control network module The expression is as follows:

[0101] ;

[0102] in, and , respectively, are the weight coefficients for the corresponding loss terms, and , are custom hyperparameters. This comprehensive optimization objective can simultaneously ensure pixel-level reconstruction accuracy of video frames, temporal dynamic consistency, and realism of embodied motion regions, thereby achieving a more stable and natural motion-conditional video generation effect.

[0103] Figure 3 This paper demonstrates the predicted video results generated by the method of this invention on the DROID dataset based on different camera perspectives. The top image shows the predicted video sequences generated under the camera perspectives within the training domain, while the bottom image shows the generated results under the unseen camera perspectives. As can be seen from the figures, this invention, through embodied mask alignment mechanism and multi-view condition constraints, enables the model to maintain action consistency and temporal stability under different camera perspectives, verifying the robustness and generalization ability of the method in cross-view scenarios.

[0104] Figure 4 This paper demonstrates the predictive video generation results of the method of this invention in unseen scenes on the DROID dataset. The experimental scenes contain backgrounds and object arrangements that differ from the distribution of the training data. This invention, through embodied mask conditional modeling and optical flow constraint mechanisms, enables the model to accurately generate embodied motion sequences consistent with actions even under changing environmental conditions, maintaining good temporal continuity and semantic consistency. This verifies the generalization ability and stability of the method in complex, unseen scenes.

[0105] Figure 5This paper demonstrates the predicted video generation results of the method of this invention on the AgiBot-G1 dual-arm embodied platform, used to verify its generalization ability under cross-embodied conditions. By sharing a unified embodied mask representation and video generation architecture with the single-arm DROID model, this invention can maintain action consistency and video temporal continuity even when there are differences in embodied structure, degrees of freedom, and camera viewpoints. Experimental results show that the method still has good generation quality and structural alignment ability in cross-embodied transfer scenarios, proving the model's versatility and scalability in multi-embodied environments.

[0106] In summary, this invention has been validated on different embodied platforms, including single-arm (DROID) and dual-arm (AgiBot-G1). The method of this invention can generate video results with consistent actions in unseen perspectives and complex scenes, significantly improving video generation quality, embodied alignment accuracy, and action semantic consistency. It can also be further applied to downstream tasks such as policy evaluation and target image planning.

[0107] The beneficial effects of the present invention are as follows:

[0108] (1) The method of the present invention can be applied to both single-arm (DROID) and dual-arm (AgiBot-G1) embodied structures. It achieves a unified representation of motion and pixel space through embodied masking, ensuring that different robotic arms can still share the same model architecture despite differences in degrees of freedom, structural dimensions, and kinematic constraints. This design avoids the problem of repeatedly training independent models for different embodiments and realizes cross-embodiment knowledge transfer and unified representation.

[0109] (2) The method of the present invention can still maintain the consistency of action and dynamic stability of the generated video in environments including multiple viewpoints, occlusion, and no background. The embodied mask constrains the embodied motion in the pixel space, so that the model can still generate physically reasonable and semantically consistent prediction results under conditions of camera viewpoint changes or complex environment, thereby significantly improving the video generation quality and cross-scene generalization ability.

[0110] (3) The method of this invention can be directly used for strategy evaluation and action planning tasks based on target images. By simulating the action execution results in a virtual environment, the performance of the strategy can be predicted and filtered. At the same time, an executable action sequence can be generated based on the target image to achieve end-to-end closed-loop verification from visual input to physical control. Experiments show that the method has strong generalization ability and execution reliability in real robot systems.

[0111] In summary, this invention significantly enhances the robot's operational strategy learning capabilities under multi-viewpoint and multi-embodiment conditions by introducing embodied mask alignment modeling, a control-net-like conditional injection mechanism, optical flow-based dynamic region constraints, and a unified embodied world model architecture. Compared with traditional video generation methods that rely on coordinate space actions or independent training, this invention has the following key advantages: First, strong representation space consistency. By mapping coordinate space actions to embodied masks aligned with the video pixel space, this invention achieves spatial unification of action control and visual generation, solving the action-video mismatch problem and enabling the model to fully utilize the visual and motion priors in the pre-trained video model. Second, excellent viewpoint robustness and cross-embodiment generalization. Embodied masks directly reflect embodied structure and pose changes in pixel space, without relying on specific camera views or embodied parameters. This allows for unified modeling and knowledge transfer across different robot structures, such as single-arm and dual-arm robots, improving the model's adaptability in multiple scenarios. Third, focus on task-related regions and improve generation quality. By introducing optical flow-based motion constraints, this invention focuses on the dynamic regions of embodied and interactive objects during model training, reducing overfitting to static backgrounds and thus improving the temporal consistency and semantic accuracy of video generation. Finally, it exhibits good scalability and practical application value. This method is compatible with various pre-trained video generation models and supports downstream tasks such as policy evaluation and target image-based action planning. This framework enables end-to-end learning from visual input to executable actions while maintaining model versatility, making it suitable for various service robots, industrial control, and human-computer interaction scenarios.

[0112] The above description is merely a few embodiments of this application and is not intended to limit this application in any way. Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any changes or modifications made by those skilled in the art without departing from the scope of the technical solution of this application using the disclosed technical content are equivalent to equivalent implementation cases and fall within the scope of the technical solution.

Claims

1. A video generation method based on embodied masking, characterized in that, The method includes: S1. Determine the embodied mask sequence based on the target video; S2. Encode the embodied mask sequence and the initial frame image of the target video respectively to obtain embodied mask features and image coding features. S3. Input the embodied mask features into the control network module to obtain the injected features, and input the image encoding features into the backbone network of the video generation model to obtain the backbone layer features; S4. The injected features and the backbone layer features are fused to obtain fused features, and a predicted video is generated based on the fused features.

2. The method according to claim 1, characterized in that, S1 specifically includes: Determine the robot's action sequence based on the target video; The embodied 3D pose information is determined based on the robot's structural description file, camera parameters, and the robot's motion sequence. The embodied 3D pose information is then projected onto the image plane to obtain the embodied mask sequence.

3. The method according to claim 1, characterized in that, S1 specifically includes: The embodied mask sequence is extracted from the target video using a segmentation model.

4. The method according to claim 1, characterized in that, Prior to S3, the method further includes: Construct a loss function; the loss function includes diffusion loss; The parameters of the control network module are determined based on the loss function.

5. The method according to claim 4, characterized in that, The loss function also includes dynamic consistency loss; The dynamic consistency loss is constructed, specifically including: A dynamic consistency loss is constructed based on the motion differences between the target video and the predicted video at multiple time offsets in the latent space.

6. The method according to claim 5, characterized in that, The loss function also includes composite optical flow loss; Constructing composite optical flow loss specifically includes: A composite optical flow loss is constructed based on the optical flow fields of the target video and the predicted video.

7. The method according to claim 6, characterized in that, A composite optical flow loss is constructed based on the optical flow fields of the target video and the predicted video, specifically including: Calculate the direction consistency term of the optical flow field of the target video and the optical flow field of the predicted video, and calculate the amplitude term of the optical flow field of the target video and the optical flow field of the predicted video; The composite optical flow loss is constructed based on the directional consistency term and the amplitude term.

8. The method according to claim 6, characterized in that, The weighting coefficient of the composite optical flow loss is 0 when the number of training rounds is less than the activation rounds threshold, and is a fixed weight value when the number of training rounds is greater than or equal to the activation rounds threshold.

9. The method according to claim 1, characterized in that, The fusion of the injected features and the backbone layer features in S4 specifically includes: The injected features and the backbone layer features are additively fused layer by layer.

Images