VLA model training method and device based on conditional video generation
By constructing robot operation scenarios in a simulation environment and generating video sequences corresponding to the desired action trajectories, and combining reinforcement learning and fine-tuning with real videos, the problem of acquiring robot operation data is solved, achieving efficient and low-cost generation of long-term consistent videos and improving the model's performance in real environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHUHAI KUWA TECHNOLOGY CO LTD
- Filing Date
- 2026-02-09
- Publication Date
- 2026-05-19
AI Technical Summary
Existing technologies struggle to efficiently and cost-effectively acquire high-quality, diverse real-world robot operation data, and a significant visual gap exists between simulation and reality, leading to performance degradation of models when transferred to the real world.
An initial operation scenario is constructed in a simulation environment, geometric feature information is extracted, the desired action trajectory is generated, and a video sequence corresponding to the desired action trajectory is synthesized through a conditional video generation model. By combining reinforcement learning and fine-tuning with real videos, a multimodal training dataset is constructed, and a diffusion model architecture and autoregressive generation technology are used to generate long-term consistent videos.
It achieves high-quality, physically consistent simulation operation video generation, reduces data acquisition costs, and improves the model's generalization ability and task success rate in real-world environments.
Smart Images
Figure CN122067045A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of robot learning and artificial intelligence, and specifically relates to a VLA model training method and apparatus based on conditional video generation. Background Technology
[0002] Vision-Language-Action (VLA) models have shown great potential in robotic manipulation tasks, but their performance heavily relies on massive amounts of high-quality, diverse real-world demonstration data. Currently, acquiring such data faces two fundamental challenges:
[0003] First, data collection is costly and its application scenarios are limited. Using real robots to collect data is a cumbersome process involving hardware debugging, security monitoring, and human intervention. To obtain high-quality data, rigorous screening is required, and due to the limitations of the laboratory's physical environment, it is difficult to cover the complex backgrounds, lighting, object appearances, and long-tail interaction scenarios in open worlds.
[0004] Secondly, there is a significant visual gap between simulation and reality. Traditional simulation data based on graphics rendering has significant domain differences from real images, and models trained directly using simulation data experience a sharp decline in performance when transferred to the real world.
[0005] Existing general-purpose video generation models have significant shortcomings in solving the above problems:
[0006] On the one hand, commercial closed-source models lack knowledge of the robotics field, cannot understand robot structure and operational tasks, and the prompts are difficult to accurately constrain the generated content. For example, they cannot generate the correct robotic arm grasping action, or they may generate a human hand incorrectly.
[0007] On the other hand, general open-source models struggle to balance generation quality and control capabilities. Models with fewer parameters produce poor quality data, exhibiting issues such as object disappearance and flickering; while large models, although highly faithful, lack the fine-grained control over robot motion sequences and 3D geometry, making it impossible to directly generate structured data suitable for policy learning.
[0008] In addition, mainstream models can usually only generate short videos of 5-10 seconds, lacking the ability to generate long sequences. Longer sequences may suffer from problems such as screen crashes and inconsistent timing, making it difficult to meet the demonstration needs of complex operation tasks.
[0009] Therefore, there is an urgent need for a technical solution that can understand robotics knowledge, support refined multimodal control, and generate long-term consistent videos, so as to automatically and cost-effectively generate training data that bridges the gap between simulation and reality. Summary of the Invention
[0010] To address the aforementioned technical problems, this invention proposes a VLA model training method and apparatus based on conditional video generation, which is used for large-scale synthesis of training data with high realism and high physical consistency.
[0011] To achieve the above objectives, the technical solution of the present invention is as follows:
[0012] In a first aspect, this invention discloses a VLA model training method based on conditional video generation, comprising:
[0013] Step S1: Construct the initial operation scenario in the simulation environment and obtain the initial reference frame containing robot state and environmental information;
[0014] Step S2: Simultaneously extract scene geometric feature information corresponding to the initial reference frame from the simulation environment;
[0015] Step S3: For the preset operation task, generate the robot's expected motion trajectory in the simulation environment through a reinforcement learning model or trajectory planner;
[0016] Step S4: The initial reference frame, geometric feature information, and desired motion trajectory are used as multimodal conditional inputs to drive the conditional video generation model finely tuned by robot data to synthesize a physically continuous sequence of subsequent frames corresponding to the desired motion trajectory, so as to form a simulation operation video for model training.
[0017] Step S5: Train the VLA model using a dataset containing simulation operation videos to obtain the target policy network.
[0018] Based on the above technical solution, the following improvements can be made:
[0019] As a preferred approach, the conditional video generation model adopts a diffusion-based architecture and is fine-tuned using real video sequences containing robot operation tasks.
[0020] As a preferred option, in step S4...
[0021] Depth information and / or surface normal information from geometric features are used as geometric control signals. These signals are processed by a control branch network that runs in parallel with the conditional video generation model, and then injected into the denoising network of the conditional video generation model to constrain the geometric structure of the generated video.
[0022] As a preferred embodiment, in step S4, a piecewise autoregressive method is used to generate the simulation operation video, specifically including:
[0023] Step S4.1: Divide the simulation operation video to be generated into multiple continuous and partially overlapping video segments on the timeline;
[0024] Step S4.2: For the current video segment to be generated, construct its generation conditions, which include at least: the starting frame of the current segment, the corresponding expected motion trajectory segment, and the processed sequence of the last frames of the previously generated segments as historical context.
[0025] The processed pre-generated fragment's last frame sequence is obtained through FramePack technology, specifically: the last frame sequence is downsampled at multiple scales and packaged together with the position encoding, and embedded as the historical context into the generation conditions;
[0026] Step S4.3: Based on the generation conditions, drive the conditional video generation model to generate the current video segment;
[0027] Furthermore, during the generation process, the conditional video generation model assigns different denoising time step IDs to different feature blocks in the latent space to achieve progressive and refined denoising.
[0028] Step S4.4: Update the end of the currently generated video segment to the historical context of the next segment, and iterate through steps S4.2 to S4.4, using the autoregressive Diffusion Forcing paradigm to generate each video segment until a simulation operation video of the preset duration is generated.
[0029] As a preferred embodiment, step S5 specifically includes:
[0030] Step S5.1: Automatically construct multimodal training sample pairs from the simulation operation video. Each sample pair contains a video frame sequence, the corresponding robot action sequence, and natural language task instructions.
[0031] Step S5.2: Train the VLA model using the constructed sample pairs in a supervised learning or imitation learning manner.
[0032] As a preferred approach, the method also includes a closed-loop iterative step:
[0033] Based on the performance of the target policy network in simulated or real-world environments, screen out failed task cases;
[0034] Based on the failed task cases, steps S1 to S4 are re-executed to generate targeted enhanced training data;
[0035] The VLA model is iteratively retrained using targeted augmented training data to optimize the performance of the target policy network.
[0036] As a preferred option, the method also includes:
[0037] Pre-train an independent conditional embedding vector for each fixed camera viewpoint;
[0038] When generating simulation operation videos, the conditional embedding vector corresponding to the target camera's viewpoint is added to the multimodal conditional input to control the viewing angle of the generated video.
[0039] Secondly, the present invention discloses a VLA model training device based on conditional video generation, comprising:
[0040] The scene construction module is used to build the initial operation scene in the simulation environment and obtain the initial reference frame containing robot state and environmental information;
[0041] The geometric feature extraction module is used to synchronously extract scene geometric feature information corresponding to the initial reference frame from the simulation environment;
[0042] The desired trajectory generation module is used to generate the robot's desired motion trajectory in a simulation environment based on a preset operation task, through a reinforcement learning model or trajectory planner.
[0043] The simulation video generation module is used to take the initial reference frame, geometric feature information and the desired motion trajectory as multimodal conditional inputs, drive the conditional video generation model finely tuned by robot data, and synthesize a physically continuous sequence of subsequent frames corresponding to the desired motion trajectory to form a simulation operation video for model training.
[0044] The model training module is used to train the VLA model using a dataset containing simulation operation videos to obtain the target policy network.
[0045] Thirdly, the present invention discloses a computing device, comprising:
[0046] One or more processors;
[0047] Memory;
[0048] And one or more programs, wherein the one or more programs are stored in memory and configured to be executed by one or more processors, and the one or more programs include instructions for any of the above-described VLA model training methods based on conditional video generation.
[0049] Fourthly, the present invention discloses a storage medium storing one or more computer-readable programs, the one or more programs including instructions adapted to be loaded by a memory and executed by any of the above-described VLA model training methods based on conditional video generation.
[0050] This invention discloses a VLA model training method based on conditional video generation, which has the following beneficial effects:
[0051] First, by fine-tuning the general video generation model with robotics data and introducing geometric consistency control technology, high-quality, physically plausible simulation operation video generation was achieved.
[0052] Second, it innovatively combines FramePack historical context packaging with the autoregressive Diffusion Forcing generation paradigm to stably generate long-time-series, highly consistent video sequences, supporting demonstrations of complex multi-step tasks.
[0053] Third, an end-to-end automated data production pipeline was built, from simulation scene initialization to video synthesis and annotation, which significantly reduced the manual cost and acquisition threshold of high-quality training data.
[0054] Fourth, the generated simulation operation video is visually close to the real world while retaining the accuracy of the simulation trajectory. The generated data is visually closer to the real world while retaining the accurate trajectory and state information of the simulation. When the VLA model trained using this video is transferred to a real robot, it shows stronger generalization ability and higher task success rate.
[0055] Fifth, the invention is highly modular, making it easy to expand and integrate new control signals, adapt to different robot platforms, and extend to a wider range of embodied intelligent task scenarios. Attached Figure Description
[0056] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0057] Figure 1 A flowchart of the VLA model training method provided in an embodiment of the present invention.
[0058] Figure 2 This is a diagram illustrating the refined control architecture of the conditional video generation model in this invention.
[0059] Figure 3 This is a schematic diagram illustrating the principle of the long sequence generation technology (FramePack) in this invention.
[0060] Figure 4 This is a schematic diagram illustrating the principle of the long sequence generation technology (Diffusion Forcing) in this invention. Detailed Implementation
[0061] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.
[0062] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0063] Using ordinal numbers such as “first,” “second,” “third,” etc. to describe ordinary objects merely indicates different instances of similar objects and is not intended to imply that the objects being described must have a given order in time, space, sequence, or any other way.
[0064] Furthermore, the expression "includes" is an "open-ended" expression, which means only that there is a corresponding component or step, and should not be interpreted as excluding additional components or steps.
[0065] To achieve the objectives of this invention, some embodiments of the VLA model training method and apparatus based on conditional video generation are provided.
[0066] like Figure 1 As shown, the VLA model training methods include:
[0067] Step S101: Construct an initial operation scenario in the simulation environment and obtain an initial reference frame containing robot state and environmental information;
[0068] Step S102: Simultaneously extract scene geometric feature information corresponding to the initial reference frame from the simulation environment;
[0069] Step S103: For the preset operation task, generate the robot's expected motion trajectory in the simulation environment through a reinforcement learning model or trajectory planner.
[0070] Step S104: The initial reference frame, geometric feature information, and desired motion trajectory are used as multimodal conditional inputs to drive the conditional video generation model finely tuned by robot data to synthesize a physically continuous sequence of subsequent frames corresponding to the desired motion trajectory, so as to form a simulation operation video for model training.
[0071] Step S105: Train the VLA model using a dataset containing simulation operation videos to obtain the target policy network.
[0072] Finally, the trained target policy network is deployed to a real robot entity.
[0073] It is worth noting that:
[0074] In step S101, the conditions for constructing the initial operation scenario include at least one of the following:
[0075] A library of replaceable 3D models of target objects relevant to the robot product line;
[0076] Variable background environment and dynamic lighting configuration that match real-world work scenarios;
[0077] Randomized initial spatial layout of the robot body, cooperative targets, and interfering objects.
[0078] Furthermore, in step S102, the geometric feature information is extracted through a pre-trained visual encoder, including at least one of the following: pixel-level depth information, pixel-level surface normal information, and semantic instance segmentation information.
[0079] Furthermore, in step S103, the reinforcement learning model can be, but is not limited to, a policy network trained with a reward system designed to guide task success in a simulation environment. The trajectory planner can be, but is not limited to, a sampling-based motion planner or a model-based optimizer.
[0080] Furthermore, in step S104, the conditional video generation model adopts a diffusion-based architecture and is fine-tuned using real video sequences containing robot operation tasks. The desired motion trajectory is encoded in the form of time-series embeddings.
[0081] This embodiment uses the complex task of "a robotic arm picking up a specific object from a cluttered table and placing it into a designated container" as an example to illustrate the implementation process of the present invention, as detailed below:
[0082] Step S101: Construct an initial operation scenario in the simulation environment and obtain an initial reference frame containing robot state and environmental information.
[0083] In this embodiment, in a physical simulation environment, the desktop texture, the model and color of objects (such as plastic bottles and gears), the container position, and the starting pose of the robotic arm are randomly initialized, and the initial reference frame I0 of the first perspective is rendered, which includes the initial pose of the robotic arm, the color, texture, position, and lighting information of all objects.
[0084] Step S102: Synchronously extract scene geometric feature information corresponding to the initial reference frame from the simulation environment.
[0085] At the same simulation moment when the initial reference frame I0 is generated, the simulation engine's interface is synchronously called to obtain the depth map D0 and surface normal map N0 corresponding to the initial reference frame.
[0086] The geometric feature information extracted in this step, consisting of {D0, N0}, accurately describes the three-dimensional geometric structure of the scene, providing spatial structural constraints for subsequent video generation.
[0087] Step S103: For the preset operation task, generate the desired motion trajectory of the robot in the simulation environment.
[0088] For the preset task of "grabbing a blue plastic bottle and placing it into a red container," the system invokes a sampling-based motion planner. This planner projects a trajectory T from the starting point to the successful placement of the end effector. This trajectory T is a time series, indicating the spatial position and orientation that the robotic arm should reach at each subsequent moment.
[0089] Step S104: Using the initial reference frame, geometric feature information, and desired motion trajectory as multimodal conditional inputs, drive the conditional video generation model finely tuned by robotics domain data to synthesize the simulation operation video.
[0090] The geometric control signal consisting of {D0, N0}, the trajectory T, and the embedding vector of the current viewpoint are used as multimodal conditional inputs and fed into a conditional video generation model (e.g., based on the Wan 2.1 architecture) that has been fine-tuned using real robotic arm operation videos. Figure 2 As shown.
[0091] The model fine-tuning process is as follows: First, approximately 200 video clips of the robotic arm performing various tasks in real-world scenarios were collected, and each video clip was manually annotated with a natural language description. Then, this was used as a proprietary dataset to perform supervised fine-tuning of the open-source WAN 2.1 (14-parameter) model. Finally, the fine-tuning enabled the model to learn the robotic arm's appearance features, typical working environment, and basic motion patterns.
[0092] In step S104, D0 and N0 in the geometric feature information are used as geometric control signals. After being processed by the control branch network in parallel with the conditional video generation model, they are injected into the denoising network of the conditional video generation model to constrain the geometric structure of the generated video.
[0093] In this embodiment, D0 and N0 are processed through an independent encoding network (i.e., the control branch network) using VACE technology, and then injected into the cross-attention layer of the WAN model backbone denoising network (DIT Blocks) in an additive manner (or in a ControlNet-like manner) to ensure that the three-dimensional spatial relationship of the robotic arm, object and container is strictly consistent with the conditional input in each generated frame.
[0094] like Figure 3-4As shown, in step S104, a piecewise autoregressive method is used to generate the simulation operation video, specifically including:
[0095] Step S104.1: Set the duration of the simulation operation video to be generated to 60 seconds (1800 frames), and divide the video into multiple continuous and partially overlapping video segments on the timeline;
[0096] Step S104.2: For the current video segment to be generated, construct its generation conditions. The conditions include at least: the starting frame of the current segment, the corresponding expected motion trajectory segment, and the processed sequence of the last frames of the previously generated segments as historical context.
[0097] The last frame sequence of the pre-generated fragments after processing is obtained through FramePack technology, which involves: downsampling the last frame sequence at multiple scales and packaging it together with the position encoding, and embedding it as historical context into the generation conditions.
[0098] Step S104.3: Based on the generation conditions, drive the conditional video generation model to generate the current video segment;
[0099] Furthermore, during the generation process, the conditional video generation model assigns different denoising time step IDs to different feature blocks in the latent space to achieve progressive and refined denoising.
[0100] Step S104.4: Update the end of the currently generated video segment to the historical context of the next segment, and iteratively execute steps S104.2 to S104.4, using the autoregressive Diffusion Forcing paradigm to generate each video segment until a simulation operation video of the preset duration is generated.
[0101] In this embodiment, to generate a continuous video of up to 60 seconds, a combination of FramePack and DiffusionForcing techniques is employed. The DiffusionForcing strategy divides the 1800 frames into multiple overlapping video segments. After generating the first video segment (e.g., frames 0-250), FramePack technology is used to perform layered downsampling on the last 32 frames of the block (the nearest frames retain 8x8 resolution, while more distant frames are compressed to 4x4 and 2x2), and this, along with the new trajectory segment and the updated initial frame, serves as a condition for generating the next video segment. This process is iterated until a complete video is generated. During the process, RoPE positional encoding is used to process the downsampled historical frame grid to ensure correct spatial correspondence.
[0102] In this step, fine-grained control is achieved through two key technologies: First, VACE technology is used to inject depth and normal information as geometric control signals into the generation process, ensuring the accuracy and consistency of the scene structure; Second, an autoregressive paradigm combining FramePack and Diffusion Forcing is used for generation, and by introducing historical frame context and progressive denoising, the temporal coherence and stability of long video sequences are guaranteed.
[0103] Step S105: Train the VLA model using a dataset containing simulation operation videos to obtain the target policy network.
[0104] Step S105 specifically includes:
[0105] Step S105.1: Automatically construct multimodal training sample pairs from the simulation operation video. Each sample pair contains a video frame sequence, the corresponding robot action sequence, and natural language task instructions.
[0106] Step S105.2: Train the VLA model using the constructed sample pairs in a supervised learning or imitation learning manner.
[0107] The generated videos automatically include precise timestamps and corresponding trajectory information. These are then packaged with the natural language instruction "grab the blue bottle and put it into the red container" to construct a multimodal training sample, which is added to the dataset. This dataset is then used for end-to-end training of the VLA model.
[0108] After training, the model policy network was deployed on a real robotic arm of the same model for testing.
[0109] The simulated operation videos generated by this invention far surpass traditional simulation rendering in terms of texture, lighting, and dynamic details, and are much closer to real-world shooting. The VLA model trained using the data generated by this invention achieves a success rate of over 35% in real-world grasping and placing tasks compared to models trained using pure simulation data or general model-generated data, and its generalization ability to object color and background changes is significantly enhanced.
[0110] This invention synthesizes high-quality, multi-view continuous frames through a video generation model under geometric and motion trajectory supervision, effectively bridging the visual differences between simulation and real-world scenes and solving the problem of degraded policy transfer performance caused by inconsistencies between simulation data and real-world image distribution. This invention can efficiently generate massive amounts of diverse and realistic training data, significantly improving the generalization ability, stability, and task success rate of robot operation strategies in real-world environments.
[0111] To continuously improve the robustness and generalization ability of the model in complex scenarios, in some other embodiments, this method further includes a closed-loop iterative optimization process.
[0112] After the target policy network is initially trained, it is placed in a simulation or real environment for extensive task testing to screen out cases of task failure.
[0113] For these failed cases, repeat steps S101 to S104 to generate targeted enhanced training data;
[0114] The VLA model is iteratively retrained using targeted augmentation training data to optimize and update the target policy network.
[0115] By repeatedly going through this closed loop of "testing-failure analysis-targeted data generation-retraining," performance bottlenecks can be automatically overcome, enabling continuous and targeted enhancement of the policy network's performance.
[0116] For complex tasks that require multi-view perception for 3D reasoning or obstacle avoidance, in some other embodiments, during the model training phase, view-specific conditional embedding vectors (e.g., Embedding_Cam1, Embedding_Cam2) are trained for two fixed cameras deployed in the environment.
[0117] When generating training data, by switching and inputting different viewpoint embeddings, the model can be driven to generate synchronized videos from different camera perspectives under the same operation task, thereby constructing multi-viewpoint aligned training data, which is convenient for downstream models to learn 3D perception.
[0118] This invention introduces a domain data adaptive fine-tuning strategy, an innovative refined control mechanism, and a long sequence generation technology to automatically generate visually realistic, physically reasonable, and highly relevant long video sequences that are strongly correlated with specific robot operation tasks. This allows for the efficient and low-cost construction of a large-scale training dataset for VLA models, significantly improving the model's generalization and transfer capabilities in real-world environments.
[0119] In other embodiments, the present invention discloses a VLA model training apparatus based on conditional video generation, comprising:
[0120] The scene construction module is used to build the initial operation scene in the simulation environment and obtain the initial reference frame containing robot state and environmental information;
[0121] The geometric feature extraction module is used to synchronously extract scene geometric feature information corresponding to the initial reference frame from the simulation environment;
[0122] The desired trajectory generation module is used to generate the robot's desired motion trajectory in a simulation environment based on a preset operation task, through a reinforcement learning model or trajectory planner.
[0123] The simulation video generation module is used to take the initial reference frame, geometric feature information and the desired motion trajectory as multimodal conditional inputs, drive the conditional video generation model finely tuned by robot data, and synthesize a physically continuous sequence of subsequent frames corresponding to the desired motion trajectory to form a simulation operation video for model training.
[0124] The model training module is used to train the VLA model using a dataset containing simulation operation videos to obtain the target policy network.
[0125] Furthermore, it should be noted that the VLA model training device based on conditional video generation provided in the above embodiments is only illustrated by the division of the above functional modules when generating training data. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the VLA model training device based on conditional video generation can be divided into different functional modules to complete all or part of the functions described above.
[0126] Furthermore, the VLA model training device based on conditional video generation provided in the above embodiments and the VLA model training method based on conditional video generation belong to the same concept. For details of its implementation process, please refer to the method embodiments, which will not be repeated here.
[0127] In other embodiments, the present invention discloses a computing device comprising:
[0128] One or more processors;
[0129] Memory;
[0130] And one or more programs, wherein the one or more programs are stored in memory and configured to be executed by one or more processors, and the one or more programs include instructions for any of the above-described VLA model training methods based on conditional video generation.
[0131] In other embodiments, the present invention discloses a storage medium storing one or more computer-readable programs, the programs including instructions adapted to be loaded by memory and execute any of the above-described VLA model training methods based on conditional video generation.
[0132] This invention discloses a VLA model training method based on conditional video generation, which has the following beneficial effects:
[0133] First, by fine-tuning the general video generation model with robotics data and introducing geometric consistency control technology, high-quality, physically plausible simulation operation video generation was achieved.
[0134] By fine-tuning general video generation models (such as the Wan series) to enable them to deeply understand the shape and motion patterns of specific robotic arms, the problem of generating incorrect structures is fundamentally avoided. Combined with VACE technology, pixel-level geometric consistency is precisely controlled, resulting in generated videos with physical plausibility far exceeding that of general models.
[0135] Second, it innovatively combines FramePack historical context packaging with the autoregressive Diffusion Forcing generation paradigm to stably generate long-time-series, highly consistent video sequences, supporting demonstrations of complex multi-step tasks.
[0136] It innovatively combines FramePack and Diffusion Forcing technologies, breaking through the limitations of mainstream model generation time. It can stably generate long operation videos of about one minute with smooth footage and stable subjects, providing complete data demonstrations for complex multi-step tasks.
[0137] Third, an end-to-end automated data production pipeline was built, from simulation scene initialization to video synthesis and annotation, which significantly reduced the manual cost and acquisition threshold of high-quality training data.
[0138] From defining tasks and trajectories in a simulation environment to automatically generating multi-view, long-sequence training videos with precise geometric alignment, the entire process requires no manual annotation or intervention, reducing the marginal cost of high-quality robot training data to an extremely low level.
[0139] Fourth, the generated simulation operation video is visually close to the real world while retaining the accuracy of the simulation trajectory. The generated data is visually closer to the real world while retaining the accurate trajectory and state information of the simulation. When the VLA model trained using this video is transferred to a real robot, it shows stronger generalization ability and higher task success rate.
[0140] Fifth, the invention is highly modular, making it easy to expand and integrate new control signals (such as tactile and audio signals), adapt to different robot platforms, and extend to a wider range of embodied intelligent task scenarios, such as desktop operation and mobile navigation.
[0141] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the present invention. Various changes and modifications can be made to the present invention without departing from its spirit and scope. All such changes and modifications fall within the scope of the present invention as claimed, which is defined by the appended claims and their equivalents.
Claims
1. A VLA model training method based on conditional video generation, characterized in that, include: Step S1: Construct the initial operation scenario in the simulation environment and obtain the initial reference frame containing robot state and environmental information; Step S2: Simultaneously extract scene geometric feature information corresponding to the initial reference frame from the simulation environment; Step S3: For the preset operation task, generate the robot's desired motion trajectory in the simulation environment through a reinforcement learning model or trajectory planner; Step S4: Using the initial reference frame, the geometric feature information, and the desired motion trajectory as multimodal conditional inputs, drive the conditional video generation model finely tuned by robot data to synthesize a physically continuous sequence of subsequent frames corresponding to the desired motion trajectory, so as to form a simulation operation video for model training. Step S5: Use the dataset containing the simulation operation video to train the VLA model to obtain the target policy network.
2. The VLA model training method according to claim 1, characterized in that, The conditional video generation model adopts a diffusion model-based architecture and is fine-tuned using real video sequences containing robot operation tasks.
3. The VLA model training method according to claim 2, characterized in that, In step S4, The depth information and / or surface normal information in the geometric feature information are used as geometric control signals. After being processed by the control branch network in parallel with the conditional video generation model, they are injected into the denoising network of the conditional video generation model to constrain the geometric structure of the generated video.
4. The VLA model training method according to claim 1, characterized in that, In step S4, the simulation operation video is generated using a piecewise autoregressive method, specifically including: Step S4.1: Divide the simulation operation video to be generated into multiple continuous and partially overlapping video segments on the timeline; Step S4.2: For the current video segment to be generated, construct its generation conditions, which include at least: the starting frame of the current segment, the corresponding expected motion trajectory segment, and the processed sequence of the last frames of the previously generated segments as historical context. The processed pre-generated fragment's last frame sequence is obtained through FramePack technology, specifically: the last frame sequence is downsampled at multiple scales and packaged together with the position encoding, and embedded as the historical context into the generation conditions; Step S4.3: Based on the generation conditions, drive the conditional video generation model to generate the current video segment; Furthermore, during the generation process, the conditional video generation model assigns different denoising time step IDs to different feature blocks in the latent space to achieve progressive and refined denoising. Step S4.4: Update the end of the currently generated video segment to the historical context of the next segment, and iterate through steps S4.2 to S4.4, using the autoregressive Diffusion Forcing paradigm to generate each video segment until a simulation operation video of the preset duration is generated.
5. The VLA model training method according to claim 1, characterized in that, Step S5 specifically includes: Step S5.1: Automatically construct multimodal training sample pairs from the simulation operation video. Each sample pair contains a video frame sequence, a corresponding robot action sequence, and natural language task instructions. Step S5.2: Train the VLA model using the constructed sample pairs in a supervised learning or imitation learning manner.
6. The VLA model training method according to claim 1, characterized in that, The method also includes a closed-loop iterative step: Based on the test performance of the target policy network in simulation or real environment, filter out failed task cases; Based on the failed task cases, steps S1 to S4 are re-executed to generate targeted enhanced training data; The VLA model is iteratively retrained using the targeted enhanced training data to optimize the performance of the target policy network.
7. The VLA model training method according to claim 1, characterized in that, The method further includes: Pre-train an independent conditional embedding vector for each fixed camera viewpoint; When generating the simulation operation video, the conditional embedding vector corresponding to the target camera's viewpoint is added to the multimodal conditional input to control the viewing angle of the generated video.
8. A VLA model training device based on conditional video generation, characterized in that, include: The scene construction module is used to build the initial operation scene in the simulation environment and obtain the initial reference frame containing robot state and environmental information; The geometric feature extraction module is used to synchronously extract scene geometric feature information corresponding to the initial reference frame from the simulation environment; The desired trajectory generation module is used to generate the desired action trajectory of the robot in the simulation environment based on a preset operation task, through a reinforcement learning model or trajectory planner. The simulation video generation module is used to take the initial reference frame, the geometric feature information and the expected motion trajectory as multimodal conditional inputs, drive the conditional video generation model finely tuned by robot data, and synthesize a physically continuous sequence of subsequent frames corresponding to the expected motion trajectory to form a simulation operation video for model training. The model training module is used to train the VLA model using a dataset containing the simulation operation videos to obtain the target policy network.
9. A computing device, characterized in that, include: One or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors, and the one or more programs include instructions for the VLA model training method based on conditional video generation as described in any of claims 1-7.
10. A storage medium, characterized in that, The storage medium stores one or more computer-readable programs, the programs including instructions adapted to be loaded by memory and executed as described in any of claims 1-7 for training a VLA model based on conditional video generation.