A method, device, medium and product for synthesizing VLA training data

Abstract

Embodiments of the present application relate to the field of embodied intelligence, and disclose a method, device, medium and product for synthesizing VLA training data. The method comprises: extracting semantic representation information from a human video to obtain a semantic action tuple; the semantic action tuple comprises a task instruction, a key frame sequence, a trajectory sequence and a contact event sequence; inputting a multi-modal signal, a current video observation frame and the semantic action tuple as a joint condition, fine-tuning a self-attention video-action world model to obtain a target self-attention video-action world model; obtaining expanded training data according to the target self-attention video-action world model; performing a three-dimensional quality filtering mechanism on the expanded training data to obtain a training triple; the three-dimensional quality filtering mechanism is used to eliminate samples that do not meet preset quality requirements; the training triple comprises a video frame sequence, a task instruction text and an action label sequence; and obtaining target training data set according to the training triple.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, medium and product for synthesizing VLA training data. Background Technology

[0002] With the rapid development of embodied intelligence technology, VLA (Vision-Language-Action) models, as the core paradigm for driving robots to perform complex tasks, have received widespread attention from academia and industry. However, the large-scale real robot interaction data required to train high-quality VLA models is extremely scarce and the acquisition cost is high—obtaining professional teaching data in vertical application scenarios is difficult, usually requiring a large investment of manpower, professional robot hardware, and scenario construction resources, which seriously restricts the implementation of embodied intelligence technology in real-world scenarios.

[0003] Currently, the interaction data required for training VLA models comes from two main sources: First, world models. World models are an important research area that has emerged in recent years. Their core idea is to internalize the dynamics of the physical world into the model, predicting future scene evolution through joint modeling of current states and actions. World models can serve as data engines, synthesizing embodied interaction sequences required for robot learning at scale. These embodied interaction sequences can then be used to train realistic robot control models, reducing reliance on real robot data. Second, human videos. Egocentric videos contain rich prior information on hand manipulations and physical contact, which can be directly used to pre-train robot policy backbone networks, significantly improving cross-embodied transfer efficiency.

[0004] However, existing methods have the following shortcomings:

[0005] 1) Existing general world models are mostly trained based on Internet videos or robot simulation data, which makes it difficult to understand the material, deformation, contact characteristics and operation logic of objects in specific scenarios. The synthetic data has a large distribution deviation from the real scene, resulting in unreliable synthetic data.

[0006] 2) There is a misalignment in perspective and action semantics between human videos and robot training data. Human videos are taken from a first-person perspective, while robot training data usually uses the perspective of a wrist or a fixed camera. Furthermore, human videos lack structured robot action annotations, making human video data unsuitable for direct use in VLA model training.

[0007] 3) The lack of a quality filtering mechanism for synthetic data leads to the introduction of large-scale general world model synthetic data to solve the problem of insufficient data volume. This results in low-quality samples with physical inconsistencies, misaligned instructions, or inconsistent timing. Furthermore, the lack of a filtering mechanism leads to chaotic robot logic. Summary of the Invention

[0008] One objective of this application is to provide a method for synthesizing VLA training data, at least to address the technical problems of unreliable synthetic data for general world models, the inability to directly use human video data, and inconsistent data quality in existing methods.

[0009] To achieve the above objectives, some embodiments of this application provide the following aspects:

[0010] In a first aspect, some embodiments of this application also provide a method for synthesizing VLA training data, comprising: extracting semantic representation information from human videos to obtain semantic action tuples; the semantic action tuples include task instructions, keyframe sequences, trajectory sequences, and contact event sequences; using multimodal signals, current video observation frames, and the semantic action tuples as joint conditional inputs to fine-tune an autoregressive video-action world model to obtain a target autoregressive video-action world model; obtaining augmented training data based on the target autoregressive video-action world model; performing a three-dimensional quality filtering mechanism on the augmented training data to obtain training triples; the three-dimensional quality filtering mechanism is used to remove samples that do not meet preset quality requirements; the training triples include video frame sequences, task instruction text, and action label sequences; and obtaining a target training dataset based on the training triples.

[0011] Secondly, some embodiments of this application also provide an electronic device, a cloud training paradigm, in which all raw data is uploaded to the cloud for unified training, which facilitates centralized management. The electronic device includes: one or more processors; and a memory storing computer program instructions, which, when executed, cause the processor to perform the steps of the method described above.

[0012] Thirdly, some embodiments of this application also provide a computer-readable medium having computer program instructions stored thereon, which can be executed by a processor to implement the method described above.

[0013] Fourthly, some embodiments of this application also provide a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the method described above.

[0014] Compared with related technologies, the solution provided in this application reduces the cost of training data collection in vertical scenarios and the dependence on professional operators and venues by collecting human videos instead of teaching data collection methods that rely on professional robot hardware. Semantic action tuple extraction transforms human videos into structured robot action labels. The extracted semantic action tuples undergo data cleaning and initial quality screening, and are then input into an autoregressive video-action world model along with pre-collected multimodal signals and the current video observation frame. This model is then fine-tuned to obtain a target autoregressive video-action world model. Using this target autoregressive video-action world model, the training data scale is systematically expanded to obtain expanded training data. This expanded training data transforms operational knowledge from human-view videos into multi-view training data that the robot can learn, solving the problem of viewpoint mismatch between human video data and robot training data in existing methods. A three-dimensional quality filtering mechanism is executed on the expanded training data, retaining only samples that pass all filters and forming a standard format to obtain training triples. The training triples are further standardized to obtain target training data, which is easily integrated into existing training pipelines. This method does not rely on a specific robot body or VLA model architecture and can be adapted to various robot forms such as robotic arms, mobile manipulation robots, and humanoid robots. It can also be used in a variety of embodied intelligence application fields, such as housekeeping, factories, autonomous driving, and warehousing and logistics. Attached Figure Description

[0015] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0016] Figure 1 An exemplary flowchart of a method for synthesizing VLA training data provided for some embodiments of this application;

[0017] Figure 2 This is a schematic diagram of the structure of an electronic device provided in some embodiments of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] The following terms are used in this document:

[0020] VLA, Vision-Language-Action, is a visual-language-action model.

[0021] The World Model comprises a visual perception module (V) that simplifies the pixels seen into abstract concepts; a dynamics module (M) that predicts what will happen in the future based on past and present states; and a controller (C) that selects the best action based on the brain's predictions.

[0022] Embodied interaction sequences are data streams that include timelines and causal chains, typically consisting of perception, action, feedback, and state evolution.

[0023] RGB-D cameras are used for 3D modeling and mapping. They capture point clouds with physical dimensions and can scan rooms in real time to generate 3D models containing geometric features and textures. They are also used to identify and segment foreground objects from the environment and to track human posture and movement.

[0024] LingBot-VA, an embodied intelligent world model open-sourced by Ant Group in early 2026, is the world's first open-source framework that deeply couples autoregressive video generation with action reasoning.

[0025] CoTracker is a long-term point tracking model introduced by Meta. Unlike ordinary box tracking, it can track specific pixels on the surface of an object and maintain a stable motion trajectory even if the object is rotated or partially occluded.

[0026] CLIP, or Contrastive Language-Image Pre-training, is a synesthetic model developed by OpenAI that translates text and images into coordinates in the same space. It scores the results by calculating the cosine similarity between two vectors, with scores typically ranging from 0 to 1.

[0027] VLM, Vision-Language Model, includes CLIP, but also includes more advanced models used to determine whether video content deviates from its original intent.

[0028] π0 (pi-zero) is an embodied model developed by Physical Intelligence, which excels at handling complex end-to-end tasks.

[0029] OpenVLA, an open-source vision-language-action model, is one of the most widely used base models in academia and industry.

[0030] LeRobot / RLDS is a data format standard. RLDS is a reinforcement learning dataset specification led by Google; LeRobot is a robot learning ecosystem launched by Hugging Face.

[0031] HoloBrain-0 is an open-source embodied pedigree model released by Horizon Robotics.

[0032] Contact grounding refers to the precise matching of action and physical contact. That is, when an object is observed to be under force, it is necessary to be able to locate the specific point of contact, rather than picking up the object from a distance.

[0033] Object Permanence refers to the fact that when an object's line of sight is obstructed, its physical properties (position, size, shape) still logically exist and it will not disappear or mutate into another object simply because it is temporarily out of sight.

[0034] Causal compliance emphasizes the sequential order of physical phenomena. The motion of an object must be driven by a force; there cannot be arbitrary displacement.

[0035] like Figure 1 As shown, one embodiment of this application relates to a method for synthesizing VLA training data, the method may include the following steps:

[0036] S1. Extract semantic representation information from human videos to obtain semantic action tuples; the semantic action tuples include task instructions, keyframe sequences, trajectory sequences, and contact event sequences;

[0037] S2. Using the multimodal signal, the current video observation frame, and the semantic action tuple as joint conditional inputs, fine-tune the autoregressive video-action world model to obtain the target autoregressive video-action world model;

[0038] S3. Obtain expanded training data based on the target autoregressive video-action world model;

[0039] S4. For the expanded training data, a three-dimensional quality filtering mechanism is executed to obtain training triples; the three-dimensional quality filtering mechanism is used to remove samples that do not meet the preset quality requirements; the training triples include video frame sequences, task instruction text, and action label sequences.

[0040] S5. Based on the training triples, obtain the target training dataset.

[0041] Before implementing the above methods, human first-person perspective data collection and other data collection can be performed. Specifically, the operator can use a wearable fisheye camera device to perform tasks in specific scenarios and record the process, obtaining human video. The frame rate should be no less than 30fps, and the resolution no less than 720p. Each captured segment corresponds to a single complete task, with a duration controlled between 10 and 120 seconds. The wearable fisheye camera device can be a head-mounted glasses camera or a fixed camera on the chest. Specific scenarios can include domestic scenarios, such as kitchen food preparation, dishwashing, clothing folding, mopping and wiping tables, or factory scenarios, such as parts assembly, screw tightening, packaging and sealing, and quality inspection.

[0042] While acquiring human video, multimodal signals can be acquired simultaneously. These multimodal signals include at least inertial measurement unit (IMU) data, depth information, and task instruction text. The IMU data is used to estimate the operator's head posture and wrist movement trajectory. The depth information is used to characterize the actual physical distance between the acquired object and the camera, and can be acquired using an RGB-D camera. The task instruction text serves as a supervisory signal for subsequent language conditions and is expressed as voice or text commands, such as "put the bowl in the bowl rack" or "insert the screw into the corresponding hole."

[0043] Specifically, before proceeding to step S1, the collected human videos can be preliminarily screened as follows: removing captured segments with blurry images, severe obstructions, or interrupted operations; using VLM for automatic scoring to remove captured segments where the task instructions do not match the content of the collected human videos; and establishing hierarchical data indexes by scene category and operation category to ensure that the data is evenly distributed across operation type and object category.

[0044] For step S1, since the human video does not contain structured robot action labels, it is necessary to extract semantic representations of the actions from the human video, which are then encapsulated to obtain the Semantic ActionTuple (SAT). The Semantic ActionTuple = {task instruction, keyframe sequence, trajectory sequence, contact event sequence}, which can serve as a unified input format for subsequent fine-tuning of the autoregressive video-action world model and obtaining the expanded training data. The keyframe sequence represents keyframes in the human video where the hand makes contact with an object, the trajectory sequence represents the three-dimensional trajectory sequence of both wrists in the human video, and the contact event sequence represents the structured task description of each operation segment, including: the operation object, the initial state, the target state, and the sequence of operation sub-steps.

[0045] Specifically, for step S2, an autoregressive video-action world model pre-trained on a large scale of internet videos can be selected, such as using LingBot-VA as the fine-tuning basis, and using the semantic action tuple, the multimodal signal, and the current video observation frame as joint conditional inputs to the pre-selected autoregressive video-action world model. After fine-tuning the autoregressive video-action world model using the corresponding fine-tuning strategy, the target autoregressive video-action world model is obtained.

[0046] Taking a housekeeping scenario as an example, in the fine-tuning stage of step S2, the task instruction of "organizing the scattered bowls and chopsticks and putting them into the dish rack" can be given, the initial frame when the operator starts to organize in the human video, as well as the semantic action tuple and the multimodal signal. The fine-tuned autoregressive video-action model should be able to predict the complete action video sequence of bowls and chopsticks being picked up, moved and put into the designated grid of the dish rack frame by frame. However, the general world model often cannot accurately model the insertion contact relationship between bowls and chopsticks and the dish rack.

[0047] Specifically, for steps S3-S5, the target autoregressive video-action world model is used to systematically expand the training data scale, resulting in expanded training data. For this expanded training data, a three-dimensional quality filtering mechanism is executed to remove samples that do not meet preset quality requirements. After three layers of filtering, only samples that pass all filters are retained. Each retained sample forms a standard format VLA training triplet: {video frame sequence, task instruction text, action label sequence}, where the action label sequence is the robot end effector target posture sequence obtained from the trajectory sequence through inverse kinematic mapping. The video frame sequence includes keyframe sequences, trajectory sequences, and contact event sequences. The training triplets are then formatted into the standard LeRobot / RLDS data format to obtain the target training dataset, ensuring compatibility with mainstream VLA training frameworks such as π0, OpenVLA, and HoloBrain-0, facilitating direct integration into existing training pipelines.

[0048] It is not difficult to see that, compared with related technologies, the solution provided in this application reduces the cost of training data collection in vertical scenarios and the dependence on professional operators and venues by collecting human videos instead of teaching data collection methods that rely on professional robot hardware. Through semantic action tuple extraction, human videos are transformed into structured robot action labels. The extracted semantic action tuples are cleaned and initially screened for quality, and then input together with pre-collected multimodal signals and the current video observation frame into an autoregressive video-action world model. This model is then fine-tuned to obtain a target autoregressive video-action world model. Using this target autoregressive video-action world model, the scale of training data is systematically expanded to obtain expanded training data. This expanded training data transforms operational knowledge from human-view videos into multi-view training data that the robot can learn, solving the problem of viewpoint mismatch between human video data and robot training data in existing methods. A three-dimensional quality filtering mechanism is executed on the expanded training data, retaining only samples that pass all filters and forming a standard format to obtain training triples. The training triples are further standardized to obtain target training data, which is convenient for direct integration into existing training pipelines. This method does not rely on a specific robot body or VLA model architecture and can be adapted to various robot forms such as robotic arms, mobile manipulation robots, and humanoid robots. It can also be used in a variety of embodied intelligence application fields, such as housekeeping, factories, autonomous driving, and warehousing and logistics.

[0049] In one embodiment, step S1, which involves extracting semantic representation information from human videos to obtain semantic action tuples, may include:

[0050] The three-dimensional trajectories of both wrists in the human video are extracted to obtain the trajectory sequence;

[0051] Identify key frames in the human video where the hand makes contact with an object, and label the grip type corresponding to the key frames to obtain the key frame sequence;

[0052] The visual language model is used to perform semantic parsing on the operation segments in the human video to obtain the contact event sequence; the contact event sequence includes the operation object, the initial state, the target state, and the operation sub-step sequence;

[0053] The task instructions, trajectory sequence, keyframe sequence, and contact event sequence are encapsulated to obtain a semantic action tuple.

[0054] Specifically, optical flow-based hand tracking models such as HandFormer can be used to extract the three-dimensional trajectories of both wrists in the human video, resulting in a discretized end-effector trajectory sequence. A contact detection model can be used to identify keyframes in the human video where the hand contacts an object, and the corresponding grasping type can be labeled. Grasp types can include: precise grasp, wrap-around grasp, and pinch. A visual language model (VLM) is used to perform semantic parsing on each operation segment in the human video, automatically generating a structured task description as the contact event sequence, specifically including: the operation object, the initial state, the target state, and the sequence of operation sub-steps. Finally, the above information is encapsulated into a Semantic Action Tuple (SAT), where SAT = {task instruction, keyframe sequence, trajectory sequence, contact event sequence}.

[0055] In this embodiment, by deconstructing human videos into trajectory sequences from a spatial perspective, keyframes from a physical interaction perspective, and contact event sequences from a logical semantic perspective, the video is transformed into ordered structured semantics and multi-dimensional feature alignment is achieved. The contact event sequence includes initial states, target states, and sub-steps, providing a clear logical chain for subsequent model fine-tuning. Combined with keyframes annotating the grasping type, this allows the model to grasp the timing of operations and interaction details while learning actions, preventing common model logic breakdowns and physical illusions. Furthermore, since human arms and robotic arms differ in morphology, directly imitating human videos often yields poor results. Extracting the three-dimensional trajectories of both wrists provides a geometric constraint independent of morphology, shielding irrelevant variables such as the demonstrator's appearance, allowing the extracted semantic action tuples to be transferred to robots with different shapes.

[0056] In one embodiment, step S2, which involves using the multimodal signal, the current video observation frame, and the semantic action tuple as joint conditional inputs to fine-tune the autoregressive video-action world model to obtain the fine-tuned target autoregressive video-action world model, may include:

[0057] The multimodal signal, the current video observation frame, and the semantic action tuple are feature-encoded and fused to obtain a conditional sequence, and time position encoding is added to the conditional sequence.

[0058] Using the latent variables of the video to be generated as the query vector, cross-attention computation is performed with the conditional sequence after adding time position encoding to obtain conditional association features containing semantic weights;

[0059] Based on the conditional association features, the prediction results of the autoregressive video-action world model are obtained; the prediction results include predicted video sequences and action sequences.

[0060] For the prediction results, the physical prior loss value is calculated in the fine-tuning loss function that includes the physical regularization term;

[0061] For the physical prior loss value, the attention layer and MLP layer of the autoregressive video-action world model are adapted in a low-rank manner using the LoRA method to obtain a fine-tuned autoregressive video-action world model.

[0062] Specifically, task instructions are presented in text form, observation frames in image form, and trajectories / contact events in numerical sequence form. These three belong to completely different modalities. Before being input into the attention layer of the autoregressive video-action world model, the feature alignment method needs to be defined, i.e., feature encoding and fusion are performed to obtain a conditional sequence, and then temporal position encoding is added to the conditional sequence. In this step, unstructured visual and textual data, as well as structured SAT data, are unified into the same feature dimension. Adding temporal position encoding meets the requirement that autoregressive models rely on temporal sequence relationships, ensuring that the model knows the order of actions.

[0063] Then, using the latent video variables to be generated as the query vector, and the conditional sequence with added temporal location encoding as the key / value pair, cross-attention computation is performed to obtain conditional association features containing semantic weights. Through cross-attention computation, the model learns how to observe specific objects or motion trends in the scene based on semantic weights.

[0064] Furthermore, when the model makes predictions, it examines conditional association features for each frame generated. The prediction result is the output of the autoregressive video-action model, which includes the predicted video sequence and action sequence. The video sequence represents the predicted visual evolution, and the action sequence represents the action changes corresponding to the visual sequence.

[0065] Furthermore, the error between the predicted result and the actual result is calculated by fine-tuning the loss function as the physical prior loss value. A physical regularization term is introduced into the fine-tuning loss function as a constraint to reduce the physical illusion of the model in vertical scenes.

[0066] Finally, instead of updating the model's full parameters based on the physical prior loss value, a Low-Rank Adaptation (LoRA) approach is used to perform low-rank adaptation on the model's attention layer and MLP layer. This involves adjusting the parameters of the attention layer and MLP (Multi-Layer Perceptron) layer to correct the illusion. This step does not require retraining the entire model's massive parameters; it only uses a small number of low-rank matrices to learn the mapping relationship between SAT conditions and video generation. Specifically, the conditional projection matrix is ​​used to learn the alignment relationship between SAT conditions and video latency, while the LoRA parameters within the temporal block are used to enhance the modulation capability of keyframe sequences, trajectory sequences, and contact event sequences for future frame generation. This efficiently injects vertical scene knowledge while retaining general generation capabilities, reducing the computational overhead of fine-tuning.

[0067] Furthermore, during the model fine-tuning process, appearance enhancements can be performed only on the visual observation channel, including background replacement, lighting perturbation, object texture transfer, and local occlusion injection; the labels of task instructions, trajectory sequences, and contact event sequences remain unchanged, thereby preventing the model from overfitting to the appearance of a specific scene without changing the action semantics.

[0068] In one embodiment, the step of feature encoding and fusing the multimodal signal, the current video observation frame, and the semantic action tuple to obtain a conditional sequence includes:

[0069] Map the task instructions to text tokens;

[0070] Map the current video observation frame and the keyframe in the semantic action tuple to a visual token;

[0071] The trajectory sequence in the semantic action tuple and the IMU sequence in the multimodal signal are compressed into a motion token;

[0072] Discretize the contact event sequence in the semantic action tuple into event tokens;

[0073] The above tokens are projected uniformly onto the same feature dimension and concatenated in the order of text token, visual token, motion token, and event token to obtain the conditional sequence.

[0074] Specifically, the task instructions are mapped to text tokens by a text encoder, the current video observation frame and the keyframes in the semantic action tuple are mapped to visual tokens by a visual encoder, the trajectory sequence in the semantic action tuple and the IMU sequence in the multimodal signal are compressed into motion tokens by a one-dimensional temporal encoder (1D convolution + Transformer), and the contact event sequence in the semantic action tuple is discretized into event tokens. The content of the event tokens can be: contact start / hold / release + object category + grasp type. After all the above modal tokens are uniformly projected to the same feature dimension, they are concatenated into the conditional sequence in the order of text token-visual token-motion token-event token.

[0075] In this embodiment, by uniformly projecting data from different modalities onto the same feature dimension, modal barriers are broken down, and a fixed splicing order of "text-vision-motion-contact" is defined. This fixed order helps the model learn the causal dependencies between different modalities more quickly during the fine-tuning process.

[0076] In one embodiment, the augmented training data is video-action pairs generated by calling the target autoregressive video-action world model, wherein the generalization dimension used in generating the video-action pairs includes at least one of the following:

[0077] Viewpoint generalization: Based on different camera poses, simulate the sensor configuration when the robot is actually deployed to obtain viewpoint generalized video-action pairs;

[0078] Object generalization: By randomly replacing operation objects under the same operation type, video-action pairs are obtained that are generalized to objects;

[0079] Background generalization: By randomly transforming the scene background, video-action pairs with a generalized background are obtained;

[0080] Command generalization: By generating commands in multiple languages ​​for the same operation task and pairing them with corresponding video sequences, a video-action pair of command generalization is formed.

[0081] Specifically, perspective generalization: Under the same operational semantics, different camera poses are randomly sampled, such as wrist camera view, top view, and side view, to simulate multiple sensor configurations when the robot is actually deployed, and to obtain perspective generalized video-action pairs;

[0082] Object generalization: Randomly replace the shape, color, texture, and size of the objects under the same operation type to obtain generalized video-action pairs of different object instances under the same operation type;

[0083] Background generalization: Randomly transform the scene background, such as the color of the kitchen counter, the factory floor, and surrounding clutter, to obtain background generalized video-action pairs, thereby improving the robustness of transferring to new scenes;

[0084] Instruction diversification: Generate multiple language expressions for the same operation task, such as "put the bowl in the bowl rack", "tidy up the bowls and chopsticks", and "put the tableware back in place", and pair them with the corresponding video sequences to form video-action pairs that generalize instructions, thereby enhancing the language generalization ability of the VLA model.

[0085] In this embodiment, a target autoregressive video-action world model is used as the data engine. Only a small amount of human video data is needed to derive a large number of synthetic samples as supplementary training data through a multi-dimensional generalization strategy. This overcomes the data scarcity problem faced in the field of embodied intelligence and reduces the human and time costs required for teaching real robots. Furthermore, the generalization generation through the target autoregressive video-action world model ensures that the supplementary training data has high learning value.

[0086] In one embodiment, the step S4, which involves performing a three-dimensional quality filtering mechanism on the augmented training data to obtain training triples, may include:

[0087] For the expanded training data, physical rationality filtering is performed to obtain a first sample set; the physical rationality filtering is used to verify whether the changes in object pose in keyframes of the generated video conform to the physical laws of operation.

[0088] For the first sample set, the instruction-video alignment filter is executed to obtain the second sample set; the instruction-video alignment filter is used to check whether the video content meets the instruction requirements.

[0089] For the second sample set, temporal consistency filtering is performed to obtain the training triplet; the temporal consistency filtering is used to determine the stability of video content through pixel-level and object-level displacement analysis.

[0090] Specifically, for the expanded training data, a lightweight physics simulation proxy model can be invoked to verify whether the object pose changes in keyframes of the generated video conform to the laws of operational physics. For example, if an object cannot move without contact, samples that do not meet the conditions are marked as low-quality samples and discarded, resulting in the first sample set. For the first sample set, the generated video content is checked to see if it meets the instruction requirements, and samples that do not meet the requirements are filtered out, resulting in the second sample set. For the second sample set, the stability of the video content is determined through pixel-level and object-level displacement analysis. Abnormal frames are further judged, and based on the judgment results, partial deletion or complete discarding of segments is performed. After the above three layers of filtering, the retained samples are formed into a standard format to obtain the training triplet.

[0091] In one embodiment, performing physical plausibility filtering on the augmented training data to obtain a first sample set includes:

[0092] Based on a preset video authenticity index, joint tracking is performed on the generated video corresponding to the expanded training data to obtain a frame-by-frame contact state map.

[0093] Based on the frame-by-frame contact state diagram, contact causal consistency determination, object persistence determination, and support and penetration constraint determination are performed respectively, and samples that do not meet any of the determination results are filtered to obtain the first sample set.

[0094] Specifically, the contact causal consistency determination is used to determine whether there is a change in hand-object contact or object-support relationship before and after the target object undergoes displacement or posture change; the object persistence determination is used to determine the trajectory continuity of the target object before and after occlusion; the support and penetration constraint determination is used to determine whether the target object is hovering without support or has mask penetration exceeding the allowable threshold.

[0095] Video realism metrics can include ContactGrounding, Object Permanence, and Causal Compliance, which are commonly used in robot operation world models. First, joint tracking of "hand-object-supporting surface" is performed on the generated video. Based on hand keypoint detection, target segmentation, and the CoTracker point trajectory tracker, the temporal trajectory of the operator's hand, the target object, and the supporting surface is obtained. Then, a frame-by-frame contact state map is constructed.

[0096] Then, based on the frame-by-frame contact state diagram, contact causality consistency determination, object persistence determination, and support and penetration constraint determination are performed respectively, and samples that do not meet any of the determination results are filtered to obtain the first sample set. First, contact causality consistency determination is performed: when the target object has a significant displacement or posture change at time t, a change in hand-object contact or object-support relationship should be detected within the k-frame window before and after it; otherwise, it is considered causeless motion. Second, object persistence determination is performed: when the target object is briefly occluded by a hand or other object, its trajectory should remain continuous and fall within the predicted motion envelope before occlusion after re-revealing, avoiding instantaneous disappearance, cross-position reappearance, or abrupt size changes of the target object. Third, support and penetration constraint determination is performed: within a single frame, the bottom of rigid objects such as tableware, tools, and parts must not be unsupported and hovering, and the mask overlap between the rigid object and the table, container, or other object must not exceed the allowable penetration threshold. Within a cross-frame frame, the state of an object from time t to time t+n must conform to physical laws. A displacement exceeding 10% of the target box diagonal length can be defined as significant displacement to determine whether the object has undergone meaningful motion. A position deviation not exceeding 15% of the velocity extrapolated from the position before occlusion can be defined as a reproducible position deviation to ensure that the motion of the object when occluded conforms to the physical trajectory. An area change exceeding 20% ​​in adjacent visible frames can be defined as an abnormal scale change to ensure that rigid objects do not deform or scale during motion.

[0097] In this embodiment, a physical compliance assessment system ensures that sample data conforms to the physical laws of the real world. Contact causality consistency determination forces the model to establish a causal link of "action causing displacement," preventing the robot from learning erroneous strategies such as grasping objects remotely. Object persistence determination solves the problem of objects flashing due to occlusion, enabling the model to learn the constancy logic that objects still exist outside the line of sight. Support constraints ensure the realism of objects being driven by gravity, while penetration constraints protect the geometric invariance of rigid objects, preventing visual illusions generated by the model during complex grasping processes. Dual verification of single-frame spatial logic and cross-frame temporal logic greatly improves the coherence of the video sequence. A fully automated data quality inspection pipeline is implemented using CoTracker trajectory tracking and contact state graphs.

[0098] In one embodiment, the step of executing instruction-video alignment filtering to obtain a second sample set for the first sample set includes:

[0099] Calculate the semantic similarity score between the task instruction of each item in the first sample set of augmented training data and the keyframe sequence in the corresponding generated video;

[0100] The expanded training data with semantic similarity scores below a preset threshold are filtered to obtain a second sample set.

[0101] Specifically, CLIP / VLM can be used to calculate the semantic similarity score between the task instruction and the corresponding keyframe of the generated video for each item of augmented training data in the first sample set; samples with semantic similarity scores below a threshold, such as similarity scores below 0.65, are filtered out to obtain the second sample, ensuring the instruction-action alignment quality of the synthesized data.

[0102] In one embodiment, performing time-series consistency filtering on the second sample set to obtain the training triples includes:

[0103] For the video sequences in the second sample set, perform pixel-level bidirectional optical flow estimation and object-level target tracking algorithms to obtain the effective consistent pixel ratio and non-occlusion jump amplitude value;

[0104] If the percentage of valid consistent pixels is lower than the first set ratio, or the non-occlusion jump value exceeds the second set ratio, the corresponding frame is determined to be an abnormal frame.

[0105] Based on the distribution pattern of abnormal frames, the video sequence is truncated or discarded entirely. The truncated compliant samples are merged with the globally normal video sequences to obtain a third sample set.

[0106] The video frame sequence, task instruction text, and action label sequence of each sample in the third sample set are extracted to form the training triplet.

[0107] Specifically, the effective consistent pixel ratio represents the proportion of pixels in the entire frame whose forward and backward consistency errors obtained from bidirectional optical flow estimation are within the allowable range. In other words, in optical flow estimation, the proportion of pixels whose distance between the predicted pixel position and the traced position from the backward optical flow is within the allowable range is the percentage of total pixels. A higher ratio indicates a more stable image, while a low ratio usually indicates flickering, content disappearing out of nowhere, or sudden visual illusions. The non-occlusion jump amplitude represents the displacement deviation of the center point of the bounding box of a key object under non-occlusion conditions. Specifically, combined with the target tracking algorithm, when the target occlusion rate is below a certain proportion, it is the ratio of the Euclidean distance between the object's center point in adjacent frames to the diagonal of the object's bounding box. It is used to determine whether a large displacement of an object's position, occurring under non-occlusion conditions, indicates a movement that defies the laws of physics.

[0108] If the percentage of valid consistent pixels is lower than a first set percentage (85%) at the pixel level, the corresponding frame can be identified as an abnormal frame. Alternatively, if the non-occlusion jump amplitude exceeds a second set percentage (20%) at the object level, the corresponding frame can be identified as an abnormal frame. After identifying all abnormal frames, the distribution pattern of abnormal frames in the segment is statistically analyzed. If abnormal frames only appear locally in the segment, sequence truncation is performed, i.e., the corresponding abnormal sequence is truncated from the abnormal point in the segment. If the percentage of abnormal frames exceeds 10% or there are at least 3 consecutive abnormal frames, the entire segment is discarded.

[0109] Furthermore, based on the truncated timestamp range, the corresponding task instruction text and action tag sequence are extracted synchronously to generate truncated compliant samples; samples whose statistical results show as globally normal are retained; the globally normal samples and the truncated compliant samples are merged to obtain the third sample set.

[0110] Finally, the video frame sequence, task instruction text, and action label sequence of each sample in the third sample set are extracted and encapsulated across modalities to form the training triplet that matches each other.

[0111] In this implementation, pixel-level bidirectional optical flow estimation accurately captures pixel motion vector deviations between consecutive frames. Using the percentage of valid consistent pixels as a criterion, it automatically identifies and eliminates image jitter and artifacts caused by model computational instability. Combining target tracking algorithms with non-occluded jump amplitude detection, temporal constraints are applied at the macroscopic object level. By setting a second set ratio, samples violating physical laws are filtered out. Furthermore, a selective strategy of sequence truncation or complete segment discarding is adopted. By analyzing the distribution patterns of abnormal frames, compliant segments in the video sequence are retained to the maximum extent possible. This ensures high data purity while improving the utilization rate of expanded training data, ensuring the scalability advantage of the training triplet set.

[0112] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this application. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this application.

[0113] Furthermore, some embodiments of this application also provide an electronic device. The electronic device can be various forms of digital computer, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, etc. The electronic device can also be various forms of mobile devices, such as cellular phones, smartphones, wearable devices, and other similar computing devices.

[0114] The electronic device includes: one or more processors; and a memory storing computer program instructions that, when executed, cause the processor to perform the steps of the methods provided in any one or more of the above embodiments. Figure 2 An exemplary structural diagram of the electronic device is disclosed. The electronic device includes one or more processors 1101, a memory 1102, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components are interconnected via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In some other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations. The components, their connections and relationships, and their functions shown herein are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0115] The electronic device may further include an input device 1103 and an output device 1104. The processor 1101, memory 1102, input device 1103 and output device 1104 may be connected by a bus or other means, as shown in the figure, which is connected by a bus.

[0116] Input device 1103 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the electronic device, such as a touch screen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 1104 may include a display device, auxiliary lighting device (e.g., LED), and haptic feedback device (e.g., vibration motor). The display device may include, but is not limited to, a liquid crystal display, a light-emitting diode display, and a plasma display. In some embodiments, the display device may be a touch screen.

[0117] To provide interaction with the user, the electronic device can be a computer. The computer has: a display device (e.g., a cathode ray tube or LCD monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback); and input from the user can be received in any form (e.g., voice input or tactile input).

[0118] In this embodiment, a computer-readable medium stores a computer program / instructions that, when executed by a processor, implement the steps of the methods provided in any one or more of the above embodiments. This computer-readable medium may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into that device. The aforementioned computer-readable medium carries one or more computer-readable instructions.

[0119] The memory 1102 can serve as a non-transitory computer-readable storage medium, used to store non-transitory software programs, non-transitory computer-executable programs, and modules. The processor 1101 executes various functional applications and data processing of the server by running the non-transitory software programs, instructions, and modules stored in the memory 1102, thereby implementing the program instructions / modules corresponding to the methods provided in any one or more of the embodiments described above in this application.

[0120] The memory 1102 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device. Furthermore, the memory 1102 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 1102 may optionally include memory remotely located relative to the processor 1101, and these remote memories can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0121] It should be noted that the computer-readable medium described in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. Computer-readable media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections having one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, optical fibers, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0122] Computer-readable media include permanent and non-permanent, removable and non-removable media, which can store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory, static random access memory, dynamic random access memory, other types of random access memory, read-only memory, electrically erasable programmable read-only memory, flash memory or other memory technologies, read-only optical discs, digital versatile optical discs or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0123] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including local area networks (LANs) or wide area networks (WANs), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0124] In the above embodiments, all or part of the implementation can be achieved through software, hardware, firmware, or any combination thereof. For example, it can be implemented using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In some embodiments, the software program of this application can be executed by a processor to implement the above steps or functions. Similarly, the software program of this application (including related data structures) can be stored in a computer-readable recording medium, such as RAM memory, magnetic or optical drives, floppy disks, and similar devices. In addition, some steps or functions of this application can be implemented in hardware, for example, as circuitry that cooperates with a processor to perform the various steps or functions.

[0125] The computer program product provided in this application includes one or more computer programs / instructions. When executed by a processor, these computer programs / instructions generate, in whole or in part, the processes or functions described in this application. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0126] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0127] The scope of this application is defined by the appended claims rather than the foregoing description, and is therefore intended to encompass all variations falling within the meaning and scope of equivalents of the claims. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. Terms such as "first," "second," etc., are used only to distinguish descriptions and do not indicate any particular order, nor should they be construed as indicating or implying relative importance.

[0128] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily made by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims, and the above embodiments should be regarded as exemplary and non-limiting.

Claims

1. A method for synthesizing VLA training data, characterized in that, include: Semantic representation information is extracted from human videos to obtain semantic action tuples; the semantic action tuples include task instructions, keyframe sequences, trajectory sequences, and contact event sequences; The multimodal signal, the current video observation frame, and the semantic action tuple are used as joint conditional inputs to fine-tune the autoregressive video-action world model, resulting in the target autoregressive video-action world model. Based on the aforementioned target autoregressive video-action world model, augmented training data is obtained; For the augmented training data, a three-dimensional quality filtering mechanism is executed to obtain training triples; the three-dimensional quality filtering mechanism is used to remove samples that do not meet the preset quality requirements; the training triples include video frame sequences, task instruction text, and action label sequences. The target training dataset is obtained based on the training triples.

2. The method according to claim 1, characterized in that, The extraction of semantic representation information from human videos to obtain semantic action tuples includes: The three-dimensional trajectories of both wrists in the human video are extracted to obtain the trajectory sequence; Identify key frames in the human video where the hand makes contact with an object, and label the grip type corresponding to the key frames to obtain the key frame sequence; The visual language model is used to perform semantic parsing on the operation segments in the human video to obtain the contact event sequence; the contact event sequence includes the operation object, the initial state, the target state, and the operation sub-step sequence; The task instructions, trajectory sequence, keyframe sequence, and contact event sequence are encapsulated to obtain a semantic action tuple.

3. The method according to claim 1, characterized in that, The step of using multimodal signals, the current video observation frame, and the semantic action tuple as joint conditional inputs to fine-tune the autoregressive video-action world model, resulting in a fine-tuned target autoregressive video-action world model, includes: The multimodal signal, the current video observation frame, and the semantic action tuple are feature-encoded and fused to obtain a conditional sequence, and time position encoding is added to the conditional sequence. Using the latent variables of the video to be generated as the query vector, cross-attention computation is performed with the conditional sequence after adding time position encoding to obtain conditional association features containing semantic weights; Based on the conditional association features, the prediction results of the autoregressive video-action world model are obtained; the prediction results include predicted video sequences and action sequences. For the prediction results, the physical prior loss value is calculated in the fine-tuning loss function that includes the physical regularization term; For the physical prior loss value, the attention layer and MLP layer of the autoregressive video-action world model are adapted in a low-rank manner using the LoRA method to obtain a fine-tuned autoregressive video-action world model.

4. The method according to claim 3, characterized in that, The step of feature encoding and fusing the multimodal signal, the current video observation frame, and the semantic action tuple to obtain a conditional sequence includes: The task instructions are mapped to text representation units; Map the current video observation frame and the keyframe in the semantic action tuple to visual representation units; The trajectory sequence in the semantic action tuple and the inertial measurement unit sequence in the multimodal signal are compressed into a motion representation unit; Discretize the contact event sequence in the semantic action tuple into event representation units; The aforementioned representation units are projected uniformly onto the same feature dimension and then concatenated in the order of text representation units, visual representation units, motion representation units, and event representation units to obtain the conditional sequence.

5. The method according to claim 1, characterized in that, The augmented training data consists of video-action pairs generated by calling the target autoregressive video-action world model. The generalization dimension used in generating the video-action pairs includes at least one of the following: Viewpoint generalization: Based on different camera poses, simulate the sensor configuration when the robot is actually deployed to obtain viewpoint generalized video-action pairs; Object generalization: By randomly replacing operation objects under the same operation type, video-action pairs are obtained that are generalized to objects; Background generalization: By randomly transforming the scene background, video-action pairs with a generalized background are obtained; Command generalization: By generating commands in multiple languages ​​for the same operation task and pairing them with corresponding video sequences, a video-action pair of command generalization is formed.

6. The method according to claim 5, characterized in that, The process of performing a three-dimensional quality filtering mechanism on the augmented training data to obtain training triples includes: For the expanded training data, physical rationality filtering is performed to obtain a first sample set; the physical rationality filtering is used to verify whether the changes in object pose in keyframes of the generated video conform to the physical laws of operation. For the first sample set, the instruction-video alignment filter is executed to obtain the second sample set; the instruction-video alignment filter is used to check whether the video content meets the instruction requirements. For the second sample set, temporal consistency filtering is performed to obtain the training triplet; the temporal consistency filtering is used to determine the stability of video content through pixel-level and object-level displacement analysis.

7. The method according to claim 6, characterized in that, The process of performing physical plausibility filtering on the expanded training data to obtain a first sample set includes: Based on a preset video authenticity index, joint tracking is performed on the generated video corresponding to the expanded training data to obtain a frame-by-frame contact state map. Based on the frame-by-frame contact state diagram, contact causal consistency determination, object persistence determination, and support and penetration constraint determination are performed respectively, and samples that do not meet any of the determination results are filtered to obtain the first sample set. The contact causal consistency determination is used to determine whether there is a change in hand-object contact or object-support relationship before and after the target object undergoes displacement or posture change. The object persistence determination is used to determine the trajectory continuity of the target object before and after occlusion. The support and penetration constraint determination is used to determine whether the target object is hovering without support or has mask penetration exceeding the allowable threshold.

8. An electronic device, characterized in that, The electronic device includes: One or more processors; and A memory storing computer program instructions, which, when executed, cause the processor to perform the steps of the method as described in any one of claims 1 to 7.

9. A computer-readable medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 7.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 7.

Images