Method and device for generating VLA model for robot

Abstract

The invention provides a method for generating a VLA model for a robot. The method comprises the steps that video streams and joint state data synchronously recorded in the process that the robot executes actions according to a preset track are acquired; for each frame of recorded image in the video stream, acquiring associated data of a current joint state and a future joint state, and a natural language instruction; and taking each frame of image and the corresponding natural language instruction as the input of a multi-modal neural network model, and taking the corresponding associated data as a supervision output target for training to obtain the VLA model. The invention further provides a corresponding device. According to the method, the VLA model has the capability of pre-judging the future joint state in combination with the visual environment information, so that the robot is supported to execute a coherent and reasonable action sequence.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot VLA model training, and in particular to a method and device for generating a VLA model for a robot. BACKGROUND

[0002] A VLA (Visual-Linguistic-Action) model is a core technology for a humanoid robot to achieve autonomous task execution, which receives natural language instructions, perceives visual information of the environment, and outputs precise continuous action sequences, so as to ultimately achieve that the robot understands the instructions, clearly sees the environment, and makes actions. The training effect of the model directly determines the adaptability of the robot to complex scenes and the accuracy of task execution.

[0003] The current mainstream training methods of the VLA model can be divided into three categories, and there are significant differences in the dependence characteristics of the data. Supervised learning training is based on large-scale labeled data for training, which is stable and converges quickly; reinforcement learning training does not need to rely on a large amount of labeled data, but requires feedback signals from the environment, and has high requirements for real-time and continuity, and is difficult to converge; self-supervised / semi-supervised learning training utilizes the internal correlation of unlabeled data to reduce the dependence on artificial labeling.

[0004] However, the inventors found in research that in reinforcement learning training, the current VLA model does not pay attention to the joint state of the robot at the next moment, and does not establish a time sequence transfer mapping of "current action-future action", the model cannot learn the continuous evolution rule of the action, resulting in a connection fault (such as a smooth transition to a moving action after a grasping action) in the output action sequence, poor action execution fluency, and easy joint motion conflict (such as posture mutation in multi-joint collaborative action). In a complex dynamic scene (such as moving target objects and changing environmental obstacles), the robot needs to adjust the current decision by predicting the effect of subsequent actions, and when the future joint state data is missing, the model can only optimize local actions based on immediate environmental feedback, and cannot form a global optimal strategy, resulting in an increased task failure rate (such as grasping a moving object, resulting in grasping deviation due to failure to predict the subsequent position of the object).

[0005] Therefore, in this context, how to enable the VLA model to have the ability to predict the future joint state in combination with visual environmental information to support the robot to perform coherent and reasonable action sequences is a technical problem to be solved. SUMMARY

[0006] In view of the above problems of the prior art, the present application provides a method and device for generating a VLA model for a robot, so as to enable the VLA model to have the ability to predict the future joint state in combination with visual environmental information to support the robot to perform coherent and reasonable action sequences.

[0007] To achieve the above objectives, the first aspect of this application provides a method for generating a VLA model for a robot, comprising: Acquire video streams and joint status data synchronously recorded during the robot's execution of actions according to a preset trajectory; For each frame of the recorded video stream, obtain the correlation data between the current joint state and the future joint state, as well as natural language instructions; Each frame of image and its corresponding natural language instruction are used as input to the multimodal neural network model, and the corresponding associated data are used as the supervised output target for training to obtain the VLA model.

[0008] As described above, by synchronously pairing the current joint state with the associated data of the future joint state for each frame of the video stream, and combining this with semantically matched natural language instructions, training samples with temporal action supervision signals are constructed. This design enables the VLA model to explicitly learn the temporal transition mapping of "current action - future action" during training, effectively overcoming the problems of action connection discontinuity, abrupt posture changes, and multi-joint motion conflicts caused by relying solely on instantaneous action labels in existing technologies. When performing complex tasks such as grasping, moving, and placing, the model can generate smooth transitional continuous action sequences, which conform to the mechanical motion characteristics of humanoid robots. In complex scenarios such as moving target objects or dynamic changes in environmental obstacles, the model can combine visual environment information with the prediction results of future joint states to predict the posture effect after the current action is executed, thereby optimizing the decision path, avoiding local optima traps, and significantly reducing the task failure rate (such as grasping deviation, untimely obstacle avoidance, etc.). This enables the VLA model to predict future joint states by combining visual environment information, supporting the robot to execute coherent and reasonable action sequences.

[0009] As one possible implementation of the first aspect, the associated data is the current motion joint state data corresponding to each frame of the video stream and the future motion joint state data corresponding to the next time step image frame. The semantics of the natural language instruction must be strongly correlated with the visual scene of the video stream and the action sequence of the executed action; each frame of the image corresponds to a unique set of current action joint state data and a unique semantically matched natural language instruction.

[0010] As described above, by precisely pairing each frame of image with its current action and its future action at the next time step, and ensuring that natural language instructions are strictly aligned semantically and temporally with the visual scene and action sequence, the training samples explicitly encode the short-term evolutionary patterns of actions. The VLA model can thus learn the inherent transition logic between consecutive actions, significantly improving the smoothness and mechanical rationality of action output. Furthermore, in scenarios such as moving objects or dynamic environmental changes, the model can adjust its current decisions in advance based on predictions of future joint states.

[0011] As one possible implementation of the first aspect, the training includes: Construct a multimodal neural network model, which includes a visual encoder, a language encoder, and a fusion module; Each image frame is input into the visual encoder, and the natural language instructions are input into the language encoder; the images are then fused through the fusion module to generate fused features. Based on the fusion features, the current motion joint state data and future motion joint state data are regressed and predicted respectively through dual task output heads; The model is trained end-to-end using a loss function, which includes a current action prediction loss term and a future action prediction loss term.

[0012] As described above, by employing a multimodal neural network architecture including a visual encoder, a language encoder, and a fusion module, and setting dual-task output heads to regress current and future action joint state data respectively, the model can represent data features across modalities. Combined with a joint loss function including current action prediction loss terms and future action prediction loss terms, the physical continuity and temporal rationality of the action trajectory are effectively constrained, avoiding amplification of future action prediction bias or gradient conflicts caused by single-task optimization. This improves training efficiency and long-term consistency of action sequences without increasing model complexity. Consequently, temporally coherent and physically reasonable action sequences are generated, significantly improving the robot's task execution stability and fluency in complex environments.

[0013] As one possible implementation of the first aspect, the joint state data includes at least two of the following: joint position, joint velocity, and joint torque.

[0014] As described above, by simultaneously including at least two of the joint position, velocity, and / or torque, the dynamic state of the robot when performing actions can be recorded more comprehensively, providing a high-dimensional, physically consistent action supervision target for the VLA model, thereby improving the accuracy and generalization ability of action prediction.

[0015] As one possible implementation of the first aspect, the robot is equipped with a first controller, a second controller, a third-party controller, and a camera; the acquisition of video streams and joint state data synchronously recorded during the robot's execution of actions according to a preset trajectory includes: The video stream is acquired through a camera, and joint state data during the execution of the action is acquired through the second controller; By deploying ROS2 forwarding nodes on the first controller, the joint status data topic from the second controller is subscribed to, and the joint status data is redistributed to a forwarding topic accessible to the third-party controller; wherein, the forwarding topic has the same QoS configuration as the joint status data topic; The third-party controller subscribes to and synchronously records the joint status data of the forwarded topic and the video stream from the camera image topic; The recorded joint state data is time-aligned based on the timestamp of each frame in the recorded video stream.

[0016] As described above, current training methods typically use training data from a first controller (such as Orin, responsible for controlling sensors like cameras and radar) and a second controller (such as an x86 industrial control board, responsible for controlling joints like arms, fingers, and legs). However, third-party controllers, when connected to the first controller, can only acquire camera video streams and cannot directly read joint state data. This is because, to ensure robot safety, manufacturers only release joint state ROS topics and prohibit third-party controllers from directly receiving data from these topics. These technical limitations prevent third-party controllers from obtaining joint state data, thus failing to meet the core data requirements of the VLA model. The lack of joint state data leads to a break in the basic "vision-action" correlation, making it impossible to form complete ternary training data. Even if joint data is obtained through methods such as granting permissions to the native controller or additional collection by external sensors, there are still significant drawbacks. The native controller has poor adaptability (interfaces are not uniform across different robot brands) and low synchronization accuracy (cross-controller communication delays cause vision and action misalignment). External sensor collection has problems such as high deployment costs (requiring additional installation of joint angle sensors and torque sensors) and large data interference (ambient lighting and mechanical vibration affect collection accuracy). Ultimately, the generated training data cannot meet the VLA model's requirements for temporal alignment and semantic consistency, which seriously restricts the improvement of model training efficiency and robot task execution capabilities.

[0017] By configuring the nodes that forward topics to match the QoS parameters of the joint state data topics published by the original second controller, the first controller, which legally obtains joint state data, forwards the joint state data collected in real time by the second controller, enabling the third-party controller to securely and efficiently acquire the joint state data from the second controller. Furthermore, the third-party controller performs temporal alignment between the joint state data and video image frames to construct temporally aligned training samples. This allows the VLA model to explicitly learn the temporal transition mapping between "current action – future action" during training. The VLA model thus learns the inherent transition logic between consecutive actions, significantly improving the smoothness and mechanical rationality of action output. Simultaneously, in scenarios such as moving target objects or dynamic environmental changes, the model can adjust current decisions in advance based on predictions of future joint states.

[0018] As one possible implementation of the first aspect, it also includes: The third-party controller has a joint data cache container for caching the joint state data received by subscribing to the forwarding topic before the first frame of the video stream arrives at the third-party controller.

[0019] As described above, the joint state data of the video data stream before it reaches the third-party controller is cached through the joint data caching container.

[0020] As one possible implementation of the first aspect, it also includes: When the first frame of the video stream arrives at the third-party controller, the joint state data cached in the joint data cache container is supplemented. The timestamps of the cached joint state data are converted based on the timestamps of the image frames of the video stream recorded by the third-party controller; Align the cached joint state data with the image frames of the video stream.

[0021] As described above, by supplementing cached data, the loss of joint state data before the video stream reaches the third-party controller is avoided. This allows for the acquisition of high-quality, time-aligned VLA model training data.

[0022] A second aspect of this application provides an apparatus for generating VLA models for robots, comprising: The data acquisition module is used to acquire video streams and joint status data that are recorded synchronously during the robot's execution of actions according to a preset trajectory; The data processing module is used to obtain the current joint state and future joint state association data, as well as natural language commands, for each frame of the recorded video stream. The model training module is used to train the VLA model by taking each frame of image and its corresponding natural language instruction as input to the multimodal neural network model and taking the corresponding associated data as the supervised output target.

[0023] A third aspect of this application provides a computing device, including: a processor and a memory storing program instructions thereon, the program instructions, when executed by the processor, causing the processor to perform the method for generating a VLA model for a robot as described in any of the first aspects.

[0024] A fourth aspect of this application provides a computer-readable storage medium having program instructions stored thereon, which, when executed by a computer, cause the computer to perform the method for generating a VLA model for a robot as described in any of the first aspects.

[0025] The fifth aspect of this application provides a computer program product including program instructions that, when executed by a computer, cause the computer to perform the method for generating a VLA model for a robot as described in any of the first aspects. Attached Figure Description

[0026] Figure 1 This is a flowchart of a method for generating a VLA model for a robot, provided in the first embodiment of this application; Figure 2 This is a flowchart of a method for generating a VLA model for a robot, provided in the second embodiment of this application; Figure 3 This is a schematic diagram of an apparatus for generating a VLA model for a robot, provided in the third embodiment of this application; Figure 4 This is a schematic structural diagram of a computing device provided in an embodiment of this application.

[0027] It should be understood that the dimensions and shapes of the blocks in the above structural diagrams are for reference only and should not constitute an exclusive interpretation of the embodiments of the present invention. The relative positions and inclusion relationships between the blocks presented in the structural diagrams are only schematic representations of the structural relationships between the blocks, and are not intended to limit the physical connection methods of the embodiments of the present invention. Detailed Implementation

[0028] The technical solutions provided in this application will be further described below with reference to the accompanying drawings and embodiments. It should be understood that the system architecture and business scenarios provided in the embodiments of this application are mainly for illustrating possible implementations of the technical solutions of this application and should not be construed as the sole limitation on the technical solutions of this application. Those skilled in the art will recognize that the technical solutions provided in this application are equally applicable to similar technical problems as system architectures evolve and new business scenarios emerge.

[0029] It should be understood that the scheme for generating VLA models for robots provided in the embodiments of this application includes generating training datasets for robot VLA models and methods, apparatus, computing devices, computer-readable storage media, and computer program products for training robot VLA models. Since these technical solutions solve problems based on the same or similar principles, some repetitions may not be repeated in the following descriptions of specific embodiments, but it should be considered that these specific embodiments have mutual references and can be combined with each other.

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. In case of any inconsistency, the meaning set forth in this specification or derived from the content described herein shall prevail. Furthermore, the terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application. To accurately describe the technical content of this application and to accurately understand the invention, the following explanations or definitions of the terms used in this specification are provided before describing specific embodiments: 1) Vision-Language-Action Model (VLA Model): A multimodal embodied intelligence model that can receive natural language commands and visual inputs, and output a continuous sequence of joint-level actions to drive the robot to complete tasks.

[0031] 2) Joint State Data: A set of physical quantities describing the motion state of each joint of the robot, typically including at least one of joint position, velocity, and torque, used to characterize the robot's real-time dynamic behavior.

[0032] 3) ROS 2 (Robot Operating System 2): An open-source middleware framework for robot applications that supports distributed node communication, real-time configuration (QoS) and cross-platform deployment, and is widely used in modern robot software architecture.

[0033] 4) QoS (Quality of Service): In ROS 2, QoS is a policy configuration used to define the reliability of message transmission. It includes parameters such as history depth, reliability (reliable / best effort), and durability, which affect the cross-node data synchronization effect.

[0034] 5) Topic: The data channel for asynchronous communication between nodes in ROS 2. Publishers send messages to topics, and subscribers receive messages from topics. It is the core mechanism for the transmission of sensor data and control commands.

[0035] 6) Timestamp Alignment: Matching and synchronizing asynchronous data streams from different sensors or controllers according to their associated timestamps to construct time-consistent multimodal samples.

[0036] 7) Dual-task Head: Two parallel regression branches at the end of the neural network, used to predict different targets (such as the current action and the future action), sharing backbone features to achieve joint optimization.

[0037] 9) Relay Node: In ROS 2 systems, this is an intermediate node used to subscribe to the original topic and redistribute messages to the new topic. It is often used for data bridging across permission domains or for QoS adaptation.

[0038] 10) Third-party controller: An external computing unit that is not part of the robot's original manufacturer's underlying control system. It is usually used for high-level task planning, data logging, or model reasoning. Due to security policies, it cannot directly access the original joint data.

[0039] The scheme for generating a VLA model for a robot provided in this application includes: acquiring a video stream and joint state data synchronously recorded during the robot's execution of actions according to a preset trajectory; for each frame of the recorded video stream, acquiring the correlation data between the current joint state and the future joint state, as well as natural language commands; using each frame of the image and its corresponding natural language command as input to a multimodal neural network model, and using the corresponding correlation data as the supervised output target for training, to obtain the VLA model. This method enables the VLA model to have the ability to predict future joint states by combining visual environment information, so as to support the robot in executing coherent and reasonable action sequences. This application embodiment can be applied to VLA model training scenarios in various industrial, robotics, drone, and embodied intelligence fields.

[0040] The first embodiment of this application provides a method for generating a VLA model for a robot, which will be described below in conjunction with... Figure 1 The implementation of each step of the method is described in detail, including steps S10-S30.

[0041] S10: Acquire the video stream and joint status data synchronously recorded during the robot's execution of actions according to the preset trajectory.

[0042] In some embodiments, the robot includes intelligent robot systems that require precise operation based on visual feedback, such as humanoid robots, robotic arms, and mobile operating platforms.

[0043] In some embodiments, the acquisition scene is built according to the VLA training requirements, including at least one of the following methods: placing target objects and obstacles, adjusting the camera installation angle, and adjusting the scene light source.

[0044] In some embodiments, the joint state data includes at least two of the following: joint position, joint velocity, and joint torque.

[0045] In some embodiments, the robot is configured with a first controller, a second controller, a third-party controller, and a camera; acquiring the video stream and joint state data synchronously recorded during the robot's execution of actions according to a preset trajectory includes: The video stream is acquired through a camera, and joint state data during the execution of the action is acquired through the second controller; By deploying ROS2 forwarding nodes on the first controller, the joint status data topic from the second controller is subscribed to, and the joint status data is redistributed to a forwarding topic accessible to the third-party controller; wherein, the forwarding topic has the same QoS configuration as the joint status data topic; The third-party controller subscribes to and synchronously records the joint status data of the forwarded topic and the video stream from the camera image topic; The recorded joint state data is time-aligned based on the timestamp of each frame in the recorded video stream.

[0046] In some embodiments, the robot adopts a two-level control architecture, with the first controller (such as NVIDIA Jetson Orin) responsible for global scheduling and the second controller (such as x86 industrial control board) responsible for real-time driving of joints such as arms and fingers.

[0047] In some embodiments, the robot adopts a ROS2-based distributed communication architecture. Joint state data is published as a ROS2 topic (e.g., / motion / control / arm_joint_state) through a second controller. Through QoS configuration, network domain isolation, or topic namespace restrictions, only the first controller is authorized to subscribe to the joint state data topic.

[0048] In some embodiments, the QoS configuration of the joint state data topic includes at least one of the following: a reliability policy of best-effort or reliable; a persistence policy of volatile; and a history policy of keep the latest (KEEP_LAST) where the depth of historical messages can be determined based on the frame rate of the acquisition hardware.

[0049] In some embodiments, the second controller has multiple publishers corresponding to multiple joint state data topics, such as an arm joint state data topic and a finger joint state data topic. The forwarding node creates multiple subscribers, subscribes to multiple joint state data topics by binding corresponding callback functions, and publishes the data to the forwarding topics through the multiple publishers.

[0050] The callback function bound to the publisher of the forwarded topic can also include the following functionality: outputting subscriber count information (i.e., outputting the number of nodes that have subscribed to the forwarded topic). For example, by setting an optional log debugging function in the forwarding node, the subscriber count information can be output once every 100 forwarded messages.

[0051] In some embodiments, the main function of the forwarding node implements complete ROS2 lifecycle management, including: ROS2 initialization, node instance creation, node spin operation and resource release.

[0052] In some embodiments, the format of the video stream data captured by the camera (including pixel encoding, resolution, and frame rate) is determined by its type and model; for example, color cameras typically output images encoded in RGB8 or BGR8, while grayscale cameras output images encoded in Mono8.

[0053] In some embodiments, the camera publishes a video stream via an image topic in the format of a ROS2 image message.

[0054] Among them, the reliability strategy for camera image topics is either best-effort or reliable; the persistence strategy is either transient local or volatile; the history strategy is keep the latest; and the depth of historical messages is determined based on the number of frames in the video stream.

[0055] In some embodiments, the third-party controller is on the same local area network as the robot and supports the ROS2 Humble operating environment, with at least 8GB of memory and 50GB of storage space (for storing videos and datasets).

[0056] In some embodiments, temporary communication parameters are configured in the terminal of the third-party controller. These parameters can be written to a user configuration file for permanent effect. The temporary communication parameters to be configured include the ROS domain identifier (ROS_DOMAIN_ID), which must be completely consistent with the configuration of the robot's first controller; the hostname of the third-party controller (ROS_HOSTNAME); and the ROS master node address of the robot's first controller (ROS_MASTER_URI), in the format "http: / / [first controller IP address]:11311", where the IP address needs to be replaced with the actual IP address of the robot's first controller obtained from the query.

[0057] In some embodiments, the core parameter configuration of the multi-source data synchronous acquisition node includes at least one of the following: the subscribed topic list (including forwarded topics, i.e., joint state data topics, and video image topics), the total recording duration, the video output frame rate (matching the camera frame rate), and the image flipping mode.

[0058] In some embodiments, a third-party controller subscribes to and synchronously records joint state data of the forwarded topic and video streams from camera image topics via a multi-source data synchronization acquisition node. The multi-source data synchronization acquisition node is also used for data storage and timing alignment of joint state data and video stream image frames.

[0059] In some embodiments, the moment when the first frame of the video stream is received is taken as the starting timestamp, and the video stream and the joint state data are synchronously recorded for a fixed duration starting from the starting timestamp.

[0060] In some embodiments, the storage path adopts a hierarchical structure of "root directory + timestamp subdirectory". Each time synchronization begins, a subdirectory is created with the current starting timestamp to avoid data overwriting.

[0061] In some embodiments, the multi-source data synchronization acquisition node subscribes to forwarding topics and camera image topics by creating multiple subscribers; the QoS configuration of the subscriber is consistent with its corresponding topic and associated with the corresponding callback function.

[0062] The callback function bound to the subscriber of the camera image topic includes at least one of the following functions: triggering the recording (synchronous recording) initialization process, including creating a start timestamp subdirectory, generating a video output path, supplementing cached joint state data, and starting timing; within the recording duration and before reaching the maximum frame rate, converting ROS image messages to OpenCV format (BGR color space), performing flip processing according to the configuration, and storing them in the image frame list, outputting the recording progress once every time a number of frames matching the output frame rate are acquired; when the recording duration or the maximum frame rate is reached, triggering the data storage process, closing the recording state, and releasing ROS2 node resources.

[0063] Among them, the multi-source data synchronous acquisition node converts ROS image messages into OpenCV format image frames by initializing the image conversion tool and stores them in the image frame list.

[0064] The image frames of the OpenCV format video stream stored in the image frame list are encoded into MP4 format video files according to the configured output frame rate through the video write interface of the computer vision library (OpenCV) and stored in the timestamp subdirectory.

[0065] In some embodiments, the callback function bound to the subscriber who forwards the topic may include at least one of the following functions: caching joint state data and outputting cached state logs.

[0066] Caching joint state data may include: setting up a joint data cache container to cache the joint state data received by subscribing to the forwarding node topic before the first frame of the video stream arrives at the third-party controller (i.e. before recording begins).

[0067] The output buffer status log may include: outputting a log prompt every 50 buffered data entries before the first frame of the video stream arrives at the third-party controller (i.e. before recording begins).

[0068] In some embodiments, it also includes: The third-party controller has a joint data cache container for caching the joint state data received by subscribing to the forwarding topic before the first frame of the video stream arrives at the third-party controller.

[0069] In some embodiments, it also includes: When the first frame of the video stream arrives at the third-party controller, the joint state data cached in the joint data cache container is supplemented. The timestamps of the cached joint state data are converted based on the timestamps of the image frames of the video stream recorded by the third-party controller; Align the cached joint state data with the image frames of the video stream.

[0070] In some embodiments, the third-party control node terminal is verified to see if it can receive forwarded topics and camera image topics.

[0071] In some embodiments, a joint data storage container is provided to store joint status data categorized by topic, including: joint name, timestamp, joint position, joint velocity, and joint torque.

[0072] Among them, the timestamp of the joint status data is a relative timestamp, which is converted based on the timestamp of the image frame of the video stream recorded by the third-party controller.

[0073] The timestamps of the joint state data stored in the joint data storage container are used to align with the timeline of the image frames in the video stream.

[0074] In some embodiments, the number of joint state data entries stored in the joint data storage container is on the same order of magnitude as the number of frames stored in the image frame list, and matches the recording duration.

[0075] In some embodiments, the method further includes verifying the synchronization between joint state data and image frames, including at least one of the following methods: checking file integrity, checking the order of magnitude, and verifying keyframe motion matching.

[0076] S20: For each frame of the recorded video stream, obtain the correlation data of the current joint state and the future joint state, as well as natural language instructions.

[0077] In some embodiments, the associated data consists of the current motion joint state data corresponding to each frame of the video stream and the future motion joint state data corresponding to the next time step image frame. The semantics of the natural language instruction must be strongly correlated with the visual scene of the video stream and the action sequence of the executed action; each frame of the image corresponds to a unique set of current action joint state data and a unique semantically matched natural language instruction.

[0078] Among them, semantically matched natural language instructions refer to natural language instructions used to describe the action to be performed, such as "grab the cylinder on the desktop and place it on the left platform".

[0079] The visual information mentioned in the instructions, such as target objects, obstacles, colors, and backgrounds, must be clearly identifiable in the visual file.

[0080] In this process, the semantic matching natural language instructions must be consistent with the task stage of the corresponding frame image. For example, the task flow of "grab → place" must be completely matched with the temporal logic of "joint extension → clamping → moving → releasing" composed of multiple frames of images. Each frame image corresponds to one of the steps, such as "joint extension", in order to avoid model misunderstanding caused by semantic and visual / action mismatch.

[0081] In some embodiments, joint state data and image frames are preprocessed. Preprocessing of the joint state data includes at least one of the following methods: removing invalid data, linear interpolation, standardization, and normalization. Preprocessing of the image frames includes at least one of the following methods: conversion to RGB color space, normalization, bilinear interpolation, and image enhancement.

[0082] In some embodiments, dual action labels are associated with each frame of image. The joint state data corresponding to the current image frame (current action joint state data) is the current action label, and the joint state data corresponding to the image frame at the next time step or the next moment (future action joint state data) is the future action label.

[0083] The dual action labels are used as supervised output targets when training the model.

[0084] Specifically, using the timestamp t of each frame as a reference, key state data at the corresponding moment is extracted from the synchronously recorded joint state data to form the current motion joint state data; simultaneously, the timestamp of the next time step image frame is extracted. The corresponding joint states serve as joint state data for future movements, where, S30: The frame interval of the video stream. Each frame image and its corresponding natural language instruction are used as input to the multimodal neural network model, and the corresponding associated data are used as the supervised output target for training to obtain the VLA model.

[0085] In some embodiments, training samples or training datasets are stored in a hierarchical structure, such as by encapsulating them in HDF5 format.

[0086] The root node of the hierarchical structure can include: a joint data group, an image data group, and a metadata group. The joint data group can further include multiple sub-data groups, such as an arm joint sub-group and a finger joint sub-group. The metadata group stores at least one of the following information: start time, recording duration, frame rate, number of joints, image size, or preprocessing parameters.

[0087] In some embodiments, natural language instructions are used to describe the semantic goals or behavioral intentions of the current operational task (the robot performs actions according to a preset trajectory), and may include, but are not limited to, at least one of the following: the attributes and location of the target object, the desired action, the target location or reference point of the action, high-level semantic expression (such as cleaning the desktop), constraints or operational requirements.

[0088] Natural language instructions are aligned with image frames and stored as text strings. They can be organized and stored using structured templates or prompt word templates.

[0089] In some embodiments, the training includes: Construct a multimodal neural network model, which includes a visual encoder, a language encoder, and a fusion module; Each image frame is input into the visual encoder, and the natural language instructions are input into the language encoder; the images are then fused through the fusion module to generate fused features. Based on the fusion features, the current motion joint state data and future motion joint state data are regressed and predicted respectively through dual task output heads; The model is trained end-to-end using a loss function, which includes a current action prediction loss term and a future action prediction loss term.

[0090] The visual encoder can employ at least one of the following models: EfficientNet, ResNet, VisionTransformer, ConvNeXt, or MobileNetV3.

[0091] The language encoder may employ at least one of the following models: BERT, RoBERTa, DeBERTa, DistilBERT, or ALBERT.

[0092] The fusion module can employ at least one of the following fusion mechanisms: cross-attention, adaptive feature weighting, MLP-based fusion, bilinear pooling, Transformer encoder, gated fusion network, or cascaded splicing followed by a fully connected layer.

[0093] In some embodiments, the model adopts a "multimodal feature extraction-cross-modal fusion-dual-task output" architecture. It uses EfficientNet-B3 and RoBERTa-base to extract visual and linguistic features respectively, achieves cross-modal alignment and fusion through cross-attention and adaptive weighting mechanisms, and then completes current action regression and future action prediction in parallel through dual output heads. The training is carried out in three stages: first, unsupervised pre-training is used to explore the intrinsic correlation of modalities; then, supervised fine-tuning is performed through weighted joint loss (MSE + smooth L1); and finally, temporal consistency loss is added to optimize the smoothness of actions.

[0094] During training, Dropout, early stopping, and L2 regularization are used to suppress overfitting. Dynamic weight adjustment and LayerNorm are used to ensure modal balance. At the same time, the model is pruned, quantized, and converted to ONNX format to adapt to real-time requirements, ensuring that the single-frame inference time is ≤30ms.

[0095] The evaluation focuses on quantitative indicators (Joint State MAE / RMSE, Temporal Consistency AD, Task Success Rate) and qualitative standards (Modal Semantic Consistency, Robot Movement Fluency). It requires that the position MAE ≤ 0.02, the future action prediction MAE ≤ 0.03, and the task success rate ≥ 85%. After meeting the standards, the optimal model will be deployed to the humanoid robot third-party controller, adapted to the ROS2 inference environment, and realize precise action execution driven by language commands.

[0096] Training can also use at least one of the following loss terms as a single loss function or a weighted combination of multiple loss terms: MSE, smoothed L1, cosine similarity, Huber loss.

[0097] The second embodiment of this application provides a method for generating a VLA model for a robot. Referring below... Figure 2 The flowchart shown illustrates that the method provided in this second embodiment includes the following steps S200-S270.

[0098] S200: Deploy ROS2 forwarding nodes on the first controller, subscribe to joint status data topics from the second controller, and publish forwarding topics.

[0099] This application provides a detailed description of a method for generating VLA models for humanoid robots. The humanoid robot in this application adopts a two-level control architecture. The first controller (such as NVIDIA Jetson Orin) is responsible for global scheduling, and the second controller (such as an x86 industrial control board) is responsible for real-time actuation of joints such as arms and fingers. The first and second controllers communicate bidirectionally through the ROS2 distributed communication architecture. Joint state data is collected and published by the second controller (e.g., / motion / control / arm_joint_state), and only the first controller is authorized to subscribe to this joint state data topic. This application uses Python as the primary development language.

[0100] This application employs a method that bypasses the access restrictions of the joint controller by deploying a ROS2 forwarding node on the first controller and configuring its forwarding topic with the same QoS configuration as the joint state data topic of the second controller, thereby enabling the third-party controller to legally obtain the native joint state data, as detailed below.

[0101] Create and initialize a ROS2 forwarding node on the first controller, and name it "joint_state_forwarder" to ensure uniqueness.

[0102] Configure the QoS parameters for the forwarding topic. This configuration must be completely consistent with the joint status data topic of the second controller to ensure data transmission reliability. For example: the reliability policy can be BEST_EFFORT or RELIABLE; the persistence policy can be VOLATILE; the history policy can be KEEP_LAST. The history message depth can be determined according to the frame rate of the acquisition hardware, for example, it can be set to 10-1000.

[0103] Based on the QoS parameters mentioned above, create two publishers, corresponding to the forwarding topics for arm and finger joint status respectively. The topic names can be set to " / transfer_arm_joint_status" and " / transfer_hand_joint_status".

[0104] At the same time, create two subscribers to subscribe to the raw arm and finger joint state data topics output by the second controller (the topic name needs to be adjusted according to the actual topic name of the robot, such as " / motion / control / arm_joint_state"), and bind the corresponding callback functions.

[0105] The core function of the callback function bound to the subscriber of the joint status data is to receive the joint status data and forward it directly to the forwarding topic through the corresponding publisher. The callback function bound to the publisher of the forwarding topic also includes outputting the number of subscribers (i.e., the number of nodes subscribed to the forwarding topic). For example, by setting an optional log debugging function in the forwarding node, the subscriber count information is output once every 100 messages forwarded.

[0106] The main function of the forwarding node implements complete ROS2 lifecycle management, including: ROS2 initialization, node instance creation, node spin operation and resource release.

[0107] Once activated, the forwarding node can automatically monitor topics and forward data without manual intervention, ensuring the real-time and completeness of joint status data.

[0108] S210: Deploy camera nodes on the first controller and publish camera image topics.

[0109] The camera node can be deployed on the first controller or an external controller. This application describes the deployment on the first controller as an example.

[0110] The camera publishes RGB video streams via camera image topics. Its QoS configuration is as follows: the reliability policy is RELIABLE; the persistence policy is TRANSIENT_LOCAL; the history policy is KEEP_LAST; and the history message depth is determined based on the number of video stream frames, for example, it can be set to 150 to ensure that the third-party controller node can immediately obtain the latest image frames after subscribing.

[0111] S220: Deploy multi-source data synchronization acquisition nodes on a third-party controller, and subscribe to forwarding topics and camera image topics.

[0112] Since the third-party controller needs to communicate with the robot's first controller via ROS2 topics, the ROS domain identifier (ROS_DOMAIN_ID) and network configuration of the two need to be unified, as follows.

[0113] To find the IP address and hostname of the robot's first controller: Execute `ifconfig` in the robot's first controller terminal to obtain the IP address (e.g., 192.168.1.100), and execute `hostname` to obtain the hostname (e.g., humanoid-orin).

[0114] Configure temporary communication parameters in the terminal of the third-party controller. For permanent effects, these parameters can be written to the user configuration file. The temporary communication parameters to be configured include the ROS domain identifier (ROS_DOMAIN_ID), which must be completely consistent with the configuration of the robot's first controller; the hostname of the third-party controller (ROS_HOSTNAME); and the ROS master node address of the robot's first controller (ROS_MASTER_URI), in the format "http: / / [first controller IP address]:11311", where the IP address needs to be replaced with the actual IP address of the robot's first controller obtained from the query.

[0115] Deploy multi-source data synchronization acquisition nodes on a third-party controller. These nodes create forwarding topic subscribers and camera image topic subscribers. The QoS configuration of each subscriber is consistent with its corresponding topic, and the corresponding callback function is associated with it.

[0116] The core function of the multi-source data synchronization acquisition node is to synchronously record, cache, and store joint state data and RGB image frames. Before actual data acquisition, the multi-source data synchronization acquisition node needs to be configured.

[0117] First, initialize the ROS2 node, and name the node "joint_image_sync_recorder" to ensure uniqueness.

[0118] Configure the core parameters, including the following: The list of topics to subscribe to includes forwarding topics, namely joint status data topics, and camera RGB video image topics such as " / aima / hal / rgbd_camera / head_front / color"; The total recording time can be adjusted according to the complexity of the action. Simple single-joint actions can be set to 10-15 seconds, while complex multi-joint coordinated actions (such as grasping-placing) can be set to 30-60 seconds. The video output frame rate needs to match the actual frame rate of the camera. In this embodiment, the camera frame rate is 30 FPS, and the output frame rate is set to 15 FPS to balance the amount of data and the clarity. Image flip mode: If you need to adjust the image flip mode (such as when the camera is mounted upside down), you can modify the flip mode parameter in the node configuration. It supports three modes: horizontal flip, vertical flip, and bidirectional flip.

[0119] The multi-source data synchronous acquisition node subscribes to forwarding topics and camera image topics by creating multiple subscribers; the QoS configuration of the subscriber is consistent with its corresponding topic and associated with the corresponding callback function.

[0120] The callback function bound to the subscriber of the forwarding topic can include at least one of the following functions: caching joint status data and outputting cached status logs.

[0121] Caching joint state data may include setting up a joint data cache container (joint_cache) to cache the joint state data received by subscribing to the forwarding node topic before the first frame of the video stream arrives at the third-party controller (i.e. before recording begins).

[0122] The output buffer status log may include: outputting a log prompt every 50 buffered data entries before the first frame of the video stream arrives at the third-party controller (i.e. before recording begins).

[0123] The callback function for camera image topic subscribers is the core of synchronous acquisition. It triggers the recording (synchronous recording) initialization process, including creating a start timestamp subdirectory, generating a video output path, supplementing cached joint state data, and starting timing. Within the recording duration and before reaching the maximum frame rate, ROS image messages are converted to OpenCV format (BGR color space), flipped according to configuration, and stored in the image frame list (frame_list). Recording progress is output every time a number of frames matching the output frame rate are acquired. When the recording duration or maximum frame rate is reached, the data storage process is triggered, recording is stopped, and ROS2 node resources are released.

[0124] The image frame list is used to store the image frames published through the image topic video stream. Since the message format published by the image topic is ROS2 image message, and the visual image processing of the training dataset mostly uses OpenCV format, the ROS image messages are converted into OpenCV format image frames by initializing the image conversion tool (CvBridge) and then stored in the image frame list.

[0125] Key status data is stored in a joint data storage container (joint_record_data) set by the multi-source data synchronization acquisition node. The joint status data recorded synchronously after the start of recording is stored by topic, including: joint name, timestamp, joint position, joint velocity, and joint torque.

[0126] S230: Set up a data collection scenario so that the robot can perform actions according to a preset trajectory within the scenario.

[0127] Before data collection, the environment needs to be prepared. The collection scene should be set up according to the VLA training requirements, such as placing the target object (such as a cube or cylinder) within the robot's operating range, and adjusting the camera installation angle to ensure that the robot's operating area and the target object can be captured completely, avoiding strong light direct rays or reflective areas in the scene that may affect image quality.

[0128] The data collection process employs a "remote control + automatic data collection" approach, with the following specific steps: First, remotely log in to the robot's first controller via the debugging terminal, start the robot joint driver program, and ensure that the robot is in a movable state.

[0129] Then log in to the third-party controller, start the multi-source data synchronous acquisition node, and confirm that the node enters the "waiting for the first frame image" state.

[0130] Send action commands through the robot control terminal or host computer to make the robot perform actions according to the preset trajectory (such as starting from the initial posture and completing the complete process of grasping, lifting, moving, placing and resetting the target object).

[0131] Once the first frame of image arrives at the multi-source data synchronization acquisition node, the node automatically begins synchronous recording. During the recording process, the progress (including recording percentage, number of acquired image frames, and number of joint data entries) can be viewed in real time through the node log.

[0132] After the action is completed, wait for the multi-source data synchronous acquisition nodes to reach the preset recording time, then automatically stop acquisition and save the data. When the log outputs the message "All data has been saved", the acquisition execution ends.

[0133] Before performing formal data collection, the forwarding nodes and multi-source data synchronous collection nodes can be debugged and verified. The debugging and verification of the forwarding node consists of two steps: First, verify the topic creation status by executing the ROS2 topic list query command in the third-party controller terminal and filtering the results that contain the forwarding topic identifier (such as " / transfer_arm_joint_status"). If the forwarding topics for arm and finger joints are displayed, it means that the topic was created successfully. Second, verify the data forwarding status by executing the ROS2 topic echo command to subscribe to the forwarded arm joint status topic. If the joint name (name field), joint position (position field), joint velocity (velocity field), and joint torque (effort field) are output normally, it proves that the forwarding function is normal.

[0134] The debugging and verification of multi-source data synchronous acquisition nodes are divided into two steps: functional verification and synchronization verification.

[0135] The functional verification phase includes: executing the ROS2 node list query command on the third-party controller terminal to confirm that the "joint_image_sync_recorder" node has started normally; then executing the topic list query command to confirm that the node has successfully subscribed to the forwarded joint topic and camera image topic, with no subscription failure prompts.

[0136] Synchronization Verification Phase: Manually trigger the robot to perform a simple action (such as an arm raising-lowering cycle). After the multi-source data synchronization acquisition nodes complete the preset duration (e.g., 15 seconds) of acquisition, check the generated files in the timestamp subdirectory. First, check the file integrity to confirm the existence of arm joint state data text files, finger joint state data text files, and video stream files, with no missing or corrupted files. Second, verify the data volume matching by counting the number of data entries in the joint state data text files and the number of image frames in the video files, ensuring that the two are of the same order of magnitude in the time dimension (e.g., for a 15-second acquisition duration, a 15FPS video corresponds to 225 frames, and the number of joint state data entries should be close to this value). Finally, verify the timeline synchronization by selecting key action frames in the video (e.g., the arm raised to its highest point), recording its approximate time point, and searching for the corresponding timestamp joint position data in the joint state data text files to confirm that the data value matches the actual action state (e.g., the arm joint position value is near the maximum stroke). If all verification items are satisfied, the synchronization acquisition function is normal.

[0137] S240: The forwarding node receives joint status data and publishes it to the forwarding topic. The multi-source data synchronous acquisition node receives joint status data from the forwarding topic and video stream from the camera image topic, and starts recording.

[0138] Before the first frame of the video stream arrives at the third-party controller, the joint state data before the arrival of the image frame is stored in a joint data cache container to avoid data loss.

[0139] When the first frame of the video stream arrives at the third-party controller, the multi-source data synchronization acquisition node controls the recording start timing through the "image first frame arrival marker" (image_first_received) to ensure that synchronous recording starts based on the arrival of image data. The joint state data received by the synchronization recording is stored as a text file, and the image frames of the video stream from the camera image topic are converted and stored in the image frame list. The image frame list is then encoded into an MP4 format video file according to the configured output frame rate through the video writing interface of the computer vision library (OpenCV) and stored in the timestamp subdirectory.

[0140] The first time it is saved, the joint name is also recorded and the number of joints is output.

[0141] The storage path adopts a hierarchical structure of "root directory + timestamp subdirectory". The root directory is set to "hdf5_result". Each time a recording is made, a subdirectory is automatically created with the current timestamp (accurate to milliseconds) to avoid data overwriting.

[0142] When the first frame of the video stream arrives at the third-party controller, the recording (synchronous recording) initialization process is triggered through the callback function of the camera image topic subscriber. This includes: creating a start timestamp subdirectory, generating a video output path, supplementing cached joint state data, and starting the timer. Within the recording duration and before reaching the maximum frame count, the ROS image message is converted to OpenCV format (BGR color space), flipped according to the configuration, and stored in the image frame list. The recording progress is output once every time the number of frames matching the output frame rate is acquired. When the recording duration or the maximum frame count is reached, the data storage process is triggered, the recording state is closed, and the ROS2 node resources are released.

[0143] S250: Based on the timeline of the video stream recorded by a third-party controller, the joint state data is time-aligned, and the current action label and future action label are associated with each frame.

[0144] The multi-source data synchronous acquisition node stores joint status data by topic / joint type, such as arm and fingers. The file header contains recording parameter descriptions, such as joint name, recording duration, amount of joint status data, and number of joints. Data rows are organized in the format of "timestamp + joint status data (joint position, joint velocity, and joint torque)".

[0145] Parse the joint data text file, extract the timestamps and joint position, joint velocity, and joint torque data for each joint, and convert them into a numerical DataFrame.

[0146] Extract all image frames and their corresponding timestamps from the video file. Using the arrival timestamp of the first image frame as a reference, convert the relative timestamps to the absolute timestamps of the third-party controller to align the timestamps of the data rows with the timestamps of the image frames, thereby achieving minimum error matching between the image frames and the joint state data.

[0147] When supplementing cached joint state data, the same relative time conversion is performed to ensure that the cached joint state data is aligned with the timeline of the image frame.

[0148] Before associating current action labels and future action labels with each image frame, the joint state data and image frames need to be preprocessed. The core of preprocessing is to remove invalid data, unify the timeline, and standardize the data format, which is divided into two parts: joint state data preprocessing and image frame preprocessing.

[0149] Joint data preprocessing includes: First, read the joint text file and, based on the joint name information in the file header, remove invalid data rows (such as rows with missing data or abnormal values; the criterion for judging abnormal values ​​can be that the position value exceeds the joint range of motion by ±5%). Subsequently, based on the timeline of the video stream, the joint state data is resampled using linear interpolation to ensure that the sampling frequency of the joint state data is consistent with the video frame rate, thus ensuring that each frame corresponds to a set of joint data. Finally, the joint position, joint velocity, and joint torque data were standardized. The min-max normalization method was used to map the data to the [0,1] interval to eliminate the influence of differences in the magnitude of data from different joints.

[0150] Image frame preprocessing includes: The computer vision library is used to read the MP4 video file, extract the image frame by frame and convert it to the RGB color space (the original camera capture format is BGR). Each frame of the image is normalized to a size of 640×480 pixels, and bilinear interpolation is used to ensure image details. Depending on the requirements of the acquisition scenario, image enhancement operations can be performed, such as brightness equalization, contrast adjustment, and noise filtering, to enhance the feature recognition of the image. Finally, the image frames are organized in the format of "image frame number - image frame data" to establish a frame-level correspondence with the joint state data.

[0151] S260: For each recorded image frame, obtain the correlation data between the current joint state and the future joint state, as well as natural language commands.

[0152] For each preprocessed image frame, obtain the correlation data between the current joint state and the future joint state, specifically: Based on the timestamp t of each preprocessed image frame, key state data corresponding to the current action joint state data, i.e., the "action" field, is extracted from the synchronously recorded joint state data; simultaneously, the timestamp of the next time step image frame is extracted. The corresponding joint state serves as the joint state data for future actions, i.e., the "next_action" field. This represents the frame interval of the video stream.

[0153] For action prediction tasks, current action data and future action data must be consecutive samples in adjacent time sequences. For example, visual data in frame t corresponds to action data at time t, and visual data in frame t+1 corresponds to action data at time t+1, forming an action transition pair of "t→t+1". Otherwise, the model will learn incorrect action planning logic.

[0154] For each frame of the training data, a natural language description (NLP) instruction is obtained. The semantics of the NLP instruction are strongly correlated with the video file and joint state data. For example, the action is "grab the red cube on the desktop and place it on the left platform." The "red cube" in the instruction must be clearly identifiable in the visual file, and the task flow of "grab → place" in the instruction must completely match the temporal logic of "joint extension → clamping → moving → releasing" composed of multiple frames of images. Each frame of image corresponds to one step, such as "joint extension," to avoid model misunderstanding due to semantic and visual / action mismatch.

[0155] The frame images, actions, next_actions, and episode index information are encapsulated into HDF5 format training data to ensure that the data structure meets the input requirements of the VLA model. This format supports efficient reading and writing of large volumes of data and is compatible with mainstream deep learning frameworks (such as TensorFlow and PyTorch). The dataset structure is hierarchically organized, with three first-level subgroups under the root node: "joint_data", "image_data", and "metadata". The functions and data organization of each subgroup are as follows: The `joint_data` group contains "arm" (arm joint subgroup) and "hand" (finger joint subgroup), with each subgroup storing standardized data for the corresponding joint. Taking the arm joint subgroup as an example, it contains four datasets: "joint_names" (string type, storing a list of joint names), "position" (floating-point array with dimensions [number of frames × number of joints], storing normalized position data), "velocity" (floating-point array with the same dimensions as "position", storing normalized velocity data), and "effort" (floating-point array with the same dimensions as "position", storing normalized torque data).

[0156] The image_data group contains two datasets: "data" (a floating-point array with dimensions [number of frames × height × width × number of channels], where height = 480, width = 640, and number of channels = 3, storing normalized RGB image data) and "frame_index" (an integer array storing frame numbers). The image data must first be converted to floating-point and normalized to the [0,1] range.

[0157] The metadata group stores the metadata of the dataset, including acquisition time (string type), recording duration (floating-point type), frame rate (integer type), number of joints (integer type), image size (integer array, storing [height, width]) and preprocessing parameters (string type, storing normalization method, interpolation method, etc.), which facilitates the reuse and traceability of the dataset in the future.

[0158] The training samples obtained by the above method include frame images, natural language instructions, current action, and future action.

[0159] Before training, the training dataset containing multiple training samples is divided into training, validation, and test sets according to a 7:2:1 episode independence principle. Each episode is treated as an independent and complete task execution unit, and data from different episodes should not be contiguous in time or dependent on each other.

[0160] Before training, further visual enhancements (random flipping, brightness adjustment, etc.), action enhancements (Gaussian noise perturbation), and language enhancements (synonym replacement) can be performed on the dataset to improve generalization ability.

[0161] S270: Use each frame of image and its corresponding natural language instruction as input to the multimodal neural network model, and use the corresponding associated data as the supervised output target for training to obtain the VLA model.

[0162] The embodiments of this application adopt a "multimodal feature extraction-cross-modal fusion-dual task output" architecture to construct a multimodal neural network model.

[0163] Specifically, visual features are extracted by the EfficientNet-B3 model: the input image is preprocessed and then fed into the visual encoder, which outputs a high-dimensional semantic feature map, and then obtains a fixed-dimensional visual representation vector through global average pooling. Language features are extracted by the RoBERTa-base model: natural language instructions are segmented and positionally encoded and then input into the language encoder. The output of the [CLS] position is taken as the semantic vector of the whole sentence, and then projected to the same dimension as the visual features through a fully connected layer.

[0164] Subsequently, the aforementioned visual and linguistic features are input into the fusion module, where cross-modal alignment fusion is achieved through a cross-attention and adaptive weighting mechanism. First, using linguistic features as the query, attention-weighted visual features are applied to generate language-guided enhanced visual representations; simultaneously, using visual features as the query, attention-weighted linguistic features are applied to generate visual-guided enhanced linguistic representations. Next, the two enhanced representations are pooled separately to obtain visual and linguistic vectors, which are then concatenated and fed into a lightweight gating network. This network dynamically calculates the fusion weights and outputs the weighted joint features. This mechanism adaptively balances the contributions of visual and linguistic information based on instruction semantics and scene content, improving cross-modal alignment accuracy.

[0165] Based on the fused joint features, the model predicts the current joint state data and future joint state data through two independent fully connected regression heads. During training, a weighted joint loss function is used to optimize the model parameters, which comprehensively considers prediction accuracy and robustness. For example, the loss function L could be:

[0166] in, , The loss weights for the current action and future actions; This is the balance coefficient between MSE loss and smoothing L1 loss; , These are the predicted values ​​for the current action and future actions. , This refers to the actual values ​​of the current action and the future action (i.e., the associated data that serves as the target of the supervision output).

[0167] The training is carried out in three stages: first, unsupervised pre-training is used to explore the intrinsic correlation of modalities; then, supervised fine-tuning is performed through weighted joint loss (MSE + smooth L1); and finally, temporal consistency loss is added to optimize the smoothness of actions.

[0168] During training, Dropout, early stopping, and L2 regularization are used to suppress overfitting. Dynamic weight adjustment and LayerNorm are used to ensure modal balance. At the same time, the model is pruned, quantized, and converted to ONNX format to adapt to real-time requirements, ensuring that the single-frame inference time is ≤30ms.

[0169] The evaluation focuses on quantitative indicators (Joint State MAE / RMSE, Temporal Consistency AD, Task Success Rate) and qualitative standards (Modal Semantic Consistency, Robot Movement Fluency). It requires that the position MAE ≤ 0.02, the future action prediction MAE ≤ 0.03, and the task success rate ≥ 85%. After meeting the standards, the optimal model will be deployed to the humanoid robot third-party controller, adapted to the ROS2 inference environment, and realize precise action execution driven by language commands.

[0170] This application provides a method for generating a VLA model for a robot based on a third-party controller.

[0171] The VLA model trained through the embodiments of this application is based on the "image frame-current action-future action" association data, which enables each frame of the video stream to be accurately matched with the corresponding joint state data and provides joint state prediction data for future moments.

[0172] By training with dual labels for "current action" and "next action," the model can predict the joint state of the (humanoid) robot's VLA model at the next moment, which can improve the smoothness of action execution. This clarifies the temporal relationship logic of actions, avoids abrupt switching between actions such as grasping, moving, and placing, and makes the output continuous action sequence transition smoothly, which is consistent with the mechanical motion characteristics of humanoid robots.

[0173] In scenarios such as moving target objects and dynamic changes in environmental obstacles, the model can combine future actions with current visual environment information to predict the posture effect after the action is executed (such as "whether the current direction of movement can accurately reach the next target position"), thereby optimizing the current decision and forming a globally optimal action strategy, significantly reducing task failures caused by local optimization (such as positional deviation when grasping moving objects, and untimely action adjustment when avoiding dynamic obstacles).

[0174] The robot's joint state data (joint position, joint velocity, joint torque), visual data, and natural language commands are strictly synchronized along the time axis. The core constraint is that "each frame of visual data corresponds to a unique set of joint state data for the current action and a unique semantically matched natural language command".

[0175] Current action data and future action data should be used as consecutive samples in adjacent time sequences (e.g., visual data of frame t corresponds to action data at time t, and visual data of frame t+1 corresponds to action data at time t+1, forming an "t→t+1" action transfer pair) to avoid the model learning incorrect action planning logic.

[0176] Addressing the technical bottlenecks of unidirectional communication between the (humanoid) robot joint controller (second controller) and the first controller, and the limitation of third-party controllers to only acquiring RGB video streams, this paper proposes a three-step core process: joint state data topic forwarding, multi-source data synchronous acquisition, and standardized HDF5 format encapsulation. This process achieves precise alignment between video image frames and joint state data, as well as prediction of future joint state data. This method solves the requirement for a one-to-one correspondence between "image frames and joint state data" and prediction of future joint state data in VLA model training, ensuring the integrity, synchronization, and standardization of data acquisition. It significantly reduces the difficulty of generating robot VLA training data and improves data adaptability and model training efficiency.

[0177] The third embodiment of this application provides an apparatus for generating a VLA model for a robot. This apparatus can be used to implement the method for generating a VLA model for a robot described in the above embodiments, such as... Figure 3 As shown, the apparatus for generating VLA models for robots includes: The data acquisition module is used to acquire video streams and joint state data synchronously recorded during the robot's execution of actions according to a preset trajectory; specifically, the data acquisition module can be used to implement step S10 in the first embodiment and its optional embodiments.

[0178] The data processing module is used to acquire current joint state and future joint state association data, as well as natural language commands, for each frame of the recorded video stream; specifically, the data processing module can be used to implement step S20 in the first embodiment and its optional embodiments.

[0179] The model training module is used to take each frame of image and its corresponding natural language instruction as input to the multimodal neural network model, and use the corresponding associated data as the supervised output target for training to obtain the VLA model; specifically, the model training module can be used to implement step S30 in the first embodiment and its optional embodiments.

[0180] Figure 4 This is a schematic structural diagram of a computing device 900 provided in an embodiment of this application. This computing device can execute various optional embodiments of the methods described above. The computing device can be a terminal, or a chip or chip system within the terminal. Figure 4 As shown, the computing device 900 includes: a processor 910, a memory 920, and a communication interface 930.

[0181] It should be understood that Figure 4 The communication interface 930 in the computing device 900 shown can be used to communicate with other devices, and may specifically include one or more transceiver circuits or interface circuits.

[0182] The processor 910 can be connected to the memory 920. The memory 920 can be used to store the program code and data. Therefore, the memory 920 can be a storage unit inside the processor 910, an external storage unit independent of the processor 910, or a component that includes both the storage unit inside the processor 910 and the external storage unit independent of the processor 910.

[0183] Optionally, the computing device 900 may also include a bus. The memory 920 and communication interface 930 can be connected to the processor 910 via the bus. The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 The symbol is represented by a line without an arrow, but this does not mean that there is only one bus or one type of bus.

[0184] It should be understood that in the embodiments of this application, the processor 910 may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor. Alternatively, the processor 910 may employ one or more integrated circuits to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0185] The memory 920 may include read-only memory and random access memory, and provides instructions and data to the processor 910. A portion of the processor 910 may also include non-volatile random access memory. For example, the processor 910 may also store device type information.

[0186] When the computing device 900 is running, the processor 910 executes computer execution instructions stored in the memory 920 to perform any of the operational steps of the above method and any of the optional embodiments thereof.

[0187] It should be understood that the computing device 900 according to the embodiments of this application can correspond to the corresponding subject in executing the methods according to the various embodiments of this application, and the above and other operations and / or functions of each module in the computing device 900 are respectively for implementing the corresponding processes of the methods of this embodiment. For the sake of brevity, they will not be described in detail here.

[0188] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0189] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0190] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0191] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0192] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0193] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0194] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is used to perform the above-described method, which includes at least one of the schemes described in the above embodiments.

[0195] The computer storage medium in this application embodiment can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. For example, a computer-readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0196] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0197] The program code contained on a computer-readable medium may be transmitted using any suitable medium, including, but not limited to, wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0198] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0199] Furthermore, the terms "first, second, third, etc." or similar terms such as module A, module B, and module C used in the specification and claims are only used to distinguish similar objects and do not represent a specific ordering of objects. It is understood that, where permissible, a specific order or sequence may be interchanged so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0200] In the above description, the labels of the steps involved, such as S110, S120, etc., do not mean that the steps will necessarily be executed. The order of the steps can be interchanged or executed simultaneously if permitted.

[0201] The term "comprising" as used in the specification and claims should not be construed as limiting itself to what follows; it does not exclude other elements or steps. Therefore, it should be interpreted as specifying the presence of the mentioned feature, integral, step, or component, but does not exclude the presence or addition of one or more other features, integrals, steps, or components, or groups thereof. Thus, the statement "device comprising means A and B" should not be limited to a device consisting solely of components A and B.

[0202] The terms "an embodiment" or "an embodiment" as used in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in at least one embodiment of this application. Therefore, the terms "in one embodiment" or "in an embodiment" appearing throughout this specification do not necessarily refer to the same embodiment, but may refer to the same embodiment. Furthermore, in one or more embodiments, the particular features, structures, or characteristics can be combined in any suitable manner, as will be apparent to those skilled in the art from this disclosure.

[0203] Note that the above are merely preferred embodiments and the technical principles employed in this application. Those skilled in the art will understand that this application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of this application, all of which fall within the scope of protection of this application.

Claims

1. A method for generating VLA models for robots, characterized in that, include: Acquire video streams and joint status data synchronously recorded during the robot's execution of actions according to a preset trajectory; For each frame of the recorded video stream, obtain the correlation data between the current joint state and the future joint state, as well as natural language instructions; Each frame of image and its corresponding natural language instruction are used as input to the multimodal neural network model, and the corresponding associated data are used as the supervised output target for training to obtain the VLA model.

2. The method according to claim 1, characterized in that, The associated data consists of the current motion joint state data corresponding to each frame of the video stream and the future motion joint state data corresponding to the next time step image frame. The semantics of the natural language instruction must be strongly correlated with the visual scene of the video stream and the action sequence of the executed action; each frame of the image corresponds to a unique set of current action joint state data and a unique semantically matched natural language instruction.

3. The method according to claim 1, characterized in that, The training includes: Construct a multimodal neural network model, which includes a visual encoder, a language encoder, and a fusion module; Each image frame is input into the visual encoder, and the natural language instructions are input into the language encoder; the images are then fused through the fusion module to generate fused features. Based on the fusion features, the current motion joint state data and future motion joint state data are regressed and predicted respectively through dual task output heads; The model is trained end-to-end using a loss function, which includes a current action prediction loss term and a future action prediction loss term.

4. The method according to claim 1, characterized in that, The joint status data includes at least two of the following: joint position, joint velocity, and joint torque.

5. The method according to claim 1, characterized in that, The robot is equipped with a first controller, a second controller, a third-party controller, and a camera; the acquisition of video streams and joint state data synchronously recorded during the robot's execution of actions according to a preset trajectory includes: The video stream is acquired through a camera, and joint state data during the execution of the action is acquired through the second controller; By deploying ROS2 forwarding nodes on the first controller, the joint status data topic from the second controller is subscribed to, and the joint status data is redistributed to a forwarding topic accessible to the third-party controller; wherein, the forwarding topic has the same QoS configuration as the joint status data topic; The third-party controller subscribes to and synchronously records the joint status data of the forwarded topic and the video stream from the camera image topic; The recorded joint state data is time-aligned based on the timestamp of each frame in the recorded video stream.

6. The method according to claim 5, characterized in that, Also includes: The third-party controller has a joint data cache container for caching the joint state data received by subscribing to the forwarding topic before the first frame of the video stream arrives at the third-party controller.

7. The method according to claim 6, characterized in that, Also includes: When the first frame of the video stream arrives at the third-party controller, the joint state data cached in the joint data cache container is supplemented. The timestamps of the cached joint state data are converted based on the timestamps of the image frames of the video stream recorded by the third-party controller; Align the cached joint state data with the image frames of the video stream.

8. An apparatus for generating VLA models for robots, characterized in that, include: The data acquisition module is used to acquire video streams and joint status data that are recorded synchronously during the robot's execution of actions according to a preset trajectory; The data processing module is used to obtain the current joint state and future joint state association data, as well as natural language commands, for each frame of the recorded video stream. The model training module is used to train the VLA model by taking each frame of image and its corresponding natural language instruction as input to the multimodal neural network model and taking the corresponding associated data as the supervised output target.

9. A computing device, characterized in that, include: processor, and A memory having stored program instructions that, when executed by the processor, cause the processor to perform the method for generating a VLA model for a robot as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores program instructions that, when executed by a computer, cause the computer to perform the method for generating a VLA model for a robot as described in any one of claims 1 to 7.

Images