A humanoid robot motion generation method, system, device and storage medium

Abstract

The present application relates to the technical field of robots, and in particular to a motion generation method, system, device and storage medium for a humanoid robot, comprising: physical simulation rendering generates a visual video containing visual representation of joint motion state of the robot; analyzing the visual video to obtain frame images, natural language instructions and motion sequences, decoding output trajectory data; and outputting control instructions through a hierarchical architecture of high-level motion planning and bottom-level dynamics tracking to drive the robot to perform actions. The present application implicitly encodes the joint state of the humanoid robot in the visual features, improves the inference efficiency and generalization ability of the model, and the hierarchical decoupling architecture of the large model high-level trajectory planning and the reinforcement learning bottom-level dynamics tracking controller pays a small performance cost, which can compress the inference time delay to within 50ms and the parameter quantity to below 2 billion, and balances the high generalization of the stateless VLA and the small model, few-step inference, edge low latency, and low power requirement.

Description

Technical Field

[0001] This invention relates to the field of robotics, and in particular to a method, system, device, and storage medium for generating motions in a humanoid robot. Background Technology

[0002] Vision-Language-Action (VLA) is the core technological paradigm for decision-making and control of humanoid robots. Its core function is to convert the robot's visual observations and language understanding into executable action commands. With the deepening of embodied intelligence research, how to enable robots to understand natural language commands and produce coordinated, natural full-body movements like humans has become a focal point of common concern in academia and industry.

[0003] Existing VLA models typically follow a multimodal input architecture design, treating the visual modality (perceiving the environment), the language modality (understanding instructions), the state modality (e.g., the robot's own joint angles, angular velocities, end-effector poses, and other bodily states), and the action modality (outputting control signals) as four independent input channels.

[0004] For example, some research uses the Transformer architecture to simultaneously process visual features, text embeddings, and robot state vectors, achieving robot control through multimodal fusion. However, the four-modal VLA model has a large number of parameters, increasing the complexity of cross-modal alignment and resulting in high inference latency, making it difficult to meet the millisecond-level real-time response requirements of humanoid robots.

[0005] Meanwhile, the acquisition of state modes requires high-precision joint encoders or inertial measurement units. Different models and configurations of humanoid robots have different numbers of joint degrees of freedom, range of motion and dynamic parameters, resulting in inconsistencies in the dimension and semantics of the state space. This can easily introduce sensor noise and domain differences during the transfer from simulation to reality, which limits the generalization ability of the model. Independent state modes become the biggest obstacle to adaptation. Usually, it is necessary to re-acquire data or fine-tune the model for the target robot, which reduces the actual deployment effect of the model. Furthermore, due to the large computational load of fusing multimodal inputs, coupled with the additional processing overhead brought by state modes, the inference latency of existing mainstream VLA models is usually above 80ms, while the motion control frequency requirement of humanoid robots is usually 20-50Hz (i.e., 20-50ms response period). There is a significant gap between the response speed of existing models and the real-time requirements of human-computer interaction.

[0006] To address the problems in the existing technology, this invention provides a method, system, device, and storage medium for generating motion in a humanoid robot. Summary of the Invention

[0007] The purpose of this invention is to provide a method, system, device, and storage medium for generating motion in a humanoid robot, in order to solve the technical problems that still exist in the existing technology of improving generalization through stateless VLA, which has high low-edge computing power threshold and high inference latency, and cannot be taken into account at the low-end edge.

[0008] The technical solution of this invention is: a method for generating motion in a humanoid robot, comprising: In the simulation environment, the humanoid robot's movements are driven by the first action command data, and a visual video containing multiple frames of joint motion states is rendered and generated; the visual video is decoded into a frame image sequence from a third-person perspective, and natural language commands and action sequences are obtained simultaneously and split into triples according to the period. A multi-layer structure model is constructed, including a visual language model and a conditional diffusion transformer. The triplet data is used as input, and the robot-executable control instructions are output. The multi-layer structure model is iteratively trained. A real-time mapping relationship between the real machine and the simulator in the simulation environment is established. Input data pairs of the simulator are obtained according to the mapping relationship, including the frame image sequence and natural language instructions. The input data pairs are input into the multi-layer structure model that has been trained. The output robot-executable control instructions are sent to the real machine drive controller to deploy the simulation to the real machine and control the execution of the real robot's actions.

[0009] Preferably, visual language fusion features are extracted through the visual language model, and the temporal change information of the frame image sequence is captured through the self-attention mechanism of the visual language model, thereby encoding the joint motion state into visual features; The intermediate layer of the visual language model obtains visual language fusion conditional features through projection compression. Guided by the fusion conditional features, the conditional diffusion transformer introduces joint degree of freedom constraints and motion smoothing constraints during the denoising process, and generates control instructions that can be executed by the robot by denoising and decoding using multi-step ordinary differential equations. The real robot drive controller is set as a reinforcement learning controller and is equipped with a residual network. The residual network freezes the multi-layer structure model, performs residual network learning, and outputs residual correction amount based on the joint motion state output by the body sensor in real time and the joint motion state of the simulator, and then corrects and outputs the final control command. The control command is sent to the servo driver of the real robot to control the real robot body to execute the action command.

[0010] Preferably, the real-time mapping relationship between the real machine and the simulator in the simulation environment is set as a spatial state mapping. Through the URDF model unification and joint zero-point calibration, a real-time bidirectional mapping table between the joint motion state of the real machine and the joint motion state in the simulation environment is established, including joint angle, deceleration ratio and offset calibration.

[0011] Preferably, the visual language model uses a pre-trained Qwen3-VL-2B model as the visual language backbone network. The data of the triples are input into the Qwen3-VL-2B model, all parameters of the visual encoder ViT in the Qwen3-VL-2B model are frozen, the parameters of the top 4 Transformer layers are adjusted, and visual language fusion features with a dimension of 2048 are extracted from the 16th Transformer layer. The visual language fusion features are linearly normalized based on the VLLN projection module, which includes a linear transformation layer and a normalization layer, compressing the 2048 dimensions to 1536 dimensions to obtain the visual language fusion conditional features. Among them, the visual changes between consecutive frames of the frame image sequence are implicitly encoded as joint velocity direction and temporal motion information through the self-attention mechanism of the Transformer.

[0012] Preferably, the conditional diffusion transformer employs an AlternateVLDiT alternating attention diffusion transformer with 32 alternating attention layers. Even-numbered layers perform self-attention of action sequences to model temporal coherence, while odd-numbered layers perform cross-attention using the fused conditional features as the key and value, thereby aligning action generation with visual language conditions. Preferably, the reinforcement learning controller employs a Teacher-Student architecture combined with a proximal policy optimization PPO algorithm for training.

[0013] Preferably, the simulation environment uses the Mujoco physics simulation engine, and the rendered visualization video has a resolution of 720P and a frame rate of 20fps.

[0014] A humanoid robot motion generation system, used to implement the aforementioned humanoid robot motion generation method, includes: The simulation data construction module is used to render and generate visual videos containing visual representations of robot joint motion states in a physical simulation environment. The training dataset generation module is used to decode the visualized video to generate a sequence of frame images, parse it to obtain natural language instructions and action sequences, and the sequence of frame images, natural language instructions and action sequences are used to construct a training dataset; The visual language feature extraction module has a built-in pre-trained visual language model, configured to freeze the ViT parameters and fine-tune the parameters of the top 4 layers, and extracts 1536-dimensional visual language fusion features from the 16th layer. The diffusion motion generation module has a built-in alternating attention diffusion transformer, which generates motion sequences by solving multi-step ordinary differential equations under the conditions of joint degrees of freedom and smoothness constraints. The hierarchical control module includes a high-level planner that outputs pure kinematic trajectories at a first frequency, a low-level tracker that runs a reinforcement learning controller at a second frequency, and a servo driver that drives the execution of the real machine. The output of the simulation data construction module is connected to the visual language feature extraction module, the output of the visual language feature extraction module is connected to the diffusion action generation module, and the output of the diffusion action generation module is connected to the high-level planner of the hierarchical control module. The underlying tracker learns data from the higher-level planner and outputs the final control commands to the servo driver, driving the actual robot to perform actions.

[0015] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the aforementioned method for generating motions of a humanoid robot.

[0016] A computer-readable storage medium storing at least one instruction or at least one program, characterized in that, when the at least one instruction or the at least one program is loaded and executed by a processor, it implements the method for generating motions of a humanoid robot.

[0017] Compared with the prior art, the advantages of the present invention are: This invention captures temporal change information between consecutive frames of images through the self-attention mechanism of a visual language model, automatically aggregating and calculating the velocity direction and temporal motion trend information of robot joints. It implicitly encodes the robot joint states into visual features, achieving implicit internalization of state data, removing redundant state modal inputs, reducing model computational complexity, accelerating model convergence, and achieving an end-to-end response cycle of less than 50 milliseconds from natural language commands to full-body robot movements. This invention not only maintains the advantages of lightweight design and low R&D costs, but also further improves the model's inference efficiency and generalization ability through modal simplification, avoiding information redundancy and inference latency problems caused by independent state modalities.

[0018] In the training and actual operation of the real robot, this invention directly uses the training dataset obtained from the simulation environment, eliminating the need for additional vision hardware. The output frame rate of the visual simulation data is customizable via software, supporting arbitrary adjustment and stable output without image transmission delay or frame loss issues, and is unaffected by hardware sampling frequency or environmental conditions. In the case of mass robot deployment, this not only saves on the hardware cost of visual information sampling but also eliminates the installation, calibration, and adjustment steps for visual acquisition. While ensuring a high frame rate data supply, it simplifies the hardware deployment process and improves data consistency and deployment efficiency from training to production application.

[0019] This invention employs a layered, decoupled architecture of a large-scale motion model for high-level trajectory planning and a low-level tracking controller. The upper-level motion model outputs only a pure kinematic real-time reference trajectory. This module does not incorporate dynamic constraints, robot body sensing observations, or participate in reinforcement learning iterative optimization; it is solely responsible for outputting high-level semantic motion targets. Based on this characteristic, this layer eliminates robot body state modal inputs, thereby ensuring low-latency real-time performance and sequential continuity of the high-level motion generation chain. The low-level controller operates at a high frequency of 50Hz, receiving the high-level reference trajectory and real-time body perception state, and converting the output joint target positions into motor execution commands, forming a high-frequency closed-loop control. This layered architecture ensures both the semantic richness and low latency of high-level motion generation and the dynamic adaptability and robustness of low-level execution.

[0020] In the deployment scenario of humanoid robot vehicle-mounted edge computing platform (16GB of video memory), with a performance cost of less than 2 percentage points, it is possible to compress inference latency to less than 50ms and parameter count to less than 2 billion, while taking into account the high generalization of the ontology of stateless VLA and the requirements of small model, few-step inference, low edge latency, and low computing power.

[0021] Visual modal data is generated in a physical simulation environment, eliminating the need for extensive data collection from real robots. Combined with an automated video data conversion engine and semantic annotation technology, a standardized visual-language-action dataset can be constructed on a large scale at low cost. Furthermore, since there are no state modalities in the model bound to specific robot configurations, only the simulated visual data needs to be replaced with rendered videos of the target robot in the simulation environment to achieve zero-fine-tuning transfer of the model across ontology or only minor fine-tuning with a small amount of adaptation data. This fundamentally simplifies the model and eliminates information redundancy and cross-ontology transfer barriers caused by state modalities. Attached Figure Description

[0022] The present invention will be further described below with reference to the accompanying drawings and embodiments: Figure 1 A flowchart illustrating a method for generating motion in a humanoid robot provided by the present invention; Figure 2 A schematic diagram of the framework of a motion generation system for a humanoid robot provided by the present invention; Figure 3 This is a schematic diagram of the hierarchical control architecture provided by the present invention, illustrating the collaborative workflow between the high-level motion planner and the low-level reinforcement learning controller. Figure 4 Flowchart for constructing the input training dataset for the humanoid robot during operation provided by this invention; Figure 5 This invention provides a complete conversion process from raw video data to adaptive robot trimodal data; Figure 6 The flowchart for feature extraction and fusion of the visual-language large model provided by this invention; Figure 7 The present invention provides a flowchart of the four-step ordinary differential equation reasoning and denoising process. Figure 8 A schematic diagram of the internal hierarchical structure of the alternating attention diffusion converter provided by the present invention; Figure 9 This is a schematic diagram comparing the task completion rate of the embodiments of the present invention with existing common models. Figure 10 This is a schematic diagram illustrating the movements of a humanoid robot in the Mujoco simulation environment provided by the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0024] This invention provides a method for generating motion in a humanoid robot. This method is based on a humanoid robot motion generation system, as shown in the attached diagram. Figure 2 The system includes a simulation data construction module, a training dataset generation module, a visual language feature extraction module, a diffusion action generation module, and a hierarchical control module. The output of the simulation data construction module is connected to the visual language feature extraction module, the output of the visual language feature extraction module is connected to the diffusion action generation module, and the output of the diffusion action generation module is connected to the high-level planner of the hierarchical control module.

[0025] The simulation data construction module is used to render and generate a visual video containing visual representations of the robot's joint motion states in a physical simulation environment. The training dataset generation module decodes the visual video to generate frame image sequences, parses them to obtain natural language commands and action sequences, and constructs the training dataset from these sequences. The visual language feature extraction module has a built-in pre-trained visual language model configured to freeze ViT parameters and fine-tune the parameters of the top four layers, extracting 1536-dimensional visual language fusion features from the 16th layer. The diffusion action generation module has a built-in alternating attention diffusion transformer that generates action sequences by solving multi-step ordinary differential equations under joint degrees of freedom and smoothness constraints. The hierarchical control module includes a high-level planner that outputs pure kinematic trajectories at a first frequency and a low-level tracker that runs a reinforcement learning controller at a second frequency. (See Appendix) Figure 1 The method includes the following steps: I. Construction of cross-modal datasets.

[0026] The process of creating this dataset is as follows: Figure 5 As shown, a cross-modal (language-vision-action) dataset was constructed through multiple steps of data acquisition, format conversion, simulation verification, and semantic parsing. The specific steps are as follows: (1) Acquisition of raw video data.

[0027] Using distributed web crawling technology, publicly available video resources from across the internet are crawled as raw data, covering human movement videos in various scenarios such as daily life services, industrial operations, sports and fitness, and etiquette. Invalid videos with blurry images, motion occlusion rates exceeding 30%, and no clear human motion are filtered out. Finally, valid video segments are retained to construct the original video dataset. At the same time, metadata such as the source, shooting angle, and motion category of each video is recorded to ensure the diversity and usability of the raw data.

[0028] (2) Structured extraction of human body movements.

[0029] Using GVHMR (World-Grounded Human Motion Recovery via Gravity-ViewCoordinates) technology, the original video dataset is processed frame by frame to accurately identify key human points (skeletal joints, torso contours, limb postures, etc.) in the video. Dynamic human movements are converted into standardized 3D human model data in the SMSLX format. This format can completely preserve detailed information such as human joint angles, limb movement trajectories, and body posture changes, realizing the structured and digital extraction of human movements and laying a data foundation for subsequent robot motion conversion.

[0030] (3) Transformation of robot action data.

[0031] By using GMR (General Motion Retargeting) technology, human motion data in SMSLX format is mapped and adapted to the robot body. Combined with the hardware characteristics of the target robot, such as joint degrees of freedom, range of motion, and dynamic parameters, the human motion data is scaled, smoothed, and its rationality is verified. Motion data that exceeds the robot's motion limits is eliminated, and finally, motion command data that the robot can directly read and execute is generated, realizing the accurate transfer of human motion to robot motion.

[0032] (4) Visual modal data generation.

[0033] The simulation data generation module is used to drive the simulated robot to complete the corresponding command actions in a physical simulation environment by taking motion command data as input, and synchronously recording the dynamic process of the robot's actions to generate a visual video.

[0034] In one feasible approach, specifically, a Mujoco high-performance simulation environment is built, and physical parameters consistent with the target humanoid robot are configured within the Mujoco high-performance simulation environment, including: gravity, friction, and collision detection threshold. (See Appendix) Figure 10 An exemplary diagram is provided showing the actions of a simulated humanoid robot performing a preset task in a simulated environment.

[0035] Preferably, the gravitational acceleration is set to -9.81 m / s², the friction coefficient is set to 0.5, and the collision detection threshold is set to 0.01. The robot's executable motion command data is imported, driving the simulated robot to complete the corresponding motion sequence. The dynamic process of the robot's actions is recorded synchronously, and a visual video is rendered. Preferably, the resolution of the visual video is set to 720P, and the frame rate is 20fps to ensure the clarity and temporal continuity of the visual information. The visual video not only presents the environmental background but also clearly shows the robot's joint movement changes, limb posture switching, and the entire motion execution process. In other words, the visual input itself implicitly contains information about the robot's current motion state: joint angles, movement speed, body posture, etc., can all be obtained through visual perception from a continuous sequence of video frames. The visual video data, as the visual modal component of the dataset, forms a correlation mapping with the motion command data.

[0036] This invention provides a method and system for converting non-standardized video data into standardized "language-vision-motion" trimodal data adapted to a robot. It can efficiently mine various publicly available video resources online, such as remote control operations, life tutorials, and industrial operations, through methods such as... Figure 5 The AI ​​algorithm demonstrated completes accurate motion trajectory extraction and structured analysis of environmental information. It transforms non-standardized video data into standardized "language-vision-motion" trimodal data adapted to the robot, enabling the model to achieve the same training effect as data collected from a real machine, while simultaneously solving the problem of insufficient training data. The entire data conversion and standardization process is automated, significantly reducing dataset production costs. This method can be used for robots of different configurations to convert network data into ontology data, providing ample data support for large-scale training of VLA models, thereby breaking down the industry's "data silo" limitations.

[0037] Language modality data construction.

[0038] Natural language instructions are used to describe the actions or interactive intentions expected of a robot, such as "wave to greet," "step aside to make way," or "nod to indicate agreement." Natural language instructions can be obtained in various ways, including speech recognition and conversion, text input, or predefined instruction sets.

[0039] The Qwen3-VL multimodal large model is used to perform segment-by-segment semantic analysis and motion description generation on the generated robot motion visualization video. The model generates accurate, standardized, and realistic natural language text by identifying the robot's motion type, joint motion trajectory, motion execution sequence, and key motion nodes in the video. The text covers information such as motion name, execution steps, and limb motion details, ensuring consistency between the language description and the robot's motion and visual video. This completes the construction of language modal data and ultimately forms a three-in-one cross-modal dataset of "language description - visual features - motion instructions".

[0040] The visual language feature extraction module and the diffusion action generation module form a dual-system structure, constituting the high-level planning structure of the language robot action generation model. Specifically, in this embodiment, the visual language model adopts Qwen3-VL-2B, using supervised learning and pre-training with labeled data. This model has 2 billion parameters, with input dimensions adapted to 224×224 pixel pre-processed images and text sequences of up to 512 characters in length. The core encoding module contains only 16 layers of Transformer encoders, with 16 attention heads per layer. The attention head dimension is 128, and the output visual-language unified feature dimension is fixed at 2048. The model simultaneously receives image and text labels and performs deep fusion through a cross-attention mechanism.

[0041] The robot motion generation model is built based on the VLA (Vision-Language-Action) model framework. It adopts a dual-module serial architecture of "Vision-Language Model Backbone (VLM Backbone) and Flow Matching Diffusion Head (Flow Matching Diffusion Head)" to realize the end-to-end complete process from multimodal input to full-body robot motion generation. It can accurately complete the entire link of "perception-fusion-generation" motion driving.

[0042] The overall information flow is as follows: RGB observation images and natural language task descriptions serve as raw inputs. Visual features are extracted by a visual encoder, and visual-language fusion conditional features are extracted by the intermediate layer of the language model. Then, dimensionality reduction and alignment are achieved through a linear projection layer. Finally, guided by these conditional features, a 32-layer Alternate Attention Diffusion Transformer (AlternateVLDiT) is driven to denoise and generate an action sequence starting from Gaussian noise using a 4-step ordinary differential equation (ODE) solution. The output shape is... The normalized action tensor, after being denormalized by the action decoder, can be directly used for robot execution. For example, in one embodiment, The value is 8, meaning one inference operation consists of 8 consecutive frames of action. The value is 30, meaning that a single inference operation contains 30 dimensions of action information.

[0043] The specific implementation is as follows: First, the input image data and natural language instructions are standardized and preprocessed to ensure the uniformity and validity of the input data.

[0044] Image data preprocessing.

[0045] See appendix Figure 6 As shown, the frame image sequence undergoes preprocessing before entering the visual encoder: each original image is first scaled to an intermediate size of 256×256 using bilinear interpolation, and then randomly cropped during the training phase, with the cropping ratio adjusted from... The parameter control was set to 0.95, resulting in a cropped size of 243×243 pixels. During the evaluation phase, center cropping was used to ensure the determinism of the result. After cropping, the size was adjusted to 224×224 pixels. Pixel values ​​were normalized and mapped to the [0,1] interval, and a Gaussian filtering algorithm was used to remove noise.

[0046] In addition, color jitter enhancement is applied to the images during the training phase, with the brightness perturbation intensity set to 0.3, the contrast perturbation intensity set to 0.4, the saturation perturbation intensity set to 0.5, and the hue perturbation intensity set to 0.08, thereby improving the model's robustness to changes in lighting and color.

[0047] The preprocessed image is used for feature extraction and encoding through the ViT network encoding module to generate fixed-dimensional image labels for further fusion of visual features.

[0048] Language instruction preprocessing includes word segmentation, stop word removal, and word embedding. The BPE (Byte Pair Encoding) word segmentation algorithm is used to segment natural language instructions. After filtering out meaningless stop words, the word segmentation results are converted into fixed-dimensional text tags (sequences) through the word embedding layer. The maximum length of the text sequence is set to 512, thereby realizing the digital encoding of language information.

[0049] After image preprocessing is completed, the VisionLanguageProcessor is responsible for combining the processed image list with the natural language task description into a multimodal input conforming to the Qwen3-VL dialogue template format.

[0050] Specifically, the dialogue is constructed using a single-turn user role, with images and text arranged in an alternating order in the content field: image entries for all perspectives are placed first, followed by text entries. This dialogue structure is converted into a text string with special image placeholder tokens (stored in the "text" field) using the `apply_chat_template` method of the AutoProcessor, while the actual list of image objects is extracted using the `process_vision_info` function (stored in the "image" field). Both serve as input for subsequent batch processing by the DataCollator. In the DataCollator phase, all images and text samples within the same batch undergo unified tokenization and padding via Qwen3-VL's AutoProcessor. Images are segmented into patches and encoded into pixel value tensors (pixel_values), and text is tokenized into integer sequences (input_ids), generating corresponding attention masks (attention_mask) and image grid information (image_grid_thw).

[0051] The training dataset generation module is set to the Visual Transformer (ViT) built into Qwen3-VL, whose input is the RGB image generated from the retargeted data. ViT encodes the patch sequence into visual token embedding vectors, whose dimensions are consistent with the hidden dimensions inside the LLM. The visual tokens are then concatenated with text tokens to form a mixed sequence, which serves as the input to the LLM. In this specific implementation, the visual tokens are specifically objects, regions, or image patches in the image, and the text tokens are specifically words or characters.

[0052] Visual-language fusion occurs in the LLM built into Qwen3-VL. In this embodiment, the LLM is set with a 28-layer Transformer Decoder structure, which receives a mixed input sequence composed of visual tokens and text tokens. The total length of the sequence, seq_len, is the sum of the number of visual tokens and the number of text tokens.

[0053] The model of this invention extracts intermediate layer features from the 16th layer of the LLM (specified by the select_layer parameter), preserving richer low-level spatial details. Simultaneously, the semantic representation of the intermediate layer features is more general, making them more suitable as conditional inputs for diffusion models. The feature tensor output from the 16th layer has a shape of [batch_size, seq_len, 2048], where 2048 corresponds to the setting value of the backbone_embedding_dim parameter, i.e., the hidden layer dimension within Qwen3-VL.

[0054] Regarding the parameter freezing and fine-tuning strategy, all parameters of the ViT visual encoder are frozen. By fine-tuning only the parameters of the top 4 layers of the LLM (tune_top_llm_layers=4), the pre-trained visual language understanding ability is fully preserved, while allowing high-level features to be adaptively adjusted to meet the needs of the action prediction task.

[0055] The conditional features output from layer 16 then enter the VLLN (Vision-Language LinearNormalization) projection module. The projection module consists of a linear transformation layer and a normalization layer, compressing the feature dimension from 2048 dimensions to 1536 dimensions to match the configuration of 32 attention heads of AlternateVLDiT with 48 dimensions per head; that is, mapping from backbone_embedding_dim to input_embedding_dim, with the output tensor shape being [batch_size, seq_len, 1536].

[0056] The projection module parameters are fully involved in training (tune_projector=True, tune_vlln=True), acting as a bridge between the VLM feature space and the diffusion model workspace. The dimension of 1536 is determined by the attention head configuration of the diffusion model: 32 attention heads (num_attention_heads=32) multiplied by the dimension of 48 per head (attention_head_dim=48) equals 1536, thus ensuring the accurate matching of the dimensions of the K and V matrices in Cross-Attention.

[0057] In the architecture design of this embodiment, robot state data is not used as an independent modal input, but is internalized within visual information. The signal flow process is as follows: After the frame image sequence enters the ViT visual encoder, each frame image is encoded as a visual token embedding vector; these visual tokens are concatenated with text tokens and input into the LLM; in the Transformer layer of the LLM, through a self-attention mechanism, image tokens at different time steps interact across frames, allowing the model to capture the differences in joint positions between consecutive frames, thereby implicitly inferring joint velocity direction and temporal motion information; this temporal motion information is continuously abstracted and fused as it propagates between layers, ultimately being encoded in the visual-language fusion features output at layer 16. Therefore, the model does not need to explicitly input dynamic and kinematic parameters such as joint angles and angular velocities, but rather learns implicit representations of the state autonomously through the temporal changes of the visual frame sequence.

[0058] The fused features encode both instruction semantic information (e.g., "nod" or "agree") and visual scene information (robot's current posture and environmental location). Therefore, they can fully preserve the scene information, target posture information, and action requirement information of the language instructions from the input image, providing accurate feature support for subsequent action generation.

[0059] Action encoding and flow matching diffusion training process.

[0060] The action decoding module uses conditional flow matching to generate action sequences. See appendix. Figure 7 An example is provided: a 4-step flowchart for denoising ordinary differential equation (ODE) inference.

[0061] The first step in the training process is to perform stream matching time-step sampling of the action data. Time step t starts from Beta( , Sampling is performed in a Beta distribution, mapping the sampling range from the theoretical interval [0,1] to [0.001,0.999]. The shape of this Beta distribution is biased towards larger t values, meaning that the model faces strong noise scenarios more frequently during training (i.e., t close to 1 corresponds to a pure noise state). The specific parameters are α=1.5, β=1.0, and the sampling result is multiplied by noise_s=0.999 for upper bound truncation. Then, the continuous t values ​​are discretized into 1000 time buckets (num_timestep_buckets=1000) to improve the denoising quality in strong noise scenarios.

[0062] The forward pass (noising process) of stream matching is defined as linear interpolation between a clean action signal and standard Gaussian noise. Let the ground truth action tensor be... The batch size is [batch_size, 8, 30] (where 8 is max_action_horizon and 30 is max_action_dim); noise of the same shape is sampled from a standard normal distribution. Then the noisy action corresponding to time t Calculated using the following formula: The formula degenerates into a clean action at t=0. It degenerates into pure noise at t=1. This forms a straight-line trajectory (flow) from the data distribution to a Gaussian distribution. Simultaneously, the model's training objective is to predict the direction of the velocity field of this flow at time t, where the target velocity field is defined as... That is, a vector pointing from the clean signal to the direction of the noise.

[0063] AlternateVLDiT is the core network of the action head in this invention, such as... Figure 8 As shown, this is a 32-layer alternating attention diffusion transformer structure, specifically enabled by the `interleave_self_attention=True` parameter. Each layer uses adaptive normalization (AdaNorm) as a preprocessing unit, dynamically generating scaling factor γ and offset factor β through time step embedding t to modulate the layer input, as shown in the formula: γ and β are calculated by MLP from time-step embeddings, and the norm_type parameter is set to "ada_norm". This allows the same network to dynamically adjust its normalization behavior according to the current noise level, applying subtle corrections when t is close to 0 (close to a clean signal) and performing large-scale feature reshaping when t is close to 1 (close to pure noise).

[0064] The 32-layer network alternates between even-numbered layers performing self-attention and odd-numbered layers performing cross-attention. The frequency of cross-attention is controlled by the parameter `attend_text_every_n_blocks=2`. In the self-attention layer, tokens at each time step of the noisy action sequence interact globally, modeling the coherence and dependencies of actions in the temporal dimension. The query, key, and value are all derived from the current hidden action state, with a shape of [batch_size, action_horizon, 1536]. In the cross-attention layer, the action token serves as the query, while the visual-linguistic conditional features output by VLLN (after fusing embodied embeddings) serve as the key and value, with a shape of [batch_size, seq_len, 1536]. The action token extracts information related to the current task and visual state from the conditional features through cross-attention, achieving explicit alignment of action generation with visual-linguistic conditions. Each layer is configured with 32 attention heads (num_attention_heads=32), each head has a dimension of 48 (attention_head_dim=48), and the total working dimension is 32×48=1536, consistent with input_embedding_dim. The dropout rate is set to 0.2 to provide regularization.

[0065] Using random noise as the initial input, the feature representation is progressively optimized through multiple rounds (preferably 4 rounds) of iterative denoising. During the denoising process, the model introduces joint degree-of-freedom constraints and motion smoothing L2 constraints to ensure that the generated motion labels are within the robot's joint motion range and are temporally smooth and continuous. The motion labels cover the angle parameters and motion timing information of the robot's head joints (neck joint yaw, pitch, roll angles), trunk joints (spinal flexion, lateral flexion, rotation), limb joints (3-DOF rotation of shoulder, elbow, wrist, hip, knee, and ankle), and finger joints (independent bending angle of each finger). This accurately represents the dynamic process of the robot's whole-body movements, balancing motion generation accuracy and inference efficiency, and avoiding inter-frame computational redundancy.

[0066] After 32 layers of alternating attention processing, the network maps the hidden state from 1536 dimensions to output_dim=1024 dimensions through the final output layer, and then restores it to the action space dimension through a linear projection, outputting the velocity field prediction. The shape is [batch_size,8,30].

[0067] Training loss.

[0068] The training loss is defined as the mean squared error (MSE) within the effective action region, and the formula is: .

[0069] Where m is a binary mask tensor with the same shape as action, both being [batch_size, 8, 30]: Within the effective time step range of the actual action (h < action_horizon) and the effective action dimension range (d < action_dimension), the mask value is 1; In the time-step padding region (h ≥ action_horizon) and the dimension-padding region (d ≥ action_dimension), the mask value is 0. This ensures that the loss is calculated only where the real action data is located, completely ignoring the zero-padding applied to align samples of different lengths within a batch, preventing the model from being misled by false zero signals.

[0070] In another embodiment, for application scenarios with limited resources or higher real-time requirements, this system provides a lightweight configuration scheme. It adopts the MobileCLIP-S2 visual-language fusion model, which has 120 million parameters and includes 8 Transformer encoder layers, each with 8 attention heads. Each attention head has a dimension of 64, and the output dimension is 512. The diffusion action generation module uses a simplified DiT-S diffusion transformer architecture with a total depth of 12 layers, including 6 visual-language cross-fusion layers and 6 action feature denoising layers. Each layer has 12 attention heads, each with a dimension of 32, and the output action tag dimension is 256. The conditional diffusion transformer generates an acceptable full-body action tag sequence through two rounds of iterative denoising. Experiments show that this lightweight configuration can reduce end-to-end inference latency to 40% of the original embodiment, making it suitable for edge computing devices or humanoid robot interaction scenarios with extremely high response speed requirements.

[0071] In another implementation, addressing the need for high-precision, long-term complex motion generation, this system provides a high-performance configuration. It employs the InternVL2-8B vision-language fusion model with 8 billion parameters, containing a 32-layer Transformer encoder, each layer configured with 32 attention heads, each attention head having a dimension of 256, and an output dimension of 4096. The diffusion motion generation module adopts the DeepDiT-XL architecture with a total depth of 64 layers, including 32 layers for vision-language cross-fusion and 32 layers for motion feature denoising, each layer configured with 64 attention heads, each attention head having a dimension of 64, and an output motion tag dimension of 2048. The conditional diffusion transformer, through 8 rounds of iterative denoising, can generate high-quality full-body motion tag sequences containing more refined finger gestures and facial expression control signals. This embodiment is suitable for scenarios with extremely high requirements for motion naturalness, such as high-precision simulation, film and television production, or complex service robot tasks.

[0072] The parameter size, network depth, number of attention heads, and feature dimensions of the vision-language multimodal fusion network are configured according to the real-time requirements and computational resource constraints of the application scenario. The total number of layers of the conditional diffusion transformer, the ratio of fusion layers to denoising layers, the number of attention heads, and the number of iterative denoising rounds are adaptively configured according to the action complexity requirements and inference latency constraints.

[0073] Training parameters.

[0074] The entire model's end-to-end training uses a fusion version of the AdamW optimizer. The learning rate is set to 4e-4, employing a cosine annealing scheduling strategy with a minimum learning rate (cosine_with_min_lr). The lower bound of the learning rate decay is 1% of the initial learning rate, i.e., 4e-6 (min_lr_rate=0.01), ensuring that the learning rate does not drop to zero at the end of training and retaining some parameter update capability. The warm-up phase accounts for 5% of the total training steps (warmup_ratio=0.05), during which the learning rate linearly increases from 0 to 4e-4. The weight decay coefficient is set to 1e-5, providing slight L2 regularization. The gradient pruning threshold is set to 1.0 (max_grad_norm=1.0) to prevent gradient explosion during the backpropagation of the streaming matching loss. The global batch size is 512 (64 x 8 GPUs per batch), the gradient accumulation step count is 1, and the model saves a checkpoint every 1000 steps, retaining a maximum of the 3 most recent versions.

[0075] Reasoning process – ODE solution.

[0076] During the inference phase, there are no ground truth actions. The model starts from pure Gaussian noise and gradually generates action sequences by solving ordinary differential equations. The model in this invention sets the inference time steps to 4 (num_inference_timesteps=4), uniformly dividing the [0, 1] interval into 4 sub-intervals with a step size of 0.25. The inference process starts from... Starting with standard normal noise of shape [batch_size, 8, 30], at t=1.0, call AlternateVLDiT to predict the velocity field v and perform a one-step Euler integral: Subsequently, the velocity field was predicted again at t=0.75, and the following was executed. Similarly, after two steps at t=0.5 and t=0.25, the final result is obtained. This refers to the predicted clean action sequence, with a shape of [batch_size, 8, 30]. The entire inference process requires only 4 AlternateVLDiT forward propagations, significantly improving inference efficiency and making it suitable for real-time robot control scenarios.

[0077] Action decoding.

[0078] Normalized action tensor obtained during the inference phase After inverse normalization by the action decoder, the model is restored to the original physical quantity range. During decoding, the model slices the 30-dimensional vector along the action dimension according to the dimensional range of each action key: Let the parameter corresponding to the current action key be params (containing two arrays, params['max'] and params['min'], both of which are of shape [action_key_dim]), then the inverse normalization formula is applied to this slice: ; The clip operation truncates the normalized action to the range [-1, 1] to prevent out-of-bounds errors. It ultimately returns a dictionary where the keys are the action modal names and the values ​​are NumPy arrays of shape [action_horizon, action_key_dim], directly corresponding to the target position sequence of each joint group of the robot over the next 8 time steps, which can be directly executed by the underlying controller.

[0079] By decoding, the digitized motion markers are converted into real-time control signals that the robot controller can recognize. The control signals include motion angle commands and timing synchronization commands for each joint, which drive the humanoid robot to complete the corresponding full-body movements. This achieves full-body control from visual-language perception to robot motion execution, ensuring that the robot's movements accurately match the input command requirements and improving the accuracy and smoothness of motion execution.

[0080] Control signals are sent to the robot (real robot or simulation environment) at a frequency of 20Hz, driving the robot to perform full-body movements. Taking the "nodding" command as an example, the robot's neck joint pitch angle changes from 0° to 15° and returns to its original position within 0.5 seconds, while the torso remains stable, completing a natural nodding response.

[0081] This invention employs a hierarchical decoupled architecture of high-level trajectory planning using a large model and a low-level dynamic tracking controller using reinforcement learning, such as... Figure 3 As shown, the upper-level motion model only outputs a pure kinematic real-time reference trajectory. This module does not connect to dynamic constraints, robot body sensing observations, or participate in reinforcement learning iterative optimization; it is only responsible for outputting high-level semantic motion targets. Based on this characteristic, this layer eliminates robot body state modal inputs to ensure low-latency real-time performance and motion sequence coherence in the high-level action generation chain. A third-person perspective observation is introduced into the architecture as an input branch of the large model, constructing an information feedback path for high-level trajectory generation. However, since this branch does not connect to the robot body state (state) closed-loop feedback, there is no correction iteration of the body dynamic state; therefore, this information path is defined as a semi-closed-loop feedback structure. The actual physical correction (dynamic compensation) is handled by the lower-level RL controller, realizing the separation of responsibilities between the high-level and lower-level layers and avoiding uncertainties in physical interaction of the end-to-end model.

[0082] High-level sports planning.

[0083] The high-level motion planner outputs a pure kinematic reference trajectory at a frequency of 20Hz. This planner receives robot-executable control commands from the diffusion motion generation module, performs motion planning, and outputs a reference command vector p_cmd, which includes kinematic targets such as root linear velocity, root z-position, root roll / pitch angle, yaw rate, and joint positions throughout the body.

[0084] The high-level motion planner is designed as a pure kinematic semantic generator, ensuring the generation of low-latency, temporally coherent motion targets. Incorporating ontology states (closed loop) would cause the high-level policy to be affected by physical delays, disrupting the smoothness of the generation chain; furthermore, the high-level planner does not participate in RL optimization, and the elimination of states ensures its generalization and versatility.

[0085] The underlying motion tracking controller, whose main structure is a reinforcement learning controller (lower-level motion tracking controller), is trained using a Teacher-Student architecture and the PPO algorithm. The training and deployment process is as follows: Figure 3 As shown.

[0086] The inputs to the network structure include: the reinforcement learning controller is trained using the PPO algorithm, and the inputs include the reference command vector p_cmd (root linear velocity, root z position, root roll / pitch angle, yaw rate, and body joint position) and the proprioceptive state s (root angular velocity / acceleration, and current joint position / velocity).

[0087] The network first uses a three-layer one-dimensional convolutional history encoder (5 / 3 / 3 kernels, 64 channels) to compress the (s, p_cmd) data from the past 16 steps into a 128-dimensional historical latent vector. This vector is then concatenated with the current s and p_cmd and fed into a three-layer MLP (512→256→128→29), outputting the desired 29-dimensional joint position q_ref (mean of action). The reward function consists of an exponential tracking reward and a penalty term for action change.

[0088] The training process is as follows: During training, the simulation environment executes the torque obtained by q_ref through the PD controller (Kp=150, Kd=5) and calculates the exponential tracking reward. The penalty is to store the experience tuple (s, p_cmd, q_ref, r, s_next).

[0089] The network is updated with PPO every 2048 steps, for example, with PPO configuration as follows: pruning ε=0.2, 5% warm-up ratio, learning rate 3e-4, batch size 256.

[0090] During network deployment, high-level policies (such as Diffusion Policy) output p_cmd at 20Hz. The reinforcement learning controller directly reuses the trained Student policy, receives p_cmd and real-time s at 50Hz, and outputs q_ref to the PD controller, forming a closed loop.

[0091] Preprocessing of robot motion dataset.

[0092] See attached document Figure 4 An example flowchart is provided for the construction and preprocessing of a humanoid robot motion dataset.

[0093] The input includes two external motion trajectories, and the two data are uniformly retargeted to adapt to the robot's joint space.

[0094] Offline large-scale human motion open-source data (the same dataset used for training the upper-level motion model) is redirected offline to batch map human kinematic trajectories to the native humanoid robot URDF skeleton, correcting joint degrees of freedom and limb size differences, and generating standardized robot reference trajectories.

[0095] Large models generate trajectory data on a small scale, which is then redirected online. Errors such as time delay during trajectory generation are introduced as training samples, enabling lightweight real-time adaptation and outputting a reference trajectory at the robot level.

[0096] The generated reference trajectory is used to provide initial supervision samples for the second-stage simulation RL, and behavioral cloning (BC) pre-trains teacher and student policies to avoid random exploration.

[0097] Framework structure for reinforcement learning in a simulation environment.

[0098] The teacher strategy inputs two types of information: the current simulated robot dynamics observation O_t (joint angles / velocities, provided by the lower-level simulation engine); and the long-term future reference sequence g_t...g_t+20, outputting the expert compensation action A_t^tea, which is then sent to the physics simulation engine for execution.

[0099] The student strategy inputs two types of information: a short-term historical observation window O_t-10...O_t (only the local dynamics observations of the past 10 frames, with small input dimensionality, low inference latency, and suitable for real-world 50Hz hard real-time simulation); and a single-step current reference trajectory g_t (only the kinematic target of the upper-level large model at the current moment, without long-term planning information). It outputs a lightweight compensation action A_t^stu, which is then sent to the physics simulation engine for execution.

[0100] Decoupled from the simulation engine, the student strategy trained in the lower layer is fully reused and connected to the real-time data stream of the upper-layer large model to form a hardware 50Hz control closed loop: the external upper-layer motion large model outputs kinematic trajectories at 50Hz in real time, which are then fed into the lower-layer built-in online redirection module and converted into robot standards. ;Real-time feedback dynamics observation window O_t-10...O_t (lower-level frame observation input, joints, IMU);Lower-level student policy reception It has a short observation window for real-time reasoning of joint target motion after dynamic compensation; the lower layer has a built-in PD servo control loop that converts strategy output into motor execution instructions to directly drive the humanoid robot; new observations generated by robot motion are transmitted back to the strategy input to complete the real-time hardware closed loop.

[0101] The work efficiency of the above methods and systems is quantitatively evaluated.

[0102] Inference speed test: Under an NVIDIA A100 GPU environment, the average inference time was calculated after 100 tests. Each inference generated 8 action frames (corresponding to an action sequence of 400ms). The average inference time of the model method in this embodiment is 41ms, while the inference time of the existing model GR00T N1.5 (an open-source humanoid robot visual language action base model) is 83ms. The 41ms inference speed can support the motion control frequency of approximately 24Hz for humanoid robots, meeting the real-time requirements of human-computer interaction (typically requiring 20-50Hz).

[0103] Task completion test: Performance of the proposed model and existing VLA models on the LIBERO benchmark set, with appendix. Figure 9 An exemplary table comparing the success rates of test models GROOT N1.5, VLA-0, UnifoLM-VLA-0, and this embodiment in four sub-tasks—space, object, target, and long-range—and their average performance is provided.

[0104] The open-source UnifoLM-VLA-0 model uses the following technical approach: In order to pursue the upper limit of performance on benchmarks such as LIBERO, the Qwen2.5-VL-7B with 7 billion parameters is selected as the backbone, combined with the Flow Matching action head. During the training phase, a pipeline that mixes Mujoco simulation rendering video with real data is used. During the inference phase, 8 to 10 steps of Flow Matching are used to solve and generate action sequences. It is equipped with a hierarchical control architecture with upper-layer VLA trajectory planning (10 to 20 Hz) and lower-layer 50 Hz reinforcement learning tracking.

[0105] In this embodiment, the success rate for spatial tasks is 97.65%, for object tasks 100%, for target tasks 100%, for long-range tasks 91%, and the average success rate is 97.2%. The GROOT N1.5 model has a spatial task success rate of 97.65%, for object tasks 98.45%, for target tasks 97.5%, for long-range tasks 94.35%, and the average success rate is 97.0%. The VLA-0 model has a spatial task success rate of 97%, for object tasks 97.8%, for target tasks 96.2%, for long-range tasks 87.6%, and the average task success rate is 94.7%. The UnifoLM-VLA-0 model has a spatial task success rate of 99%, for object tasks 100%, for target tasks 99.4%, for long-range tasks 96.2%, and the average task success rate is 98.7%.

[0106] Experimental results show that the existing conventional large model UnifoLM-VLA-0 performs best in all indicators, with the highest average success rate of 98.7%, especially in space and object tasks, where it achieves or approaches full marks.

[0107] However, the open-source UnifoLM-VLA-0 model requires more than 14GB of video memory (FP16) for the VLM part alone. With the inference overhead of the 32-layer Flow Matching action head, the entire model needs to be equipped with an NVIDIA Orin AGX-level automotive computing platform with 64GB of video memory to run in real time. However, the actual automotive edge computing power of current mainstream humanoid robots is generally Jetson Orin NX 16GB or equivalent, which cannot support the real-time inference of a VLA model with 7 billion parameters. As a result, the cross-body advantage of "stateless VLA" cannot be realized on low-end edge platforms.

[0108] The Flow Matching motion head used requires 8-10 steps to converge to a usable motion sequence, with a single VLA inference latency of approximately 66ms. Even with lower-level RL tracking reaching 50Hz, upper-level planning still operates at 10-20Hz. The overall system's end-to-end latency of "perception-planning-execution" is still higher than the 50ms response period required for humanoid robot dynamic interaction scenarios (such as human-robot collaboration and dance following) (generally requiring 20-50ms). This poses a risk of lag when humanoid robots perform fast-response tasks. The latency introduced by the existing multi-step Flow Matching inference configuration still cannot meet the requirements of high-frequency control.

[0109] If the number of Flow Matching steps in the existing technical solution is directly reduced to less than 4 steps in exchange for a decrease in latency, due to the convergence characteristics of Flow Matching itself, the average success rate of LIBERO will usually drop from 98.7% to below 90%, a drop of more than 8 percentage points. The performance loss is unacceptable and it is impossible to balance fewer steps and high performance.

[0110] This embodiment provides a method and system for generating humanoid robot motion. Under the stateless VLA paradigm, through the collaborative design of a lightweight VLM backbone (Qwen3-VL-2B, 2 billion parameters), 4-step constrained ODE denoising (AlternateVLDiT combined with joint degree-of-freedom constraints and motion smoothing L2 constraints), and hierarchical RL closed loop (upper layer 20Hz / lower layer 50Hz), it achieves a perfect score of 100% on both object tasks and target tasks, with an average score of 97.2%, slightly lower than the average success rate of UnifoLM-VLA-0 (98.7%).

[0111] With an average success rate of LIBERO decreasing by only 1.5 percentage points (97.2% vs. 98.7% for D3), inference latency is reduced to 41ms and the number of parameters is reduced to 2 billion. It can be deployed on the Jetson Orin NX 16GB-level edge platform, taking into account both cross-ontology generalization capabilities and the low latency and low computing power requirements of the edge.

[0112] Cross-body transfer test: When the robot model in the simulation visual data is switched from type A robot (with different joint length ratios) to type B robot, only the URDF file needs to be replaced in the Mujoco environment and the visual data needs to be re-rendered. The model does not need to be retrained or only needs a small amount (<100) of adaptation data for fine-tuning. It can generate reasonable action sequences on the new robot, which verifies the strong generalization ability brought by stateless design.

[0113] This invention collects simulated visual data containing robot joint motion states as model input, breaking the technical paradigm of treating robot state modalities as independent modal inputs. It internalizes state information within the model architecture of the visual modality, enabling the visual modality to simultaneously perform the dual functions of environmental perception and state representation. A multimodal fusion model performs cross-attention fusion of images and language commands to generate a unified feature representation; a conditional diffusion transformer is used to generate full-body joint motion markers for the robot through iterative denoising, which are ultimately decoded into real-time control signals.

[0114] By normalizing continuous actions into integer text strings, the VLM (Virtual Machine Learning) directly predicts the action text. Combined with techniques such as masked action enhancement and ensemble prediction, it achieves excellent performance without the need for large-scale robot pre-training. For example, it achieves an average success rate of 94.7% on the LIBERO benchmark and is 12.5% ​​faster than SmolVLA on the SO-100 platform, demonstrating significant potential for inference speed. It can seamlessly integrate the vision, language, and action chains without complex modifications. Specifically, the model features a clear and efficient three-modal division of labor: The image modality is responsible for visual perception of the environment and targets, capturing key scene information, and also assumes the function of state perception, providing the model with all the necessary state information. The text modality carries instruction input and semantic understanding, bridging human needs and model actions. The action modality, as the output, directly transforms visual (including state information) and text information into specific executable actions. The three work together without the need for redundant intermediate transmission from the state modality. This design not only continues the advantages of VLA-0 in terms of lightweightness and low development cost, but also further improves the model's inference efficiency and generalization ability through modality simplification. It avoids the problems of information redundancy and inference delay that may be caused by independent state modalities, allowing the model to focus more on accurately understanding the visual scene (including state), interpreting language instructions, and efficiently outputting actions adapted to the scene. This provides a simpler and more focused new path for the development of lightweight embodied intelligent models.

[0115] This invention breaks through the limitations of shallow interaction in traditional VLA models. It can achieve end-to-end millisecond-level generation of natural language commands to full-body robot movements without the need for independent state modalities. It achieves a response cycle of less than 50 milliseconds for language understanding and full-body limb coordination, reduces the number of model parameters and inference latency, and improves generalization ability.

[0116] The method and system provided by this invention can run on a robot's local computing unit equipped with a GPU or a cloud inference server, possessing excellent real-time performance and generalization capabilities, and can be widely applied in cultural and tourism robot scenarios. In cultural and tourism scenarios, the robot can act as an intelligent tour guide, generating corresponding full-body motion responses based on tourists' natural language commands (such as "Please introduce this painting" or "Please take me to the next exhibition area"), including precise gestures pointing to paintings, torso turns to guide forward movement, and head postures to accompany explanations, achieving a natural and smooth human-computer interaction experience.

[0117] The high-level motion planning module eliminates the robot's body state modal input to ensure low latency real-time performance and motion sequence continuity in the high-level motion generation link; the simulation rendering video serves as a third-person perspective observation input, forming a semi-closed-loop feedback structure, without being connected to the body state closed-loop feedback, and does not participate in the dynamic state correction iteration.

[0118] In the deployment scenario of humanoid robot vehicle-mounted edge computing platform (16GB VRAM level), it only pays an acceptable performance cost (<2pct drop) on the LIBERO benchmark to compress inference latency to less than 50ms and parameter number to less than 2 billion, while taking into account the high generalization of stateless VLA and small model, few inference steps and low edge computing power threshold.

[0119] This invention also provides an electronic device, which includes a processor and a memory; the memory stores one or more instructions, which are adapted for the processor to load and execute, to implement a method for generating motion of a humanoid robot as described in the above method embodiments.

[0120] Memory is used to store software programs and modules. The processor executes these stored software programs and modules to perform various functional applications and data processing. Memory mainly includes a program storage area and a data storage area. The program storage area stores the operating system, application programs required for functions, etc.; the data storage area stores data created based on device usage, etc. Furthermore, memory may include high-speed random access memory (RAM) and non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, memory may also include a memory controller to provide the processor with access to the memory.

[0121] The internal structure of the electronic device provided in the embodiments of the present invention may include, but is not limited to, a processor, a memory, and a communication interface. The processor, memory, and communication interface in the electronic device may be connected by a bus or other means. In the embodiments of this specification, a connection via a bus is taken as an example.

[0122] The processor (or CPU, Central Processing Unit) is the computing and control core of the electronic device. A communication interface is used for communication between the memory and the processor. The memory stores programs and data. It is understood that the memory here can be a high-speed RAM storage device, or a non-volatile memory device, such as at least one disk storage device; optionally, it can also be at least one storage device located remotely from the aforementioned processor. The memory provides storage space, which stores the operating system of the electronic device, and may include, but is not limited to, Windows (an operating system), Linux (an operating system), etc. This invention does not limit this; furthermore, the storage space also stores computer programs (including program code) suitable for loading and execution by the processor. In the embodiments of this specification, the processor loads and executes the computer program stored in the memory to implement the humanoid robot motion generation method provided in the above method embodiments.

[0123] This invention also provides a computer-readable storage medium, which can be disposed in an electronic device to store at least one instruction, at least one program, code set, or instruction set related to implementing a humanoid robot motion generation method in the method embodiments. The at least one instruction, at least one program, code set, or instruction set can be loaded and executed by the processor of the electronic device to implement the humanoid robot motion generation method provided in the above method embodiments.

[0124] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0125] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, the above description focuses on specific embodiments, while other embodiments fall within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than those shown in the embodiments and still achieve the desired results. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0126] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0127] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0128] The above description is merely a preferred embodiment of the present invention and should not be construed as limiting the scope of the invention. Therefore, any equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A method for generating motion in a humanoid robot, characterized in that, include: In the simulation environment, the simulated humanoid robot's movements are driven by the first action command, and a visual video containing multiple frames of joint motion states is rendered. The visualized video is decoded into a sequence of frame images from a third-person perspective, and natural language instructions and action sequences are obtained through synchronous parsing and split into triples according to the cycle. A multi-layer structure model is constructed, including a visual language model and a conditional diffusion transformer. The triplet data is used as input, and the robot-executable control instructions are output. The multi-layer structure model is iteratively trained. A real-time mapping relationship between the real robot and the simulator in the simulation environment is established. Input data pairs of the simulator are obtained according to the mapping relationship, including the frame image sequence and natural language instructions. The input data pairs are input into the multi-layer structure model that has been trained. The output robot-executable control instructions are sent to the real robot drive controller to control the execution of the real robot's actions.

2. The method for generating motion in a humanoid robot according to claim 1, characterized in that, Visual language fusion features are extracted through the visual language model, and temporal change information of frame image sequences is captured through the self-attention mechanism of the visual language model, thereby encoding joint motion states into visual features. The intermediate layer of the visual language model obtains visual language fusion conditional features through projection compression. Guided by the fusion conditional features, the conditional diffusion transformer introduces joint degree of freedom constraints and motion smoothing constraints during the denoising process, and generates control instructions that can be executed by the robot by denoising and decoding using multi-step ordinary differential equations. The real robot drive controller is set as a reinforcement learning controller and is equipped with a residual network. The residual network freezes the multi-layer structure model, performs residual network learning, and outputs residual correction amount based on the joint motion state output by the body sensor in real time and the joint motion state of the simulator, and then corrects and outputs the final control command. The control command is sent to the servo driver of the real robot to control the real robot body to execute the action command.

3. The method for generating motion in a humanoid robot according to claim 1, characterized in that, The real-time mapping relationship between the real machine and the simulator in the simulation environment is set as a spatial state mapping. Through the URDF model and joint zero-point calibration, a real-time bidirectional mapping table is established between the joint motion state of the real machine and the joint motion state in the simulation environment, including joint angle, deceleration ratio and offset calibration.

4. The method for generating motion in a humanoid robot according to claim 2, characterized in that, The visual language model uses a pre-trained Qwen3-VL-2B model as the visual language backbone network. The data of the triples are input into the Qwen3-VL-2B model, all parameters of the visual encoder ViT in the Qwen3-VL-2B model are frozen, the parameters of the top 4 Transformer layers are adjusted, and visual language fusion features with a dimension of 2048 are extracted from the 16th Transformer layer. Based on the VLLN projection module including linear transformation layers and normalization layers, the visual language fusion features are linearly normalized to compress the 2048 dimensions to 1536 dimensions, resulting in visual language fusion conditional features. Among them, the visual changes between consecutive frames of the frame image sequence are implicitly encoded as joint velocity direction and temporal motion information through the self-attention mechanism of the Transformer.

5. The method for generating motion in a humanoid robot according to claim 1, characterized in that, The conditional diffusion transformer employs an AlternateVLDiT alternating attention diffusion transformer with 32 alternating attention layers. Even-numbered layers perform self-attention of action sequences to model temporal coherence, while odd-numbered layers perform cross-attention with the fused conditional features as the key and value, thereby aligning action generation with visual language conditions.

6. The method for generating motion in a humanoid robot according to claim 2, characterized in that, The reinforcement learning controller employs a Teacher-Student architecture combined with a proximal policy to optimize the PPO algorithm training.

7. The method for generating motion in a humanoid robot according to claim 1, characterized in that, The simulation environment uses the Mujoco physics simulation engine, and the rendered visualization video has a resolution of 720P and a frame rate of 20fps.

8. A humanoid robot motion generation system, characterized in that, A method for generating motion for a humanoid robot as described in any one of claims 1-7, comprising: The simulation data construction module is used to render and generate visual videos containing visual representations of robot joint motion states in a physical simulation environment. The training dataset generation module is used to decode the visualized video to generate a sequence of frame images, parse it to obtain natural language instructions and action sequences, and the sequence of frame images, natural language instructions and action sequences are used to construct a training dataset; The visual language feature extraction module has a built-in pre-trained visual language model, configured to freeze the ViT parameters and fine-tune the parameters of the top 4 layers, and extracts 1536-dimensional visual language fusion features from the 16th layer. The diffusion motion generation module has a built-in alternating attention diffusion transformer, which generates motion sequences by solving multi-step ordinary differential equations under the conditions of joint degrees of freedom and smoothness constraints. The hierarchical control module includes a high-level planner that outputs pure kinematic trajectories at a first frequency, a low-level tracker that runs a reinforcement learning controller at a second frequency, and a servo driver that drives the execution of the real machine. When running on a real machine, the output of the simulation data construction module is connected to the visual language feature extraction module, the output of the visual language feature extraction module is connected to the diffusion action generation module, and the output of the diffusion action generation module is connected to the high-level planner of the hierarchical control module. The underlying tracker learns data from the higher-level planner and outputs the final control commands to the servo driver, driving the actual robot to perform actions.

9. An electronic device, the device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the motion generation method for a humanoid robot as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing at least one instruction or at least one program, characterized in that, When the at least one instruction or the at least one program segment is loaded and executed by the processor, the motion generation method for a humanoid robot as described in any one of claims 1 to 7 is implemented.

Images