Humanoid robot action training method, device, equipment and medium

Abstract

The invention discloses a humanoid robot action training method, device and equipment and a medium. Firstly, a wearable data acquisition system is adopted to replace traditional teleoperation hardware, laboratory scene limitation is broken through, and high-efficiency data acquisition of a field environment is achieved; secondly, based on depth estimation and a three-dimensional re-projection technology, converting a human first person view angle image to the height of a robot camera, filling a shielding hole through generative repair, and solving the problem of visual distribution mismatching caused by human body height difference; thirdly, constructing a layered decoupling unified action space, mapping the upper limbs into relative pose increments, mapping the lower limbs into discrete motion primitives, mapping the clamping jaws into a binary state, and avoiding action semantic confusion caused by form differences; and finally, man-machine data fusion is realized through timestamp synchronization, so that the robot can realize zero-sample generalization deployment in an environment without field acquisition, and generalization and reliability of a strategy are remarkably improved.

Description

Technical Field

[0001] This application relates to the field of robot training technology, and in particular to a method, apparatus, equipment and medium for training humanoid robot movements. Background Technology

[0002] The application of humanoid robots in human-centric environments such as home assistance and outdoor services is becoming an important development direction in the field of robotics, and data-driven learning methods have gradually become the mainstream paradigm for achieving complex motion control. However, the development of humanoid robot whole-body loco-manipulation has long been constrained by the core bottleneck of the scarcity of high-quality demonstration data. Whole-body loco-manipulation requires robots to coordinate movement navigation and fine manipulation while maintaining dynamic balance. This tightly coupled task characteristic places extremely high demands on the diversity and scale of training data.

[0003] Traditional solutions primarily rely on three data acquisition methods: first, directly controlling the robot to perform tasks and recording data through a teleoperation system; second, training control strategies in a simulation environment using reinforcement learning; and third, using motion redirection technology to map human motion to robot joint configurations. These solutions have significant limitations: robot teleoperation requires expensive hardware and specialized operators, resulting in low data acquisition efficiency; transporting humanoid robot platforms and teleoperation devices to real-world scenarios is often impractical, limiting data acquisition to laboratory environments; laboratory data lacks sufficient coverage in terms of object types, scene layout, lighting conditions, and perspective changes, making it difficult to generalize training strategies to outdoor environments; and simulation training suffers from a simulation-reality gap and requires modeling specific objects and designing complex reward functions.

[0004] In the research field of using human demonstration data for robot learning, the most representative work is the Universal Manipulation Interface (UMI) series published by the Stanford University team at the RSS conference in 2024. Its core idea is to design handheld hardware devices that are compatible with the robot's body shape during the human data acquisition phase, thereby transforming human demonstrations into training data that can be used by the robot.

[0005] However, from the perspective of learning full-body motion in humanoid robots, existing UMI-like solutions still have the following significant limitations: The "time-of-acquisition alignment" strategy implemented through handheld devices in the UMI solution is only applicable to robotic arm operations with fixed bases. For humanoid robots, handheld devices cannot simulate the robot's full-body kinematics—humanoid robots (such as the Unitree G1, 1.3 meters tall, 29 degrees of freedom) differ significantly from normal adults (1.6-1.8 meters tall) in limb proportions, joint constraints, and mass distribution. First-person perspective observation is affected by both camera height and body movement. The camera height and perspective change patterns when humans walk are completely different from those of small humanoid robots, making the observation differences between human hands and the robot's metal robotic arm even more pronounced. This morphological gap is further amplified in full-body movement operations, with full-body movement causing drastic changes in perspective, and differences in body swaying and balance strategies introducing a large amount of observation space noise. Human data collected using handheld devices cannot be directly used to train humanoid robot strategies, leading to severe out-of-distribution problems and high task failure rates during deployment.

[0006] Existing solutions typically use the absolute pose of the end effector as the motion representation, which is effective in fixed-base scenarios but faces fundamental difficulties in full-body motion operations. There are systematic differences in the distance from the human hand to the camera between humans and humanoid robots; the human arm is generally longer relative to the torso, resulting in vastly different absolute positional distributions even for functionally equivalent movements. More critically, the correspondence between human hand poses (26 degrees of freedom) and robotic gripper poses (binary opening and closing state) is semantically ambiguous, making it impossible to reliably infer the expected direction of the robotic gripper from a first-person perspective. This ambiguity cannot be resolved through visual observation in the absence of proprioceptive input because morphological differences lead to incompatible proprioceptive distributions between humans and robots. This results in a significant decrease in the success rate of fine manipulation tasks, unreliable rotational control, and requires 2-3 times the amount of demonstration data compared to fixed-base operations to achieve similar performance.

[0007] The UMI (Unified Viewpoint Integration) approach achieves consistent perspective by fixing a camera on a handheld device, but this static alignment strategy cannot handle dynamic perspective changes introduced by full-body movements. In motion-based tasks, robots rarely stop in exactly the same position; the variability in stopping position during each trial leads to highly diverse perspectives in subsequent operational phases. Human demonstration data shows that camera height continuously changes during actions such as walking, bending, and squatting, and humanoid robots of different heights have completely different perspectives when observing the same scene. Existing solutions lack a systematic perspective conversion mechanism, failing to convert human first-person observation into an equivalent representation from the robot's perspective, and also needing to address dynamic perspective distribution shifts introduced by variations in movement trajectories. This results in a significant drop in performance when the trained strategy generalizes to new scenarios, especially in tasks where object placement height changes, leading to operational failures when the strategy encounters unfamiliar perspective configurations encountered during training. Summary of the Invention

[0008] This application provides a method, apparatus, device, and medium for training humanoid robot movements to solve the aforementioned technical problems.

[0009] In view of the above, the first aspect of this application provides a method for training humanoid robot movements, the method comprising: S1. Obtain human demonstration data based on a wearable data acquisition system, wherein the human demonstration data includes first-person image sequences, human action data, and natural language task descriptions; S2. Obtain robot teaching data through the robot teleoperation system. The robot teaching data includes robot body state, end effector pose, gripper state, and discrete motion commands. S3. A visual alignment method based on depth estimation and 3D reprojection converts the first-person image sequence into an equivalent viewpoint image data stream of the target robot. S4. Perform action alignment processing on the human action data and the robot teaching data by constructing a hierarchical and decoupled unified action space; S5. Based on a unified timestamp synchronization mechanism, the first-person image sequence after viewpoint alignment is fused with the equivalent viewpoint image data stream, and the human action data after action alignment is fused with the robot teaching data to construct a human-machine hybrid training dataset. S6. The vision-language-action model is collaboratively trained using the human-machine hybrid training dataset to generate a humanoid robot's full-body motion operation strategy.

[0010] Optionally, step S3 specifically includes: Reconstructing 3D point clouds of a scene based on a monocular depth estimation network, obtaining the 3D point cloud coordinates and validity mask corresponding to each pixel; Calculate the rigid body transformation matrix from the human camera coordinate system to the target robot camera coordinate system. The rigid body transformation matrix includes rotational and translational components. The vertical value of the translational component is used to reflect the height difference between the human and the robot. The three-dimensional point cloud coordinates are batch transformed to the target robot camera coordinate system according to the rigid body transformation matrix, and perspective projection is performed using the intrinsic parameter matrix of the target robot camera to generate a reprojected image with a hole mask. The hole regions in the reprojected image are filled based on the potential diffusion repair model, generating an equivalent viewpoint image data stream of the target robot after viewpoint alignment.

[0011] Optionally, the human motion data specifically includes whole-body joint posture data, fine hand posture data, and pelvic movement trajectory data.

[0012] Optionally, step S4 specifically includes: S41. By extracting human wrist posture data from the whole-body joint posture data, the human upper limb operation is mapped to the six-degree-of-freedom relative pose increment of the target robot. S42. By performing velocity analysis and threshold quantization on the pelvic trajectory data, the human lower limb navigation is mapped into discrete motion commands for the target robot. S43. By extracting binary opening and closing instructions from the fine hand posture data, the human gripper operation is mapped to the binary opening and closing state of the target robot.

[0013] Optionally, step S41 specifically includes: Extract the three-dimensional position coordinates and rotation quaternions of the wrist joint in the pelvic coordinate system from the whole-body joint posture data; The three-dimensional position coordinates are smoothed by a preset filtering parameter; The rotation quaternion is converted into a rotation vector by logarithmic mapping. After smoothing in the vector space according to the preset filtering parameters, it is restored to a rotation quaternion by exponential mapping. Together with the smoothed and filtered three-dimensional position coordinates, they form a wrist posture sequence. After downsampling the wrist pose sequence to the target robot's control frequency, the position difference and rotation difference between adjacent frames are calculated to form the six-degree-of-freedom relative pose increment of the target robot.

[0014] Optionally, step S42 specifically includes: Extract the three-dimensional world coordinate sequence of pelvic joints from the pelvic trajectory data and perform smoothing filtering; The instantaneous velocity vector of the pelvic joint in the horizontal plane and the rate of change of height in the vertical direction were calculated using the central difference method. The instantaneous velocity vector and the rate of change of height are quantified into discrete motion commands by using a first preset threshold. The discrete motion commands include forward and backward movement, left and right translation, rotation, and rise and fall.

[0015] Optionally, step S43 specifically includes: Extract the finger joint sequence from the fine hand posture data and perform smoothing filtering; The curvature value at the midpoint of the quadratic polynomial curve fitted to the finger joints of each finger is calculated based on the finger joint sequence, and the average curvature of the five fingers is used as the overall bending scalar of the hand. The curvature scalar is binarized into a closed or open state by using a second preset threshold.

[0016] A second aspect of this application provides a humanoid robot motion training device, the device comprising: The first acquisition unit is used to acquire human demonstration data based on a wearable data acquisition system. The human demonstration data includes first-person image sequences, human action data, and natural language task descriptions. The second acquisition unit is used to acquire robot teaching data through the robot teleoperation system. The robot teaching data includes robot body state, end effector pose, gripper state, and discrete motion commands. A visual alignment unit is used to convert the first-person image sequence into an equivalent viewpoint image data stream of the target robot using a visual alignment method based on depth estimation and 3D reprojection. The action alignment unit is used to perform action alignment processing on the human action data and the robot teaching data by constructing a hierarchical and decoupled unified action space. The training dataset construction unit is used to perform multimodal data fusion based on a unified timestamp synchronization mechanism, combining the first-person image sequence after viewpoint alignment with the equivalent viewpoint image data stream, and the human action data after action alignment with the robot teaching data, to construct a human-machine hybrid training dataset. The strategy generation unit is used to collaboratively train the vision-language-action model using the human-machine hybrid training dataset to generate a full-body motion operation strategy for the humanoid robot.

[0017] A third aspect of this application provides a humanoid robot motion training device, the device comprising a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the steps of the humanoid robot motion training method described in the first aspect above, according to the instructions in the program code.

[0018] A fourth aspect of this application provides a computer-readable storage medium for storing program code for performing the method described in the first aspect above.

[0019] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: This application provides a method, apparatus, device, and medium for training humanoid robot movements. First, a wearable data acquisition system replaces traditional remote-operated hardware, overcoming the limitations of laboratory settings and enabling high-efficiency data acquisition in field environments. Second, based on depth estimation and 3D reprojection technology, human first-person perspective images are converted to the robot's camera height, and generative inpainting fills in occlusion gaps, resolving the visual distribution mismatch caused by height differences between humans and robots. Third, a hierarchical, decoupled, unified motion space is constructed, mapping the upper limbs to relative pose increments, the lower limbs to discrete motion primitives, and the gripper to binary states, avoiding semantic confusion caused by morphological differences. Finally, human-robot data fusion is achieved through timestamp synchronization, enabling the robot to achieve zero-sample generalization deployment in environments without prior field data collection, significantly improving the generalization and reliability of the strategy. Attached Figure Description

[0020] Figure 1 This is a flowchart of a humanoid robot motion training method according to an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a humanoid robot motion training device in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a humanoid robot motion training device in an embodiment of this application. Detailed Implementation

[0021] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0022] For easier understanding, please refer to Figure 1 , Figure 1 This is a flowchart of a humanoid robot motion training method according to an embodiment of this application, such as... Figure 1 As shown, specifically: S1. Human demonstration data is acquired based on a wearable data acquisition system. The human demonstration data includes first-person image sequences, human action data, and natural language task descriptions. It should be noted that the wearable data acquisition system used in this application embodiment specifically includes a head-mounted virtual reality device, a body posture tracking system, a first-person visual acquisition device, and a language execution annotation interface, wherein: As the core computing and perception platform of the system, the head-mounted virtual reality device integrates a six-degree-of-freedom spatial positioning system, and has a built-in inertial measurement unit (IMU) to capture head acceleration and angular velocity, assisting visual odometry and providing immersive virtual environment feedback, enabling the operator to intuitively perceive the task execution status. The body posture tracking system includes five wireless inertial measurement units, which are fixed to both wrists, both ankles and waist respectively. The sampling frequency is 100Hz. The local posture is calculated in real time through sensor fusion algorithm and mapped to the three-dimensional coordinates of 24 whole-body joints based on inverse kinematics model, covering the neck, shoulders, elbows, wrists, spine, hips, knees and ankles. The first-person visual acquisition device is mounted on a fixed bracket at the front of the head-mounted device, aligned with the direction of the line of sight; The language instruction annotation interface provides a graphical interface that allows operators to input task descriptions (such as "move the pillow from the sofa to the bed"). The text is encoded into a 512-dimensional semantic vector by the BERT tokenizer and associated with the trajectory data.

[0023] All multimodal data streams are transmitted to the data synchronization workstation via communication middleware. The workstation ensures that the clock error of each sensor system is less than 1 millisecond based on the hardware clock synchronization protocol, and adds a unified UNIX timestamp to each frame of data. It maintains a 100-millisecond sliding time window buffer to perform nearest neighbor matching, and packages RGB-D images, full-body pose, hand state, and time alignment into synchronized data frames. Finally, it writes the data to storage in HDF5 format, forming a complete human demonstration dataset that includes first-person image sequences, full-body joint pose data, fine hand pose data, pelvic motion trajectory data, and natural language task descriptions.

[0024] The core advantage of human demonstration data lies in its ability to cover real-world environments. Unlike remote operation in a laboratory, wearable devices enable operators to perform tasks naturally in real-world settings such as homes, outdoors, and commercial spaces.

[0025] S2. Obtain robot teaching data through the robot teleoperation system. The robot teaching data includes robot body state, end effector pose, gripper state, and discrete motion commands. It should be noted that the robot's remote control terminal uses the same VR hardware configuration as the human data acquisition terminal to ensure the consistency of the sensors used for data acquisition.

[0026] The operator controls the humanoid robot using a VR controller to achieve: Upper limb manipulation: The six-DOF pose of the handle is mapped to the target pose of the robot's end effector; Lower limb navigation: The joystick and buttons generate discrete motion commands (forward / backward / left / right translation / rotation / lifting). Gripper control: The trigger button controls the binary opening and closing state.

[0027] During teleoperation, the system simultaneously records the angles, angular velocities, and torques of the robot's 29 joints, the six-DOF pose data of the end effector, the opening and closing width or binary state of the gripper, and the discrete motion commands input by the operator via the handle. This data serves not only as supervisory signals for policy learning but, more importantly, as a means to verify the kinematic feasibility of the human demonstration data after motion alignment transformation, ensuring that cross-morphological mapping does not violate the robot's joint constraints and dynamic constraints.

[0028] S3. A visual alignment method based on depth estimation and 3D reprojection converts a first-person image sequence into an equivalent viewpoint image data stream for the target robot. It should be noted that there is a height difference of 0.3-0.5m between humans (1.6-1.8m tall) and humanoid robots (1.3m tall), which causes a systematic shift in the spatial distribution observed from a first-person perspective. Traditional 2D image transformations (such as homography transformations) cannot handle changes in occlusion relationships and struggle to maintain geometric consistency.

[0029] Furthermore, step S3 specifically includes: S31. Reconstruct the 3D point cloud of the scene based on a monocular depth estimation network, and obtain the 3D point cloud coordinates and validity mask corresponding to each pixel; It should be noted that this application first initiates the MoGe monocular depth estimation module, which loads a pre-trained depth inference neural network model and focal length recovery parameters. The MoGe model is based on affine invariant depth estimation theory and can infer the relative depth relationships of a scene from geometric cues in a single frame of RGB image. Its core advantage lies in simultaneously predicting pixel-level depth and camera intrinsic parameters, thereby achieving metric 3D reconstruction of monocular images.

[0030] This network receives a single-frame RGB image as input and extracts multi-scale features through an encoder-decoder architecture. During the decoding stage, it outputs the 3D point cloud coordinates and validity mask for each pixel. The 3D point cloud coordinates are defined in the camera coordinate system, and the validity mask is a binary matrix that identifies which pixels have reliable depth estimates (typically based on reconstruction error and geometric consistency). Simultaneously, the network utilizes reprojection optimization techniques to recover the true scale and camera intrinsics, converting affine-invariant depth to metric depth to ensure the geometric accuracy of subsequent point cloud transformations. The output of this sub-step is a partial 3D point cloud representation of the scene, providing the geometric basis for subsequent spatial perspective transformations.

[0031] S32. Calculate the rigid body transformation matrix from the human camera coordinate system to the target robot camera coordinate system. The rigid body transformation matrix includes rotational and translational components. The vertical value of the translational component is used to reflect the height difference between the human and the robot. It should be noted that the point cloud conversion module reads the pre-calibrated extrinsic parameter matrices of the human camera and the target robot camera, both defined in the world coordinate system. The system calculates the rigid body transformation matrix from the source coordinate system (human camera) to the target coordinate system (robot camera). This transformation matrix includes rotation and translation components. The vertical value of the translation component directly reflects the height difference between the human and the robot. For the robot and a typical adult operator, this value is usually in the range of 0.3 to 0.5 meters.

[0032] To enhance the robustness of the training strategy to changes in viewpoint, the system applies random perturbations to the target pose during the training phase. The perturbation range is set to ±0.1 meters for position and ±15 degrees for rotation angle to simulate the variability of the robot's stopping position and changes in body orientation in real tasks. This data augmentation strategy significantly improves the strategy's ability to generalize to new viewpoint configurations.

[0033] The point cloud conversion module transforms the coordinates of three-dimensional points from the source viewpoint to the target coordinate system through batch multiplication of rigid body transformation matrices, with a computational complexity of O(N), where N is the number of effective pixels.

[0034] S33. Based on the rigid body transformation matrix, batch transform the coordinates of the three-dimensional point cloud to the coordinate system of the target robot camera, and use the intrinsic parameter matrix of the target robot camera to perform perspective projection to generate a reprojected image with a hole mask. It should be noted that, using the intrinsic parameter matrix of the target robot's camera (including focal length and principal point coordinates), the system performs perspective projection on the transformed 3D point cloud, calculating the 2D coordinates of each 3D point on the target image plane. The projection formula follows the pinhole camera model, converting the 3D homogeneous coordinates into 2D pixel coordinates while retaining the depth value for occlusion detection.

[0035] This process generates a sparse correspondence between pixels in the source image and pixels in the target image, that is, the pixel coordinates in the source image correspond to the new coordinates in the target image after a three-dimensional spatial transformation.

[0036] Due to the uncertainty of depth estimation, the occlusion effect caused by changes in viewpoint, and the sparsity of point clouds, there are a large number of uncovered pixel areas in the target image. These areas form irregular hole masks, which typically account for 20% to 40% of the image area. The specific proportion depends on the geometric complexity of the scene and the size of the viewpoint difference.

[0037] S34. Based on the potential diffusion repair model, fill the hole areas in the reprojected image to generate the equivalent viewpoint image data stream of the target robot after viewpoint alignment.

[0038] It should be noted that, to fill in the hole regions in the reprojected image, the system invokes a pre-trained latent diffusion inpainting model. This model is built upon denoising diffusion probability theory, with its core architecture being an improved Stable Diffusion, specifically optimized for viewpoint synthesis tasks.

[0039] The restoration process receives three inputs: the reprojected incomplete RGB image, a binarized hole mask (1 represents a hole region, 0 represents a valid region), and a contextual feature vector extracted from the original image by a VAE encoder.

[0040] The model first performs an iterative denoising process in the latent feature space, capturing the global scene structure through a multi-scale attention mechanism and generating fine texture details using local convolutional layers, ensuring that the filled content maintains a high degree of semantic and textural consistency with the known regions. The number of denoising iterations is typically set to 50 steps, and the DDIM sampler is used to accelerate inference. After restoration, the image is restored to its original resolution using a VAE decoder, and Poisson fusion is applied to the edges of the restored region to eliminate stitching artifacts between the old and new regions, ensuring the realism and visual coherence of the generated image.

[0041] The entire viewpoint alignment pipeline operates in batch processing mode, processing 32 frames of images per batch. The generated viewpoint alignment images are sent to the data storage module and updated in real time as visual observation input for human demonstration data during policy training, forming a continuous robot equivalent viewpoint image data stream. This effectively solves the problem of visual distribution offset caused by the fixed camera height of traditional handheld devices, which cannot adapt to the short stature of humanoid robots, as well as the problem of difficulty in modeling dynamic viewpoint changes introduced by whole-body movements.

[0042] S4. Perform motion alignment processing on human motion data and robot teaching data by constructing a hierarchical and decoupled unified motion space; It should be noted that this step aims to solve the core problem of the difficulty in joint training of heterogeneous human-machine motion representations. By decomposing human whole-body motion into three functionally independent and semantically clear subspaces, cross-morphological motion semantic alignment is achieved, avoiding motion semantic confusion caused by differences in limb proportions, mismatch of degrees of freedom, and reference frame drift.

[0043] Furthermore, human motion data specifically includes whole-body joint posture data, fine hand posture data, and pelvic movement trajectory data.

[0044] Furthermore, step S4 specifically includes: S41. By extracting human wrist pose data from whole-body joint pose data, the human upper limb manipulation is mapped to the six-degree-of-freedom relative pose increment of the target robot. Furthermore, step S41 specifically includes: Extract the three-dimensional position coordinates and rotation quaternions of the wrist joint in the pelvic coordinate system from the whole-body joint posture data; The three-dimensional position coordinates are smoothed by a preset filtering parameter. The rotation quaternion is converted into a rotation vector through logarithmic mapping. After smoothing in the vector space according to preset filtering parameters, it is restored to a rotation quaternion through exponential mapping. Together with the smoothed and filtered three-dimensional position coordinates, it forms a wrist posture sequence. After downsampling the wrist pose sequence to the target robot's control frequency, the position and rotation differences between adjacent frames are calculated to form the six-degree-of-freedom relative pose increment of the target robot.

[0045] It should be noted that this sub-step maps the fine manipulation of the human upper limbs to the six degrees of freedom relative pose increment of the target robot.

[0046] The system first extracts the three-dimensional position coordinates and rotation quaternions of the wrist joint in the pelvic coordinate system from the whole-body joint posture data. Choosing the pelvic coordinate system instead of the world coordinate system as the reference system can eliminate the influence of global position drift, which is in line with the human kinematics control strategy centered on the trunk, and makes the motion representation more robust.

[0047] The three-dimensional position coordinates are smoothed by applying a Savitzky-Golay filter with a filter window length of 5 frames (corresponding to 50 milliseconds, calculated at a raw sampling rate of 100 Hz) and a polynomial order of 3. This configuration effectively suppresses high-frequency tracking noise while preserving motion trend characteristics.

[0048] For rotated quaternions, in order to avoid the multivalue problem caused by direct interpolation in the quaternion space (i.e., q and -q represent the same rotation), the system adopts a smoothing strategy on the SO(3) manifold: the rotated quaternion is converted into a rotated vector (dimension 3) through logarithmic mapping, the Savitzky-Golay filter with the same parameters is applied in the vector space, and then the rotated quaternion is restored through exponential mapping, thereby ensuring the geometric correctness of rotation smoothing.

[0049] The processed wrist pose sequence is downsampled to 20 Hz to match the robot's control frequency. The six-DOF relative pose increments of the target robot are calculated by determining the position and rotation differences between adjacent frames. Position increments are expressed in meters, typically ranging from ±0.05 meters per control cycle; rotation increments are expressed in axis angles, typically ranging from ±0.2 radians per control cycle. The core advantage of this relative representation lies in its shape independence: functionally equivalent tasks (such as grasping a cup) exhibit similar increment patterns in the local coordinate system under different arm lengths, thus avoiding the problem of absolute position distribution offset caused by differences in human-robot arm lengths.

[0050] S42. By performing velocity analysis and threshold quantization on pelvic trajectory data, human lower limb navigation is mapped into discrete motion commands for the target robot. Furthermore, step S42 specifically includes: Extract the three-dimensional world coordinate sequence of pelvic joints from the pelvic trajectory data and perform smoothing filtering; The instantaneous velocity vector of the pelvic joints on the horizontal plane and the rate of change of height in the vertical direction are calculated using the central difference method. The instantaneous velocity vector and the rate of change of height are quantized into discrete motion commands by using a first preset threshold. The discrete motion commands include forward and backward movement, left and right translation, rotation, and rise and fall.

[0051] It should be noted that this sub-step maps human lower limb movement behavior into discrete motion commands compatible with the robot control interface.

[0052] The system first extracts the three-dimensional world coordinate sequence of pelvic joints from the pelvic trajectory data, and then applies Savitzky-Golay filtering for smoothing to eliminate the accumulated noise of the inertial tracker.

[0053] The instantaneous velocity vector (including forward and lateral components) of the pelvic joint points on the horizontal plane and the rate of change of height in the vertical direction are calculated using the central difference method. The difference step size is set to 3 frames (30 milliseconds), which balances temporal resolution and sensitivity to high-frequency noise. Simultaneously, the system calculates the instantaneous orientation angle, defined as the angle between the forward vector of the pelvic local coordinate system and the X-axis of the world coordinate system. A continuity constraint is applied when calculating the orientation difference between adjacent frames, and orientation jumps of ±180 degrees are prevented by judging the direction of angle change.

[0054] Once continuous speed and orientation change rates are obtained, the system converts them into discrete commands using preset quantization rules: forward speed is mapped to three states—forward, backward, or stationary—based on a threshold of ±0.15 meters per second; lateral speed is mapped to moving left, moving right, or stationary based on a threshold of ±0.10 meters per second; orientation change rate is mapped to turning left, turning right, or going straight based on a threshold of ±0.3 radians per second; and vertical height change rate is mapped to crouching, standing, or holding position based on a threshold of ±0.08 meters per second.

[0055] These discrete states strictly correspond to the VR controller instruction set of the robot's teleoperation interface, ensuring complete consistency between human demonstration data and robot data in terms of action semantics. At the same time, discretization reduces the complexity of policy learning, transforming the lower limb navigation task into a classifiable decision problem.

[0056] S43. Extract binary opening and closing commands from the fine hand posture data and map the human gripper operation to the target robot's binary opening and closing state.

[0057] Furthermore, step S43 specifically includes: Extract finger joint sequences from fine hand posture data and perform smoothing filtering; The curvature value at the midpoint of the quadratic polynomial curve fitted to the finger joints of each finger is calculated based on the finger joint sequence, and the average curvature of the five fingers is used as the scalar of the overall bending degree of the hand. The curvature scalar is binarized into a closed or open state by using a second preset threshold.

[0058] It should be noted that this sub-step maps human hand grasping behavior to the binary opening and closing state of the robot gripper.

[0059] The system extracts finger joint sequences from fine hand posture data, first applying low-pass filtering to remove high-frequency jitter, and then performing Savitzky-Golay smoothing.

[0060] To accurately identify grasping intent, the system calculates the curvature characteristics of each finger: a quadratic polynomial curve is fitted to three joints of the finger, and the curvature value at the midpoint of the curve is calculated. This feature is sensitive to the overall curvature of the finger and robust to local joint noise.

[0061] The average curvature values ​​of the five fingers are used to obtain a scalar of the overall hand curvature. This scalar is then binarized into a closed (grasping) or open (releasing) state using a preset threshold of 0.15. This threshold is determined through statistical analysis of the curvature distribution of human grasping actions in the labeled dataset, ensuring that 95% of grasping events are correctly identified while maintaining a low false detection rate.

[0062] This curvature-based binarization method effectively solves the semantic mapping ambiguity between the 26 degrees of freedom of the human hand and the 1 degree of freedom of the robot gripper, transforming the gripper orientation problem, which cannot be reliably inferred from the first-person perspective, into a robust problem of grasping intent recognition.

[0063] Ultimately, the upper limb six-degree-of-freedom increment (6-dimensional), lower limb discrete commands (8-dimensional one-hot encoding), and gripper binary state (1-dimensional) are concatenated into a unified motion vector (15-dimensional). All human demonstration data and robot teleoperation data are converted into this unified format and sent to the policy training module for end-to-end learning.

[0064] S5. Based on a unified timestamp synchronization mechanism, the first-person image sequence after viewpoint alignment and the equivalent viewpoint image data stream, as well as the human action data after action alignment and the robot teaching data, are fused together to construct a human-machine hybrid training dataset. It should be noted that this step aims to address the alignment and fusion issues of multi-source heterogeneous data in both the temporal and modal dimensions, construct a structured cross-morphological training dataset, and provide a data foundation for the joint training of human demonstration data and robot teaching data.

[0065] The system employs a distributed time alignment mechanism based on hardware clock synchronization. Through a network time protocol, it ensures that the system clock error of the VR device, tracking system, and camera is less than 1 millisecond, guaranteeing time reference consistency from the source. All hardware data is appended with a UNIX timestamp during generation, ensuring data comparability between different sensors within the same timeframe. The data fusion node maintains a sliding time window buffer with a window length of 100 milliseconds (covering the maximum expected latency). When a new camera image frame (33-millisecond period) is received, the node retrieves the body pose data (100 Hz) and hand pose data (30 Hz) with the closest timestamps from the buffer, combining the three into a multimodal data sample.

[0066] If data for a certain modality is missing or delayed by more than 50 milliseconds, the sample at that time step is marked as invalid and discarded. Through this quality control mechanism, the system maintains a data synchronization success rate of over 95% throughout the entire acquisition process.

[0067] The synchronized data frames are written to storage in HDF5 layered data format. The file structure includes image groups, pose groups, and metadata groups, supporting subsequent random access and efficient reading.

[0068] During the data fusion phase, the first-person image sequence after viewpoint alignment is matched and integrated with the equivalent viewpoint image data stream, and the human action data after action alignment is matched with the robot teaching data according to a unified timestamp.

[0069] The human demonstration data includes: RGB images of the robot's equivalent viewpoint (1920×1080 resolution) after depth estimation and diffusion restoration, six-DOF relative pose increments of the upper limbs (6-dimensional continuous values), discrete motion commands of the lower limbs (8-dimensional one-hot encoding), binary states of the gripper (1-dimensional), and 512-dimensional semantic embeddings for natural language task descriptions.

[0070] Robot teleoperation data includes: raw acquired RGB images, six-DOF pose of the end effector, discrete motion commands, gripper status, and 29-dimensional body perception status (joint angles).

[0071] When constructing the human-machine hybrid training dataset, a preset sampling ratio is adopted (usually the ratio of human data to robot data is 3:1 to 5:1). This fully utilizes the advantages of the large scale and diversity of human data to cover a wide range of scenarios and task types, while retaining enough robot data to ensure that the kinematic feasibility constraints are strictly met.

[0072] Simultaneously, data augmentation strategies were implemented, including introducing random perturbations (position ±0.1 meters, rotation ±15 degrees), color jitter, Gaussian noise, and random occlusion during viewpoint alignment, to improve the model's robustness to sensor noise and environmental changes, ultimately forming a high-quality, highly consistent, cross-morphological hybrid training dataset.

[0073] S6. The vision-language-action model is trained collaboratively using a human-machine hybrid training dataset to generate a full-body motion operation strategy for a humanoid robot.

[0074] Please see Figure 2 , Figure 2 This is a schematic diagram of the structure of a humanoid robot motion training device according to an embodiment of this application, as shown below. Figure 2 As shown, specifically: The first acquisition unit 201 is used to acquire human demonstration data based on the wearable data acquisition system. The human demonstration data includes first-person image sequences, human action data, and natural language task descriptions. The second acquisition unit 202 is used to acquire robot teaching data through the robot teleoperation system. The robot teaching data includes robot body state, end effector pose, gripper state and discrete motion commands. The visual alignment unit 203 is used to convert a first-person image sequence into an equivalent viewpoint image data stream of the target robot using a visual alignment method based on depth estimation and 3D reprojection. The motion alignment unit 204 is used to perform motion alignment processing on human motion data and robot teaching data by constructing a hierarchical and decoupled unified motion space. Training dataset construction unit 205 is used to perform multimodal data fusion based on a unified timestamp synchronization mechanism, combining the first-person image sequence after viewpoint alignment with the equivalent viewpoint image data stream, and the human action data after action alignment with the robot teaching data to construct a human-machine hybrid training dataset. The strategy generation unit 206 is used to collaboratively train the vision-language-action model using a human-machine hybrid training dataset to generate a full-body motion operation strategy for a humanoid robot.

[0075] Another embodiment of the present invention provides a humanoid robot motion training device, such as... Figure 3 As shown, device 10 includes: One or more processors 110 and memory 120, Figure 3 The following description uses a processor 110 as an example. The processor 110 and the memory 120 can be connected via a bus or other means. Figure 3 Taking the example of a connection between China and Israel via a bus.

[0076] Processor 110 is used to perform various control logics of device 10, and can be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), microcontroller, ARM (Acorn RISC Machine) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination of these components. Furthermore, processor 110 can also be any conventional processor, microprocessor, or state machine. Processor 110 can also be implemented as a combination of computing devices, such as a combination of DSP and microprocessor, multiple microprocessors, one or more microprocessors combined with DSP and / or any other such configuration.

[0077] The memory 120, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions corresponding to the method for constructing the multilingual phoneme representation model in this embodiment of the invention. The processor 110 executes various functional applications and data processing of the device 10 by running the non-volatile software programs, instructions, and units stored in the memory 120, thereby implementing the method for constructing the multilingual phoneme representation model in the above-described method embodiment.

[0078] The memory 120 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created according to the use of the device 10. Furthermore, the memory 120 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 120 may optionally include memory remotely located relative to the processor 110, and these remote memories may be connected to the device 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0079] One or more units are stored in memory 120, and when executed by one or more processors 110, perform the following steps: S1. Human demonstration data is acquired based on a wearable data acquisition system. The human demonstration data includes first-person image sequences, human action data, and natural language task descriptions. S2. Obtain robot teaching data through the robot teleoperation system. The robot teaching data includes robot body state, end effector pose, gripper state, and discrete motion commands. S3. A visual alignment method based on depth estimation and 3D reprojection converts a first-person image sequence into an equivalent viewpoint image data stream for the target robot. S4. Perform motion alignment processing on human motion data and robot teaching data by constructing a hierarchical and decoupled unified motion space; S5. Based on a unified timestamp synchronization mechanism, the first-person image sequence after viewpoint alignment and the equivalent viewpoint image data stream, as well as the human action data after action alignment and the robot teaching data, are fused together to construct a human-machine hybrid training dataset. S6. The vision-language-action model is trained collaboratively using a human-machine hybrid training dataset to generate a full-body motion operation strategy for a humanoid robot.

[0080] This invention provides a non-volatile computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are executed by one or more processors, they implement any one of the implementation methods of the XX method described in the above embodiments.

[0081] As examples, non-volatile storage media can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) as external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms such as synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The memory components or memories disclosed in the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.

[0082] This application provides a method, apparatus, device, and medium for training humanoid robot movements. First, a wearable data acquisition system replaces traditional remote-operated hardware, overcoming the limitations of laboratory settings and enabling efficient data acquisition in field environments. Second, based on depth estimation and 3D reprojection technology, human first-person perspective images are converted to the robot's camera height, and generative repair fills in occlusion gaps, resolving the visual distribution mismatch caused by height differences between humans and robots. Third, a hierarchical, decoupled, unified motion space is constructed, mapping the upper limbs to relative pose increments, the lower limbs to discrete motion primitives, and the gripper to binary states, avoiding semantic confusion caused by morphological differences. Finally, human-robot data fusion is achieved through timestamp synchronization, enabling the robot to achieve zero-sample generalization deployment in environments without prior field data collection, significantly improving the generalization and reliability of the strategy.

[0083] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0084] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0085] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0086] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0087] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0088] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0089] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0090] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

[0091] It should be noted that if any software tools or components not belonging to our company appear in the embodiments of this application, they are merely for illustrative purposes and do not represent actual use.

Claims

1. A method for training humanoid robot movements, characterized in that, include: S1. Obtain human demonstration data based on a wearable data acquisition system, wherein the human demonstration data includes first-person image sequences, human action data, and natural language task descriptions; S2. Obtain robot teaching data through the robot teleoperation system. The robot teaching data includes robot body state, end effector pose, gripper state, and discrete motion commands. S3. A visual alignment method based on depth estimation and 3D reprojection converts the first-person image sequence into an equivalent viewpoint image data stream of the target robot. S4. Perform action alignment processing on the human action data and the robot teaching data by constructing a hierarchical and decoupled unified action space; S5. Based on a unified timestamp synchronization mechanism, the first-person image sequence after viewpoint alignment is fused with the equivalent viewpoint image data stream, and the human action data after action alignment is fused with the robot teaching data to construct a human-machine hybrid training dataset. S6. The vision-language-action model is collaboratively trained using the human-machine hybrid training dataset to generate a humanoid robot's full-body motion operation strategy.

2. The humanoid robot motion training method according to claim 1, characterized in that, Step S3 specifically involves: Reconstructing 3D point clouds of a scene based on a monocular depth estimation network, obtaining the 3D point cloud coordinates and validity mask corresponding to each pixel; Calculate the rigid body transformation matrix from the human camera coordinate system to the target robot camera coordinate system. The rigid body transformation matrix includes rotational and translational components. The vertical value of the translational component is used to reflect the height difference between the human and the robot. The three-dimensional point cloud coordinates are batch transformed to the target robot camera coordinate system according to the rigid body transformation matrix, and perspective projection is performed using the intrinsic parameter matrix of the target robot camera to generate a reprojected image with a hole mask. The hole regions in the reprojected image are filled based on the potential diffusion repair model, generating an equivalent viewpoint image data stream of the target robot after viewpoint alignment.

3. The humanoid robot motion training method according to claim 1, characterized in that, The human motion data specifically includes whole-body joint posture data, fine hand posture data, and pelvic movement trajectory data.

4. The humanoid robot motion training method according to claim 3, characterized in that, Step S4 specifically involves: S41. By extracting human wrist posture data from the whole-body joint posture data, the human upper limb operation is mapped to the six-degree-of-freedom relative pose increment of the target robot. S42. By performing velocity analysis and threshold quantization on the pelvic trajectory data, the human lower limb navigation is mapped into discrete motion commands for the target robot. S43. By extracting binary opening and closing instructions from the fine hand posture data, the human gripper operation is mapped to the binary opening and closing state of the target robot.

5. The humanoid robot motion training method according to claim 4, characterized in that, Step S41 specifically involves: Extract the three-dimensional position coordinates and rotation quaternions of the wrist joint in the pelvic coordinate system from the whole-body joint posture data; The three-dimensional position coordinates are smoothed by a preset filtering parameter; The rotation quaternion is converted into a rotation vector by logarithmic mapping. After smoothing in the vector space according to the preset filtering parameters, it is restored to a rotation quaternion by exponential mapping. Together with the smoothed and filtered three-dimensional position coordinates, they form a wrist posture sequence. After downsampling the wrist pose sequence to the target robot's control frequency, the position difference and rotation difference between adjacent frames are calculated to form the six-degree-of-freedom relative pose increment of the target robot.

6. The humanoid robot motion training method according to claim 4, characterized in that, Step S42 specifically involves: Extract the three-dimensional world coordinate sequence of pelvic joints from the pelvic trajectory data and perform smoothing filtering; The instantaneous velocity vector of the pelvic joint in the horizontal plane and the rate of change of height in the vertical direction were calculated using the central difference method. The instantaneous velocity vector and the rate of change of height are quantified into discrete motion commands by using a first preset threshold. The discrete motion commands include forward and backward movement, left and right translation, rotation, and rise and fall.

7. The humanoid robot motion training method according to claim 4, characterized in that, Step S43 specifically involves: Extract the finger joint sequence from the fine hand posture data and perform smoothing filtering; The curvature value at the midpoint of the quadratic polynomial curve fitted to the finger joints of each finger is calculated based on the finger joint sequence, and the average curvature of the five fingers is used as the overall bending scalar of the hand. The curvature scalar is binarized into a closed or open state by using a second preset threshold.

8. A humanoid robot motion training device, characterized in that, include: The first acquisition unit is used to acquire human demonstration data based on a wearable data acquisition system. The human demonstration data includes first-person image sequences, human action data, and natural language task descriptions. The second acquisition unit is used to acquire robot teaching data through the robot teleoperation system. The robot teaching data includes robot body state, end effector pose, gripper state, and discrete motion commands. A visual alignment unit is used to convert the first-person image sequence into an equivalent viewpoint image data stream of the target robot using a visual alignment method based on depth estimation and 3D reprojection. The action alignment unit is used to perform action alignment processing on the human action data and the robot teaching data by constructing a hierarchical and decoupled unified action space. The training dataset construction unit is used to perform multimodal data fusion based on a unified timestamp synchronization mechanism, combining the first-person image sequence after viewpoint alignment with the equivalent viewpoint image data stream, and the human action data after action alignment with the robot teaching data, to construct a human-machine hybrid training dataset. The strategy generation unit is used to collaboratively train the vision-language-action model using the human-machine hybrid training dataset to generate a full-body motion operation strategy for the humanoid robot.

9. A humanoid robot motion training device, characterized in that, The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the humanoid robot motion training method according to any one of claims 1-7 according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program code for executing the humanoid robot motion training method according to any one of claims 1-7.

Images