Robot whole body control method and system based on real-time inertial motion capture and reinforcement learning

Abstract

The application discloses a kind of robot whole body control method and system based on real-time inertial motion capture and reinforcement learning, belong to robot control field.The system includes the motion capture module, data processing and forward kinematics module, motion redirection and optimization module, general motion strategy control module and real-time control and driving module in turn link, form end-to-end closed loop control flow line by shared memory and high-speed communication protocol.The method is realized high-quality mapping from human posture to robot model by collecting human joint data through inertial motion capture equipment, pose solution, non-uniform scaling and two-stage differential inverse kinematics optimization, and generates stable joint instructions combined with reinforcement learning strategy trained by PPO algorithm.The application solves the problems of lack of feedback loop, joint deadlock and contradiction between realism and stability in the prior art, has the characteristics of strong robustness and high expansibility, and can be widely applied to remote operation, motion reproduction and other scenes.

Description

Technical Field

[0001] This invention relates to the field of robot control and human-computer interaction technology, specifically to a robot whole-body control method and system based on real-time inertial motion capture and reinforcement learning. Background Technology

[0002] Motion control of full-size, multi-degree-of-freedom humanoid robots has always been a major challenge and research hotspot in the field of robotics. Teleoperation technology based on motion capture can endow robots with the dexterity and complex movement capabilities that humans possess when performing tasks, making it an effective technical approach for robots to complete complex motion tasks. However, existing motion capture control schemes still face several technical bottlenecks in achieving efficient, natural, and stable human-robot interaction: First, they lack a feedback closed-loop mechanism based on the robot's own state. Existing teleoperation systems mostly use human movements as feedforward command inputs, lacking real-time feedback and adjustment capabilities for the robot's own state (such as joint states, body posture, balance, foot contact, etc.). This leads to the robot being prone to instability or movement failure when model errors, external disturbances, or differences in human-robot structure exist, making it difficult to operate stably in real-world environments for extended periods. Second, Euler angle alignment-based solutions can lead to "joint deadlock" at singular joint positions. Existing motion retargeting methods, when mapping human poses to robot joint space, rely on forward kinematics solvers to obtain joint angles, which can easily get stuck in joint angle limits or cause solution failures or abrupt changes near singular points—a phenomenon known as "joint deadlock." This can cause robot movements to stutter, jitter, or even become unstable, making it impossible for robots to safely reproduce many smooth and continuous human movements. Third, existing systems face an irreconcilable contradiction between "fidelity of imitation" and "motion stability." To ensure stability, systems often tend to use conservative motion trajectories, while pursuing highly realistic imitation increases latency and sacrifices the real-time nature of motion capture to the robot. How to design a control architecture that can accurately reproduce human intentions while ensuring stable robot execution in real physical environments through the underlying controller remains a key problem that current technology has not adequately solved. In summary, how to construct a new control system that can both maintain high-fidelity reproduction of human intentions and ensure the dynamic stability of robots in real environments, thereby achieving natural, safe, and low-latency human-computer interaction, is a key issue that current technology urgently needs to overcome. Summary of the Invention

[0003] The purpose of this invention is to provide a robot whole-body control method and system based on real-time inertial motion capture and reinforcement learning, so as to solve the pain points of the prior art such as lack of feedback closed loop, joint deadlock and contradiction between realism and stability.

[0004] The technical solution of the present invention to solve the above-mentioned technical problems is as follows:

[0005] A humanoid robot full-body control system based on inertial motion capture and reinforcement learning includes a data processing and execution pipeline that forms an end-to-end closed-loop control through shared memory and a high-speed communication protocol. This pipeline comprises five core modules connected sequentially:

[0006] Motion capture module: configured to collect raw rotation data of the operator's whole body joints through the inertial motion capture device Rebocap, the data is represented in quaternion form, establish a data transmission channel and write the skeletal joint rotation information into shared memory;

[0007] Data processing and forward kinematics module: configured to extract joint local rotation information from shared memory and convert it into a unified rotation matrix format; based on the human skeleton topology, it recursively calculates the global pose from the root node to the end effector using the principle of forward kinematics to obtain the complete pose information in the motion capture coordinate system.

[0008] Motion redirection and optimization module: Configured to use differential inverse kinematics and constraint optimization methods to map the global human body posture to the target humanoid robot model and generate a reference joint motion sequence that satisfies kinematic constraints;

[0009] General motion strategy control module: configured with a general motion strategy based on reinforcement learning training, receiving reference motion sequences and robot proprioceptive state, and outputting stable and robust joint target angles;

[0010] Real-time control and drive module: configured to send the joint target angle to the robot's main control board via CANFD bus, and generate joint drive torque through the controller to drive the motor to perform motion.

[0011] Employing a pipeline architecture of "acquisition-processing-mapping-optimization-execution," a closed-loop control system is formed, specifically addressing three major pain points of existing technologies: lack of feedback loops, joint deadlock, and the contradiction between realism and stability. Each module performs its specific function while data flows seamlessly, ensuring the continuity of signal transmission and processing. This achieves end-to-end low-latency control, balancing high-fidelity reproduction of human movements with robot dynamic stability, thereby improving the safety and reliability of human-computer interaction.

[0012] In a further proposed solution, the motion capture module is developed based on C++ SDK v3. It calls the `rebocap_ws_sdk_new` interface to create an SDK instance and configures it for Unity right-handed coordinate system output. A WebSocket connection is established via `rebocap_ws_sdk_open` to acquire data from 24 joints conforming to the SMPL specification at a 60Hz frame rate. Data is transmitted via a dual-channel approach using UDP protocol and shared memory, with a data latency of less than 10ms. The VMC port number is configured as 39539. The choice of C++ SDK ensures efficient data processing; the Unity right-handed coordinate system adapts to the robot's kinematic calculation habits; the 60Hz frame rate matches the human motion sampling requirements; the 24 joints in the SMPL specification cover key motion areas; dual-channel transmission eliminates the risk of latency or packet loss from a single channel; and the fixed VMC port ensures communication stability. This achieves high-precision, low-latency acquisition and transmission of human motion data, providing reliable raw input for subsequent modules and avoiding motion reproduction distortion due to data loss or latency.

[0013] In a further embodiment, the data processing and forward kinematics module will use quaternions. Convert to a 3×3 rotation matrix (where w is the real part of the quaternion, Let q be the imaginary part of the quaternion and q be the unit quaternion, satisfying the condition... The conversion formula is: The global pose is calculated based on the skeleton topology recursion. The global rotation recursion formula for non-root node i is as follows: (in Let be the global rotation matrix for joint i. Let be the global rotation matrix of the parent node of joint i. (where i is the local rotation matrix of joint i), the global position recursive formula is: (in Let be the global position coordinates of joint i. Let be the global position coordinates of the parent node of joint i. (This refers to the fixed bone offset vector of joint i relative to its parent node). It supports coordinate system transformation and outputs unified pos (position coordinates) and quat (quaternion) data in a relative coordinate system with the pelvis as the origin. Quaternion-to-rotation matrix conversion avoids Euler angle calculation problems in singular positions, and unit quaternions ensure the standardization of rotation data. Forward kinematics recursion is based on skeleton topology, ensuring the logical consistency of global pose calculation. Unified coordinate system output provides standardized input for subsequent motion retargeting. This achieves standardized processing of human pose data, ensuring the accuracy and consistency of global pose calculation, providing high-quality pose input for subsequent modules, and reducing mapping errors.

[0014] In a further embodiment, the motion redirection and optimization module uses a mapping relationship (in It is a collection of human joints. Joint matching is achieved for a robot joint set, and rotational offset is applied. (Restless attitude alignment rotation matrix) Completes resting attitude alignment using a locally non-uniform scaling matrix. (Where b represents a body part) are the scaling factors of body part b in the X, Y, and Z axes, respectively. This approach matches the structural differences between humans and robots, generating joint sequences through a two-stage optimization process: "initial feasible solution solving - full constraint refinement." Motion artifacts are eliminated through temporal continuity and ground consistency correction. Joint matching establishes the human-robot correspondence, resting posture alignment eliminates initial posture deviations, and local non-uniform scaling specifically addresses the differences in human-robot size ratios. The two-stage optimization first obtains feasible solutions to avoid local optima, then refines the optimization to satisfy multiple constraints. Temporal and ground consistency correction ensures the physical rationality of the motion sequence. This effectively solves the "joint deadlock" problem of traditional forward mapping, reduces motion artifacts such as foot slippage and ground penetration, and generates smooth, executable robot joint sequences.

[0015] In a further embodiment, the general motion strategy control module is trained using the Proximal Policy Optimization (PPO) algorithm. The training dataset contains 18 MoCap-OP action sequences (15 dance sequences, 1 walking sequence, 1 lying / standing sequence, and 1 other sequence). The training process incorporates simulated scenarios such as various terrains (flat ground, slopes, steps, etc.) and random external force disturbances, with at least 20,000 iterations. The input includes robot joint angles, joint velocities, foot contact states, and a reference action target. The PPO algorithm balances training stability and sample utilization, covering common human movement patterns with multiple action sequences, and multi-scenario simulation training enhances the strategy's resilience to disturbances. The input includes the robot's body state and the reference target, forming a feedback loop. The strategy possesses dynamic equilibrium maintenance and disturbance resistance capabilities, automatically compensating for model errors and environmental disturbances, ensuring the robot's stability and robustness during complex action execution.

[0016] In a further embodiment, the real-time control and drive module adopts a multi-level architecture of "60Hz command reception - 500Hz intermediate frequency processing - 2000Hz low-level drive," configured with an RK3588 main control board and a PD controller. The expected torque calculation formula for the PD controller is as follows: (in To achieve the desired joint drive torque, This is the proportionality coefficient. The differential coefficients are... For the desired joint position, This refers to the actual joint position. For the desired joint velocity, (actual joint velocity); obtained through the first-order difference formula. (in Let the joint velocity be the velocity of the k-th frame. The joint position in frame k. For the first Frame joint position, The joint velocity is calculated at a sampling time interval, and high-frequency control commands are generated using linear interpolation. A multi-level architecture matches the processing frequencies of different modules, resolving the mismatch between command and drive frequencies. The RK3588 main control board ensures real-time processing capabilities, while the PD controller balances position tracking accuracy and motion smoothness. First-order differential calculation speed and linear interpolation generate high-frequency commands, ensuring the continuity of motion states. This achieves low-latency, high-precision command execution, smooth and stable joint movements, avoids motion stuttering caused by frequency mismatch, and improves the fidelity of robot motion reproduction.

[0017] A method for controlling the whole body of a humanoid robot based on inertial motion capture and reinforcement learning includes the following steps: (1) Motion capture step: The original data of the operator's whole body joints are collected by the Rebocap device at a frequency of 60Hz and transmitted in the form of quaternions via UDP protocol and shared memory with low latency;

[0018] (2) Data processing and forward kinematics steps: convert the original data into a rotation matrix, recursively calculate the global pose and normalize it to a unified coordinate system;

[0019] (3) Motion redirection and optimization steps: The human posture is mapped to the robot reference joint sequence through posture alignment, non-uniform scaling and two-stage differential inverse kinematics optimization;

[0020] (4) General motion strategy control steps: Input the reference sequence and robot body state into the strategy model trained by PPO to generate stable joint target angles;

[0021] (5) Real-time control and drive steps: Commands are sent through the CANFD bus, and the drive torque is calculated by the PD controller to achieve low-latency reproduction of human body movements. The steps are progressively arranged in the order of "data acquisition - processing - mapping - optimization - execution" to form a complete closed loop. Each step addresses the problems of the preceding steps to ensure the continuity of data flow and processing. The process is clear and the steps work together to achieve low-latency, high-fidelity conversion of human body movements from acquisition to robot execution, taking into account both the realism of the movements and the stability of the motion.

[0022] In a further step, the non-uniform scaling strategy described in step (3) sets scale factors for the upper limbs, lower limbs, and trunk respectively. In the two-stage optimization, the first stage aims to obtain an initial solution by matching the rotational continuity of the body chain and the position of the end effector. In the second stage, the objective function is expanded to include the position and rotational errors of all key parts. Damping and priority strategies are introduced near singular points to ensure stable solution. Setting scale factors for different parts accurately matches the differences in human-machine structure. The two-stage optimization first solves the feasibility problem and then improves the accuracy. Damping and priority strategies near singular points avoid solution jumps. The motion redirection effect is further optimized to completely solve the "joint deadlock" problem. The generated joint sequence is kinematically natural and physically reliable.

[0023] In a further step, the training process of the strategy model described in step (4) includes simulated scenarios such as multi-terrain, random external force disturbances, and sensor noise. The reward function is steadily converged through at least 20,000 iterations, enabling the model to autonomously maintain the robot's dynamic balance while tracking reference actions. Multi-scenario simulations cover interference factors that may be encountered in practical applications, and extensive iterative training ensures the strategy fully learns stable control laws. The convergence of the reward function verifies the reliability of the strategy. The model has strong generalization and anti-disturbance capabilities, adapting to complex and ever-changing application environments and ensuring stable robot operation in different scenarios.

[0024] In a further step, step (5) uses independent high real-time processes to achieve data synchronization and decoupling at different frequency levels. A motion state reconstruction algorithm is used to derive the position and velocity information required for high-frequency control from low-frequency pose data, ensuring smooth and stable joint movement with a total latency of less than 10ms. Independent high real-time processes avoid mutual interference between different frequency levels, and the motion state reconstruction algorithm compensates for the deficiencies in low-frequency data, ensuring the accuracy of high-frequency drive. This further reduces system latency, improves the smoothness and continuity of joint movement, achieves near real-time reproduction of human movements, and optimizes the human-computer interaction experience.

[0025] The present invention has the following beneficial effects:

[0026] The technical solution of this invention constructs a full-process control framework of "motion capture perception → posture calculation → retargeting optimization → feedback-based reinforcement learning control → real-time drive execution", which has the following significant advantages compared with the prior art: It realizes a complete, real-time, and low-latency human-computer interaction closed loop: by adopting shared memory and high-speed data channels, it realizes the millisecond-level transmission of motion capture data and sends the robot's body state back to the strategy module to form a real-time closed loop, effectively reducing the risks of motion instability, posture deviation and environmental collision, and improving the safety and reliability of human-computer interaction.

[0027] Effectively solves the "joint deadlock" problem in traditional forward kinematics retargeting: Through the motion retargeting and constraint optimization method of differential inverse kinematics, the robot joint posture is continuously solved under multiple constraints. Near singular poses, the solution is kept stable by damping and priority constraints, avoiding sudden changes in joint angles, solution failures or joint jamming.

[0028] Balancing motion realism and stability: By combining structural consistency constraints of human posture with a feedback-based reinforcement learning-based stabilization control strategy, the robot can reproduce human motion details with high fidelity while meeting its own dynamic stability requirements, without sacrificing the naturalness of the motion for stability.

[0029] Reinforcement learning strategies enhance the robustness and generalization ability of complex actions: Through large-scale multi-scene training, RL strategies can automatically compensate for model inaccuracies, terrain disturbances and sensing errors, enabling the robot to remain stable when performing complex actions such as fast movement, bending and rotating, and is more adaptable to the environment than traditional controllers.

[0030] The system is highly scalable and suitable for multiple scenarios: its modular architecture supports seamless replacement of different motion capture devices, robot models and strategy networks, and can be widely used in various scenarios such as remote teleoperation, motion acquisition, imitation learning training and natural interaction, with significant engineering value and application prospects. Attached Figure Description

[0031] Figure 1 A flowchart illustrating the full-body control process of a humanoid robot based on inertial motion capture and reinforcement learning;

[0032] Figure 2 This is a schematic diagram of Rebocap inertial motion capture.

[0033] Figure 3 Motion redirection for the Nix1 robot;

[0034] Figure 4 To train a multi-policy general policy reward graph 20,000 times;

[0035] Figure 5 To dynamically capture Tracker strategy deployment. Detailed Implementation

[0036] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0037] System Overall Architecture

[0038] The humanoid robot full-body control system based on inertial motion capture and reinforcement learning proposed in this invention is fundamentally about constructing an end-to-end closed-loop control system from human intention perception to stable robot execution. The system's five core modules execute in a strict sequence: "motion capture module → data processing and forward kinematics module → motion redirection and optimization module → general motion strategy control module → real-time control and drive module." The output of one module serves as the input for the next, achieving seamless conversion from raw motion data to robot joint torques. The overall architecture is as follows: Figure 1 As shown.

[0039] Specific implementation of each module

[0040] The motion capture module uses the Rebocap inertial motion capture system as the core data acquisition device, and uses its software development kit (SDK) to achieve real-time capture and data transmission of human motion.

[0041] The system is developed based on C++ SDKv3. It calls the rebocap_ws_sdk_new interface to create an SDK instance and configures it to output local coordinates in Unity's right-handed coordinate system. This ensures that all rotation data are relative to the parent skeleton coordinate system of the T-pose, which meets the requirements of robot kinematics calculation.

[0042] A WebSocket client connection is established using rebocap_ws_sdk_open, and a real-time data callback for a 60Hz frame rate is registered using rebocap_ws_sdk_set_pose_msg_callback.

[0043] The collected skeletal data strictly follows the SMPL specification's naming system of 24 joints, covering all major joints of the human body from "Pelvis" (pelvis) to "R_Hand" (right hand). The rotation information of each joint is represented in quaternion form, and the displacement unit is metric.

[0044] At the data output level, the module constructs a low-latency transmission channel based on the UDP protocol. The parsed skeletal rotation data is broadcast to a designated network address via local area network Wi-Fi at a frequency of 60Hz. Simultaneously, based on a predefined joint mapping table, the local rotation information of each joint is written into the shared memory data area, and fast data retrieval is achieved by setting a unique joint index. The shared memory mechanism effectively eliminates the serialization overhead of inter-process communication, ensuring that motion data is read by subsequent processing modules with an extremely low latency of less than 10ms. The specific parameter configuration of the module is as follows: Figure 2 As shown.

[0045] Data Processing and Forward Kinematics Module

[0046] This module is responsible for unifying the local rotation data output by the motion capture device into a standard rotation matrix format, and constructing the global pose (position, quaternion) of the entire human skeleton based on the topology of the human skeleton. The core includes three parts: rotation format conversion, forward kinematics modeling, and coordinate transformation.

[0047] Rotational format standardization conversion: The joint poses output by the motion capture equipment are in quaternion format. The formal representation is given by w, where w is the real part. The part is imaginary, and q is a unit quaternion (satisfying...) Convert it to a rotation matrix. The calculation formula is: The transformed rotation matrix is ​​used for subsequent forward kinematics recursive calculations.

[0048] Forward kinematic modeling based on skeleton topology: Assume the human skeleton is a tree with a known topology. The position of the root node (pelvis) in the motion capture coordinate system is given by the motion capture system, denoted as […]. Its global rotation matrix has an initial value equal to its local rotation matrix. For non-root node joint i (parent node is...) Its global pose is derived recursively using the following formula: Global rotation recursion: Global position recursion: Through the above recursion, the global pose of all joints can be obtained. .

[0049] Coordinate system transformation: To adapt to robot control, the global pose of the joints in the motion capture coordinate system is transformed. Transform to the target control coordinate system using the following formula: ,in, These represent the rotation and translation of the target coordinate system relative to the motion capture coordinate system, respectively.

[0050] Motion Redirection and Optimization Module

[0051] This module is the core of bridging the human-machine "embodiment gap." Through local non-uniform bone segment scaling, resting posture alignment, and multi-objective constraint optimization using differential inverse kinematics, it achieves high-quality redirection of human motion to robot motion, effectively suppressing problems such as foot slippage, ground penetration, and joint lockup. Joint matching relationships are as follows: Figure 3 As shown. (1) Matching of key body parts: based on mapping relationship ( It is a collection of human joints. (This is a robot joint assembly), which matches key human body parts (pelvis, spine, hands and feet, etc.) with corresponding robot parts one by one.

[0052] Resting posture alignment: by applying rotational offset To align the orientation of key parts of the human body in its resting posture (T-Pose) with the robot's zero-position posture, the local rotation matrix of joint i of the human body after alignment is: .

[0053] Local non-uniform bone segment scaling: Set a 3D scaling matrix for different body parts b such as the upper limb, lower limb, and trunk. ( The target position of the key part b after scaling is: The root node is simplified to ,in This is the global scaling factor.

[0054] Two-stage optimization solution: The first stage aims to achieve approximate matching of the body chain rotational continuity and the end effector position, obtaining an initial feasible solution through rapid iteration; the second stage, based on the initial solution, expands the objective function to include the position and rotational errors of all key components, optimizing the objective function as follows: ,in, These are the weighting coefficients for the rotation and position error terms, respectively. For rotation difference operators, This represents the set of joint mapping pairs for the end effector. It is solved iteratively using a differential inverse kinematics method, and after each update, the mapping is projected onto the joint constraint. .

[0055] Temporal continuity and ground consistency correction: The motion sequence is processed frame by frame, using the optimized solution of the previous frame as the initial value of the current frame to ensure smooth and coherent joint sequences; the minimum sequence of robot foot height is calculated. By vertical translation amount Correct the vertical coordinates of the root node to eliminate floating and ground penetration.

[0056] General motion strategy control module

[0057] This module constructs a general whole-body motion control strategy based on deep reinforcement learning. Through training with a large number of action sequences and multi-scenario physical simulations, it achieves unified control and dynamic balance maintenance of multiple action modes.

[0058] The training dataset contains 18 MoCap-OP action sequences cleaned and aligned by BeyondMimic, which are divided into four categories according to semantics: 15 dance sequences (to improve the ability to model complex whole-body coordinated movements), 1 walking sequence (to supplement the learning of periodic lower limb movement stability), 1 lying down / standing up sequence (to enhance the ability to transition between large postures and balance recovery), and 1 other sequence (to improve the robustness of non-periodic movements).

[0059] The Proximal Policy Optimization (PPO) algorithm was used for training. The training process incorporated simulated scenarios such as diverse terrains (flat land, slopes, steps, etc.), random external force disturbances, random contact loss, reference action randomization, and random sensor noise. The number of iterations was no less than 20,000. The convergence curve of the training reward function is shown below. Figure 4 As shown, this demonstrates that the strategy has learned consistent and reliable whole-body control.

[0060] The strategy input is the robot's body state (joint angles, speed, contact state, etc.) and the reference action target. The output is the target joint angle or joint displacement increment that can be directly executed. It can autonomously maintain the robot's center of gravity stability while tracking the reference action and resist external force disturbances and terrain uncertainties.

[0061] Table 1: Composition of multi-policy trajectories

[0062]

[0063] Real-time control and drive module

[0064] This module, as the final execution unit of the control chain, adopts a hierarchical processing architecture and a multi-rate control mechanism to achieve high-frequency, low-latency whole-body motion control. The strategy deployment process is as follows: Figure 5 As shown.

[0065] The module adopts a multi-level architecture of "60Hz command reception - 500Hz intermediate frequency processing - 2000Hz underlying driver": the command reception layer runs on the RK3588 main control board, receives the target joint angle command at a frequency of 60Hz via UDP protocol and writes it into the shared memory command pool; the intermediate frequency control layer reads the latest command from the shared memory at a frequency of 500Hz; the underlying servo drive layer sends drive signals to the joint motor driver at a frequency of 2000Hz via CANFD bus.

[0066] To address the frequency mismatch problem, a motion state reconstruction algorithm is employed: using the first-order difference formula... ( The joint velocity is calculated, and then high-frequency control commands are generated using linear interpolation to avoid uneven motion. The underlying layer uses a PD controller to achieve precise joint position tracking. The desired torque calculation formula is as follows: ,in, To achieve the desired joint position and velocity, The encoder provides feedback on the actual joint position and speed.

[0067] Implementation effect verification

[0068] Through the collaborative work of the above modules, the system has achieved the following key indicators:

[0069] Data transmission latency: The latency from data capture to robot joint execution is less than 10ms.

[0070] Motion reproduction quality: It can reproduce complex human movements such as dancing, walking, lying down / getting up with high fidelity, and the movements are smooth and without lag;

[0071] Stability: In scenarios with diverse terrains and random external forces, the robot can continuously maintain dynamic balance without instability or joint lock-up.

[0072] Robustness: It has a good ability to compensate for model errors and sensor noise, and can adapt to humanoid robot models of different sizes and degrees of freedom.

[0073] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A robot whole-body control system based on real-time inertial motion capture and reinforcement learning, characterized in that, include, Motion capture module: configured to collect raw rotation data of the operator's joints throughout the body through the inertial motion capture device Rebocap, the data being represented in quaternion form, establishing a data transmission channel and writing the skeletal joint rotation information into shared memory; Data processing and forward kinematics module: configured to extract joint local rotation information from shared memory and convert it into a unified rotation matrix format; based on the human skeleton topology, it recursively calculates the global pose from the root node to the end effector using the principle of forward kinematics to obtain the complete pose information in the motion capture coordinate system. Motion redirection and optimization module: Configured to use differential inverse kinematics and constraint optimization methods to map the global human body posture to the target humanoid robot model and generate a reference joint motion sequence that satisfies kinematic constraints; General motion strategy control module: configured with a general motion strategy based on reinforcement learning training, receiving reference motion sequences and robot proprioceptive state, and outputting stable and robust joint target angles; Real-time control and drive module: configured to send the joint target angle to the robot main control board via CANFD bus, and generate joint drive torque through the controller to drive the motor to perform motion; The motion redirection and optimization module uses a mapping relationship To achieve joint matching, where It is a collection of human joints. A collection of robot joints; Apply rotation offset To achieve resting pose alignment, a locally non-uniform scaling matrix was used. Matching human-machine structural differences, where b represents a body part. These are the scaling factors for body part b in the X, Y, and Z axes, respectively. The joint sequence is generated through a two-stage optimization process of "initial feasible solution solving - full constraint refinement", and motion artifacts are eliminated after time continuity and ground consistency correction.

2. The robot whole-body control system based on real-time inertial motion capture and reinforcement learning according to claim 1, characterized in that, The motion capture module calls the rebocap_ws_sdk_new interface to create an SDK instance and configure it for Unity right-handed coordinate system output. It establishes a WebSocket connection through rebocap_ws_sdk_open, and captures 24 joint data conforming to the SMPL specification at a frame rate of 60Hz. The data is transmitted via dual channels of UDP protocol and shared memory, with a data latency of less than 10ms. The VMC port number is configured as 39539.

3. The robot whole-body control system based on real-time inertial motion capture and reinforcement learning according to claim 1, characterized in that, The data processing and forward kinematics module will use quaternions. Convert to a 3×3 rotation matrix ; Where w is the real part of the quaternion. q is the imaginary part of the quaternion, and q is the unit quaternion; The global pose is calculated based on the skeleton topology recursion. The global rotation recursion formula for non-root node i is as follows: ,in Let be the global rotation matrix for joint i. Let be the global rotation matrix of the parent node of joint i. Let be the local rotation matrix of joint i; The global position recursion formula is ,in Let be the global position coordinates of joint i. Let be the global position coordinates of the parent node of joint i. Let i be the fixed bone offset vector of joint i relative to its parent node; It supports coordinate system transformation and outputs unified pos and quat data in a relative coordinate system with the pelvis as the origin. pos is the position coordinate and quat is the quaternion.

4. The robot whole-body control system based on real-time inertial motion capture and reinforcement learning according to claim 1, characterized in that, The general motion strategy control module is trained using the PPO algorithm. The training dataset contains 18 MoCap-OP action sequences. The training process introduces multi-terrain and random external force disturbance simulation scenarios, with no less than 20,000 iterations. The input includes robot joint angles, speeds, contact states, and reference action targets.

5. The robot whole-body control system based on real-time inertial motion capture and reinforcement learning according to claim 1, characterized in that, The real-time control and drive module adopts a multi-level architecture of "60Hz command reception - 500Hz intermediate frequency processing - 2000Hz low-level drive", configured with an RK3588 main control board and a PD controller. It calculates joint speed using first-order differential and generates high-frequency commands using linear interpolation. The PD controller's desired torque calculation formula is as follows: ,in To achieve the desired joint drive torque, This is the proportionality coefficient. The differential coefficients are... For the desired joint position, This refers to the actual joint position. For the desired joint velocity, This represents the actual joint velocity.

6. A robot whole-body control method based on real-time inertial motion capture and reinforcement learning, characterized in that, Includes the following steps: (1) Motion capture steps: The Rebocap device collects raw data of the operator's whole body joints at a frequency of 60Hz and transmits them in quaternion form via UDP protocol with low latency to shared memory; (2) Data processing and forward kinematics steps: convert the original data into a rotation matrix, recursively calculate the global pose and normalize it to a unified coordinate system; (3) Motion redirection and optimization steps: The human posture is mapped to the robot reference joint sequence through posture alignment, non-uniform scaling and two-stage differential inverse kinematics optimization; (4) General motion strategy control steps: Input the reference sequence and robot body state into the strategy model trained by PPO to generate stable joint target angles; (5) Real-time control and drive steps: commands are sent through the CANFD bus, and the drive torque is calculated by the PD controller to realize low-latency reproduction of human body movements; In step (3), the non-uniform scaling strategy sets scale factors for the upper limbs, lower limbs and trunk respectively. In the two-stage optimization, the first stage aims to obtain the initial solution with the goal of body chain rotation continuity and end effector position matching. In the second stage, the objective function is extended to the position and rotation errors of all key parts. Damping and priority strategies are introduced near singular points to ensure solution stability.

7. The control method according to claim 6, characterized in that, The training process of the strategy model described in step (4) includes multi-terrain, random external force disturbance, and sensor noise simulation scenarios. The reward function is steadily converged through no less than 20,000 iterations, and the model has the ability to maintain the robot's dynamic balance autonomously while tracking the reference action.

8. The control method according to claim 7, characterized in that, In step (5), data synchronization and decoupling at different frequency levels are achieved through independent high real-time processes. The motion state reconstruction algorithm is used to derive the position and velocity information required for high-frequency control from low-frequency pose data, ensuring smooth and stable joint movement with a total delay of less than 10ms.

Images