Real-time whole-body remote operation control system for humanoid robot

Abstract

The invention discloses a real-time whole-body remote operation control system for a humanoid robot, and the system comprises a motion instruction generation subsystem which is located at an operation end and is used for receiving and analyzing input instructions of various modes, and carrying out the unified conversion and complementation of the input instructions into a whole-body joint motion instruction which is suitable for a skeleton structure of a target humanoid robot; the body motion control subsystem is located at the robot end and used for receiving the whole-body motion instruction, adaptively generating a whole-body motion coordination control signal capable of ensuring the stability of the robot in combination with the state of the robot and environment information, and driving and executing the whole-body motion coordination control signal; through cooperation of the two subsystems, the system provides a dual guarantee path for realizing whole-body teleoperation: a complete whole-body instruction can be generated based on the instruction generation subsystem, and whole-body motion can be adaptively generated by the body motion control system according to a received local instruction; and the robustness, the flexibility and the reliability of the system under different conditions are ensured.

Description

Technical Field

[0001] This invention belongs to the field of robot control technology, and more specifically, relates to a real-time full-body remote control system for a humanoid robot. Background Technology

[0002] With the rapid development of computer technology, sensor technology, artificial intelligence, and robotics, humanoid robot technology has continuously progressed and improved. Breakthroughs, particularly in deep learning and computer vision, have provided humanoid robots with more powerful perception, learning, and decision-making capabilities. Humanoid robots have broad application prospects in many fields, such as home services, medical care, and public safety. Due to rising living standards and increasing labor costs, the demand for robots capable of replacing humans in repetitive, tedious, or dangerous tasks is increasing. Many countries have introduced relevant policies to encourage and support the development of the humanoid robot industry.

[0003] Humanoid robot control systems need to simultaneously control dozens of joint motors throughout the body and process various types of sensor data. The movement of a humanoid robot requires all motors to work in coordination, making the system extremely complex and demanding high real-time control performance. Therefore, improving humanoid robot manipulation technology is a crucial branch of technological development in the field of robotics.

[0004] For humanoid robot control technology, existing technologies include teleoperation to achieve remote control of humanoid robots. Teleoperation technology refers to a technology that allows operators to remotely control humanoid robots to perform tasks through a remote control system. Traditional teleoperation methods generally convert the operator's actions into control signals for the humanoid robot through a controller (such as a joystick or button). This method usually has good intuitiveness, but its control precision and flexibility are poor. When facing complex or dangerous tasks, it is impossible to flexibly and accurately control every joint of the humanoid robot, which may lead to uncertainty and safety hazards in the operator's operation.

[0005] Furthermore, current research and applications of teleoperation technologies for humanoid robots largely focus on the upper body, aiming to achieve precise control of the robotic arm and dexterous hand. These solutions typically rely on deep learning models or inverse kinematics methods to establish a mapping from the operator's hand movements to the robot's hand joint movements. However, for the robot's lower body and foot movements, they often depend on pre-trained motion models for adaptive generation, rather than direct real-time control by the operator. This approach makes it difficult to achieve truly simultaneous teleoperation of the robot's upper and lower body, limiting its application in complex tasks. In addition, at the command interaction level, existing solutions mostly rely on traditional remote control devices such as joysticks and handles for motion command input. This interaction method is not only unintuitive and has a high learning cost, but it also struggles to understand and interpret multimodal commands such as voice, gestures, and vision, failing to meet the interaction needs of diverse and complex task scenarios.

[0006] In summary, existing humanoid robot teleoperation technologies have the following three main shortcomings: First, at the command interaction and parsing level, existing systems cannot effectively receive and understand multimodal and diverse high-level commands (such as natural language commands, action commands, etc.), and their command forms are singular, the interaction is not intuitive, and the flexibility is insufficient.

[0007] Secondly, at the system architecture level, there is a lack of a unified framework or middleware capable of universally parsing and converting the aforementioned multimodal instructions into commands suitable for the underlying control interfaces of different humanoid robot bodies. Control interfaces vary among robot manufacturers, and existing motion control systems are typically designed to accept a single, fixed control mode, lacking a universal instruction access and conversion system capable of adapting to and parsing instructions of any form.

[0008] Finally, and most importantly, lies the limitation of the underlying motion control algorithms. Due to the extreme complexity of whole-body motion control in humanoid robots, particularly the dynamic balance control of the lower limbs, existing algorithms suffer from deficiencies in real-time performance, stability, and generalization ability. They struggle to quickly generate coordinated motion responses that simultaneously ensure overall robot stability and accuracy after receiving real-time transmitted whole-body motion commands. Therefore, this algorithmic bottleneck means that real-time, stable, and reliable whole-body teleoperation of humanoid robots remains an unresolved technical challenge.

[0009] Based on this, the purpose of this invention is to address the shortcomings of the prior art by providing a real-time full-body remote telemanipulation control system for humanoid robots. This system overcomes the deficiencies of existing humanoid robot telemanipulation technologies, such as the single command format, lack of a universal command conversion framework, and insufficient real-time performance and stability of the underlying control algorithms. It enables a method and system that can parse multimodal motion commands and drive any humanoid robot to perform full-body collaborative real-time telemanipulation, thereby enhancing the feasibility and reliability of "human in the loop" in future robot control and representing a completely new robot control paradigm. Summary of the Invention

[0010] To address the shortcomings of existing technologies, the purpose of this invention is to provide a real-time full-body remote control system for humanoid robots, overcoming the core problems of existing humanoid robot teleoperation technologies, such as single command modes, lack of a universal command conversion framework, and insufficient real-time performance and stability of the underlying motion control algorithm.

[0011] Specifically, the technical problem to be solved by the present invention is to address the shortcomings of the prior art. In the first aspect, the present invention provides a real-time whole-body remote control system for a humanoid robot, including: a motion command generation subsystem, used to receive and process various forms of source motion commands, and map and complete them into complete whole-body joint motion commands for the target robot.

[0012] The body motion control subsystem is used to convert received whole-body joint motion commands into low-level joint motor control commands.

[0013] The motion command generation subsystem supports various input methods, including operator handle input, visual motion capture data, inertial measurement unit data, inertial motion capture data, predefined human joint motion data, pre-recorded human motion video data, human joint animation data, and high-level language commands.

[0014] It outputs motion control commands for all joints of the body, including an inverse kinematics solution module, a deep learning remapping model, and a whole-body motion generation module.

[0015] The body motion control subsystem is a humanoid robot whole-body motion control subsystem developed and designed based on deep reinforcement learning, and it is deployed in the humanoid robot body.

[0016] In one embodiment of the real-time full-body remote control system for humanoid robots according to the present invention, the motion command generation subsystem includes: an inverse kinematics calculation module, used to convert the input motion signal into joint motion commands that conform to the mechanical structure of the target robot.

[0017] Deep learning remapping models are used to eliminate morphological and dynamic differences between humans and machines, and generate coordinated whole-body joint control signals.

[0018] The whole-body motion generation module is a data generation module that completes the motion data of all joints in the whole body by supplementing partial joint motion data.

[0019] In one embodiment of the real-time whole-body remote control system for humanoid robots according to the present invention, the generated whole-body joint movement commands are transmitted in real time to the humanoid robot at any remote location via a communication network.

[0020] In one embodiment of the real-time full-body remote control system for humanoid robots according to the present invention, after receiving the instruction, the body motion control subsystem deeply integrates the received target instruction with the robot's own real-time state and environmental perception information, and makes appropriate motion control to ensure the robustness and sturdiness of the control actions.

[0021] In one embodiment of the real-time full-body remote control system for humanoid robots according to the present invention, the body motion control subsystem is driven by one or more control models trained based on reinforcement learning and imitation learning strategies. The model is used to process the fused information, generate control signals to drive the motor, and enable the humanoid robot to reproduce the action.

[0022] In one embodiment of the real-time whole-body remote control system for humanoid robots according to the present invention, during the execution of actions, the humanoid robot transmits its own state data and environmental perception information back to the remote operator interface in real time.

[0023] Secondly, the present invention provides a multimodal command conversion and whole-body motion generation method, based on a real-time whole-body remote control system for a humanoid robot as described above, including the following steps: receiving local or abstract motion signals from different input sources, local motion signals from different input sources, including but not limited to center of mass motion commands and motion control commands for some joints.

[0024] Abstract motion signals include robot motion described by human language / speech and video signals during motion.

[0025] By using a whole-body motion data generation algorithm, inverse kinematics calculation, and deep learning remapping, the input signal is converted into complete whole-body joint motion commands.

[0026] The output contains coordinated whole-body joint control signals for all joints of the target robot.

[0027] In one embodiment of the multimodal command conversion and whole-body motion generation method according to the present invention, the input sources include: local joint motion of the human body provided by motion capture data; single joint velocity commands defined by the keyboard; 2D / 3D human posture captured by the camera; and abstract task commands, including action language descriptions and voice.

[0028] In one embodiment of the multimodal instruction conversion and full-body motion generation method according to the present invention, a game development engine is used as a starting tool to develop and build the entire instruction generation system software based on the game engine. That is, the method can be implemented with the help of a game engine such as Unreal Engine, which can quickly and efficiently engineer the instruction generation system and provide rich interaction modes.

[0029] Thirdly, the present invention provides a data-driven intelligent body motion control method, based on a real-time full-body remote control system for a humanoid robot as described above, including the following steps: training the control model by combining imitation learning and deep reinforcement learning.

[0030] Using human motion data for imitation learning allows the model to learn natural movement patterns.

[0031] Optimize the stability and anti-interference ability of the model in a simulation environment through deep reinforcement learning.

[0032] The received whole-body joint movement commands are converted into joint motor control commands in real time.

[0033] In one embodiment of a data-driven intelligent ontology motion control method according to the present invention, the learning phase of the control model training uses more than ten hours of human motion data.

[0034] The control model can receive motion commands from one or more joints and output whole-body motor motion commands.

[0035] The motor control commands output by the control model include control signals such as torque, position, and speed.

[0036] Compared with the prior art, the positive effects of the present invention are: 1. The humanoid robot real-time whole-body remote telecontrol control system proposed in the present invention realizes true whole-body coordinated teleoperation: it supports seamless switching from single-joint fine control to whole-body coordinated movement, breaking through the limitation of the prior art that can only control the upper body.

[0037] 2. The humanoid robot real-time full-body remote control system proposed in this invention has multi-modal command adaptation capability: through a general command conversion framework, it can receive and parse various input forms such as motion capture data, keyboard commands, and abstract control commands to adapt to different operation scenarios.

[0038] 3. The humanoid robot real-time full-body remote control system proposed in this invention can improve motion stability and intelligence: the control system can adaptively generate full-body movements to maintain balance, ensuring stable execution in complex environments.

[0039] 4. The humanoid robot real-time full-body remote control system proposed in this invention can achieve efficient closed-loop control: real-time status feedback enables the operator to adjust in a timely manner according to the execution effect and environmental changes, forming a complete perception-decision-execution closed loop.

[0040] 5. The humanoid robot real-time full-body remote control system proposed in this invention can provide a dual-path guarantee mechanism: it can generate full-body motion through the command system, and it can also adaptively complete the command by the body control system, which significantly enhances the robustness of the system. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the system structure of the real-time full-body remote control system for the humanoid robot in this invention.

[0042] Figure 2 This is a system function diagram of the motion command generation subsystem in this invention.

[0043] Figure 3 This is a system function diagram of the body motion control subsystem in this invention.

[0044] Figure 4 This is a flowchart illustrating the real-time full-body remote manipulation of the humanoid robot in this invention. Detailed Implementation

[0045] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0046] It is understood that the term "a" should be understood as "at least one" or "one or more," meaning that in one embodiment, the number of an element can be one, while in another embodiment, the number of the element can be multiple. The term "a" should not be construed as a limitation on the quantity. "Multiple" means two or more.

[0047] While ordinal numbers such as “first,” “second,” etc., will be used to describe various components, there is no limitation on which components are used herein. The term is used only to distinguish one component from another. For example, a first component may be referred to as a second component, and similarly, a second component may be referred to as a first component, without departing from the teachings of this application. The term “and / or” as used herein includes any and all combinations of one or more of the associated listed items.

[0048] The terminology used herein is for the purpose of describing various embodiments only and is not intended to be limiting. As used herein, the singular form also includes the plural form, unless the context clearly indicates otherwise. It will also be understood that the terms “comprising” and / or “having” as used in this specification specify the presence of the described features, numbers, operations, components, elements or combinations thereof, without excluding the presence or addition of one or more other features, numbers, operations, components, elements or combinations thereof.

[0049] The following is combined with Figures 1-4 The present invention will be further described with reference to specific embodiments.

[0050] The basic concept of this invention is to construct a universal, flexible, and stable real-time teleoperation system for the whole body of a humanoid robot. This system aims to solve the core problems of existing humanoid robot teleoperation technologies, such as the single command modality, lack of a universal command conversion framework, and insufficient real-time performance and stability of the underlying motion control algorithms, through the collaborative work of two core subsystems.

[0051] 1. Core Objective: To allow users to control any humanoid robot to perform any action at any spatial distance using any method (such as motion capture, keyboard commands, trajectory planning, abstract control commands, etc.). Users can freely choose to control some joints of the robot (with the motion of the remaining joints adaptively generated by the system) or all joints, thereby meeting the needs of various application scenarios from fine local operations to whole-body coordinated movements.

[0052] 2. System Composition: The system consists of two major subsystems forming a complete control closed loop: such as... Figure 1 The diagram shows the system structure of the real-time full-body remote control system for humanoid robots in this invention. The system in this embodiment mainly consists of the following parts: motion command generation subsystem: located at the operation end, its core function is to receive and parse input commands of multiple modalities, and convert and complete them into a set of full-body joint motion commands suitable for the target humanoid robot skeleton structure.

[0053] Body motion control subsystem: Located at the robot's end, its core function is to receive the aforementioned whole-body motion commands (whether for all or some joints), combine the robot's own state and environmental information, adaptively generate coordinated whole-body motion control signals that ensure robot stability, and drive their execution, such as... Figure 4 The diagram shown is a flowchart of real-time full-body remote manipulation of a humanoid robot.

[0054] 3. Coordination Mechanism: Through the coordinated operation of two subsystems, this system provides a dual-safety path for achieving whole-body teleoperation: it can generate complete whole-body commands based on the command generation subsystem, or the body motion control system can adaptively generate whole-body motion based on received local commands. This design ensures the system's robustness, flexibility, and reliability under different conditions.

[0055] Specifically: A real-time full-body remote control system for a humanoid robot includes: a motion command generation subsystem, used to receive and process various forms of source motion commands, mapping and completing them into complete full-body joint motion commands for the target robot, such as... Figure 2 The diagram shows the system function of the motion command generation subsystem.

[0056] The body motion control subsystem is used to convert received whole-body joint motion commands into low-level joint motor control commands, such as... Figure 3 The diagram shown is a system function diagram of the body motion control subsystem.

[0057] The motion command generation subsystem supports multiple input methods, including operator handle input, visual motion capture data, inertial motion capture data, and high-level language commands.

[0058] The motion command generation subsystem includes an inverse kinematics solution module, which converts the input motion signal into joint motion commands that conform to the mechanical structure of the target robot.

[0059] Deep learning remapping models are used to eliminate morphological and dynamic differences between humans and machines, and generate coordinated whole-body joint control signals.

[0060] The whole-body motion generation module generates whole-body joint motion data based on the received partial joint commands.

[0061] The generated full-body joint movement commands are transmitted in real time to a humanoid robot at any remote location via a communication network.

[0062] After receiving the command, the body motion control subsystem will deeply integrate the received target command with the robot's real-time status and environmental perception information.

[0063] The body motion control subsystem is driven by one or more control models trained based on deep reinforcement learning and imitation learning strategies. These models are used to process the fused information, generate control signals to drive motors, and enable the humanoid robot to reproduce actions.

[0064] During the execution of actions, the humanoid robot transmits its own status data and environmental perception information back to the remote operator interface in real time.

[0065] Based on the humanoid robot real-time whole-body remote control system described above, the present invention also provides a multimodal command conversion and whole-body motion generation method, including the following steps: receiving local or abstract motion signals from different input sources; converting the input signals into complete whole-body joint motion commands through whole-body motion data generation algorithms, inverse kinematics calculation and deep learning remapping; and outputting coordinated whole-body joint control signals containing all joints of the target robot.

[0066] The input sources include: local joint motion of the human body provided by motion capture data; single joint velocity commands defined by the keyboard; 2D / 3D human posture captured by the camera; pre-recorded human motion data (video or animation); predefined human joint motion data; and abstract task instructions.

[0067] Based on the humanoid robot real-time full-body remote control system described above, the present invention also provides a data-driven intelligent body motion control method, including the following steps: training the control model by combining imitation learning and deep reinforcement learning.

[0068] Using human motion data for imitation learning allows the model to learn natural movement patterns.

[0069] Optimize stability and anti-interference capabilities in a simulation environment through deep reinforcement learning.

[0070] The received whole-body joint movement commands are converted into joint motor control commands in real time.

[0071] In one embodiment of a data-driven intelligent ontology motion control method according to the present invention, the imitation learning phase uses more than ten hours of human motion data.

[0072] The control model can receive motion commands from one or more joints and output whole-body motor motion commands.

[0073] The motor control commands output by the control model include torque, position, and speed control signals.

[0074] Example: The following describes the workflow of this system in detail through a typical application scenario: Example: An operator wears an inertial motion capture system to control a humanoid robot.

[0075] Step 1: Complete command generation and conversion in motion command generation software developed based on the UE game engine: Command input: The remote operator wears an inertial measurement unit (IMU) suit and performs full-body movements such as walking, waving, or squatting. The motion capture suit generates motion data of all joints in the operator's body.

[0076] Command Processing and Conversion: This motion data is input into the motion command generation software. The software first parses the input commands. Since there may be differences in skeletal size and degrees of freedom between the operator and the robot, the command generation software uses inverse kinematics (IK) calculations or deep learning models to remap the natural human movements into physically achievable full-body joint motion commands that conform to the mechanical structure of the target humanoid robot. In this example, the operator performed a full-body movement, so the command generation software outputs a complete set of control commands containing all joint target states (such as position and angle). If the operator only wears motion capture devices on their arms, that is, only sends arm movement commands to the command generation software, the software can similarly complete this partial command into a full-body command that includes leg balance posture.

[0077] Step 2: Command transmission: The generated standardized whole-body joint movement commands are transmitted in real time to the humanoid robot at any remote location via communication networks (such as 5G, Wi-Fi).

[0078] Step 3: Command Execution and Control (completed in the humanoid robot's body motion control subsystem): Command Reception and Fusion: The robot's body motion control subsystem receives commands. This system deeply fuses the received target commands with the robot's real-time state (such as joint encoder feedback and body posture angles measured by the IMU) and environmental perception information (such as obstacles seen by the camera).

[0079] Intelligent motion generation: The control subsystem is driven by one or more control models trained based on deep reinforcement learning and imitation learning strategies. This model processes the fused information, and even when the received command is an absolute joint angle, it outputs coordinated whole-body motor torque or speed control signals capable of responding to uneven ground or external disturbances, ensuring the accuracy, stability, and safety of motion execution.

[0080] Action execution: The generated control signals drive the motors, enabling the humanoid robot to accurately reproduce the operator's actions such as walking, waving, or squatting.

[0081] Step 4: Closed-loop feedback: During execution, the robot transmits its own state data (such as whether it has lost its balance) and environmental perception information (such as obstacles being detected ahead) back to the remote operator interface in real time.

[0082] Based on this feedback information, the operator determines whether the instructions need to be adjusted (such as stopping walking or detouring), thus forming an efficient "perception-decision-execution-feedback" control closed loop, ultimately achieving accurate, safe and stable control of the robot.

[0083] In summary, this embodiment demonstrates how a system can convert a specific multimodal input (inertial motion capture) into robot motion. This system is also applicable to other input methods (such as VR controllers, keyboard commands, abstract control instructions (motion language descriptions), etc.). Its core innovation lies in its universal instruction conversion and intelligent adaptive execution capabilities, thereby truly achieving full-body, real-time, and stable teleoperation. A game engine is used as a practical example to demonstrate the engineering implementation of this system.

[0084] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0085] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems or apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section.

[0086] It should be understood that in this invention, "at least one (item)" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0087] It should also be noted that, in this invention, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0088] The steps of the methods or algorithms described in conjunction with the embodiments disclosed in this invention can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0089] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined in this invention may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A humanoid robot real-time full-body remote teleoperation control system, characterized by, Comprise: Motion instruction generation subsystem for receiving and processing a variety of forms of source motion instructions, mapping and completing them into complete full-body joint motion instructions of target robot; Body motion control subsystem for converting received full-body joint motion instructions into bottom layer joint motor control instructions; Wherein, the motion instruction generation subsystem supports a variety of input forms including operator handle input, visual motion capture data, inertial motion capture data, pre-defined human joint motion data, pre-recorded human motion video data, human joint animation data and high-level language commands; Output full-body motion joint motion control instruction, including inverse kinematics solving module, deep learning remapping model and full-body motion generation module; The body motion control subsystem is a humanoid robot full-body motion control subsystem developed and designed based on deep reinforcement learning, which is deployed in the humanoid robot body.

2. The humanoid robot real-time full-body remote teleoperation control system according to claim 1, wherein The motion instruction generation subsystem comprises: an inverse kinematics solving module for converting input motion signals into joint motion instructions conforming to the mechanical structure of the target robot; A deep learning remapping model is used to eliminate the differences in morphology and dynamics between man and machine, and to generate coordinated full-body joint control signals; The full-body motion generation module is a data generation module that completes partial joint motion data into full-body joint motion data.

3. The humanoid robot real-time full-body teleoperation control system of claim 1, wherein, The generated full-body joint motion instructions are transmitted in real time to a humanoid robot at any remote location through a communication network.

4. The humanoid robot real-time full-body teleoperation control system of claim 1, wherein, After receiving the instructions, the body motion control subsystem deeply fuses the received target instructions with the real-time state of the robot itself and the environmental perception information, and makes appropriate motion control to ensure the robustness and robustness of the control action.

5. The humanoid robot real-time full-body teleoperation control system of claim 4, wherein, The body motion control subsystem is driven by one or more control models trained based on reinforcement learning and imitation learning strategy, which is used to process the fused information, generate control signals to drive the motor, and make the humanoid robot reproduce the action.

6. The humanoid robot real-time full-body teleoperation control system of claim 5, wherein, During the execution of the action, the humanoid robot will return the body state data and environmental perception information to the remote operator interface in real time.

7. A multi-modal instruction conversion and whole body motion generation method based on a real-time whole body remote teleoperation control system for a humanoid robot as claimed in any one of claims 1-6, characterized in that, Comprise the following steps: Receive partial or abstract motion signals from different input sources, including partial motion signals from different input sources, including center of mass motion instructions and partial joint motion control instructions; Wherein, the abstract motion signal includes human language / voice description of robot motion, video signal in motion; Through full-body motion data generation algorithm, inverse kinematics solving and deep learning remapping, the input signal is converted into complete full-body joint motion instruction; Output contains coordinated full-body joint control signal of all joints of target robot.

8. A multi-modal instruction conversion and full-body motion generation method as claimed in claim 7, characterized by, The input source includes: human partial joint motion provided by motion capture data; Keyboard defined single joint speed command; 2D / 3D human posture captured by camera; Abstract task instruction, including action language description, voice.

9. The multi-modal instruction conversion and whole-body motion generation method of claim 7 or 8, wherein, Use game development engine as a starting implementation tool, develop and build the entire instruction generation system software based on game engine.

10. A data-driven intelligent body motion control method based on a real-time full-body remote teleoperation control system for a humanoid robot as claimed in any one of claims 1-6, characterized in that, Comprise the following steps: Adopt the method of combining imitation learning with deep reinforcement learning to train the control model; Using human motion data for imitation learning, the model learns natural motion patterns; Optimize the stability and anti-interference ability of the model in the simulation environment through deep reinforcement learning; convert the received whole-body joint motion instructions into joint motor control instructions in real time.

11. The data-driven based intelligent body motion control method of claim 10, wherein, The imitation learning stage uses more than ten hours of human motion data; the control model can receive single or multiple joint motion instructions and output whole-body motor motion commands; the motor control instructions output by the control model include torque, position and speed control signals.

Images