End-to-end humanoid robot operation behavior and skill training method and digital simulation system

Abstract

The invention discloses an end-to-end humanoid robot operation behavior and skill training method and a digital simulation system, by developing a humanoid robot in the digital simulation system, efficient generalization learning, skill training and migration capabilities are obtained, various non-preset task scenes are quickly adapted, and the training efficiency is improved. The problems that the humanoid robot is complex in development environment and high in development cost are solved. The method comprises the following steps: creating a humanoid robot digital prototype model based on a simulation engine; collecting data and generating a data set through the humanoid robot digital prototype model; constructing a learning algorithm library of operation behaviors of the humanoid robot; based on the learning algorithm library, designing an operation behavior and skill model of the humanoid robot; based on the simulation data set and the learning algorithm library, training operation behaviors and skill models of the humanoid robot, and creating skill libraries of different tasks and scenes; and calling the skill library, executing tasks of the humanoid robot, and performing iterative optimization on the skill library.

Description

Technical Field

[0001] The present invention relates to the field of robot simulation systems, and in particular to an end-to-end humanoid robot operation behavior and skill training method and a digital simulation system. Background Art

[0002] Due to their versatility and versatility, humanoid robots need to handle multiple tasks such as visual perception, voice interaction, motion control, and dexterous operation. At the same time, the types of scenarios and tasks that need to be considered are extremely rich, making it extremely complex and expensive to develop key technologies such as stable walking and dexterous operation of humanoid robots in real environments. Building a simulation platform and skill library to develop humanoid robots in a digital simulation system can effectively solve this problem. By simulating humanoid robots and various complex environments in a simulation environment, the effect of the algorithm can be verified efficiently, safely, and at low cost. This research method can not only save a lot of resources, but also iteratively optimize key technologies and continuously improve the performance of humanoid robots. Summary of the invention

[0003] The purpose of the present invention is to provide an end-to-end humanoid robot operation behavior and skill training method and a digital simulation system in order to solve the above problems.

[0004] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0005] The first aspect of the present invention provides an end-to-end humanoid robot operation behavior and skill training method and digital simulation system, comprising the following steps:

[0006] Step S01: creating a digital prototype model of a humanoid robot based on a simulation engine;

[0007] Step S02: Collect data and generate a data set through a humanoid robot digital prototype model;

[0008] Step S03: constructing a learning algorithm library for humanoid robot operation behavior;

[0009] Step S04: designing the operation behavior and skill model of the humanoid robot based on the learning algorithm library;

[0010] Step S05: Based on the simulation data set and the learning algorithm library, the operation behavior and skill model of the humanoid robot are trained to create a skill library for different tasks and scenarios;

[0011] Step S06: calling the skill library, executing the task of the humanoid robot, and iteratively optimizing the skill library.

[0012] As a preferred embodiment of the present invention, the "simulation engine" in step S01 includes simulation and game engines such as Isaac Sim, Gazebo, Webots, Mujoco, and Unity3D.

[0013] As a preferred embodiment of the present invention, the "creating a humanoid robot digital prototype model" in step S01 specifically includes a dual-arm operation model, a lower limb walking model, and an application scenario model. The dual-arm operation model includes the robot's upper limb trunk and arms, the lower limb walking model includes the robot's lower limb trunk and feet, and the application scenario model includes the robot body and scenario information, and the scenario information includes different types of infrastructure, target objects, and operation tool models.

[0014] As a preferred embodiment of the present invention, the "collecting data" in step S02 specifically includes collecting and recording the video of the robot operation, the interaction data between the robot and the environment, and the operation log through the interaction between the two-arm operation model, the lower limb walking model, and the application scenario model using various sensors; the "generating a data set" in step S02 includes performing data cleaning, feature extraction, standardization and other steps on the collected data to establish a simulation data set.

[0015] As a preferred embodiment of the present invention, the "construction of a learning algorithm library for humanoid robot operation behavior" in step S03 includes optimizing and designing learning algorithms based on classic algorithms in machine learning frameworks such as reinforcement learning and imitation learning, and forming a learning algorithm library suitable for humanoid robot operation behavior and skill training. The classic algorithms based on machine learning frameworks such as reinforcement learning and imitation learning include Q-learning, Sarsa, policy gradient algorithm, behavior cloning algorithm, etc.

[0016] As a preferred embodiment of the present invention, the step S04 of "designing the operation behavior and skill model of the humanoid robot based on the learning algorithm library" specifically includes selecting a learning algorithm from the learning algorithm library according to different tasks, and forming the operation behavior and skill model of the humanoid robot based on the selected learning algorithm. The forming the operation behavior and skill model of the humanoid robot based on the selected learning algorithm includes forming a deep Q-network (DQN), a policy network, etc. based on an algorithm in a machine learning framework.

[0017] As a preferred embodiment of the present invention, the "training the operating behavior and skill model of the humanoid robot based on the simulation data set and the learning algorithm library" in step S05 specifically includes using the simulation data set and the learning algorithm in the learning algorithm library to train the operating behavior and skill model of the humanoid robot, and forming a skill library for different tasks and scenarios of the humanoid robot based on the trained operating behavior and skill model.

[0018] As a preferred embodiment of the present invention, the "calling the skill library to execute the task of the humanoid robot" in step S06 specifically includes calling the skill library to execute the task operation of the humanoid robot according to different tasks and scenarios.

[0019] As a preferred embodiment of the present invention, the "iterative optimization of the skill library" in step S06 specifically includes executing the tasks of the humanoid robot by calling the skill library, collecting relevant data during the task execution, updating the model parameters and optimizing the strategy based on the relevant data, and realizing iterative optimization of the skill library to meet different task requirements. The relevant data collected during the task execution includes the video of the robot operation, the interaction data between the robot and the environment, and the operation log, etc.

[0020] A second aspect of the present invention provides a digital simulation system for implementing an end-to-end humanoid robot operation behavior and skill training method, comprising a model library module, a database module, an algorithm library module and a skill library module.

[0021] As a preferred embodiment of the present invention, the model library module includes a dual-arm working model, a lower limb walking model and an application scenario model. The dual-arm working model includes an upper limb trunk and arm model of a humanoid robot; the lower limb walking model includes a lower limb trunk and bipedal model of a humanoid robot; and the application scenario model includes an interactive environment and object model in which the humanoid robot is located.

[0022] As a preferred embodiment of the present invention, the database module includes a data acquisition module and a data set generation module. The data acquisition module is used to collect and record videos of robot operations, interaction data between the robot and the environment, and operation logs; the data set generation module is used to process the collected data through data cleaning, feature extraction, standardization and other steps to establish a simulation data set.

[0023] As a preferred embodiment of the present invention, the algorithm library module includes machine learning algorithm modules such as reinforcement learning algorithm and imitation learning algorithm. The reinforcement learning algorithm module is used to realize reinforcement learning training of robot operation behavior through reinforcement learning algorithm; the imitation learning algorithm module is used to realize imitation learning training of humanoid robot's task skills through imitation learning algorithm.

[0024] As a preferred embodiment of the present invention, the skill library module includes a skill construction module, a skill training module and a training visualization module. The skill construction module is used to construct the operating behavior and skill model of the humanoid robot; the skill training module is used to learn and train the operating behavior and skill model of the humanoid robot; and the training visualization module is used to visualize the training process of the operating behavior and skill model of the humanoid robot.

[0025] The beneficial effects of the present invention are: creating a humanoid robot digital prototype model based on a simulation engine; collecting data and generating a data set through the humanoid robot digital prototype model; building a learning algorithm library for the humanoid robot's operating behavior; designing the operating behavior and skill model of the humanoid robot based on the learning algorithm library; training the operating behavior and skill model of the humanoid robot based on the simulation data set and the learning algorithm library, and creating a skill library for different tasks and scenarios; calling the skill library to perform the tasks of the humanoid robot, and iteratively optimizing the skill library. The present invention provides an end-to-end humanoid robot operating behavior and skill training method and a digital simulation system, which can obtain efficient generalized learning, skill training and migration capabilities by developing a humanoid robot in a digital simulation system, quickly adapt to a variety of non-preset task scenarios, and solve the problems of complex development environment and expensive development costs of humanoid robots. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 The present invention is a flowchart of an end-to-end humanoid robot operation behavior and skill training method.

[0027] Figure 2 This is an architecture diagram of a digital simulation system for realizing an end-to-end humanoid robot operation behavior and skill training method according to the present invention. DETAILED DESCRIPTION

[0028] In order to make the objectives, technical solutions and advantages of the present invention clearer, the implementation method of the present invention will be described in detail below in conjunction with the accompanying drawings and embodiments, so that the implementation process of how the present invention applies technical means to solve technical problems and achieve technical effects can be fully understood and implemented accordingly.

[0029] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0030] Example 1

[0031] like Figure 1 As shown, this embodiment provides an end-to-end humanoid robot operation behavior and skill training method, including the following steps:

[0032] Step S01: creating a digital prototype model of a humanoid robot based on a simulation engine.

[0033] In the specific implementation, a digital prototype model of a humanoid robot is created based on a simulation engine, and the simulation engine includes but is not limited to simulation and game engines such as Isaac Sim, Gazebo, Webots, Mujoco, and Unity3D. The created digital prototype model of the humanoid robot includes a dual-arm operation model, a lower limb walking model, and an application scenario model. The dual-arm operation model includes the upper limb torso and arms of the robot, the lower limb walking model includes the lower limb torso and both feet of the robot, and the application scenario model includes the robot body and scene information. The scene information includes but is not limited to models of different types of infrastructure, target objects, operating tools, etc.

[0034] Step S02: Collect data and generate a data set through the humanoid robot digital prototype model.

[0035] In specific implementation, data is collected and a data set is generated through a digital prototype model of a humanoid robot. The collected data includes but is not limited to, through the interaction between a dual-arm operation model, a lower limb walking model and an application scenario model, using various sensors to collect and record videos of robot operations, interaction data between the robot and the environment, and operation logs, etc. The generated data set includes but is not limited to processing the collected data through steps such as data cleaning, feature extraction, and standardization to establish a simulation data set.

[0036] Step S03: Building a learning algorithm library for humanoid robot operation behaviors.

[0037] In the specific implementation, the learning algorithm library for constructing the humanoid robot operation behavior includes but is not limited to optimizing and designing learning algorithms based on classic algorithms in machine learning frameworks such as reinforcement learning and imitation learning to form a learning algorithm library suitable for humanoid robot operation behavior and skill training. The classic algorithms in machine learning frameworks such as reinforcement learning and imitation learning include but are not limited to Q-learning, Sarsa, policy gradient algorithm, behavior cloning algorithm, etc.

[0038] Step S04: Based on the learning algorithm library, design the operation behavior and skill model of the humanoid robot.

[0039] In specific implementation, based on the learning algorithm library, designing the operation behavior and skill model of the humanoid robot includes but is not limited to, according to different tasks, selecting a learning algorithm from the learning algorithm library, and forming the operation behavior and skill model of the humanoid robot based on the selected learning algorithm. The operation behavior and skill model of the humanoid robot formed based on the selected learning algorithm includes but is not limited to forming a deep Q-network (DQN), a policy network, etc. based on the algorithm in the machine learning framework.

[0040] Step S05: Based on the simulation data set and the learning algorithm library, the operation behavior and skill model of the humanoid robot are trained to create a skill library for different tasks and scenarios.

[0041] In specific implementation, based on the simulation data set and the learning algorithm library, the operation behavior and skill model of the humanoid robot are trained, and the skill library for different tasks and scenarios is created, including but not limited to, using the simulation data set and the learning algorithm in the learning algorithm library to train the operation behavior and skill model of the humanoid robot, and based on the trained operation behavior and skill model, a skill library for different tasks and scenarios of the humanoid robot is formed.

[0042] Step S06: calling the skill library, executing the task of the humanoid robot, and iteratively optimizing the skill library.

[0043] In a specific implementation, calling the skill library to execute the task of the humanoid robot includes calling the skill library to execute the task operation of the humanoid robot according to different tasks and scenarios. Iterative optimization of the skill library includes collecting performance data and training result feedback of the humanoid robot by calling the skill library to execute the task of the humanoid robot, and continuously updating the model parameters and strategies according to the feedback to achieve iterative optimization of the skill library to meet different task requirements.

[0044] Embodiment 2:

[0045] like Figure 2 As shown, this embodiment provides a digital simulation system for realizing an end-to-end humanoid robot operation behavior and skill training method, including:

[0046] Model library module, used to build digital prototype models of humanoid robots, including dual-arm operation models, lower limb walking models, and application scenario models;

[0047] A database module, used to collect the interaction data between the humanoid robot and the environment and establish a simulation data set, including a data collection module and a data set generation module;

[0048] Algorithm library module, used to build a learning algorithm library for humanoid robot operation behaviors, including machine learning algorithm modules such as reinforcement learning algorithm and imitation learning algorithm;

[0049] The skill library module is used to build a skill library for humanoid robots in different tasks and scenarios, including a skill building module, a skill training module, and a training visualization module.

[0050] According to the method and system provided by the embodiment of the present invention, an end-to-end humanoid robot operation behavior and skill training method and digital simulation system are used to create a humanoid robot digital prototype model based on a simulation engine; data is collected and a data set is generated through the humanoid robot digital prototype model; a learning algorithm library for the operation behavior of the humanoid robot is constructed; the operation behavior and skill model of the humanoid robot is designed based on the learning algorithm library; the operation behavior and skill model of the humanoid robot are trained based on the simulation data set and the learning algorithm library, and a skill library for different tasks and scenarios is created; the skill library is called to perform the tasks of the humanoid robot, and the skill library is iteratively optimized. By developing a humanoid robot in a digital simulation system, efficient generalized learning, skill training and migration capabilities are obtained, and a variety of non-preset task scenarios are quickly adapted to solve the problems of complex development environment and high development costs of humanoid robots.

[0051] It should be noted that the modules (or units) in this embodiment are logical in nature. In specific implementation, multiple modules (or units) can be combined into one module (or unit), and one module (or unit) can also be split into multiple modules (or units).

[0052] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0053] Although the present invention is described herein with reference to the illustrative embodiments of the present invention, the above embodiments are only preferred embodiments of the present invention, and the embodiments of the present invention are not limited to the above embodiments. It should be understood that those skilled in the art can design many other modifications and embodiments, which will fall within the scope and spirit of the principles disclosed in this application.

Claims

1. An end-to-end humanoid robot manipulation behavior and skill training method, Features: The steps include: Step S01: creating a digital prototype model of a humanoid robot based on a simulation engine; Step S02: Collect data and generate a data set through a humanoid robot digital prototype model; Step S03: constructing a learning algorithm library for humanoid robot operation behavior; Step S04: designing the operation behavior and skill model of the humanoid robot based on the learning algorithm library; Step S05: Based on the simulation data set and the learning algorithm library, the operation behavior and skill model of the humanoid robot are trained to create a skill library for different tasks and scenarios; Step S06: calling the skill library, executing the task of the humanoid robot, and iteratively optimizing the skill library.

2. An end-to-end humanoid robot operation behavior and skill training method according to claim 1, Features: The "humanoid robot digital prototype model" in step S01 specifically includes a dual-arm working model, a lower limb walking model and an application scenario model; the dual-arm working model includes the robot's upper limb trunk and arms, the lower limb walking model includes the robot's lower limb trunk and feet, and the application scenario model includes the robot body and scene information, and the scene information includes different types of infrastructure, target objects, and operating tool models.

3. An end-to-end humanoid robot operation behavior and skill training method according to claim 1 or 2, Features: The "data collection" in step S02 specifically includes collecting and recording the robot operation video, the interaction data between the robot and the environment, and the operation log through the interaction between the dual-arm operation model, the lower limb walking model and the application scenario model using various sensors.

4. The end-to-end humanoid robot operation behavior and skill training method according to claim 3, Features: The "generating a data set" in step S02 includes processing the collected data through steps such as data cleaning, feature extraction, and standardization to establish a simulation data set.

5. An end-to-end humanoid robot operation behavior and skill training method according to any one of claims 1-2 and 4, Features: The "constructing a learning algorithm library for humanoid robot operation behavior" in step S03 includes forming a learning algorithm library suitable for humanoid robot operation behavior and skill training based on classic algorithms in machine learning frameworks such as reinforcement learning and imitation learning.

6. An end-to-end humanoid robot operation behavior and skill training method according to claim 5, Features: The step S04 of "designing the operation behavior and skill model of the humanoid robot based on the learning algorithm library" specifically includes selecting a learning algorithm from the learning algorithm library according to different tasks, forming a deep Q-network and a policy network based on the algorithm in the machine learning framework, and forming the operation behavior and skill model of the humanoid robot based on the selected learning algorithm.

7. An end-to-end humanoid robot operation behavior and skill training method according to any one of claims 1-2, 4, and 6, Features: The "training the operating behavior and skill model of the humanoid robot based on the simulation data set and the learning algorithm library" in step S05 specifically includes using the simulation data set and the learning algorithm in the learning algorithm library to train the operating behavior and skill model of the humanoid robot, and forming a skill library for different tasks and scenarios of the humanoid robot based on the trained operating behavior and skill model.

8. An end-to-end humanoid robot operation behavior and skill training method according to claim 7, Features: The "calling the skill library to execute the task of the humanoid robot" in step S06 specifically includes calling the skill library to execute the task operation of the humanoid robot according to different tasks and scenarios.

9. The end-to-end humanoid robot operation behavior and skill training method according to claim 8, Features: The "iterative optimization of the skill library" in step S06 specifically includes executing the tasks of the humanoid robot by calling the skill library, collecting relevant data during the task execution, updating model parameters and optimizing strategies based on the relevant data, and implementing iterative optimization of the skill library to meet different task requirements; the relevant data collected during the task execution includes videos of robot operations, interaction data between the robot and the environment, and operation logs, etc.

10. A digital simulation system for implementing the end-to-end humanoid robot operation behavior and skill training method as claimed in any one of claims 1 to 9, Features: Including model library module, database module, algorithm library module and skill library module; The model library module includes a dual-arm operation model, a lower limb walking model and an application scenario model. The dual-arm operation model includes an upper limb trunk and arm model of a humanoid robot; the lower limb walking model includes a lower limb trunk and bipedal model of a humanoid robot; the application scenario model includes an interactive environment and object model in which the humanoid robot is located; The database module includes a data acquisition module and a data set generation module. The data acquisition module is used to collect and record videos of robot operations, interaction data between the robot and the environment, and operation logs; the data set generation module is used to process the collected data through data cleaning, feature extraction, standardization and other steps to establish a simulation data set; The algorithm library module includes machine learning algorithm modules such as reinforcement learning algorithm and imitation learning algorithm. The reinforcement learning algorithm module is used to implement reinforcement learning training of robot operation behavior through reinforcement learning algorithm; the imitation learning algorithm module is used to implement imitation learning training of humanoid robot for task skills through imitation learning algorithm; The skill library module includes a skill construction module, a skill training module and a training visualization module. The skill construction module is used to construct the operation behavior and skill model of the humanoid robot; the skill training module is used to learn and train the operation behavior and skill model of the humanoid robot; The training visualization module is used to visualize the operation behavior and training process of the skill model of the humanoid robot.

Images