Robot learning data set construction method and related equipment

Abstract

The invention provides a robot learning data set construction method and related equipment, and relates to the technical field of humanoid robots. According to the method, a robot operator remotely operates the humanoid robot to execute various operation tasks based on inertial motion capture equipment, and real-time operation images and real-time joint motion data of the humanoid robot in the process of executing the operation tasks under manual guidance are collected to construct a data set; through organic combination of a human body posture direct extraction characteristic of inertial motion capture equipment and task execution flexibility and task execution reliability of robot teleoperation, construction efficiency, data set validity and data set integrity of a task learning data set are improved; and generalization and accuracy of simulation learning of the humanoid robot can be further improved.

Description

Technical Field

[0001] The present application relates to the technical field of humanoid robots, and in particular to a method for constructing a robot learning data set and related equipment. Background Art

[0002] With the continuous development of science and technology, humanoid robot technology is increasingly being used in various industries. However, with the diversification of robot application scenarios, the working environment of humanoid robots is becoming more and more complex. The motion control strategies designed for single tasks are essentially unable to adapt to a large number of flexible and complex working scenarios, which will limit the further development of robot technology. For this reason, robot imitation learning technology is of great significance to the realization of robot intelligence and autonomy because it can give humanoid robots the flexible operation ability to perform tasks in complex scenarios. It has gradually become an important research direction in the application process of humanoid robots.

[0003] At present, in the actual application of robot imitation learning technology, the quality and scale of the task learning dataset directly determine the effect of the robot task execution control model learned by imitation learning. However, it is worth noting that today's task learning datasets are mainly constructed by manually designing control algorithms to control humanoid robots to complete various tasks of interacting with the environment. It is usually necessary to repeatedly debug the control algorithm to ensure the successful execution of the task. It is actually difficult to quickly collect the task learning dataset required for humanoid robot imitation learning, and the validity and integrity of the dataset cannot be guaranteed. Summary of the invention

[0004] In view of this, the purpose of the present application is to provide a robot learning dataset construction method, computer equipment, robot teleoperation system and readable storage medium, which can enable a robot operator (i.e., robot teleoperator) to teleoperate a humanoid robot to perform various operation tasks based on an inertial motion capture device, and collect real-time operation images and real-time joint motion data of the humanoid robot performing operation tasks under manual guidance to construct a dataset, so as to improve the construction efficiency, dataset validity and dataset integrity of the task learning dataset by organically combining the direct extraction characteristics of the human body posture of the inertial motion capture device with the task execution flexibility and task execution reliability of the robot teleoperation, so as to further improve the generalization and accuracy of the imitation learning of the humanoid robot.

[0005] In order to achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0006] In a first aspect, the present application provides a method for constructing a robot learning dataset, the method comprising:

[0007] During any task execution process of the learning operation task, real-time joint motion data of the humanoid robot is obtained, and real-time image data captured by the humanoid robot for the robot operating environment and task target is obtained for display;

[0008] Acquire human motion posture data collected in real time by an inertial motion capture device for a robot teleoperator, and convert the human motion posture data into expected joint motion data adapted to the humanoid robot;

[0009] Based on the real-time joint motion data, the humanoid robot is controlled to move according to the expected joint motion data, and the step of obtaining the real-time joint motion data of the humanoid robot is skipped to continue execution until the humanoid robot successfully completes a task execution operation for the task target;

[0010] The real-time joint motion data and the real-time image data related to the task execution operation are sorted to obtain a learning data sample in the task learning data set for the operation task to be learned, wherein each learning data sample corresponds to a task execution operation of the operation task to be learned.

[0011] In an optional embodiment, the inertial motion capture device includes a plurality of inertial posture sensors, and the plurality of inertial posture sensors are respectively worn on different skeletal parts of the human body of the robot teleoperator, and the human body motion posture data includes skeletal motion posture data respectively collected by the plurality of inertial posture sensors in the motion capture coordinate system, and the step of converting the human body motion posture data into expected joint motion data adapted to the humanoid robot includes:

[0012] According to the human skeleton data of the robot teleoperator, human joint motion state estimation is performed based on the skeleton motion posture data of each of the multiple inertial posture sensors in the motion capture coordinate system to obtain the human joint motion data of the robot teleoperator in the motion capture coordinate system;

[0013] The human joint motion data in the motion capture coordinate system is mapped to the robot coordinate system of the humanoid robot to obtain the expected joint motion data.

[0014] In an optional embodiment, the robot coordinate system of the humanoid robot includes the body workspace coordinate systems of each of the multiple moving body parts of the humanoid robot, and the expected joint motion data includes the target joint motion data of each of the multiple moving body parts in the corresponding body workspace coordinate system, then the step of mapping the human joint motion data in the motion capture coordinate system to the robot coordinate system of the humanoid robot to obtain the expected joint motion data includes:

[0015] For each moving body part of the humanoid robot, extracting effective joint motion data corresponding to the moving body part from the human joint motion data;

[0016] According to the coordinate system transformation relationship between the motion capture coordinate system and the body workspace coordinate system of the moving body part, the effective joint motion data corresponding to the moving body part is transformed to obtain the target joint motion data of the moving body part in the corresponding body workspace coordinate system.

[0017] In an optional embodiment, the real-time joint motion data includes actual joint motion data of each of the multiple moving body parts in the corresponding body workspace coordinate system, and the step of controlling the humanoid robot to move according to the expected joint motion data based on the real-time joint motion data includes:

[0018] For each moving body part of the humanoid robot, a target planning strategy matching the moving body part is selected from the pre-stored smooth trajectory planning strategies of the plurality of moving body parts;

[0019] In the body workspace coordinate system of the moving body part, smooth motion trajectory planning is performed based on the actual joint motion data and target joint motion data of the moving body part according to the target planning strategy to obtain the expected smooth motion trajectory of the moving body part;

[0020] All robot joints included in the moving body part are controlled to move separately according to the desired smooth motion trajectory.

[0021] In an optional embodiment, the step of collating the real-time joint motion data and the real-time image data related to the task execution operation to obtain a learning data sample in the task learning data set for the operation task to be learned includes:

[0022] The real-time joint motion data and the real-time image data are sampled at preset time intervals to obtain robot joint sampling data and observation image sampling data at respective sampling time points;

[0023] A time series is constructed for the robot joint sampling data and the observation image sampling data at each of the plurality of sampling time points to obtain a learning data sample for the operation task to be learned.

[0024] In an optional embodiment, the method further comprises:

[0025] The coordinate system transformation relationship between the body workspace coordinate system of each of the multiple moving body parts of the humanoid robot and the motion capture coordinate system of the inertial motion capture device is calibrated and configured, and the smooth trajectory planning strategy of each of the multiple moving body parts is strategically configured.

[0026] In a second aspect, the present application provides a computer device, the computer device is communicatively connected to a humanoid robot and an inertial motion capture device, wherein the inertial motion capture device includes a plurality of inertial posture sensors respectively worn on different human skeleton parts of a robot teleoperator;

[0027] The computer device includes a processor and a memory, the memory stores a computer program that can be executed by the processor, and the processor can execute the computer program to implement the robot learning data set construction method described in any one of the aforementioned embodiments.

[0028] In a third aspect, the present application provides a robot teleoperation system, the system comprising an inertial motion capture device and a computer device, wherein the computer device is communicatively connected to a humanoid robot and the inertial motion capture device, respectively, and the inertial motion capture device comprises a plurality of inertial posture sensors respectively worn on different human skeleton parts of the robot teleoperator;

[0029] The computer device stores a computer program and can run the computer program to cooperate with the inertial motion capture device to implement the robot learning data set construction method described in any one of the aforementioned embodiments.

[0030] In an optional embodiment, the system further comprises an image display device;

[0031] The image display device is deployed in front of the robot teleoperator and is communicatively connected with the computer device to display the real-time image acquired by the computer device from the humanoid robot.

[0032] In a fourth aspect, the present application provides a readable storage medium having a computer program stored thereon. When the computer program is executed by a computer device, the robot learning dataset construction method described in any one of the aforementioned embodiments is implemented.

[0033] In this case, the beneficial effects of the embodiments of the present application may include the following:

[0034] In the present application, during any task execution process of a learning operation task, real-time joint motion data of a humanoid robot is obtained, and real-time image data captured by the humanoid robot for the robot operating environment and task targets are displayed, and then human motion posture data collected in real time by an inertial motion capture device for a robot remote operator is obtained, and the human motion posture data is converted into expected joint motion data adapted to the humanoid robot, so as to control the humanoid robot to move according to the expected joint motion data based on the real-time joint motion data, so that the humanoid robot can successfully complete a task execution operation for the task target under the action of robot remote operation, and then a learning data sample in a task learning data set of the operation task to be learned is obtained by data sorting of the real-time joint motion data and real-time image data related to the task execution operation, thereby organically combining the direct human posture extraction characteristics of the inertial motion capture device with the task execution flexibility and task execution reliability of the robot remote operation, thereby improving the construction efficiency, data set effectiveness and data set integrity of the task learning data set, and facilitating further improving the generalization and accuracy of the imitation learning of the humanoid robot.

[0035] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 A schematic diagram of the system composition of a robot teleoperation system provided in an embodiment of the present application;

[0038] Figure 2 A schematic diagram of the device composition of a computer device provided in an embodiment of the present application;

[0039] Figure 3 One of the flowcharts of the method for constructing a robot learning dataset provided in an embodiment of the present application;

[0040] Figure 4 A schematic diagram of the inertial motion capture device being worn by a robot teleoperator for upper body manipulation tasks;

[0041] Figure 5 for Figure 3 A schematic flow chart of the sub-steps included in step S220;

[0042] Figure 6 for Figure 3 A schematic flow chart of the sub-steps included in step S230;

[0043] Figure 7 The second flowchart of the method for constructing a robot learning dataset provided in an embodiment of the present application.

[0044] Icons: 10 - robot teleoperation system; 11 - inertial motion capture device; 100 - computer device; 12 - image display device; 110 - memory; 120 - processor; 130 - communication unit; 20 - humanoid robot. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0046] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0047] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0048] In the description of the present application, it should be understood that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, or are the orientations or positional relationships in which the product of the application is conventionally placed when in use, or are the orientations or positional relationships conventionally understood by those skilled in the art. They are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0049] In the description of this application, it should also be noted that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0050] In addition, in the description of the present application, it is understood that the relational terms such as the term "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also includes other elements that are not clearly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements defined by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood by specific circumstances.

[0051] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0052] Please refer to Figure 1 , Figure 11 is a schematic diagram of the system composition of the robot teleoperation system 10 provided in the embodiment of the present application. In the embodiment of the present application, the robot teleoperation system 10 may include an inertial motion capture device 11 and a computer device 100, wherein the computer device 100 is respectively connected to the humanoid robot 20 and the inertial motion capture device 11 for communication; the inertial motion capture device 11 includes a plurality of inertial posture sensors respectively worn on different human skeleton parts of the robot teleoperator (for example, corresponding to the head bones, shoulder bones, upper arm bones, lower arm bones, palm bones, back spine bones, waist hip joint bones, thigh bones, calf femurs, sole bones, etc.) of the human body, so as to monitor the human motion posture of the robot teleoperator in real time; the computer device 100 can synchronously adjust the robot motion posture of the humanoid robot 20 based on the human motion posture data monitored in real time by the inertial motion capture device 11, so that the humanoid robot 20 performs various operation tasks correspondingly under the robot teleoperation of the robot teleoperator.

[0053] In an embodiment of the present application, the computer device 100 may be an electronic device with an image display unit (for example, a personal computer, a laptop computer, etc.). The computer device 100 may obtain real-time images captured by the humanoid robot 20 for the robot operating environment and the respective task objectives of different operation tasks (for example, the "target object to be grasped" corresponding to the "grasping operation task", the "goods placement position for different item categories" corresponding to the "sorting operation task", the "workpiece to be welded" corresponding to the "welding operation task", the "balance car to be driven" corresponding to the "balance car driving operation task", etc.), and display the real-time images using the image display unit included in it, so as to guide the robot remote operator to adjust his or her own human body movement posture in time to ensure that the corresponding operation task can be successfully executed as soon as possible. Among them, the humanoid robot 20 can be equipped with at least one camera on the robot head, so as to take real-time images of the robot's operating environment and the task objectives of the corresponding operation tasks through the camera located on the robot's head; in addition, the humanoid robot 20 can also be additionally equipped with at least one camera on the robot wrist and / or robot knee, so as to assist the robot head camera in real-time image capture through the camera located on the robot wrist or robot knee, thereby ensuring that the image data acquired and displayed by the computer device 100 can reveal more task execution details.

[0054] Optionally, in an embodiment of the present application, the computer device 100 may also be an electronic device (for example, a server) that does not have an image display unit, and the robot teleoperation system 10 may also include an image display device 12, which is deployed in front of the robot teleoperator and is communicatively connected to the computer device 100 to display the real-time image obtained by the computer device 100 from the humanoid robot 20, so as to guide the robot teleoperator to adjust his or her own human movement posture in time to ensure that the corresponding operation task can be successfully executed as soon as possible.

[0055] In an embodiment of the present application, the computer device 100 can collect real-time operation images and real-time joint motion data of the humanoid robot 20 performing operation tasks under manual guidance during the process of the robot teleoperator remotely operating the humanoid robot 20 to perform various operation tasks based on the inertial motion capture device 11, and construct a data set, so as to utilize the direct human posture extraction characteristics of the inertial motion capture device 11 and the task execution flexibility and task execution reliability of the robot teleoperation, so as to effectively improve the construction efficiency, data set validity and data set integrity of the task learning data set suitable for the humanoid robot 20, so as to further improve the generalization and accuracy of the imitation learning of the humanoid robot.

[0056] Optionally, see Figure 2 , Figure 2 : is a schematic diagram of the device composition of the computer device 100 provided in an embodiment of the present application. In an embodiment of the present application, the computer device 100 is respectively connected to the humanoid robot 20 and the inertial motion capture device 11 for communication, and the computer device 100 may include a memory 110, a processor 120 and a communication unit 130. Among them, the memory 110, the processor 120 and the communication unit 130 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, the memory 110, the processor 120 and the communication unit 130 can be electrically connected to each other through one or more communication buses or signal lines.

[0057] In this embodiment, the memory 110 may be, but not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc. The memory 110 is used to store a computer program, and the processor 120 may execute the computer program accordingly after receiving an execution instruction.

[0058] In this embodiment, the processor 120 may be an integrated circuit chip with signal processing capability. The processor 120 may be a general-purpose processor, including a central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or at least one of other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc., which may implement or execute the disclosed methods, steps, and logic block diagrams in the embodiments of the present application.

[0059] In this embodiment, the communication unit 130 is used to establish a communication connection between the computer device 100 and other electronic devices through a network, and to send and receive data through the network, wherein the network includes a wired communication network and a wireless communication network. For example, the computer device 100 can be connected to the image display device 12 through the communication unit 130 to display the specific task execution details of the humanoid robot 20 for various operation tasks to the robot teleoperator through the image display device 12, so that the robot teleoperator can adjust his own human body movement posture in time to teleoperate the humanoid robot 20 to successfully perform the corresponding operation task as soon as possible.

[0060] In an embodiment of the present application, the computer device 100 may pre-store a specific computer program related to the robot learning dataset construction function in the memory 110, and by driving the processor 120 to execute the specific computer program stored in the memory 110, while ensuring that the robot remote operator can remotely operate the humanoid robot 20 to perform various operation tasks based on the inertial motion capture device 11, collect real-time operation images and real-time joint motion data of the humanoid robot 20 performing operation tasks under manual guidance to construct a dataset, so as to effectively improve the construction efficiency, validity and integrity of the task learning dataset, and facilitate further improving the generalization and accuracy of the humanoid robot imitation learning.

[0061] Understandably, Figure 2 The block diagram shown is only a schematic diagram of a composition of the computer device 100. The computer device 100 may also include Figure 2 More or fewer components as shown, or with Figure 2 Different configurations shown. Figure 2 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0062] In the present application, in order to ensure that the computer device 100 can quickly construct a task learning data set with strong corresponding data set validity and data set integrity for the humanoid robot 20 while ensuring that the robot teleoperator can remotely operate the humanoid robot 20 based on the inertial motion capture device 11 to perform various operation tasks, so as to improve the generalization and accuracy of the imitation learning of the humanoid robot, the embodiment of the present application provides a robot learning data set construction method to achieve the above purpose. The robot learning data set construction method provided in the present application is described in detail below.

[0063] Please refer to Figure 3 , Figure 3 This is one of the flow charts of the method for constructing a robot learning dataset provided in the embodiment of the present application. In the embodiment of the present application, the method for constructing a robot learning dataset may include steps S210 to S240.

[0064] Step S210, during any task execution process of the learning operation task, real-time joint motion data of the humanoid robot is obtained, and real-time image data captured by the humanoid robot for the robot operating environment and task target is obtained and displayed.

[0065] In this embodiment, the operation task to be learned is any operation task that needs to be imitated and learned. The operation task to be learned can be an upper body operation task involving only the upper body of the humanoid robot (for example, "grasping operation task", "sorting operation task", "welding operation task", etc.), or a lower body operation task involving only the lower body of the humanoid robot (for example, "balancing car driving operation task", "jumping over obstacles operation task", etc.), or a whole-body collaborative operation task involving the whole body of the humanoid robot (for example, "bicycle driving task", "climbing operation task", etc.); the task target is the target object corresponding to the operation task to be learned, and different operation tasks to be learned correspond to different task targets. The task targets involved in the same operation task to be learned during different task execution processes may be the same or different.

[0066] In this embodiment, the real-time joint motion data of the humanoid robot 20 includes the actual joint motion data of each of the multiple moving body parts of the humanoid robot 20 in the corresponding body workspace coordinate system, and the actual joint motion data of each moving body part includes the actual joint position, actual joint angle, actual joint angular velocity, actual joint angular acceleration, actual joint torque and other joint data of all robot joints involved in the moving body part in the corresponding body workspace coordinate system. Among them, the moving body parts of the humanoid robot 20 are composed of the robot structure of the humanoid robot 20, each moving body part corresponds to a body workspace coordinate system, and each moving body part can move independently relative to other moving body parts. For example, the moving body parts of the humanoid robot 20 may consist of a "robot head", "left robotic arm", "right robotic arm", "left robotic hand", "right robotic hand" and "lower limb structure", wherein the robot joints involved in the "robot head" include head joints, the robot joints involved in the "left robotic arm" and the "right robotic arm" respectively include shoulder joints, elbow joints and wrist joints, the robot joints involved in the "left robotic hand" and the "right robotic hand" respectively include finger joints or claw joints, and the robot joints involved in the "lower limb structure" include hip joints, knee joints, ankle joints, etc.

[0067] In the present embodiment, each operation task to be learned corresponds to an initial posture for task execution; for each operation task to be learned, before each execution of the operation task to be learned, the humanoid robot 20 needs to be initialized (including unifying the internal clocks of the various device structures included in the humanoid robot 20) so that the humanoid robot 20 maintains the initial posture for task execution corresponding to the operation task to be learned. At the same time, the robot remote operator also needs to maintain the initial posture for task execution corresponding to the operation task to be learned to eliminate the wearing error of the inertial motion capture device 11 on the robot remote operator. Only then can the robot remote operator remotely operate the humanoid robot 20 to execute the operation task to be learned by adjusting the human body posture.

[0068] Step S220, obtaining the human body motion posture data collected in real time by the inertial motion capture device for the robot teleoperator, and converting the human body motion posture data into expected joint motion data adapted to the humanoid robot.

[0069] In this embodiment, the inertial motion capture device 11 includes a plurality of inertial posture sensors respectively worn on different human skeleton parts of the robot teleoperator, wherein the wearing position of the inertial posture sensors of the inertial motion capture device 11 is related to the task type of the operation task to be learned; Figure 4 For example, the 11 inertial gesture sensors (such as Figure 4 The wearing positions of the inertial motion capture device 11 on the upper body of the human body are the head bones, the shoulders bones, the upper arms bones, the lower arms bones, the palm bones, the back spine bones, and the waist hip joint bones. At this time, the human motion posture data collected by the inertial motion capture device 11 will include the skeletal motion posture data collected by the multiple inertial posture sensors in the motion capture coordinate system.

[0070] In this embodiment, after obtaining the human motion posture data of the robot teleoperator during the execution of a task to be learned, the computer device 100 will map the human motion posture data to the robot coordinate system of the humanoid robot 20 to obtain the expected joint motion data that the robot teleoperator expects the humanoid robot 20 to achieve, so as to ensure that the humanoid robot 20 and the robot teleoperator achieve a motion posture synchronization effect through the expected joint motion data.

[0071] Optionally, see Figure 5 , Figure 5 yes Figure 3Schematic diagram of the flow chart of the sub-steps included in step S220. In the embodiment of the present application, the step of "converting the human motion posture data into the expected joint motion data adapted to the humanoid robot 20" in step S220 may include sub-steps S221 to S222, so as to convert the human motion posture data of the robot teleoperator into the expected joint motion data achievable by the humanoid robot 20.

[0072] Sub-step S221, according to the human skeleton data of the robot teleoperator, the human joint motion state is estimated based on the skeleton motion posture data of each of multiple inertial posture sensors in the motion capture coordinate system to obtain the human joint motion data of the robot teleoperator in the motion capture coordinate system.

[0073] In this embodiment, each inertial posture sensor included in the inertial motion capture device 11 can be built-in with a high-dynamic three-axis accelerometer, a three-axis gyroscope and a three-axis magnetometer, then the computer device 100 can be based on the human skeleton data of the robot remote operator (including the bone lengths of various human bones of the robot remote operator, and the motion transmission relationship between the various human bones, etc.), through the nine-axis data fusion algorithm and the Kalman filtering algorithm, to perform human joint posture estimation on the skeletal motion posture data collected by each of the multiple inertial posture sensors, and obtain the human joint motion data of the robot remote operator in the motion capture coordinate system (including the human joint position, human joint angle, human joint angular velocity, human joint angular acceleration, human joint torque and other joint data of all human joints of the robot remote operator related to the operation task to be learned in the motion capture coordinate system).

[0074] Sub-step S222, mapping the human joint motion data in the motion capture coordinate system to the robot coordinate system of the humanoid robot to obtain expected joint motion data.

[0075] In this embodiment, the robot coordinate system of the humanoid robot 20 includes the body workspace coordinate systems of the multiple moving body parts of the humanoid robot 20, then the computer device 100 can classify the human joint motion data according to the moving body parts of the humanoid robot 20, and map the human joint motion data corresponding to different moving body parts to the corresponding body workspace coordinate system, and obtain the expected joint motion data related to different moving body parts. At this time, the expected joint motion data adapted to the humanoid robot 20 can actually include the target joint motion data of the multiple moving body parts of the humanoid robot 20 in the corresponding body workspace coordinate system (including the target joint position, target joint angle, target joint angular velocity, target joint angular acceleration, target joint torque and other joint data of all robot joints involved in the corresponding moving body parts in the corresponding body workspace coordinate system).

[0076] In this case, the sub-step S222 may include: for each moving body part of the humanoid robot 20, extracting effective joint motion data corresponding to the moving body part from the human joint motion data; performing coordinate system transformation on the effective joint motion data corresponding to the moving body part according to the coordinate system transformation relationship between the motion capture coordinate system and the body workspace coordinate system of the moving body part, and obtaining the target joint motion data of the moving body part in the corresponding body workspace coordinate system. For a single moving body part, the effective joint motion data corresponding to the moving body part includes the joint data of all human joints associated with the moving body part in the motion capture coordinate system.

[0077] Therefore, the present application can convert the human motion posture data of the robot teleoperator into the expected joint motion data that can be achieved by the humanoid robot 20 by executing the above sub-steps S221 to S222.

[0078] Step S230: Control the humanoid robot to move according to the expected joint motion data based on the real-time joint motion data.

[0079] In this embodiment, after obtaining the expected joint motion data of the humanoid robot 20 that is synchronized with the posture of the robot teleoperator, the computer device 100 will generate an adaptive motion control instruction for the humanoid robot 20 based on the real-time joint motion data of the humanoid robot 20, and then send the motion control instruction to the humanoid robot 20 for execution to drive the humanoid robot 20 to change from the robot motion posture represented by the real-time joint motion data to the robot motion posture represented by the expected joint motion data. At the same time, the computer device 100 will also jump to the above-mentioned step S210 to continue execution until the humanoid robot 20 successfully completes a task execution operation of the to-be-learned operation task for the task target under the manual guidance and teleoperation of the robot teleoperator.

[0080] Optionally, see Figure 6 , Figure 6 yes Figure 3 Schematic diagram of the flow chart of the sub-steps included in step S230. In the embodiment of the present application, the real-time joint motion data of the humanoid robot 20 includes the actual joint motion data of each of the multiple moving body parts of the humanoid robot 20 in the corresponding body workspace coordinate system, and the expected joint motion data adapted to the humanoid robot 20 includes the target joint motion data of each of the multiple moving body parts in the corresponding body workspace coordinate system, then the step S230 may include sub-steps S231 to 233, so as to achieve smooth motion control effects in different workspaces for the humanoid robot 20.

[0081] Sub-step S231 , for each moving body part of the humanoid robot, a target planning strategy matching the moving body part is selected from a plurality of pre-stored smooth trajectory planning strategies for the moving body parts.

[0082] In this embodiment, the smooth trajectory planning strategies of different moving body parts can be completely the same, partially the same, or completely different. For example, for the upper body operation task, the moving body components involved can be "robot head", "left mechanical arm", "right mechanical arm", "left mechanical hand" and "right mechanical hand", wherein the smooth trajectory planning strategies of "left mechanical arm" and "right mechanical arm" respectively adopt "5th spline curve interpolation trajectory planning algorithm", the smooth trajectory planning strategies of "left mechanical hand" and "right mechanical hand" respectively adopt "5th spline curve interpolation trajectory planning algorithm" or "linear interpolation trajectory planning algorithm", and the smooth trajectory planning strategy of "robot head" adopts "5th spline curve interpolation trajectory planning algorithm" or "linear interpolation trajectory planning algorithm".

[0083] Sub-step S232, in the body workspace coordinate system of the moving body part, smooth motion trajectory planning is performed based on the actual joint motion data and target joint motion data of the moving body part according to the target planning strategy to obtain the expected smooth motion trajectory of the moving body part.

[0084] Sub-step S233, controlling all robot joints included in the moving body part to move separately according to the desired smooth motion trajectory.

[0085] Therefore, the present application can achieve smooth motion control effects in different workspaces for the humanoid robot 20 by executing the above sub-steps S231 to S233.

[0086] Step S240, when the humanoid robot successfully completes a task execution operation for the task target, the real-time joint motion data and real-time image data related to the task execution operation are sorted to obtain a learning data sample in the task learning data set for the operation task to be learned.

[0087] In this embodiment, each task learning data set corresponds to only one operation task to be learned, and different learning data samples in the same task learning data set correspond to one task execution operation of the operation task to be learned. For any one learning data sample corresponding to the operation task to be learned, the learning data sample implemented in the HDF5 format can be obtained by arranging all real-time joint motion data and all real-time image data involved in the corresponding task execution operation at the time dimension level. The computer device 100 can perform the above steps S210 to S240 multiple times for the same operation task to be learned to construct a task learning data set that matches the operation task to be learned.

[0088] Optionally, in one implementation of the present embodiment, in order to avoid retaining a large amount of unnecessary data related to unnecessary human motion postures of the robot remote operator in the constructed learning data sample, the step S240 may include: sampling the real-time joint motion data and real-time image data related to the operation of the task execution at preset time intervals to obtain robot joint sampling data and observation image sampling data for each of the multiple sampling time points; constructing a time series of the robot joint sampling data and observation image sampling data for each of the multiple sampling time points to obtain a learning data sample for the operation task to be learned.

[0089] Therefore, the present application can ensure that the constructed learning data samples can retain as little unnecessary data related to unnecessary human motion postures as possible by executing the specific step process of the above-mentioned step S240, so as to facilitate improving the data set effectiveness of the task learning data set, as well as the generalization and accuracy of humanoid robot imitation learning.

[0090] The present application can execute the above-mentioned steps S210 to S240 to ensure that the robot remote operator remotely operates the humanoid robot 20 to perform various operation tasks based on the inertial motion capture device 11, while quickly constructing a task learning data set with high corresponding data set validity and data set integrity for the humanoid robot 20, so as to facilitate improving the generalization and accuracy of the humanoid robot's imitation learning.

[0091] Optionally, see Figure 7 , Figure 7 This is the second flow chart of the method for constructing a robot learning data set provided in the embodiment of the present application. Figure 3 Compared with the robot learning dataset construction method shown in Figure 7 The robot learning data set construction method shown may further include step S250.

[0092] Step S250, calibrating and configuring the coordinate system transformation relationship between the body workspace coordinate system of the multiple moving body parts of the humanoid robot and the motion capture coordinate system of the inertial motion capture device, and configuring the smooth trajectory planning strategy of the multiple moving body parts.

[0093] In this embodiment, as the robot construction conditions of the humanoid robot 20 are different, the composition of the moving body parts of the humanoid robot 20 also has some differences. For example, a humanoid robot 20 with a waist joint has one more moving body part "robot waist" than a humanoid robot 20 without a waist joint. Therefore, the task learning data set constructed by the computer device 100 needs to be adapted to the humanoid robot construction conditions. Before executing the learning data set construction operation, the computer device 100 needs to first establish the coordinate system transformation relationship between the body workspace coordinate system of each of the multiple moving body parts of the humanoid robot 20 and the motion capture coordinate system of the inertial motion capture device 11 based on the robot construction conditions of the humanoid robot 20 for which the learning data set needs to be constructed, and at the same time obtain the smooth trajectory planning strategies configured by the operator for the multiple moving body parts of the humanoid robot 20 and save them, so as to ensure that any learning data set finally constructed is adapted to the robot construction conditions of the humanoid robot 20.

[0094] Therefore, the present application can ensure that any learning data set finally constructed can be adapted to the robot structural condition of the remotely operated humanoid robot 20 by executing the above step S250.

[0095] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic, for example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and a module, a program segment or a part of a code contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0096] In addition, each functional module in each embodiment of the present application can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part. If the various functions provided by the present application are implemented in the form of software functional modules and sold or used as independent products, they can be stored in a storage medium. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a readable storage medium, including several instructions to enable a computer device 100 (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method recorded in each embodiment of the present application. The aforementioned readable storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0097] The above are only various implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for constructing a robot learning dataset, characterized in that: The method comprises: During any task execution process of the learning operation task, real-time joint motion data of the humanoid robot is obtained, and real-time image data captured by the humanoid robot for the robot operating environment and task target is obtained for display; Acquire human motion posture data collected in real time by an inertial motion capture device for a robot teleoperator, and convert the human motion posture data into expected joint motion data adapted to the humanoid robot; Based on the real-time joint motion data, the humanoid robot is controlled to move according to the expected joint motion data, and the step of obtaining the real-time joint motion data of the humanoid robot is skipped to continue execution until the humanoid robot successfully completes a task execution operation for the task target; The real-time joint motion data and the real-time image data related to the task execution operation are sorted to obtain a learning data sample in the task learning data set for the operation task to be learned, wherein each learning data sample corresponds to a task execution operation of the operation task to be learned.

2. The method according to claim 1, characterized in that The inertial motion capture device includes a plurality of inertial posture sensors, and the plurality of inertial posture sensors are respectively worn on different human skeleton parts of the robot teleoperator, and the human motion posture data includes the skeleton motion posture data respectively collected by the plurality of inertial posture sensors in the motion capture coordinate system, and the step of converting the human motion posture data into the expected joint motion data adapted to the humanoid robot includes: According to the human skeleton data of the robot teleoperator, human joint motion state estimation is performed based on the skeleton motion posture data of each of the multiple inertial posture sensors in the motion capture coordinate system to obtain the human joint motion data of the robot teleoperator in the motion capture coordinate system; The human joint motion data in the motion capture coordinate system is mapped to the robot coordinate system of the humanoid robot to obtain the expected joint motion data.

3. The method according to claim 2, characterized in that The robot coordinate system of the humanoid robot includes the body workspace coordinate systems of the multiple moving body parts of the humanoid robot, and the expected joint motion data includes the target joint motion data of the multiple moving body parts in the corresponding body workspace coordinate system. The step of mapping the human joint motion data in the motion capture coordinate system to the robot coordinate system of the humanoid robot to obtain the expected joint motion data includes: For each moving body part of the humanoid robot, extracting effective joint motion data corresponding to the moving body part from the human joint motion data; According to the coordinate system transformation relationship between the motion capture coordinate system and the body workspace coordinate system of the moving body part, the effective joint motion data corresponding to the moving body part is transformed to obtain the target joint motion data of the moving body part in the corresponding body workspace coordinate system.

4. The method according to claim 3, characterized in that The real-time joint motion data includes actual joint motion data of each of the plurality of moving body parts in the corresponding body workspace coordinate system, and the step of controlling the humanoid robot to move according to the expected joint motion data based on the real-time joint motion data includes: For each moving body part of the humanoid robot, a target planning strategy matching the moving body part is selected from the pre-stored smooth trajectory planning strategies of the plurality of moving body parts; In the body workspace coordinate system of the moving body part, smooth motion trajectory planning is performed based on the actual joint motion data and target joint motion data of the moving body part according to the target planning strategy to obtain the expected smooth motion trajectory of the moving body part; All robot joints included in the moving body part are controlled to move separately according to the desired smooth motion trajectory.

5. The method according to claim 1, characterized in that: The step of arranging the real-time joint motion data and the real-time image data related to the task execution operation to obtain a learning data sample in the task learning data set for the operation task to be learned comprises: The real-time joint motion data and the real-time image data are sampled at preset time intervals to obtain robot joint sampling data and observation image sampling data at respective sampling time points; A time series is constructed for the robot joint sampling data and the observation image sampling data at each of the plurality of sampling time points to obtain a learning data sample for the operation task to be learned.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: The coordinate system transformation relationship between the body workspace coordinate system of each of the multiple moving body parts of the humanoid robot and the motion capture coordinate system of the inertial motion capture device is calibrated and configured, and the smooth trajectory planning strategy of each of the multiple moving body parts is strategically configured.

7. A computer device, characterized in that: The computer device is in communication connection with the humanoid robot and the inertial motion capture device, wherein the inertial motion capture device comprises a plurality of inertial posture sensors respectively worn on different human skeleton parts of the robot teleoperator; The computer device includes a processor and a memory, the memory stores a computer program that can be executed by the processor, and the processor can execute the computer program to implement the robot learning data set construction method described in any one of claims 1-6.

8. A robot teleoperation system, characterized in that: The system comprises an inertial motion capture device and a computer device, wherein the computer device is respectively connected to the humanoid robot and the inertial motion capture device for communication, and the inertial motion capture device comprises a plurality of inertial posture sensors respectively worn on different human skeleton parts of the robot teleoperator; The computer device stores a computer program and can run the computer program to cooperate with the inertial motion capture device to implement the robot learning data set construction method described in any one of claims 1-6.

9. The system according to claim 8, characterized in that The system also includes an image display device; The image display device is deployed in front of the robot teleoperator and is communicatively connected with the computer device to display the real-time image acquired by the computer device from the humanoid robot.

10. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a computer device, the robot learning data set construction method described in any one of claims 1 to 6 is implemented.

Images