Method and system for training motion control model and robot

Abstract

The invention provides a method, a system and a robot for training a motion control model. The method comprises the following steps: acquiring walking state information and image information of a humanoid robot; the image information is acquired when the humanoid robot performs head action according to the head posture control instruction in the walking process, and the image information is in one-to-one correspondence with the walking state information; determining corresponding second terrain information according to the image information; constructing a motion control model and carrying out model training; wherein the training sample of the operation control model comprises walking state information, image information and second terrain information of the humanoid robot.

Description

Technical Field

[0001] The present disclosure relates to the field of robots, and in particular to a method, system and robot for training a motion control model. Background Art

[0002] Traditional humanoid robot multi-terrain training is carried out in simulation, and it is difficult to introduce visual information. As a result, after the algorithm is deployed on the humanoid robot, there are problems such as bumps and large energy loss when moving on complex terrain (such as rugged roads, stairs, and climbing slopes). Even if visual information is added to the simulation, due to the difficulty of simulating visual information and the large difference between simulation and actual scenes, after the trained model is deployed on the real machine, it is easy to have problems such as poor algorithm stability and difficulty for the robot to walk stably. Summary of the invention

[0003] The present disclosure provides a method, system and robot for training a motion control model to at least solve the above technical problems existing in the prior art.

[0004] According to a first aspect of the present disclosure, a method for training a motion control model is provided, wherein the motion control model is used to control a humanoid robot to walk; the head of the humanoid robot can collect images; the humanoid robot is deployed with a head posture supervision model, and the head posture supervision model is used to output a head posture control instruction of the humanoid robot based on input information of the head posture supervision model; the input information of the head posture supervision model includes: walking state information of the humanoid robot and first terrain information; the first terrain information is determined based on the walking state information of the humanoid robot;

[0005] The method includes:

[0006] Acquire walking state information and image information of the humanoid robot; the image information is image information collected when the humanoid robot performs head movements according to the head posture control instruction during walking, and the image information corresponds to the walking state information one by one;

[0007] determining corresponding second terrain information according to the image information;

[0008] Construct a motion control model and perform model training; wherein, the training samples of the motion control model include: walking state information, image information, and second terrain information of the humanoid robot.

[0009] In one possible implementation manner, before the step of obtaining walking state information and image information of the humanoid robot, the method further includes:

[0010] Obtain walking state information and head posture information of the human body;

[0011] Determining corresponding first terrain information according to the walking state information of the human body;

[0012] A head posture supervision model is constructed and the model is trained; wherein the training samples of the head posture supervision model include: human walking state information, head posture information, and first terrain information.

[0013] In one possible implementation, the humanoid robot is further deployed with a simulation motion control model, which is used to output joint control instructions of the humanoid robot based on input information of the simulation motion control; the input information of the simulation motion control model includes: walking state information of the humanoid robot and first terrain information; the first terrain information is determined based on the walking state information of the humanoid robot;

[0014] The walking state information of the humanoid robot is collected when the humanoid robot moves according to the joint control instructions.

[0015] In one possible implementation manner, before the step of obtaining walking state information and image information of the humanoid robot, the method further includes:

[0016] Build a humanoid robot simulation model;

[0017] Obtain walking state information of the humanoid robot simulation model;

[0018] Determine corresponding third terrain information according to walking state information of the humanoid robot simulation model;

[0019] A simulation motion control model is constructed and model training is performed; wherein the training samples of the simulation motion control model include: walking state information of the humanoid robot simulation model and third terrain information.

[0020] In one embodiment, the steps of constructing a simulation motion control model and performing model training include:

[0021] Set the reward function J(π):

[0022]

[0023] τ={(x 0 ,z 0 ),(x 1 ,z 1 ),…,(x T-1 ,z T-1 )}

[0024] Where E is the expected value; τ is the trajectory when executing the simulated motion control model π; T is the reward period; r t For reward; t is the reward discount; p(τ|π) is the maximum likelihood of the simulated motion control model π; x 0 ~x T-1The walking state information of the humanoid robot simulation model; 0 ~z T-1 is the third terrain information;

[0025] The model is trained and the training is completed when the expected value of the reward reaches the expected threshold.

[0026] In one possible implementation manner, the walking state information of the humanoid robot simulation model includes: the linear velocity v of the humanoid robot simulation model t (t), v y (t), v z (t), horizontal turning speed ω of the humanoid robot simulation model z (t), joint angle q of the humanoid robot simulation model i (t), the sliding speed of the sole of the supporting leg of the humanoid robot simulation model on the ground

[0027] In one embodiment, the reward r is calculated in the following manner: t ,

[0028]

[0029] in,

[0030]

[0031]

[0032] Among them, r t The reward for each training cycle; The linear speed reward for the robot moving in three-dimensional space; The speed reward for the robot turning in the horizontal direction; Tracking rewards for the robot’s joint angles in joint space; A bonus for the sliding speed of the sole of the supporting leg on the ground; v ,a ω ,a q ,a s are the weight coefficients of the rewards corresponding to each motion parameter of the robot; v x (t), v y (t), v z (t) are the linear velocities along the three coordinate axes in three-dimensional space; are the expected values ​​of the linear velocity of the humanoid robot along the three coordinate axes in three-dimensional space; ω z (t) is the horizontal turning speed of the current humanoid robot simulation model; is the expected value of the horizontal turning speed of the humanoid robot simulation model; q i (t) is the joint angle of the current humanoid robot simulation model; is the expected value of the joint angle of the humanoid robot simulation model; diag(g) t is a diagonal function; It is the sliding speed of the sole of the supporting leg of the current humanoid robot simulation model on the ground.

[0033] In one embodiment, the head posture supervision model uses a supervised learning neural network.

[0034] According to a second aspect of the present disclosure, a training motion control model system is provided, wherein the motion control model is used to control a humanoid robot to walk; the head of the humanoid robot can collect images; the humanoid robot is deployed with a head posture supervision model, and the head posture supervision model is used to output a head posture control instruction of the humanoid robot based on input information of the head posture supervision model; the input information of the head posture supervision model includes: walking state information of the humanoid robot and first terrain information; the first terrain information is determined based on the walking state information of the humanoid robot;

[0035] The system includes:

[0036] A humanoid robot information acquisition module is used to acquire walking state information and image information of the humanoid robot; the image information is image information collected when the humanoid robot performs head movements according to the head posture control instructions during walking, and the image information corresponds to the walking state information one by one;

[0037] A second terrain determination module, used to determine corresponding second terrain information according to the image information;

[0038] The real motion training module is used to build a motion control model and perform model training; wherein, the training samples of the motion control model include: walking state information, image information, and second terrain information of the humanoid robot.

[0039] According to a third aspect of the present disclosure, there is provided a robot, comprising:

[0040] at least one processor; and

[0041] a memory communicatively connected to at least one processor; wherein,

[0042] The memory stores instructions that can be executed by at least one processor. The instructions are executed by the at least one processor so that the at least one processor can perform the method of the present disclosure.

[0043] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause a computer to execute the method of the present disclosure.

[0044] The method, system and robot for training motion control models disclosed in the present invention install a head posture supervision model on a humanoid robot, so that the humanoid robot outputs head posture control instructions of the humanoid robot based on the input information of the head posture supervision model, and collects image information when performing head movements according to the head posture control instructions during walking, determines the second terrain information according to the image information, and trains the motion control model according to the walking state information, image information and second terrain information of the humanoid robot. The present invention adds visually generated image information to the training process of the operation control model, which solves the problem of poor walking effect and bumps caused by the lack of visual information in the reinforcement learning training of traditional humanoid robots. It enables the humanoid robot to walk on multiple terrains in a real environment, improves the terrain adaptability of the humanoid robot, and reduces energy consumption.

[0045] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:

[0047] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts.

[0048] Figure 1 The schematic diagram shows the implementation process of the method for training the motion control model in the embodiment of the present disclosure. Figure 1 ;

[0049] Figure 2 The schematic diagram shows the implementation process of the method for training the motion control model in the embodiment of the present disclosure. Figure 2 ;

[0050] Figure 3 The schematic diagram shows the implementation process of the method for training the motion control model in the embodiment of the present disclosure. Figure 3 ;

[0051] Figure 4 The schematic diagram shows the implementation process of the method for training the motion control model in the embodiment of the present disclosure. Figure 4 ;

[0052] Figure 5A schematic diagram of the structure of a system for training a motion control model according to an embodiment of the present disclosure is shown;

[0053] Figure 6 A schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0054] In order to make the purpose, features, and advantages of the present disclosure more obvious and easy to understand, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0055] A method, system and robot for training a motion control surface model provided by the present disclosure are described below in conjunction with the accompanying drawings.

[0056] Figure 1 A method for training a motion control model provided by some embodiments of the present disclosure is shown, wherein the motion control model is used to control a humanoid robot to walk; the head of the humanoid robot can collect images; the humanoid robot is deployed with a head posture supervision model, and the head posture supervision model is used to output a head posture control instruction of the humanoid robot based on input information of the head posture supervision model; the input information of the head posture supervision model includes: walking state information of the humanoid robot and first terrain information; the first terrain information is determined based on the walking state information of the humanoid robot;

[0057] The motion control model provided by the present disclosure can be applied to a humanoid robot, so that the humanoid robot can walk under the control of the motion control model. The humanoid robot can be a bionic robot, which usually has a humanoid structure such as a head, a torso and four fingers. The humanoid robot can simulate the movement of human joints and achieve bipedal walking by driving a motor and a control system. Optionally, a suitable sensor is equipped on the humanoid robot to collect at least one information such as the posture, joint state, and movement speed of the humanoid robot. Collecting various types of sensor information during the walking state of the humanoid robot is conducive to a comprehensive understanding of the state of the robot, improving the accuracy and reliability of state estimation, and making the movement of the robot closer to the target movement.

[0058] The head of the humanoid robot in the disclosed embodiment has an image acquisition function, so that the humanoid robot has image perception ability. The image captured by the head of the humanoid robot can be an image within the framing range taken under the current head posture, and the image may include the terrain of the robot's environment. Optionally, an image acquisition device (such as a camera) can be set on the head of the humanoid robot, and the image acquisition device is rigidly connected to the head of the humanoid robot, and image information is collected through the image acquisition device. Optionally, the image acquisition device can be set to imitate the position of the human eye, so as to realize most of the visual functions of the human eye.

[0059] In the disclosed embodiment, the humanoid robot is deployed with a head posture supervision model, which can output the head posture control instructions of the humanoid robot according to the input walking state information of the humanoid robot and the first terrain information, and the humanoid robot controls the head posture of the humanoid robot according to the control instructions. It should be noted that the head posture of the humanoid robot determines the posture of the image acquisition device on the head, that is, it determines the field of view of the humanoid robot. It is understandable that when humans actually walk, they will adjust their field of view according to the state and terrain during walking, so as to avoid obstacles and prevent bumps. By deploying the head posture supervision model, the humanoid robot can imitate the field of view adjustment action of humans when walking.

[0060] In an optional implementation, the walking state information of the humanoid robot may include the body posture, joint state, and travel speed of the humanoid robot.

[0061] The first terrain information is the terrain information of the actual walking environment of the humanoid robot, and the first terrain information can be inferred based on the walking state information of the humanoid robot. In an optional implementation, the humanoid robot collects the joint mobility of the humanoid robot when walking, including the range of motion of the hip, knee, and ankle during walking, the body posture, and the travel speed, through the sensor, and can infer the terrain information of the current walking environment. For example, when the sensor collects that the knee joint mobility of the humanoid robot is about 90 degrees when walking, the body posture is forward, and the travel speed is slower than the normal walking speed, it is judged that the terrain of the current walking environment is stairs. Or when the sensor collects that the knee joint mobility of the humanoid robot is less than 70 degrees when walking, the body posture is backward, and the travel speed is faster than the normal walking speed, it is judged that the terrain of the current walking environment is downhill.

[0062] like Figure 1 As shown, some embodiments of the present disclosure provide a method for training a motion control model, including:

[0063] S101, obtaining walking state information and image information of the humanoid robot; the image information is image information collected when the humanoid robot performs head movements according to the head posture control instruction during walking, and the image information corresponds to the walking state information one by one;

[0064] In the disclosed embodiment, the walking state information of the humanoid robot can be collected by sensors. The head posture supervision model takes the walking state information as input and outputs the head posture control instructions of the humanoid robot. During the walking process of the humanoid robot, the humanoid robot performs head movements according to the head posture control instructions. In this way, the walking state information and image information are obtained, and the image information corresponds to the walking state information one by one.

[0065] It can be understood that the walking state information of the humanoid robot is obtained through the sensor and then input into the head posture supervision model, thereby obtaining the head posture control instruction of the humanoid robot, and the humanoid robot controls the head rotation according to the head posture control instruction to obtain the corresponding image information. Therefore, the walking state information, the head posture control instruction and the image information data correspond one to one.

[0066] S102, determining corresponding second terrain information according to the image information;

[0067] In the actual walking process of the humanoid robot in the disclosed embodiment, the humanoid robot performs head movements based on the head posture control command. The image captured by the image acquisition device on the head of the humanoid robot usually contains environmental information, for example, it may include at least one information such as the environment in front, the environment under the feet, and objects in the environment. The terrain information can be obtained by using an image algorithm. Among them, the image algorithm can adopt an existing image processing method.

[0068] S103, constructing a motion control model and performing model training; wherein the training samples of the motion control model include: walking state information, image information, and second terrain information of the humanoid robot.

[0069] The motion control model provided by the embodiments of the present disclosure may adopt a neural network algorithm, for example, a convolutional neural network, a recurrent neural network, and a deep learning neural network.

[0070] The present disclosure uses walking state information, image information and second terrain information obtained by the humanoid robot during actual walking as training data to train a neural network, thereby obtaining a motion control model, and then using the motion control model to control the walking of the humanoid robot.

[0071] The method for training a motion control model provided by the present disclosure, during the walking process of a humanoid robot, first obtains the walking state information of the humanoid robot, and uses the head posture supervision model pre-arranged in the humanoid robot to obtain the head posture control instruction according to the walking state information of the humanoid robot; the humanoid robot performs head movements according to the head posture control instruction, and when performing the head movement, uses the camera on the head to collect image information, and obtains the second terrain information through the image information; finally, the walking state information, image information and the second terrain information of the humanoid robot are used to train the operation control model. The method for training a motion control model provided by the present disclosure obtains visual information through a human-machine robot, and uses the visual information for the training of the operation control model, which solves the problem of poor walking effect and collision caused by the lack of visual information in the reinforcement learning training of the traditional humanoid robot deployment model. Compared with the operation control model obtained by adding visual information to the simulation and performing model training, the operation control accuracy and stability are improved. The embodiment of the present disclosure deploys the trained motion control model in the humanoid robot, which can control the humanoid robot to walk stably on multiple terrains, improve the robot's terrain adaptability, reduce collisions, and reduce energy consumption.

[0072] In addition, the technical solution disclosed in the present invention utilizes the humanoid robot to collect images as visual information through its head during actual walking. Compared with the visual information added through the simulation scene, the acquired visual data is more consistent with the terrain information. Then the one-to-one corresponding image information, walking state information and second terrain information are input into the model for training. Not only is the introduction of visual information into training highly operational, but it can also improve the walking effect of the humanoid robot and ensure the humanoid robot's ability to adapt to complex terrain.

[0073] In some embodiments, Figure 2 As shown, before the step of obtaining the walking state information and image information of the humanoid robot, the method further includes:

[0074] S201, obtaining walking state information and head posture information of a human body;

[0075] S202, determining corresponding first terrain information according to the walking state information of the human body;

[0076] S203, constructing a head posture supervision model and performing model training; wherein the training samples of the head posture supervision model include: human walking state information, head posture information, and first terrain information.

[0077] In the disclosed embodiments, the head posture supervision model and the motion control model are applied to the humanoid robot, and the walking state information of the humanoid robot may include, for example, the joint state, travel speed, body posture, etc. of the humanoid robot. Therefore, in order to make the movements of the humanoid robot highly anthropomorphic so as to achieve human-like movement, the human expert teaching data can be collected and used for model training, and then applied to the humanoid robot to achieve the anthropomorphic movement of the humanoid robot. Among them, the obtained human expert teaching data can be obtained from a human body with a similar height and proportion to the humanoid robot, or it can be collected from human bodies of different heights and proportions, and then obtained through mapping transformation to obtain human data for training.

[0078] In the present disclosure, the joint state and travel speed of the human expert during the movement process are collected, and the head posture information of the human expert due to observing the first terrain information is collected by using an inertial sensor. The joint state, travel speed, head posture and first terrain information collected by the human expert walking on different terrains are used as training samples, and the model is trained by supervised learning to obtain the relationship between the human body movement state information, the external control command, the first terrain information and the head posture of the human expert, that is, the head posture supervision model.

[0079] It is understandable that human experts can wear motion capture suits to collect walking status information; and head posture information can be obtained by setting an inertial measurement unit (IMU) on the head of the human expert, such as three-axis rotation angle information.

[0080] In an optional embodiment, information about each joint of a human body during movement is obtained. And the rotation angle information of the head along the three axes of its own coordinate system Derivation of the first topographic information through joint kinematics information And human walking speed information Using the above information to construct a data set,

[0081]

[0082] In an optional embodiment, the head posture supervision model can be trained by supervised learning. The training sample includes a set of training examples, which include input and correct output. For example, when performing supervised learning training, there are 400 samples in total, and the 400 samples can be divided into 300 training samples and 100 test samples. After the model is trained with 300 training samples, the trained model is tested with 100 test samples. When the accuracy of the test results reaches 90%, the head posture supervision model is output. The head posture supervision model is based on the input joint information. Derivation of the first terrain information through joint information And human walking speed information Output head posture control instructions, that is, the rotation angle information of the head along the three axes of its own coordinate system

[0083] The present invention collects data through actual multi-terrain motion of the human body, trains a head posture supervision model, and obtains the relationship between body data, terrain information and image data, i.e., head field of view, when humans walk on multiple terrains. This facilitates the introduction of image data into the neural network during subsequent motion control model training. This can solve the problems of bumps and large energy losses caused by the lack of visual information in the reinforcement learning training of traditional humanoid robots, and improve the walking effect of humanoid robots.

[0084] In some embodiments of the present disclosure, the humanoid robot is further deployed with a simulation motion control model, which is used to output joint control instructions of the humanoid robot based on input information of the simulation motion control model; the input information of the simulation motion control model includes: walking state information of the humanoid robot and first terrain information; the first terrain information is determined based on the walking state information of the humanoid robot;

[0085] The walking state information of the humanoid robot is collected when the humanoid robot moves according to the joint control instructions.

[0086] It can be understood that in the present disclosure, a simulation motion control model can be deployed in a humanoid robot, and the simulation motion control model can output joint control instructions of the humanoid robot according to the input walking state information of the humanoid robot and the first terrain information to control the walking of the humanoid robot.

[0087] In an optional embodiment, the simulation motion control model in the present disclosure may adopt a multi-layer perceptron network. For example, the input information of the simulation motion control model is the robot walking state information x t , the first terrain information at the current moment γ t , the simulation motion control model outputs the angle control instructions a for each joint of the robot t Or, the input information of the simulation motion control model can be the robot walking state information x t , the first terrain information at the current moment γ t And the current environmental information m t , the simulation motion control model outputs the angle control instructions a for each joint of the robot t Optionally, the robot walking state information x tIt may include the robot's body posture, joint status, travel speed, etc. The environmental information may include at least one of friction force and contact force. The first terrain information may include at least one of ground height and terrain type (such as steps, flat bottom, steep slope).

[0088] The present disclosure deploys a simulation motion control model in a humanoid robot, so that the humanoid robot has the basic function of walking on various terrains.

[0089] In some embodiments, Figure 3 As shown, before the step of obtaining the walking state information and image information of the humanoid robot, the method further includes:

[0090] S301, constructing a humanoid robot simulation model;

[0091] S302, obtaining walking state information of the humanoid robot simulation model;

[0092] S303, determining corresponding third terrain information according to the walking state information of the humanoid robot simulation model;

[0093] S304, constructing a simulation motion control model and performing model training; wherein the training samples of the simulation motion control model include: walking state information of the humanoid robot simulation model and third terrain information.

[0094] In the embodiments provided by the present disclosure, a simulation model of a humanoid robot can be first constructed in a simulation environment, and walking state information and third terrain information corresponding to the walking state information can be obtained through the simulation model. The walking state information and the corresponding third terrain information are input into the constructed simulation motion control model for training to obtain a simulation motion control model. The simulation motion control model thus obtained has a multi-terrain walking strategy. The simulation motion control model is trained through the walking state information of the humanoid robot simulation model and the third terrain information corresponding to the walking state information, so that the obtained simulation motion model can obtain corresponding joint control instructions according to the terrain environment, and realize the multi-terrain walking strategy.

[0095] In an optional implementation, the walking state information and the corresponding third terrain information and environmental information are input into the constructed simulation motion control model for training to obtain the simulation motion control model. Thus, the simulation motion control model can predict the angle control instructions of each joint of the robot based on the body state information, the current moment environmental information, and the current moment terrain information. Inputting the current moment environmental information as a training sample into the simulation motion control model to be trained can improve the prediction accuracy of the simulation motion model.

[0096] In some embodiments, the steps of constructing a simulated motion control model and performing model training include:

[0097] Set the reward function J(π):

[0098]

[0099] τ={(x 0 ,z 0 ),(x 1 ,z 1 ),…,(x T-1 ,z T-1 )}

[0100] Where E is the expected value; τ is the trajectory when executing the simulated motion control model π; T is the reward period; r t For reward; t is the reward discount; p(τ|π) is the maximum likelihood of the simulated motion control model π; x 0 ~x T-1 The walking state information of the humanoid robot simulation motion control model; 0 ~z T-1 is the third terrain information;

[0101] The model is trained and the training is completed when the expected value of the reward reaches the expected threshold.

[0102] Optionally, the input information also includes environmental information m t , then τ={(x 0 ,m 0 , z 0 ),(x 1 ,m 1 , z 1 )…(x T-1 ,m T-1 , z T-1 In an optional embodiment, the walking state information of the humanoid robot simulation model includes: the linear velocity v of the humanoid robot simulation model x (t), horizontal turning speed ω of the humanoid robot simulation model z (t), joint angles q of the humanoid robot simulation model i (t), the sliding speed of the sole of the supporting leg of the humanoid robot simulation model on the ground

[0103] It is understandable that when training the simulation motion control model, a reward function is set, and the sum of all rewards in the network is the total reward. When training the model, the total reward obtained needs to be maximized, and when the expected value of the total reward reaches the expected threshold, the motion control model is output. It is understandable that the reward function assigns a reward to each output generated by the model, and a higher reward indicates that the output is closer to the desired goal. The present disclosure uses rewards for multiple motion parameters of a humanoid robot to provide feedback to guide model learning, so that the actions of the humanoid robot are more in line with the expected goals, thereby conducting model training more reasonably and comprehensively.

[0104] In some embodiments, the reward r is calculated in the following manner: t ,

[0105]

[0106] in,

[0107]

[0108] Among them, r t The reward for each training cycle; The linear speed reward for the robot moving in three-dimensional space; The speed reward for the robot turning in the horizontal direction; Tracking rewards for the robot’s joint angles in joint space; A bonus for the sliding speed of the sole of the supporting leg on the ground; v ,a ω ,a q ,a s are the weight coefficients of the rewards corresponding to each motion parameter of the robot (where α v is the weight coefficient of the linear speed reward of the robot moving in three-dimensional space; α w is the weight coefficient of the robot’s joint angle tracking reward in the joint space; α q The weight coefficient of the robot's speed reward for turning horizontally; α s is the weight coefficient of the sliding speed reward of the sole of the robot's supporting leg on the ground); v x (t), v y (t), v z (t) are the linear velocities of the humanoid robot along the three coordinate axes in three-dimensional space; are the expected values ​​of the linear velocity of the humanoid robot along the three coordinate axes in three-dimensional space; ω z (t) is the horizontal turning speed of the current humanoid robot simulation model; is the expected value of the horizontal turning speed of the humanoid robot simulation model; q i(t) is the joint angle of the current humanoid robot simulation model; is the expected value of the joint angle of the humanoid robot simulation model; diag(g) t is a diagonal function; It is the sliding speed of the sole of the supporting leg of the current humanoid robot simulation model on the ground.

[0109] Finally, when the expected value of the reward reaches the expected threshold, the training is completed and the simulation motion control model is output. When the reward reaches the expected threshold, the performance of the simulation motion control model is good. By continuously optimizing the reward, the model performance can be updated and improved, thereby improving the walking ability of the humanoid robot and reducing the energy consumption of the humanoid robot.

[0110] like Figure 4 As shown, the present disclosure can pre-train a simulation motion control model and a head posture supervision model. The simulation motion control model is trained according to the input walking state information of the humanoid robot simulation model and the corresponding third terrain information, and can output the joint control instructions of the humanoid robot. The head posture supervision model is trained according to the input walking state information, head posture information and first terrain information of the human body, and can output the head posture control instructions. The trained simulation motion control model and head posture supervision model are deployed to the humanoid robot, so that the humanoid robot actually walks, thereby obtaining the walking state information x of the robot body. t , first terrain information γ t , the collected image information I t and the second terrain information inferred by the image information at the current moment The walking state x t , the second terrain information And the image information collected by the camera I t As the input of the motion control model, the model training is performed. Among them, the environmental information corresponding to the walking state information can also be used as the input of the motion control model to improve the prediction accuracy of the model. The motion control model is deployed in a humanoid robot, and the motion control model can control the walking of the humanoid robot.

[0111] The motion control model provided by the embodiment of the present disclosure can adopt a multilayer perceptron network. The multilayer perceptron network (Multilayer Perceptron, MLP) is a feedforward neural network, which includes at least three layers of nodes: an input layer, a hidden layer, and an output layer. In addition to the input node, each node is a neuron with a nonlinear activation function. During the training process of MLP, the output is compared with the expected output to generate an error value. This error is then back-propagated to the network, and the weights are updated accordingly.

[0112] The motion control model provided by the present disclosure introduces the image data actually collected by the humanoid robot into the neural network, solving the problems of easy collision and high energy consumption caused by the lack of visual information in the reinforcement learning training of traditional humanoid robots. The present disclosure deploys the head posture supervision model and the simulation motion control model to the humanoid robot, solving the problem of difficulty in introducing visual information into the training, and has strong operability. The motion control model obtained by the present disclosure can make the humanoid robot walk smoothly when walking on multiple terrains, improving the terrain adaptability of the humanoid robot.

[0113] like Figure 5 As shown, the embodiment of the present disclosure provides a system for training a motion control model, the motion control model is used to control a humanoid robot to walk; the head of the humanoid robot can collect images; the humanoid robot is deployed with a head posture supervision model, the head posture supervision model is used to output a head posture control instruction of the humanoid robot based on input information of the head posture supervision model; the input information of the head posture supervision model includes: walking state information of the humanoid robot and first terrain information; the first terrain information is determined based on the walking state information of the humanoid robot;

[0114] The system includes:

[0115] The humanoid robot information acquisition module 501 is used to acquire walking state information and image information of the humanoid robot; the image information is image information collected when the humanoid robot performs head movements according to the head posture control instruction during walking, and the image information corresponds to the walking state information one by one;

[0116] A second terrain determination module 502, configured to determine corresponding second terrain information according to the image information;

[0117] The real motion training module 503 is used to construct a motion control model and perform model training; wherein the training samples of the motion control model include: walking state information, image information, and second terrain information of the humanoid robot.

[0118] The training motion control model system provided by the present disclosure inputs the walking state information and image information of the humanoid robot into the head posture supervision model to obtain the head posture control instruction of the humanoid robot. The image information is collected when the humanoid robot performs head movements according to the head posture control instruction during walking, so as to determine the second terrain information according to the image information. The walking state information, image information, and second terrain information are used as training samples to train the motion control model to obtain the motion control model. The motion control model is deployed in the humanoid robot, and the motion control model controls the walking of the humanoid robot. The motion control model obtained by the present disclosure can avoid the problems of bumps and high energy consumption when the humanoid robot walks on multiple terrains, which is conducive to improving the terrain adaptability of the humanoid robot.

[0119] In some embodiments, the system further comprises:

[0120] A human body information acquisition module is used to acquire walking state information and head posture information of the human body;

[0121] A first terrain determination module, used to determine corresponding first terrain information according to walking state information of a human body;

[0122] The head posture training module is used to construct a head posture supervision model and perform model training; wherein, the training samples of the head posture supervision model include: human walking state information, head posture information, and first terrain information.

[0123] In some embodiments, the humanoid robot is further deployed with a simulation motion control model, which is used to output joint control instructions of the humanoid robot based on input information of the simulation motion control; the input information of the simulation motion control model includes: walking state information of the humanoid robot and first terrain information; the first terrain information is determined based on the walking state information of the humanoid robot;

[0124] The walking state information of the humanoid robot is collected when the humanoid robot moves according to the joint control instructions.

[0125] In some embodiments, the system further comprises:

[0126] A simulation model building module, used to build a simulation model of a humanoid robot;

[0127] A simulation model information acquisition module is used to obtain walking state information of a humanoid robot simulation model;

[0128] A third terrain information determination module, used to determine corresponding third terrain information according to the walking state information of the humanoid robot simulation model;

[0129] The simulation motion training module is used to construct a simulation motion control model and perform model training; wherein the training samples of the simulation motion control model include: walking state information of the humanoid robot simulation model and third terrain information.

[0130] In some embodiments, the simulation motion training module includes: a function setting module, the function setting module is used to set the reward function J(π):

[0131]

[0132] τ={(x 0 ,z 0 ),(x 1 ,z 1 ),…,(x T-1 ,z T-1 )}

[0133] Where E is the expected value; τ is the trajectory when executing the simulated motion control model π; T is the reward period; r t For reward; t is the reward discount; p(τ|π) is the maximum likelihood of the simulated motion control model π; x 0 ~x T-1 The walking state information of the humanoid robot simulation motion control model; 0 ~z T-1 is the third terrain information;

[0134] The model is trained and the training is completed when the expected value of the reward reaches the expected threshold.

[0135] In some embodiments, the walking state information of the humanoid robot simulation model includes: the linear velocity v of the humanoid robot simulation model x (t), v y (t), v z (t), horizontal turning speed ω of the humanoid robot simulation model z (t), joint angle q of the humanoid robot simulation model i (t), the sliding speed of the sole of the supporting leg of the humanoid robot simulation model on the ground

[0136] In some embodiments, the reward r is calculated in the following manner: t ,

[0137]

[0138] in,

[0139]

[0140] Among them, r t The reward for each training cycle; The linear speed reward for the robot moving in three-dimensional space; The speed reward for the robot turning in the horizontal direction; Tracking rewards for the robot’s joint angles in joint space; A bonus for the sliding speed of the sole of the supporting leg on the ground; v ,a ω ,a q ,a s are the weight coefficients of the rewards corresponding to each motion parameter of the robot; v x (t), v y (t), v z (t) are the linear velocities of the humanoid robot along the three coordinate axes in three-dimensional space; are the expected values ​​of the linear velocity of the humanoid robot along the three coordinate axes in three-dimensional space; ω z (t) is the horizontal turning speed of the current humanoid robot simulation model; is the expected value of the horizontal turning speed of the humanoid robot simulation model; q i (t) is the joint angle of the current humanoid robot simulation model; is the expected value of the joint angle of the humanoid robot simulation model; diag(g) t is a diagonal function; It is the sliding speed of the sole of the supporting leg of the current humanoid robot simulation model on the ground.

[0141] In some embodiments, the head posture supervision model uses a supervised learning neural network.

[0142] The present disclosure provides a robot, comprising:

[0143] at least one processor; and

[0144] a memory communicatively connected to at least one processor; wherein,

[0145] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor so that the at least one processor can execute the method of any of the above embodiments.

[0146] An embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute a method of any of the above embodiments.

[0147] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0148] Figure 6 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0149] like Figure 6As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0150] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0151] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as a method for training a motion control model. For example, in some embodiments, the method for training a motion control model may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the method for training a motion control model described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform a method for training a motion control model in any other appropriate manner (e.g., by means of firmware).

[0152] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0153] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0154] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0155] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0156] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0157] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0158] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0159] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0160] The above is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A method for training a motion control model, characterized in that: The motion control model is used to control the walking of the humanoid robot; the head of the humanoid robot can collect images; the humanoid robot is deployed with a head posture supervision model, and the head posture supervision model is used to output the head posture control instruction of the humanoid robot based on the input information of the head posture supervision model; The input information of the head posture supervision model includes: the walking state information of the humanoid robot and the first terrain information; the first terrain information is determined based on the walking state information of the humanoid robot; The method comprises: Acquire walking state information and image information of the humanoid robot; the image information is image information collected when the humanoid robot performs head movements according to the head posture control instruction during walking, and the image information corresponds to the walking state information one by one; Determine corresponding second terrain information according to the image information; Constructing a motion control model and performing model training; wherein the training samples of the motion control model include: walking state information, image information, and second terrain information of the humanoid robot.

2. The method according to claim 1, characterized in that Before the step of obtaining the walking state information and image information of the humanoid robot, the method further includes: Obtain walking status information and head posture information of the human body; Determining corresponding first terrain information according to the walking state information of the human body; A head posture supervision model is constructed and model training is performed; wherein the training samples of the head posture supervision model include: walking state information of the human body, head posture information, and first terrain information.

3. The method according to claim 1, characterized in that The humanoid robot is also equipped with a simulation motion control model, and the simulation motion control model is used to output joint control instructions of the humanoid robot based on input information of the simulation motion control; The input information of the simulation motion control model includes: walking state information of the humanoid robot and first terrain information; the first terrain information is determined based on the walking state information of the humanoid robot; The walking state information of the humanoid robot is collected when the humanoid robot moves according to the joint control instructions.

4. The method according to claim 3, wherein: Before the step of obtaining the walking state information and image information of the humanoid robot, the method further includes: Build a humanoid robot simulation model; Acquiring walking state information of the humanoid robot simulation model; Determining corresponding third terrain information according to the walking state information of the humanoid robot simulation model; Constructing a simulation motion control model and performing model training; wherein the training samples of the simulation motion control model include: walking state information of the humanoid robot simulation model and third terrain information.

5. The method according to claim 4, characterized in that The steps of constructing a simulation motion control model and performing model training include: Set the reward function J(π): τ={(x0,z0),(x1,z1),…,(x T-1 ,z T-1 )} Where E is the expected value; τ is the trajectory when executing the simulated motion control model π; T is the reward period; r t For reward; t is the reward discount; p(τ|π) is the maximum likelihood of the simulation motion control model π; x0~x T-1 It is the walking state information of the humanoid robot simulation motion control model; Z0~Z T-1 is the third terrain information; The model is trained and the training is completed when the expected value of the reward reaches the expected threshold.

6. The method according to claim 5, characterized in that The walking state information of the humanoid robot simulation model includes: the linear velocity v of the humanoid robot simulation model x (t), v y (t), v z (t), the horizontal turning speed ω of the humanoid robot simulation model z (t), the joint angle q of the humanoid robot simulation model i (t), the sliding speed of the sole of the supporting leg of the humanoid robot simulation model on the ground 7. The method according to claim 6, characterized in that The reward r is calculated in the following way t , in, Among them, r t The reward for each training cycle; The linear speed reward for the robot moving in three-dimensional space; The speed reward for the robot turning in the horizontal direction; Tracking rewards for the robot’s joint angles in joint space; A bonus for the sliding speed of the sole of the supporting leg on the ground; v ,a ω ,a q ,a s are the weight coefficients of the rewards corresponding to each motion parameter of the robot; v x (t), v y (t), v z (t) are the linear velocities of the humanoid robot along the three coordinate axes in three-dimensional space; are the expected values ​​of the linear velocity of the humanoid robot along the three coordinate axes in three-dimensional space; ω z (t) is the horizontal turning speed of the current humanoid robot simulation model; is the expected value of the horizontal turning speed of the humanoid robot simulation model; q i (t) is the joint angle of the current humanoid robot simulation model; is the expected value of the joint angle of the humanoid robot simulation model; diag(g) t is a diagonal function; It is the sliding speed of the sole of the supporting leg of the current humanoid robot simulation model on the ground.

8. The method according to claim 1, characterized in that The head posture supervision model adopts a supervised learning neural network.

9. A system for training a motion control model, characterized in that: The motion control model is used to control the walking of the humanoid robot; the head of the humanoid robot can collect images; the humanoid robot is deployed with a head posture supervision model, and the head posture supervision model is used to output the head posture control instruction of the humanoid robot based on the input information of the head posture supervision model; The input information of the head posture supervision model includes: the walking state information of the humanoid robot and the first terrain information; the first terrain information is determined based on the walking state information of the humanoid robot; The system comprises: A humanoid robot information acquisition module, used to acquire walking state information and image information of the humanoid robot; the image information is image information collected when the humanoid robot performs head movements according to the head posture control instruction during walking, and the image information corresponds to the walking state information one by one; A second terrain determination module, used to determine corresponding second terrain information according to the image information; The real motion training module is used to construct a motion control model and perform model training; wherein the training samples of the motion control model include: walking state information, image information, and second terrain information of the humanoid robot.

10. A robot, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

Images