Method, system and robot for training motion control model

Abstract

The present disclosure provides a method, system and robot for training a motion control model, the method comprising obtaining walking state information and image information of a humanoid robot; the image information is image information collected when the humanoid robot performs head movements according to head posture control instructions during walking, and the image information corresponds one-to-one to the walking state information; determining corresponding second terrain information based on the image information; constructing a motion control model and performing model training; wherein the training samples of the motion control model include: the walking state information, image information and second terrain information of the humanoid robot.

Description

Technical Field

[0001] The present disclosure relates to the field of robotics, and in particular to a method, system, and robot for training a motion control model. Background Art

[0002] Traditional multi-terrain training for humanoid robots is conducted in simulation, making it difficult to incorporate visual information. This leads to issues such as collisions and high energy loss when deploying algorithms on complex terrain (such as rough roads, stairs, and slopes). Even when visual information is incorporated into simulation, the difficulty of simulating visual information and the significant differences between simulation and actual scenarios can lead to poor algorithm stability and difficulty in stable robot movement when the trained model is deployed on a real machine. Summary of the Invention

[0003] The present disclosure provides a method, system, and robot for training a motion control model to at least solve the above technical problems existing in the prior art.

[0004] According to a first aspect of the present disclosure, a method for training a motion control model is provided, wherein the motion control model is used to control a humanoid robot to walk; the humanoid robot's head is capable of capturing images; the humanoid robot is deployed with a head posture supervision model, the head posture supervision model being used to output head posture control instructions for the humanoid robot based on input information of the head posture supervision model; the input information of the head posture supervision model includes: walking state information of the humanoid robot and first terrain information; the first terrain information is determined based on the walking state information of the humanoid robot;

[0005] The method includes:

[0006] Acquiring walking state information and image information of the humanoid robot; the image information is image information collected when the humanoid robot performs head movements according to the head posture control instructions during walking, and the image information corresponds to the walking state information one by one;

[0007] determining corresponding second terrain information according to the image information;

[0008] Construct a motion control model and perform model training; wherein, the training samples of the motion control model include: walking state information, image information, and second terrain information of the humanoid robot.

[0009] In one embodiment, before the step of obtaining walking state information and image information of the humanoid robot, the method further includes:

[0010] Obtaining human walking state information and head posture information;

[0011] determining corresponding first terrain information according to walking state information of the human body;

[0012] A head posture supervision model is constructed and trained; wherein the training samples of the head posture supervision model include: walking state information of the human body, head posture information, and first terrain information.

[0013] In one embodiment, the humanoid robot is further equipped with a simulation motion control model, the simulation motion control model being configured to output joint control instructions of the humanoid robot based on input information of the simulation motion control; the input information of the simulation motion control model comprising: walking state information of the humanoid robot and first terrain information; the first terrain information being determined based on the walking state information of the humanoid robot;

[0014] The walking state information of the humanoid robot is collected when the humanoid robot moves according to the joint control instructions.

[0015] In one embodiment, before the step of obtaining walking state information and image information of the humanoid robot, the method further includes:

[0016] Build a humanoid robot simulation model;

[0017] Obtain walking state information of the humanoid robot simulation model;

[0018] determining corresponding third terrain information according to walking state information of the humanoid robot simulation model;

[0019] A simulation motion control model is constructed and model training is performed; wherein the training samples of the simulation motion control model include: walking state information of the humanoid robot simulation model and third terrain information.

[0020] In one embodiment, the steps of constructing a simulation motion control model and performing model training include:

[0021] Set the reward function J(π):

[0022]

[0023] τ={(x0,z0),(x1,z1),…,(x T-1 ,z T-1 )}

[0024] Where E is the expected value; τ is the trajectory when executing the simulated motion control model π; T is the reward period; r t For reward; t is the reward discount; p(τ|π) is the maximum likelihood of the simulation motion control model π; x0~x T-1 is the walking state information of the humanoid robot simulation model; z0~z T-1 is the third terrain information;

[0025] The model is trained and the training is completed when the expected value of the reward reaches the expected threshold.

[0026] In one embodiment, the walking state information of the humanoid robot simulation model includes: the linear velocity v of the humanoid robot simulation model t (t), v y (t), v z (t), horizontal turning speed ω of the humanoid robot simulation model z (t), joint angle q of the humanoid robot simulation model i (t), the sliding speed of the sole of the supporting leg of the humanoid robot simulation model on the ground

[0027] In one embodiment, the reward r is calculated as follows: t ,

[0028]

[0029] in,

[0030]

[0031]

[0032] Among them, r t The reward for each training cycle; The linear speed reward for the robot moving in three-dimensional space; The speed reward for the robot turning in the horizontal direction; Tracking rewards for the robot's joint angles in joint space; A bonus for the sliding speed of the sole of the supporting leg on the ground; v ,a ω ,a q ,a s are the weight coefficients of the rewards corresponding to each motion parameter of the robot; v x (t), v y (t), v z (t) are the linear velocities along the three coordinate axes in three-dimensional space; are the expected linear velocities of the humanoid robot along the three coordinate axes in three-dimensional space; ω z (t) is the horizontal turning speed of the current humanoid robot simulation model; is the expected value of the horizontal turning speed of the humanoid robot simulation model; q i (t) is the joint angle of the current humanoid robot simulation model; is the expected value of the joint angle of the humanoid robot simulation model; diag(g)t is a diagonal function; It is the sliding speed of the sole of the supporting leg of the current humanoid robot simulation model on the ground.

[0033] In one embodiment, the head posture supervision model uses a supervised learning neural network.

[0034] According to a second aspect of the present disclosure, a system for training a motion control model is provided, wherein the motion control model is used to control a humanoid robot to walk; the humanoid robot's head is capable of capturing images; the humanoid robot is deployed with a head posture supervision model, the head posture supervision model being used to output head posture control instructions for the humanoid robot based on input information of the head posture supervision model; the input information of the head posture supervision model includes: walking state information of the humanoid robot and first terrain information; the first terrain information is determined based on the walking state information of the humanoid robot;

[0035] The system includes:

[0036] A humanoid robot information acquisition module is used to acquire walking state information and image information of the humanoid robot; the image information is image information collected when the humanoid robot performs head movements according to the head posture control instructions during walking, and the image information corresponds to the walking state information one by one;

[0037] A second terrain determination module, configured to determine corresponding second terrain information according to the image information;

[0038] The real motion training module is used to build a motion control model and perform model training; wherein, the training samples of the motion control model include: walking state information, image information, and second terrain information of the humanoid robot.

[0039] According to a third aspect of the present disclosure, there is provided a robot comprising:

[0040] at least one processor; and

[0041] a memory communicatively connected to at least one processor; wherein,

[0042] The memory stores instructions that can be executed by at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method of the present disclosure.

[0043] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause a computer to execute the method of the present disclosure.

[0044] The present disclosure discloses a method, system, and robot for training a motion control model. The present disclosure installs a head posture supervision model on a humanoid robot, so that the humanoid robot outputs a head posture control instruction of the humanoid robot based on the input information of the head posture supervision model, and collects image information when performing head movements according to the head posture control instruction during walking, determines the second terrain information based on the image information, and trains the motion control model based on the walking state information, image information, and second terrain information of the humanoid robot. The present disclosure adds visually generated image information to the training process of the operation control model, solving the problems of poor walking effect and bumps caused by the lack of visual information in the reinforcement learning training of traditional humanoid robots. It enables the humanoid robot to walk on multiple terrains in a real environment, improves the terrain adaptability of the humanoid robot, and reduces energy consumption.

[0045] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings, in which several embodiments of the present disclosure are shown by way of example and not limitation, wherein:

[0047] In the drawings, the same or corresponding reference numerals denote the same or corresponding parts.

[0048] Figure 1 The schematic diagram shows the implementation process of the method for training the motion control model according to the embodiment of the present disclosure. Figure 1 ;

[0049] Figure 2 The schematic diagram shows the implementation process of the method for training the motion control model according to the embodiment of the present disclosure. Figure 2 ;

[0050] Figure 3 The schematic diagram shows the implementation process of the method for training the motion control model according to the embodiment of the present disclosure. Figure 3 ;

[0051] Figure 4 The schematic diagram shows the implementation process of the method for training the motion control model according to the embodiment of the present disclosure. Figure 4 ;

[0052] Figure 5 A schematic diagram of the structure of a motion control model training system according to an embodiment of the present disclosure is shown;

[0053] Figure 6A schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0054] To make the purposes, features, and advantages of the present disclosure more apparent and understandable, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present disclosure without creative work shall fall within the scope of protection of the present disclosure.

[0055] The following describes a method, system, and robot for training a motion control surface model provided by the present disclosure in conjunction with the accompanying drawings.

[0056] Figure 1 A method for training a motion control model provided by some embodiments of the present disclosure is shown. The motion control model is used to control a humanoid robot to walk. The humanoid robot's head can capture images. The humanoid robot is deployed with a head posture supervision model, which is used to output head posture control instructions for the humanoid robot based on input information of the head posture supervision model. The input information of the head posture supervision model includes: walking state information of the humanoid robot and first terrain information. The first terrain information is determined based on the walking state information of the humanoid robot.

[0057] The motion control model provided by the present disclosure can be applied to a humanoid robot, so that the humanoid robot can walk under the control of the motion control model. The humanoid robot can be a bionic robot, typically having a humanoid structure such as a head, a torso, and four fingers. The humanoid robot can simulate the movement of human joints and achieve bipedal walking by driving a motor and a control system. Optionally, the humanoid robot is equipped with suitable sensors to collect at least one piece of information such as the posture, joint state, and movement speed of the humanoid robot. Collecting various types of sensor information during the walking state of the humanoid robot is conducive to a comprehensive understanding of the robot's state, improving the accuracy and reliability of state estimation, and making the robot's movement closer to the target movement.

[0058] The head of the humanoid robot in the embodiment of the present disclosure has an image acquisition function, so that the humanoid robot has image perception capabilities. The image captured by the head of the humanoid robot can be an image within the viewing range taken in the current head posture, and the image may include the terrain of the environment in which the robot is located. Optionally, an image acquisition device (such as a camera) can be set on the head of the humanoid robot, and the image acquisition device is rigidly connected to the head of the humanoid robot to collect image information through the image acquisition device. Optionally, the image acquisition device can be set to mimic the position of the human eye, thereby realizing most of the visual functions of the human eye.

[0059] In the disclosed embodiment, the humanoid robot is deployed with a head posture supervision model, which can output the head posture control instructions of the humanoid robot based on the input walking state information of the humanoid robot and the first terrain information, and the humanoid robot controls the head posture of the humanoid robot according to the control instructions. It should be noted that the head posture of the humanoid robot determines the posture of the image acquisition device on the head, that is, it determines the field of view of the humanoid robot. It is understandable that when humans actually walk, they will adjust their field of view according to the state and terrain during walking, so as to avoid obstacles and prevent bumps. By deploying the head posture supervision model, the humanoid robot can imitate the field of view adjustment movements of humans when walking.

[0060] In an optional embodiment, the walking state information of the humanoid robot may include the body posture, joint state, and travel speed of the humanoid robot.

[0061] The first terrain information is the terrain information of the actual walking environment of the humanoid robot, and the first terrain information can be inferred based on the walking state information of the humanoid robot. In an optional embodiment, the humanoid robot collects the joint mobility of the humanoid robot when walking, including the range of motion of the hip, knee, and ankle during walking, the body posture, and the travel speed through the sensor, and can infer the terrain information of the current walking environment. For example, when the sensor collects that the knee joint mobility of the humanoid robot is about 90 degrees, the body posture is leaning forward, and the travel speed is slower than the normal walking speed, it is judged that the terrain of the current walking environment is stairs. Or when the sensor collects that the knee joint mobility of the humanoid robot is less than 70 degrees, the body posture is leaning backward, and the travel speed is faster than the normal walking speed, it is judged that the terrain of the current walking environment is downhill.

[0062] like Figure 1 As shown, some embodiments of the present disclosure provide a method for training a motion control model, including:

[0063] S101, obtaining walking state information and image information of the humanoid robot; the image information is image information collected when the humanoid robot performs head movements according to head posture control instructions during walking, and the image information corresponds to the walking state information one by one;

[0064] In the disclosed embodiments, sensors can be used to collect walking state information of a humanoid robot. A head posture supervision model uses this walking state information as input and outputs head posture control instructions for the humanoid robot. During walking, the humanoid robot performs head movements according to the head posture control instructions. In this way, walking state information and image information are acquired, and the image information corresponds one-to-one with the walking state information.

[0065] As can be understood, the humanoid robot's walking state information is acquired through sensors and then input into the head posture supervision model, thereby generating head posture control instructions for the humanoid robot. The humanoid robot then controls head rotation according to the head posture control instructions, generating corresponding image information. Therefore, there is a one-to-one correspondence between the walking state information, head posture control instructions, and image information.

[0066] S102, determining corresponding second terrain information according to the image information;

[0067] During the actual walking process of the humanoid robot in the disclosed embodiments, the humanoid robot performs head movements based on head posture control commands. Images captured by the image acquisition device on the humanoid robot's head typically contain environmental information, such as at least one of the following: the environment ahead, the environment below, and objects within the environment. Terrain information can be obtained using an image algorithm. The image algorithm can employ existing image processing methods.

[0068] S103, constructing a motion control model and performing model training; wherein, the training samples of the motion control model include: walking state information, image information, and second terrain information of the humanoid robot.

[0069] The motion control model provided by the embodiments of the present disclosure may adopt a neural network algorithm, such as a convolutional neural network, a recurrent neural network, and a deep learning neural network.

[0070] The present disclosure uses walking state information, image information and second terrain information obtained by the humanoid robot during actual walking as training data to train a neural network, thereby obtaining a motion control model, and then using the motion control model to control the walking of the humanoid robot.

[0071] The present disclosure provides a method for training a motion control model. During a humanoid robot's walking process, the method first obtains the robot's walking state information. A head posture supervision model pre-installed in the humanoid robot is used to obtain head posture control instructions based on the robot's walking state information. The humanoid robot then performs head movements based on the head posture control instructions. During the head movements, the robot uses a head-mounted camera to capture image information, and then obtains second terrain information from the image information. Finally, the motion control model is trained using the humanoid robot's walking state information, image information, and second terrain information. The present disclosure provides a method for training a motion control model. Visual information is obtained from the humanoid robot and used to train the motion control model. This method addresses the issues of poor walking performance and collisions caused by the lack of visual information in reinforcement learning training of traditional humanoid robot deployment models. Compared to motion control models obtained by incorporating visual information into simulation and performing model training, the present disclosure improves motion control accuracy and stability. Deploying the trained motion control model in a humanoid robot can control the humanoid robot to walk stably on multiple terrains, improving the robot's terrain adaptability, reducing collisions, and lowering energy consumption.

[0072] In addition, the technical solution disclosed in the present invention utilizes the humanoid robot to collect images as visual information through its head during actual walking. Compared with the visual information added through the simulation scene, the obtained visual data is more consistent with the terrain information. Then, the one-to-one corresponding image information, walking state information and second terrain information are input into the model for training. Not only is the introduction of visual information into training highly operational, but it can also improve the walking effect of the humanoid robot and ensure the humanoid robot's ability to adapt to complex terrain.

[0073] In some embodiments, such as Figure 2 As shown, before the step of obtaining the walking state information and image information of the humanoid robot, the method further includes:

[0074] S201, obtaining walking state information and head posture information of a human body;

[0075] S202, determining corresponding first terrain information according to the walking state information of the human body;

[0076] S203, constructing a head posture supervision model and performing model training; wherein, the training samples of the head posture supervision model include: human walking state information, head posture information, and first terrain information.

[0077] In the disclosed embodiments, the head posture supervision model and motion control model are applied to a humanoid robot. The walking state information of the humanoid robot may include, for example, the joint state, travel speed, and body posture of the humanoid robot. Therefore, in order to make the humanoid robot's movements highly anthropomorphic and thus achieve human-like movement, human expert teaching data can be collected and used for model training, and then applied to the humanoid robot to achieve anthropomorphic movements of the humanoid robot. The obtained human expert teaching data can be obtained from a human of similar height and proportion to the humanoid robot, or it can be collected from humans of different heights and proportions and then transformed through mapping to obtain human data for training.

[0078] In this disclosure, the joint states and travel speeds of human experts during their movements are collected, and inertial sensors are used to collect head posture information from observing first terrain information. The joint states, travel speeds, head postures, and first terrain information collected while the human experts walk on different terrains are used as training samples. A supervised learning model is trained to obtain the relationship between human motion state information, external control commands, first terrain information, and the human experts' head posture, thus forming a head posture supervision model.

[0079] It is understandable that human experts can wear motion capture suits to collect walking status information; and inertial measurement units (IMUs) can be set on the heads of human experts to obtain head posture information, such as three-axis rotation angle information.

[0080] In an optional embodiment, information about each joint of the human body during movement is obtained. And the rotation angle information of the head along the three axes of its own coordinate system Derivation of the first topographic information through joint kinematics information and human walking speed information Using the above information to construct a data set,

[0081]

[0082] In an optional embodiment, the head posture supervision model can be trained in a supervised learning manner. The training sample includes a set of training examples, which include input and correct output. For example, when performing supervised learning training, there are 400 samples in total, and the 400 samples can be divided into 300 training samples and 100 test samples. After the model is trained with 300 training samples, the trained model is tested with 100 test samples. When the accuracy of the test results reaches 90%, the head posture supervision model is output. The head posture supervision model is based on the input joint information. Derivation of the first terrain information through joint information and human walking speed information Output head posture control instructions, that is, the rotation angle information of the head along the three axes of its own coordinate system

[0083] The present invention collects data from actual multi-terrain motion of the human body, trains a head posture supervision model, and obtains the relationship between body data, terrain information, and image data, i.e., head field of view, when humans walk on multiple terrains. This facilitates the introduction of image data into the neural network during subsequent motion control model training. This can solve problems such as bumps and large energy loss caused by the lack of visual information in traditional humanoid robot reinforcement learning training, and improve the walking effect of humanoid robots.

[0084] In some embodiments of the present disclosure, the humanoid robot is further equipped with a simulation motion control model, which is used to output joint control instructions of the humanoid robot based on input information of the simulation motion control model; the input information of the simulation motion control model includes: walking state information of the humanoid robot and first terrain information; the first terrain information is determined based on the walking state information of the humanoid robot;

[0085] The walking state information of the humanoid robot is collected when the humanoid robot moves according to the joint control instructions.

[0086] It can be understood that in the present disclosure, a simulation motion control model can be deployed in a humanoid robot. The simulation motion control model can output joint control instructions of the humanoid robot based on the input humanoid robot walking state information and first terrain information to control the walking of the humanoid robot.

[0087] In an optional embodiment, the simulation motion control model in the present disclosure may adopt a multi-layer perceptron network. For example, the input information of the simulation motion control model is the robot walking state information x t , the first terrain information at the current moment γ t , the simulation motion control model outputs the angle control instructions a for each joint of the robot t Alternatively, the input information of the simulation motion control model can be the robot walking state information x t , the first terrain information at the current moment γ t And the current environmental information m t , the simulation motion control model outputs the angle control instructions a for each joint of the robot t Optionally, the robot walking state information x tIt may include the robot's body posture, joint status, travel speed, etc. The environmental information may include at least one of friction force and contact force. The first terrain information may include at least one of ground height and terrain type (such as steps, flat bottom, steep slope).

[0088] The present disclosure deploys a simulation motion control model in a humanoid robot, so that the humanoid robot has the basic function of walking on various terrains.

[0089] In some embodiments, such as Figure 3 As shown, before the step of obtaining the walking state information and image information of the humanoid robot, the method further includes:

[0090] S301, constructing a humanoid robot simulation model;

[0091] S302, obtaining walking state information of the humanoid robot simulation model;

[0092] S303, determining corresponding third terrain information according to the walking state information of the humanoid robot simulation model;

[0093] S304, constructing a simulation motion control model and performing model training; wherein, the training samples of the simulation motion control model include: walking state information of the humanoid robot simulation model and third terrain information.

[0094] In the embodiments provided herein, a simulation model of a humanoid robot can be first constructed in a simulation environment. Walking state information and third terrain information corresponding to the walking state information can be obtained through the simulation model. The walking state information and the corresponding third terrain information are then input into the constructed simulation motion control model for training to obtain a simulation motion control model. The resulting simulation motion control model has a multi-terrain walking strategy. The simulation motion control model is trained using the walking state information of the humanoid robot simulation model and the third terrain information corresponding to the walking state information, so that the resulting simulation motion model can obtain corresponding joint control instructions based on the terrain environment, thereby implementing a multi-terrain walking strategy.

[0095] In an optional embodiment, the walking state information, along with the corresponding third terrain information and environmental information, is input into a constructed simulation motion control model for training, thereby obtaining a simulation motion control model. Thus, the simulation motion control model can predict the angle control commands for each joint of the robot based on the robot's state information, the current environmental information, and the current terrain information. Inputting the current environmental information as training samples into the simulation motion control model to be trained can improve the prediction accuracy of the simulation motion model.

[0096] In some embodiments, the steps of constructing a simulation motion control model and performing model training include:

[0097] Set the reward function J(π):

[0098]

[0099] τ={(x0,z0),(x1,z1),…,(x T-1 ,z T-1 )}

[0100] Where E is the expected value; τ is the trajectory when executing the simulated motion control model π; T is the reward period; r t For reward; t is the reward discount; p(τ|π) is the maximum likelihood of the simulation motion control model π; x0~x T-1 is the walking state information of the humanoid robot simulation motion control model; z0~z T-1 is the third terrain information;

[0101] The model is trained and the training is completed when the expected value of the reward reaches the expected threshold.

[0102] Optionally, the input information also includes environmental information m t , then τ={(x0,m0,z0),(x1,m1,z1)…(x T-1 ,m T-1 , z T-1 In an optional embodiment, the walking state information of the humanoid robot simulation model includes: the linear velocity v of the humanoid robot simulation model x (t), horizontal turning speed ω of the humanoid robot simulation model z (t), joint angle q of the humanoid robot simulation model i (t), the sliding speed of the sole of the supporting leg of the humanoid robot simulation model on the ground

[0103] It is understandable that when training the simulation motion control model, a reward function is set, and the total reward is the sum of all rewards in the network. When training the model, the total reward obtained needs to be maximized, and when the expected value of the total reward reaches the expected threshold, the motion control model is output. It is understandable that the reward function assigns a reward to each output generated by the model, and a higher reward indicates that the output is closer to the desired target. The present disclosure uses rewards for multiple motion parameters of the humanoid robot to provide feedback to guide model learning, so that the actions of the humanoid robot are more in line with the expected goals, thereby conducting model training more reasonably and comprehensively.

[0104] In some embodiments, the reward r is calculated as follows: t ,

[0105]

[0106] in,

[0107]

[0108] Among them, r t The reward for each training cycle; The linear speed reward for the robot moving in three-dimensional space; The speed reward for the robot turning in the horizontal direction; Tracking rewards for the robot's joint angles in joint space; A bonus for the sliding speed of the sole of the supporting leg on the ground; v ,a ω ,a q ,a s They are the weight coefficients of the rewards corresponding to the various motion parameters of the robot (where α v is the weight coefficient of the linear velocity reward of the robot moving in three-dimensional space; α w is the weight coefficient of the robot's joint angle tracking reward in the joint space; α q The weight coefficient of the robot's speed reward in the horizontal direction; α s is the weight coefficient of the sliding speed reward of the sole of the robot's supporting leg on the ground); v x (t), v y (t), v z (t) are the linear velocities of the humanoid robot along the three coordinate axes in three-dimensional space; are the expected linear velocities of the humanoid robot along the three coordinate axes in three-dimensional space; ω z (t) is the horizontal turning speed of the current humanoid robot simulation model; is the expected value of the horizontal turning speed of the humanoid robot simulation model; q i (t) is the joint angle of the current humanoid robot simulation model; is the expected value of the joint angle of the humanoid robot simulation model; diag(g) t is a diagonal function; It is the sliding speed of the sole of the supporting leg of the current humanoid robot simulation model on the ground.

[0109] Finally, when the expected reward reaches a desired threshold, training is complete and the simulation motion control model is output. When the reward reaches the desired threshold, the simulation motion control model's performance is good. By continuously optimizing the reward, the model's performance can be updated and improved, thereby enhancing the humanoid robot's walking ability and reducing its energy consumption.

[0110] like Figure 4As shown, the present disclosure can pre-train a simulation motion control model and a head posture supervision model. The simulation motion control model is trained based on the input walking state information of the humanoid robot simulation model and the corresponding third terrain information, and can output joint control instructions for the humanoid robot. The head posture supervision model is trained based on the input walking state information, head posture information and first terrain information of the human body, and can output head posture control instructions. The trained simulation motion control model and head posture supervision model are deployed to the humanoid robot, and the humanoid robot is made to actually walk, thereby obtaining the walking state information x of the robot body. t , the first terrain information γ t , the collected image information I t and the second terrain information inferred by image information at the current moment The walking state x t , second terrain information And the image information collected by the camera I t The motion control model is trained using the data as input. Environmental information corresponding to the walking state can also be used as input to the motion control model to improve the model's prediction accuracy. The motion control model is then deployed in a humanoid robot, enabling it to control the robot's walking.

[0111] The motion control model provided by the embodiments of the present disclosure can use a multilayer perceptron network. A multilayer perceptron (MLP) is a feedforward neural network that includes at least three layers of nodes: an input layer, a hidden layer, and an output layer. Except for the input node, each node is a neuron with a nonlinear activation function. During the training process of the MLP, the output is compared with the expected output to generate an error value. This error is then back-propagated to the network, and the weights are updated accordingly.

[0112] The motion control model provided by this disclosure incorporates image data actually collected by a humanoid robot into a neural network, addressing issues such as susceptibility to collisions and high energy consumption caused by the lack of visual information in traditional humanoid robot reinforcement learning training. This disclosure also addresses the difficulty of integrating visual information into training by deploying a head posture supervision model and a simulation motion control model within the humanoid robot, enhancing its operability. The resulting motion control model enables smooth traversal across multiple terrains, improving the robot's terrain adaptability.

[0113] like Figure 5As shown, an embodiment of the present disclosure provides a system for training a motion control model, wherein the motion control model is used to control a humanoid robot to walk; the head of the humanoid robot is capable of capturing images; the humanoid robot is deployed with a head posture supervision model, and the head posture supervision model is used to output a head posture control instruction of the humanoid robot based on input information of the head posture supervision model; the input information of the head posture supervision model includes: walking state information of the humanoid robot and first terrain information; the first terrain information is determined based on the walking state information of the humanoid robot;

[0114] The system includes:

[0115] The humanoid robot information acquisition module 501 is used to acquire walking state information and image information of the humanoid robot; the image information is image information collected when the humanoid robot performs head movements according to the head posture control instructions during walking, and the image information corresponds to the walking state information one by one;

[0116] A second terrain determining module 502 is configured to determine corresponding second terrain information based on the image information;

[0117] The real motion training module 503 is used to construct a motion control model and perform model training; wherein, the training samples of the motion control model include: walking state information, image information, and second terrain information of the humanoid robot.

[0118] The training motion control model system provided by the present disclosure inputs the walking state information and image information of the humanoid robot into the head posture supervision model to obtain the head posture control instructions of the humanoid robot. When the humanoid robot performs head movements according to the head posture control instructions during walking, image information is collected, and second terrain information is determined based on the image information. The walking state information, image information, and second terrain information are used as training samples to train the motion control model to obtain a motion control model. The motion control model is deployed in the humanoid robot, and the motion control model controls the walking of the humanoid robot. The motion control model obtained by the present disclosure can avoid the problems of bumps and high energy consumption that exist when the humanoid robot walks on multiple terrains, and is conducive to improving the terrain adaptability of the humanoid robot.

[0119] In some embodiments, the system further comprises:

[0120] A human body information acquisition module is used to obtain walking state information and head posture information of the human body;

[0121] A first terrain determination module is used to determine corresponding first terrain information according to walking state information of a human body;

[0122] The head posture training module is used to build a head posture supervision model and perform model training; wherein, the training samples of the head posture supervision model include: human walking state information, head posture information, and first terrain information.

[0123] In some embodiments, the humanoid robot is further equipped with a simulation motion control model, which is configured to output joint control instructions for the humanoid robot based on input information of the simulation motion control; the input information of the simulation motion control model includes: walking state information of the humanoid robot and first terrain information; the first terrain information is determined based on the walking state information of the humanoid robot;

[0124] The walking state information of the humanoid robot is collected when the humanoid robot moves according to the joint control instructions.

[0125] In some embodiments, the system further comprises:

[0126] A simulation model building module, used to build a humanoid robot simulation model;

[0127] A simulation model information acquisition module is used to obtain walking state information of the humanoid robot simulation model;

[0128] A third terrain information determination module, configured to determine corresponding third terrain information according to walking state information of the humanoid robot simulation model;

[0129] The simulation motion training module is used to construct a simulation motion control model and perform model training; wherein, the training samples of the simulation motion control model include: walking state information of the humanoid robot simulation model and third terrain information.

[0130] In some embodiments, the simulation motion training module includes: a function setting module, the function setting module is used to set the reward function J(π):

[0131]

[0132] τ={(x0,z0),(x1,z1),…,(x T-1 ,z T-1 )}

[0133] Where E is the expected value; τ is the trajectory when executing the simulated motion control model π; T is the reward period; r t For reward; t is the reward discount; p(τ|π) is the maximum likelihood of the simulation motion control model π; x0~x T-1 is the walking state information of the humanoid robot simulation motion control model; z0~z T-1 is the third terrain information;

[0134] The model is trained and the training is completed when the expected value of the reward reaches the expected threshold.

[0135] In some embodiments, the walking state information of the humanoid robot simulation model includes: the linear velocity v of the humanoid robot simulation model x (t), v y (t), v z (t), horizontal turning speed ω of the humanoid robot simulation model z (t), joint angle q of the humanoid robot simulation model i (t), the sliding speed of the sole of the supporting leg of the humanoid robot simulation model on the ground

[0136] In some embodiments, the reward r is calculated as follows: t ,

[0137]

[0138] in,

[0139]

[0140] Among them, r t The reward for each training cycle; The linear speed reward for the robot moving in three-dimensional space; The speed reward for the robot turning in the horizontal direction; Tracking rewards for the robot's joint angles in joint space; A bonus for the sliding speed of the sole of the supporting leg on the ground; v ,a ω ,a q ,a s are the weight coefficients of the rewards corresponding to each motion parameter of the robot; v x (t), v y (t), v z (t) are the linear velocities of the humanoid robot along the three coordinate axes in three-dimensional space; are the expected linear velocities of the humanoid robot along the three coordinate axes in three-dimensional space; ω z (t) is the horizontal turning speed of the current humanoid robot simulation model; is the expected value of the horizontal turning speed of the humanoid robot simulation model; q i (t) is the joint angle of the current humanoid robot simulation model; is the expected value of the joint angle of the humanoid robot simulation model; diag(g) t is a diagonal function; It is the sliding speed of the sole of the supporting leg of the current humanoid robot simulation model on the ground.

[0141] In some embodiments, the head posture supervision model uses a supervised learning neural network.

[0142] The present disclosure provides a robot, comprising:

[0143] at least one processor; and

[0144] a memory communicatively connected to at least one processor; wherein,

[0145] The memory stores instructions that can be executed by at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method of any of the above embodiments.

[0146] An embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method of any of the above embodiments.

[0147] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0148] Figure 6 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0149] like Figure 6 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. Various programs and data required for the operation of the device 800 can also be stored in the RAM 803. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0150] Various components in device 800 are connected to I / O interface 805, including an input unit 806, such as a keyboard, mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, optical disk, etc.; and a communication unit 809, such as a network card, modem, wireless communication transceiver, etc. The communication unit 809 allows device 800 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0151] The computing unit 801 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as the method for training a motion control model. For example, in some embodiments, the method for training a motion control model can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the method for training a motion control model described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform the method for training a motion control model by any other appropriate means (e.g., by means of firmware).

[0152] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0153] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0154] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0155] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0156] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0157] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0158] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0159] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. Throughout the present disclosure, "plurality" means two or more, unless otherwise specifically defined.

[0160] The above description is merely a specific embodiment of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this disclosure should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be based on the scope of protection of the claims.

Claims

1. A method for training a motion control model, characterized in that: The motion control model is used to control the walking of the humanoid robot; the head of the humanoid robot is capable of collecting images; the humanoid robot is deployed with a head posture supervision model, and the head posture supervision model is used to output a head posture control instruction of the humanoid robot based on input information of the head posture supervision model; The input information of the head posture supervision model includes: the walking state information of the humanoid robot and first terrain information; the first terrain information is determined based on the walking state information of the humanoid robot; The method comprises: Acquiring walking state information and image information of the humanoid robot; the image information is image information collected when the humanoid robot performs head movements according to the head posture control instruction during walking, and the image information corresponds one-to-one to the walking state information; determining corresponding second terrain information according to the image information; Constructing a motion control model and performing model training; wherein the training samples of the motion control model include: walking state information, image information, and second terrain information of the humanoid robot; The humanoid robot is further equipped with a simulation motion control model, the simulation motion control model being configured to output joint control instructions for the humanoid robot based on input information of the simulation motion control; the input information of the simulation motion control model comprising: walking state information of the humanoid robot and first terrain information; the first terrain information being determined based on the walking state information of the humanoid robot; The walking state information of the humanoid robot is collected when the humanoid robot moves according to the joint control instructions; Before the step of obtaining the walking state information and image information of the humanoid robot, the method further includes: Build a humanoid robot simulation model; Acquiring walking state information of the humanoid robot simulation model; determining corresponding third terrain information according to walking state information of the humanoid robot simulation model; Constructing a simulation motion control model and performing model training; wherein the training samples of the simulation motion control model include: walking state information of the humanoid robot simulation model and third terrain information; The walking state information of the humanoid robot simulation model includes: the linear speed of the humanoid robot simulation model 、 、 , the horizontal turning speed of the humanoid robot simulation model , the joint angles of the humanoid robot simulation model , the sliding speed of the sole of the supporting leg of the humanoid robot simulation model on the ground .

2. The method according to claim 1, characterized in that Before the step of obtaining the walking state information and image information of the humanoid robot, the method further includes: Obtaining human walking state information and head posture information; Determining corresponding first terrain information according to the walking state information of the human body; A head posture supervision model is constructed and model training is performed; wherein the training samples of the head posture supervision model include: walking state information of the human body, head posture information, and first terrain information.

3. The method according to claim 1, characterized in that The steps of constructing a simulation motion control model and performing model training include: Setting up the reward function : in, is the expected value; To perform simulation of motion control models The trajectory of time; T is the reward cycle; For rewards; Discounts for incentives; To simulate motion control models Maximum likelihood of; ~ It is the walking state information of the humanoid robot simulation motion control model; ~ is the third terrain information; The model is trained and the training is completed when the expected value of the reward reaches the expected threshold.

4. The method according to claim 3, characterized in that The rewards are calculated in the following way , in, in, The reward for each training cycle; The linear speed reward for the robot moving in three-dimensional space; The speed reward for the robot turning in the horizontal direction; Tracking rewards for the robot's joint angles in joint space; Rewards are given for the sliding speed of the sole of the supporting leg on the ground; are the weight coefficients of the rewards corresponding to each motion parameter of the robot; 、 、 are the linear velocities of the humanoid robot along the three coordinate axes in three-dimensional space; 、 、 are the expected linear velocities of the humanoid robot along the three coordinate axes in three-dimensional space; is the horizontal turning speed of the current humanoid robot simulation model; is the expected value of the horizontal turning speed of the humanoid robot simulation model; is the joint angle of the current humanoid robot simulation model; is the expected value of the joint angle of the humanoid robot simulation model; is a diagonal function; It is the sliding speed of the sole of the supporting leg of the current humanoid robot simulation model on the ground.

5. The method according to claim 1, wherein The head posture supervision model adopts a supervised learning neural network.

6. A system for training a motion control model, characterized in that: The motion control model is used to control the walking of the humanoid robot; the head of the humanoid robot is capable of collecting images; the humanoid robot is deployed with a head posture supervision model, and the head posture supervision model is used to output a head posture control instruction of the humanoid robot based on input information of the head posture supervision model; The input information of the head posture supervision model includes: the walking state information of the humanoid robot and first terrain information; the first terrain information is determined based on the walking state information of the humanoid robot; The system comprises: a humanoid robot information acquisition module, configured to acquire walking state information and image information of the humanoid robot; the image information being image information collected when the humanoid robot performs head movements according to the head posture control instructions during walking, and the image information corresponding to the walking state information; a second terrain determining module, configured to determine corresponding second terrain information according to the image information; A real motion training module, used to construct a motion control model and perform model training; wherein the training samples of the motion control model include: walking state information, image information, and second terrain information of the humanoid robot; The humanoid robot is further equipped with a simulation motion control model, the simulation motion control model being configured to output joint control instructions for the humanoid robot based on input information of the simulation motion control; the input information of the simulation motion control model comprising: walking state information of the humanoid robot and first terrain information; the first terrain information being determined based on the walking state information of the humanoid robot; The walking state information of the humanoid robot is collected when the humanoid robot moves according to the joint control instructions; Before the step of obtaining the walking state information and image information of the humanoid robot, the method further includes: Build a humanoid robot simulation model; Acquiring walking state information of the humanoid robot simulation model; determining corresponding third terrain information according to walking state information of the humanoid robot simulation model; Constructing a simulation motion control model and performing model training; wherein the training samples of the simulation motion control model include: walking state information of the humanoid robot simulation model and third terrain information; The walking state information of the humanoid robot simulation model includes: the linear speed of the humanoid robot simulation model 、 、 , the horizontal turning speed of the humanoid robot simulation model , the joint angles of the humanoid robot simulation model , the sliding speed of the sole of the supporting leg of the humanoid robot simulation model on the ground .

7. A robot, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

Images