Robot motion control method and device

Abstract

The embodiment of the present application provides a robot motion control method and device, the method includes obtaining the current state of the robot, the current state includes the current position and target position of the robot, inputting the current state into the motion control model, obtaining the current action that the robot needs to perform, the motion control model is obtained by multi-task training of the reinforcement learning model, the multi-task training includes path planning task training, and at least one of gait control task training and balance control task training, controlling the robot to perform the current action, and completing the current motion task if the robot reaches the target position after completing the current action. The method provided by the embodiment of the present application enables the robot to improve stability on the basis of being able to efficiently and reasonably plan the path.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of humanoid robots, and in particular to a robot motion control method and device. Background Art

[0002] With the rapid development of science and technology, humanoid robots have shown great application potential in many fields. In order to improve the performance of humanoid robots, it is necessary to focus on improving the motion control level of humanoid robots.

[0003] In the related art, during the movement of a humanoid robot, path planning is usually performed based on a geometric algorithm or a search algorithm.

[0004] However, in the process of implementing this application, the inventors found that there are at least the following problems in the prior art: the effectiveness of the path planning in the above-mentioned manner is poor, and the stability of the robot during movement cannot be guaranteed. Summary of the invention

[0005] The embodiments of the present application provide a robot motion control method and device to improve the efficiency and stability of motion control.

[0006] In a first aspect, an embodiment of the present application provides a robot motion control method, comprising:

[0007] Acquire the current state of the robot; the current state includes the current position and target position of the robot;

[0008] The current state is input into a motion control model to obtain a current action that the robot needs to perform; the motion control model is obtained by performing multi-task training on a reinforcement learning model; the multi-task training includes path planning task training, and at least one of gait control task training and balance control task training; the gait control task training and the balance control task training are performed on the basis of the path planning task training;

[0009] The robot is controlled to execute the current action. If the robot reaches the target position after completing the current action, the current motion task is completed.

[0010] In a possible design, the current state also includes: grid properties of the grid to which the current position belongs, joint angles, joint angular velocities, stride, step frequency, external force, body inclination angle, body angular velocity, and plantar pressure value.

[0011] In one possible design, the method further includes:

[0012] A first reinforcement learning model corresponding to the path planning task training is constructed based on the initial strategy function and the initial value function; the first reinforcement learning model includes a first reward function; the first reward function is related to whether a collision with an obstacle occurs and whether the distance to the end point is shortened;

[0013] Training the first reinforcement learning model to obtain a trained first reinforcement learning model;

[0014] Constructing a second reinforcement learning model corresponding to the gait control task based on the trained first reinforcement learning model; the second reinforcement learning model includes a second reward function; the second reward function is related to the uniformity of walking stride and the uniformity of step frequency;

[0015] Training the second reinforcement learning model to obtain a trained second reinforcement learning model;

[0016] The motion control model is determined according to the trained second reinforcement learning model.

[0017] In a possible design, training the first reinforcement learning model includes:

[0018] The value function is updated based on the following expression:

[0019] ,

[0020] Where Q is the value function, is the current state, For the state of the next moment, is the action at the current moment, is the learning rate, is the discount factor, Is the action that executes the current moment After the reward, Indicates the action at the next moment.

[0021] In a possible design, the training of the second reinforcement learning model includes:

[0022] The policy update is based on the following expression:

[0023]

[0024] in, are the parameters of the policy network, is a strategic network, is the advantage function, , is the action-value function, is the state-value function.

[0025] In a possible design, determining the motion control model according to the trained second reinforcement learning model includes:

[0026] Constructing a third reinforcement learning model corresponding to the balance control task training based on the trained second reinforcement learning model; the third reinforcement learning model includes a third reward function; the third reward function is related to whether to fall down when subjected to external force, and the speed of restoring balance when subjected to external force;

[0027] Training the third reinforcement learning model to obtain a trained third reinforcement learning model;

[0028] The motion control model is determined according to the trained third reinforcement learning model.

[0029] In a possible design, determining the motion control model according to the balance strategy function and the balance value function includes:

[0030] Constructing a fourth reinforcement learning model based on the trained third reinforcement learning model; a fourth reward function corresponding to the fourth reinforcement learning model is determined according to the first reward function, the second reward function and the third reward function;

[0031] The fourth reinforcement learning model is trained to obtain the motion control model.

[0032] In a possible design, training the fourth reinforcement learning model to obtain the motion control model includes:

[0033] Modeling the environment in which the robot is located to obtain an environment model; the environment model includes a starting position and a target position;

[0034] Placing the robot at the starting position to obtain an initial environment state;

[0035] For each time step, according to the current state, determine the current action to be performed, and after performing the current action, obtain the next state and reward value; the reward value is determined according to the fourth reward function;

[0036] Storing the current state, the current action, the next state, and the reward value in an experience replay buffer;

[0037] Randomly extracting a preset amount of training data from the experience replay buffer;

[0038] According to the training data, gradient descent training is performed on the fourth reinforcement learning model to obtain the motion control model.

[0039] In a possible design, the fourth reinforcement learning model is a deep Q network; and the training of the fourth reinforcement learning model to obtain the motion control model includes:

[0040] For each time step, generate a random number M, M is greater than or equal to 0 and M is less than or equal to 1;

[0041] If M is less than , then randomly select an action from the action space to execute;

[0042] If M is greater than or equal to , then select the action corresponding to the maximum Q value among the Q values ​​output by the Q network in the current state as the target action, and execute the target action; is a probability value between 0 and 1, indicating the degree of random exploration; The previous iteration round The attenuation factor is greater than 0 and less than 1.

[0043] In a second aspect, an embodiment of the present application provides a robot motion control device, including:

[0044] An acquisition module, used to acquire the current state of the robot; the current state includes the current position and target position of the robot;

[0045] An input module, used for inputting the current state into a motion control model to obtain a current action that the robot needs to perform; the motion control model is obtained by performing multi-task training on a reinforcement learning model; the multi-task training includes path planning task training, and at least one of gait control task training and balance control task training; the gait control task training and the balance control task training are performed on the basis of the path planning task training;

[0046] The control module is used to control the robot to perform the current action. If the robot reaches the target position after completing the current action, the current motion task is completed.

[0047] In a third aspect, an embodiment of the present application provides a robot motion control device, including: at least one processor and a memory;

[0048] The memory stores computer-executable instructions;

[0049] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the method described in the first aspect and various possible designs of the first aspect.

[0050] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the method described in the first aspect and various possible designs of the first aspect are implemented.

[0051] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the method described in the first aspect and various possible designs of the first aspect.

[0052] The robot motion control method and device provided in the present embodiment include obtaining the current state of the robot, the current state includes the current position and target position of the robot, inputting the current state into the motion control model, obtaining the current action that the robot needs to perform, the motion control model is obtained by multi-task training of the reinforcement learning model, the multi-task training includes path planning task training, and at least one of gait control task training and balance control task training, controlling the robot to perform the current action, and completing the current motion task if the robot reaches the target position after completing the current action. The robot motion control method provided in the embodiment of the present application combines gait control training or balance control training with path planning training based on a reinforcement learning algorithm, and obtains a motion control model through training, so that the robot can improve stability on the basis of being able to efficiently and reasonably plan the path. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0054] Figure 1 A schematic diagram of a flow chart of a robot motion control method provided in an embodiment of the present application;

[0055] Figure 2 Schematic diagram of the training process of the motion control model provided in the embodiment of the present application Figure 1 ;

[0056] Figure 3 Schematic diagram of the training process of the motion control model provided in the embodiment of the present application Figure 2 ;

[0057] Figure 4 A schematic diagram of the structure of a robot motion control device provided in an embodiment of the present application;

[0058] Figure 5 A schematic diagram of the hardware structure of the robot motion control device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0060] It should be noted that the robot motion control method provided in the present application can be used in the field of humanoid robot technology, and can also be used in any field other than the field of humanoid robots. The application field of the robot motion control method provided in the present application is not limited.

[0061] With the rapid development of science and technology, humanoid robots have shown great application potential in many fields, such as industrial manufacturing, medical care, home services, and rescue. However, achieving efficient, stable, and intelligent motion control of humanoid robots in complex environments has always been a major challenge in the field of robotics.

[0062] In related technologies, path planning is usually based on geometric algorithms or search algorithms. For example, The algorithm finds the path from the starting point to the end point by calculating the heuristic cost of the nodes in a known map environment. However, although this method can achieve certain results in simple, static environments, it has obvious limitations for complex and changing actual scenarios. When dynamic obstacles appear in the environment or the map information is inaccurate, traditional path planning algorithms are difficult to adjust the path quickly and effectively, which can easily cause the robot to get into trouble or collide, and affect the stable balance of the robot during walking.

[0063] In response to the above technical problems, the inventors of the present application have discovered that a reinforcement learning algorithm can be used to enable the robot to continuously trial and error in the environment and learn the optimal behavior strategy based on the reward signal fed back from the environment to deal with obstacles in the environment or inaccurate map information. In addition, based on the reinforcement learning algorithm, gait control training or balance control training can be carried out together with path planning training, so that the robot can improve stability on the basis of being able to efficiently and reasonably plan the path.

[0064] The technical solution of the present application is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0065] Figure 1 The following is a flow chart of a robot motion control method provided in an embodiment of the present application. Figure 1 As shown, the method includes:

[0066] 101. Obtain a current state of the robot; the current state includes a current position and a target position of the robot.

[0067] The execution subject of this embodiment may be a humanoid robot, a control module in the humanoid robot, etc. The control module may be hardware, software, or a combination of hardware and software.

[0068] In some embodiments, the current state may also include at least one of the following: grid properties of the grid to which the current position belongs, joint angles, joint angular velocities, strides, step frequency, external force, body inclination angle, body angular velocity, and plantar pressure value.

[0069] Specifically, during the robot's movement, the robot's position, posture, joint status, sensor data and other information are obtained in real time, and this information is fed back to the robot's motion control module (for path planning, and at least one of gait control and balance control) as the current state.

[0070] 102. Input the current state into the motion control model to obtain the current action that the robot needs to perform; the motion control model is obtained by performing multi-task training on the reinforcement learning model; the multi-task training includes path planning task training, and at least one of gait control task training and balance control task training; the gait control task training and the balance control task training are performed on the basis of the path planning task training.

[0071] Specifically, the motion control model is used to determine the current action to be performed corresponding to the current state based on the trained strategy function according to the current state.

[0072] In this embodiment, in the process of training the motion control model, only two tasks can be trained. Specifically, path planning task training is first performed, and then gait control task training or balance control task training is performed to complete the training. Three tasks can also be trained. Specifically, considering the different degrees of influence of the three task trainings on the stability of the robot, the training can be performed in order from high to low according to the degree of influence. For example, path planning task training can be performed first, then gait control task training can be performed, and finally balance control task training can be performed.

[0073] 103. Control the robot to execute the current action. If the robot reaches the target position after completing the current action, the current motion task is completed.

[0074] Specifically, after executing the current action, the robot can determine whether it has reached the target position. If it has reached the target position, the task is completed. If it has not reached the target position, the robot returns to repeatedly execute steps 101 to 103 until it reaches the target position or receives a stop command.

[0075] The robot motion control method provided in this embodiment combines gait control training or balance control training with path planning training based on a reinforcement learning algorithm, and obtains a motion control model through training, so that the robot can improve stability while being able to efficiently and reasonably plan paths.

[0076] Figure 2 Schematic diagram of the training process of the motion control model provided in the embodiment of the present application Figure 1 .like Figure 2 As shown, the method includes:

[0077] 201. Constructing a first reinforcement learning model corresponding to the path planning task training based on an initial strategy function and an initial value function; the first reinforcement learning model includes a first reward function; the first reward function is related to whether a collision with an obstacle occurs and whether the distance to the end point is shortened.

[0078] 202. Train the first reinforcement learning model to obtain a trained first reinforcement learning model.

[0079] Specifically, firstly, the parameters related to robot motion control can be defined, including but not limited to learning rate, discount factor, and environmental parameters (such as the grid size of path planning, the number of joints and related physical parameters of gait control, the number of sensors and related thresholds of balance control, etc.). The value function and strategy function in the first reinforcement learning model are initialized to obtain the initial strategy function and initial value function.

[0080] For path planning task training, we can first set up the environment: discretize or parameterize the robot's environment, for example, divide its activity space into small grid areas, each with different properties (such as whether there are obstacles, ground flatness, etc.), and use this as a reinforcement learning environment. The position of the robot in this environment and the position of the target end point constitute the state space. Secondly, we can define the action: the actions that the robot can take can include moving a certain step length in different directions (such as moving a short distance forward, backward, left, and right), and multiple step length actions constitute the action space. Finally, we can design rewards: when the robot moves toward the end point and does not collide with an obstacle, a certain positive reward is given; if it collides with an obstacle, a large negative reward is given; when it successfully reaches the end point, a high positive reward is given. By constantly trial and error in this environment (that is, constantly trying different action sequences), the robot can learn the optimal action strategy from the starting point to the end point, thereby achieving more efficient and safer path planning, and obtaining the principle of the optimal control strategy through continuous sampling (trial and error).

[0081] Exemplarily, the training process of the first reinforcement learning model corresponding to the path planning task training may include the following steps:

[0082] Step 2021, define the state representation of the first reinforcement learning model: Assume the position of the robot in the two-dimensional plane is , the target location is , the environment is discretized into The grid, with Represents the state, where Indicates the grid where the robot is located Attributes (such as whether there are obstacles, 0 means no obstacles, 1 means there are obstacles).

[0083] Step 2022: Define the action space of the first reinforcement learning model: define the action set ,in, Indicates the distance to move forward one grid unit , Indicates the distance to move one grid unit backward , Indicates the distance to move one grid unit to the left , Indicates the distance to move one grid unit to the right .

[0084] Step 2023, calculate the reward function of the first reinforcement learning model: Assume the current position of the robot is , the target location is , the distance function is .

[0085] If the robot moves without hitting an obstacle and is closer to the target, it will be rewarded .in, is the distance after moving, is the distance before moving, is a positive reward less than a preset value, which can be less than or equal to 0.2. It can be 0.1.

[0086] If you hit an obstacle, you will be rewarded =-10.

[0087] If you reach the target location, you will be rewarded =100.

[0088] Step 2024: Calculate the state transition probability of the first reinforcement learning model: Assuming that the environment is deterministic (the robot's actions can be performed accurately), the state transition probability for:

[0089] If the action is legal (will not cause the robot to exceed the boundaries of the environment or hit obstacles), then It is based on action The status after updating the location, .

[0090] If the action is illegal, then , .

[0091] Step 2025: Calculate the value function update process of the first reinforcement learning model: Use the Q-learning algorithm to update the value function , the update formula is:

[0092] (1)

[0093] Where Q is the value function, is the current state, For the state of the next moment, is the action at the current moment, is the learning rate, is the discount factor, Is the action that executes the current moment After the reward, Indicates the action at the next moment.

[0094] Repeat steps 2023 to 2025 until the robot reaches the target position or reaches the maximum number of iterations, obtain the optimal path planning strategy corresponding to the maximum reward value from the starting point to the end point, and obtain the first reinforcement learning model after training.

[0095] 203. Construct a second reinforcement learning model corresponding to the gait control task based on the trained first reinforcement learning model; the second reinforcement learning model includes a second reward function; the second reward function is related to the uniformity of walking stride and the uniformity of step frequency.

[0096] Specifically, after obtaining the trained first reinforcement learning model, the policy function and value function of the trained first reinforcement learning model can be used as the initial policy function and initial value function of the second reinforcement learning model, and a reward function specifically for the gait control task can be designed for the second reinforcement learning model.

[0097] 204. Train the second reinforcement learning model to obtain a trained second reinforcement learning model.

[0098] Specifically, for the gait control task, we can first define the environment and state representation: the robot's current motion state is described by the angle, angular velocity, torque and other parameters of each joint of the robot and the body posture (such as inclination, center of gravity position, etc.) as the state, and the entire motion scene (such as terrain conditions, whether there is external force interference, etc.) as the environmental information. Secondly, the action setting can be performed: the action can be the torque adjustment applied to each joint, the change of joint angle and other operations, which will affect the robot's gait. Finally, the reward mechanism can be designed: if the robot's gait can remain stable and conform to the expected motion pattern (such as uniform stride and stable rhythm when walking), a positive reward is given; if there is an unstable situation such as gait imbalance or fall, a negative reward is given. Using the reinforcement learning algorithm, the robot can learn the optimal action strategy that can achieve a stable gait in different environments (such as flat ground, rugged terrain, etc.) through continuous experiments, that is, by continuously sampling (trying the effects of different actions on gait) to optimize the gait control based on the dynamic model.

[0099] Exemplarily, the training process of the second reinforcement learning model corresponding to the gait control task training may include the following steps:

[0100] Step 2041, define state representation: Assume the robot has joints, use X=(X 1 ,X 2 ,…,X n ) represents the joint angle, represents the joint angular velocity, F=(F 1 , F 2,…, F j ) represents the external force acting on the robot body (such as gravity, friction, etc.), n and j are both positive integers greater than 1, Indicates status.

[0101] Step 2042, define action space: action represents the torque adjustment applied to n joints, is the torque applied to the i-th joint.

[0102] Step 2043, calculate the reward value based on the reward function: Let It is a function that measures the gait stability. For example, it can be defined based on the stride deviation, body shaking, etc. when the robot walks.

[0103] award , if the robot does not fall within a time step and has a more stable gait (smaller G value), it will get a higher reward.

[0104] Step 2044, define the dynamic model (state transition): according to the robot's dynamic equation ,in It is a function based on the physical structure and mechanical principles of the robot, describing the change of state over time. For example, for a simple single-joint robot, the following expression can be used:

[0105] (2)

[0106] in, is the moment of inertia, It's quality. is the acceleration due to gravity, is the connecting rod length, is the damping coefficient, represents the joint angular velocity, for The first derivative of , which represents the joint angular acceleration.

[0107] Step 2045, policy update (policy gradient method): the policy gradient algorithm can be used to update the policy , the gradient update formula is:

[0108] (3)

[0109] in, are the parameters of the policy network, is a strategic network, is the advantage function, , is the action-value function, is the state-value function.

[0110] Steps 2043 to 2045 are repeatedly executed until the stopping condition is met (for example, the iteration number threshold is reached). When the robot moves along the planned path, the gait control strategy is continuously adjusted so that the robot can walk stably.

[0111] 205. Construct a third reinforcement learning model corresponding to the balance control task training based on the trained second reinforcement learning model; the third reinforcement learning model includes a third reward function. The third reward function is related to whether to fall down when subjected to external force and the speed of restoring balance when subjected to external force.

[0112] Specifically, after obtaining the trained second reinforcement learning model, the policy function and value function of the trained second reinforcement learning model can be used as the initial policy function and initial value function of the third reinforcement learning model, and a reward function specifically for the balance control task can be designed for the third reinforcement learning model.

[0113] 206. Train the third reinforcement learning model to obtain a trained third reinforcement learning model.

[0114] Specifically, for balance control task training, first define the environment and state: combine the robot's posture (such as the body's tilt angle, angular velocity, etc.), the contact state between the sole and the ground (such as pressure distribution, etc.), and external interference forces (if any) as state information to describe the robot's current balance state. The surrounding terrain conditions, whether there are sudden external forces, etc. constitute the environment. Secondly, define the action: the action can be fine-tuning different parts of the robot's body (such as leg joints, waist, etc.) to adjust the center of gravity, change the support force, etc., the purpose is to maintain balance. Finally, you can set rewards: when the robot can maintain balance under external interference (for example, it can still stand firm after being slightly pushed), give a positive reward; if it loses balance and falls, give a negative reward. Through the reinforcement learning algorithm, the robot can continuously trial and error in such an environment (try different balance adjustment actions), learn the optimal action strategy to maintain balance in various situations, and realize the transition from continuous sampling trial and error learning in a simulated environment to effective balance control in real scenes (that is, using the sim-to-real method, first train the balance control strategy in a simulated environment, and then apply it to the actual scene).

[0115] Exemplarily, the training process of the third reinforcement learning model corresponding to the balance control task may include the following steps:

[0116] Step 2061, define state representation: Assume the inclination angle of the robot body is , the angular velocity is The foot pressure sensor measures Assume the robot has Foot pressure sensor), status is .

[0117] Step 2062, define action space: action Indicates the robot body Fine-tune key parts of the body, such as adjusting the leg joint angle, changing the waist torque, etc., to adjust the center of gravity and support force.

[0118] Step 2063, calculate the reward value based on the reward function: Let is a function that measures the state of balance. For example, it can be defined based on whether the tilt angle is within the stable range, whether the pressure distribution on the sole of the foot is uniform, etc. is the inclination angle of the robot body, is the angular velocity of the robot, and P is the measurement value of the robot's foot pressure sensor.

[0119] Rewards , if the robot can recover its balance faster after being disturbed (i.e. The smaller the value of ), the higher the reward.

[0120] Step 2064, calculate the state transition probability (in the presence of external interference): Assume that the external interference is a random variable, the state transition probability Depends on the robot's physical structure, control actions and external disturbance d. For example, according to the rigid body kinematics and dynamics equations of the robot, it can be established , where α t is the tilt angle of the robot body at the current moment, α t+1 is the predicted value of the robot's tilt angle at the next moment, ω t is the angular velocity of the robot at the current moment, is the time interval, It is a function obtained according to physical principles.

[0121] Step 2065: Perform policy update: The asynchronous advantage actor-critic (A3C) algorithm can be used to update the policy and value functions in multiple parallel threads. For the policy update part, the gradient of the policy can be updated according to the advantage function, and the value function can be updated to better estimate the state value. The specific formula involves the policy network and value network Gradient update of the policy network, such as the gradient update formula: (in, is the parameter of the policy network), r is the reward, is the discount factor, V() is the state-value function, S t is the current state, S t+1 The state at the next moment.

[0122] Repeat steps 2063 to 2065 until the stop condition is met (for example, the iteration number threshold is reached). During the movement of the robot along the planned path, the robot continuously monitors external interference and its own state, and adjusts its actions according to the balance control strategy to cope with various terrain changes and external interference and maintain balance.

[0123] 207. Determine the motion control model according to the trained third reinforcement learning model.

[0124] The robot motion control method provided in this embodiment combines gait control training or balance control training with path planning training based on a reinforcement learning algorithm, and obtains a motion control model through training, so that the robot can improve stability while being able to efficiently and reasonably plan paths.

[0125] In some embodiments, Figure 3 As shown, based on the above embodiment, for example, in the above Figure 2 Based on the illustrated embodiment, step 207 may include:

[0126] 2071. Construct a fourth reinforcement learning model based on the trained third reinforcement learning model; a fourth reward function corresponding to the fourth reinforcement learning model is determined based on the first reward function, the second reward function and the third reward function.

[0127] Specifically, after obtaining the trained third reinforcement learning model, the policy function and value function of the trained third reinforcement learning model can be used as the initial policy function and initial value function of the fourth reinforcement learning model, and the fourth reward function corresponding to the fourth reinforcement learning model can be determined based on the first reward function, the second reward function and the third reward function.

[0128] 2072. Train the fourth reinforcement learning model to obtain the motion control model.

[0129] In some embodiments, training the fourth reinforcement learning model to obtain the motion control model may include: modeling the environment in which the robot is located to obtain an environment model; the environment model includes a starting position and a target position; placing the robot at the starting position to obtain an initial environment state; for each time step, determining the current action to be performed according to the current state, and after performing the current action, obtaining the next state and reward value; the reward value is determined according to the fourth reward function; storing the current state, the current action, the next state and the reward value in an experience replay buffer; randomly extracting a preset number of training data from the experience replay buffer; and performing gradient descent training on the fourth reinforcement learning model according to the training data to obtain the motion control model.

[0130] Specifically, after obtaining the trained third reinforcement learning model, considering the mutual influence among path planning, gait control and balance control, in order to further balance the three, a comprehensive reward function, namely the fourth reward function, can be designed to take into account the reward factors of path planning, gait control and balance control.

[0131] Exemplary design of the fourth reward function: the fourth reward function can be defined as . Among them, R{path} is the reward related to path planning. For example, for every meter the robot approaches the target position, R{path} increases by 10 points. R{gait} is the reward related to gait control. A normal and stable gait is rewarded with 2 points for each step, and points are deducted if there is an abnormal gait. R{balance} is the balance control reward. The robot is rewarded with 3 points per second for maintaining an upright balance during movement, and points are deducted if the tilt angle exceeds a certain threshold. w1, w2, and w3 are weights, which are allocated according to the focus of the task, such as w1=0.4, w2=0.3, w3=0.3.

[0132] During the reinforcement learning training process, the robot takes actions based on the state of the environment, and after each action, the reward value is calculated based on this reward function. The agent will learn how to obtain high path planning rewards while also ensuring good gait and balance. Specifically, the following steps may be included:

[0133] Environment construction: Build a simulation environment for the humanoid robot, including terrain, such as flat ground, slopes, obstacles, etc., starting position and target position.

[0134] Agent settings: Define a humanoid robot as an agent, whose action space includes discrete or continuous actions such as moving forward, backward, turning left, turning right, adjusting stride, changing step frequency, etc. The state space covers the robot's position information (x, y coordinates), speed, attitude angle (pitch angle, yaw angle, roll angle), surrounding environment information sensed by sensors (such as distance to obstacles), and current gait parameters (stride length, step frequency, etc.).

[0135] Reinforcement learning algorithm selection: The fourth reinforcement learning model is determined on the basis of the third reinforcement learning model. Therefore, the fourth reinforcement learning model can use algorithms such as deep Q network (DQN) or proximal policy optimization (PPO) in accordance with the third reinforcement learning model. Taking DQN as an example, the parameters of the Q network are initialized based on the trained third reinforcement learning model, including the weights and biases of the neural network.

[0136] The training cycle process can specifically include the following steps:

[0137] For each training round:

[0138] Reset environment: Place the robot at the starting position and initialize the environment state s0.

[0139] For each time step t:

[0140] According to the current status , the agent selects an action via the Q network (in DQN) or the policy network (in PPO) For example, an action is selected based on the size of the Q value, or an action is sampled based on the action probability distribution output by the policy network.

[0141] Execute an action After that, the robot generates a new state in the environment , and receive a reward , by the fourth reward function For example, if the robot moves a certain distance toward the target position while maintaining balance and good gait, it will get a positive reward; if the robot falls or deviates from the path planning, it will get a negative reward.

[0142] The data Stored in the experience replay buffer. In DQN, the experience replay buffer is used to store a certain amount of historical experience for subsequent random sampling for network training.

[0143] Randomly sample a batch of data from the experience replay buffer , for the DQN algorithm, calculate the target Q value: ,in is a discount factor that weighs the importance of current rewards and future rewards, are the parameters of the target Q network (DQN uses the target network to stabilize the training process). Then according to the loss function Perform gradient descent training on the Q network and update the network parameters θ 1 For policy gradient-based algorithms such as PPO, the gradient is calculated according to the policy gradient theorem and the parameters of the policy network are updated to maximize the expectation of the cumulative reward.

[0144] A training round ends when the robot reaches the target position or exceeds the maximum time step.

[0145] In order to improve the performance of the model, training evaluation and optimization can be performed. Specifically, the performance of the agent can be evaluated every certain number of training rounds. For example, calculate the average number of steps, average rewards, and number of falls of the robot to the target position in multiple test training rounds. Adjust the training parameters according to the evaluation results, such as the learning rate, discount factor, and the weights w1, w2, and w3 of the reward function. If it is found that the robot pays too much attention to path planning and ignores balance, resulting in frequent falls, the weight w3 of the balance reward can be appropriately increased.

[0146] The above training cycle is repeated until the performance of the agent reaches a satisfactory level, that is, the robot can efficiently and stably reach the target position from the starting position in various complex environments while maintaining good gait and balance control.

[0147] In order to clearly illustrate the training process, the following is an example of the training process of the Q network:

[0148] First, collect interaction data. Let the robot act in the environment according to the current policy, which selects actions based on the output of the Q network. For example, the robot is in the initial state Next, according to the Q network's estimation of the Q values ​​of different actions, the action with the highest Q value is selected. Execution, such as moving forward a certain distance in a straight line.

[0149] Record the status of each interaction ,action ,award and the next state , and store these experience data in the experience playback buffer.

[0150] Sampling from experience replay: Randomly sample a batch of experience data from the experience replay buffer, denoted as , where N is the number of samples.

[0151] Secondly, the target Q value is calculated. For each sampled empirical data, the target Q value is calculated using the target Q network: The target value is used to guide the training of the Q network. is a discount factor that weighs the importance of current rewards and future rewards, is the weight parameter of the target Q network, S i+1 is the next state, a' is the next state S i+1 The best action among all possible actions. If the current state is a terminal state, then The target Q network has the same structure as the trained Q network, and is used to provide a relatively stable target Q value estimate. The trained Q network is used to collect empirical data.

[0152] Again, calculate the loss function and update the Q network. Use the mean square error loss function to calculate the error between the predicted Q value of the current Q network and the target Q value: .in, is the target Q value, is the predicted Q value.

[0153] The gradient of the loss function to the Q network parameters is calculated by the back propagation algorithm , and use an optimization algorithm (such as Adam) to update the parameters , the update formula is: in is the learning rate. During the update process, the weight parameters in the target Q network Remain unchanged.

[0154] Again, the strategy is updated. According to the updated Q network, Greedy strategy or other strategy improvement methods update the strategy. For example, in In the greedy strategy, with probability 1- Select the action with the largest Q value, with probability Randomly select an action. As training progresses, gradually reduce The value of makes the robot choose the optimal action based more on the estimation of the Q network. It should be noted that The greedy strategy is a simple and commonly used method to balance between exploring new actions and using known optimal actions. is a probability value between 0 and 1, indicating the degree of random exploration.

[0155] The specific process of strategy update may include: Action selection: At each time step, when the robot needs to select an action, it first generates a random number M uniformly distributed in the interval [0,1]. Then the random number M is compared with Compare: If M is less than , indicating that for exploration, the robot will randomly select an action from the action space to perform. For example, if the robot's action space contains 5 actions (move forward, turn left, turn right, adjust stride, adjust balance torque), then it will randomly select one of the 5 actions with equal probability, regardless of the current Q network's evaluation of the Q value of these actions. If M is greater than or equal to , which means that the robot will select the action with the largest Q value output by the Q network in the current state to execute. That is, by inputting the current state s into the updated Q network In the above example, we can get the corresponding different actions. The Q value of each action is then selected to take the action with the largest Q value.

[0156] In some embodiments, the Value: In the early stages of training, a relatively large Values, such as = 0.5 or even higher to ensure that the robot has a greater probability of exploring different actions in the environment, trying various possible path planning methods, gait control adjustments, and balance control operations, so as to collect more data from different situations and help the Q network better learn the value of different state-action pairs. As training continues, that is, after multiple rounds of interaction with the environment and updates to the Q network, it will gradually decrease. The value of the current iteration round The previous iteration round The decay factor is greater than 0 and less than 1. For example, a certain decay rule can be used, such as after a certain number of training rounds, Multiply it by a decay factor less than 1, such as 0.99, to make it gradually decrease. When it becomes very small, such as close to 0.05 or smaller, the robot will rely more on the optimal action estimated by the Q network to make decisions, that is, it will make more use of it. Because at this time the Q network has been through a lot of data learning and parameter updates, its estimate of the action value in each state is relatively more accurate, so the robot will choose the optimal action to achieve path planning, maintain a good gait and balance control. The probability is greater.

[0157] Finally, continuous training and optimization can be performed. Repeat the above steps continuously, allowing the robot to continuously interact in the environment and collect experience data, and update the Q network and strategy. After multiple iterations, the robot's Q network will gradually learn the value of each action in different states, and the strategy will continue to optimize, so that it performs better and better in path planning, gait control, and balance control.

[0158] Figure 4 This is a schematic diagram of the structure of the robot motion control device provided in the embodiment of the present application. Figure 4 As shown, the robot motion control device 40 includes: an acquisition module 401 and an input module 402 .

[0159] The acquisition module 401 is used to acquire the current state of the robot; the current state includes the current position and target position of the robot.

[0160] The input module 402 is used to input the current state into the motion control model to obtain the current action that the robot needs to perform; the motion control model is obtained by multi-task training of the reinforcement learning model; the multi-task training includes path planning task training, and at least one of gait control task training and balance control task training; the gait control task training and the balance control task training are performed on the basis of the path planning task training.

[0161] The control module 403 is used to control the robot to perform the current action. If the robot reaches the target position after completing the current action, the current motion task is completed.

[0162] The robot motion control device provided in the embodiment of the present application performs gait control training or balance control training together with path planning training based on a reinforcement learning algorithm, and obtains a motion control model through training, so that the robot can improve stability while being able to efficiently and reasonably plan paths.

[0163] The robot motion control device provided in the embodiment of the present application can be used to execute the above-mentioned method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0164] In some embodiments, the current state also includes: grid properties of the grid to which the current position belongs, joint angles, joint angular velocities, stride length, step frequency, external force, body inclination angle, body angular velocity, and sole pressure value.

[0165] In some embodiments, the device 40 also includes a training module (not shown), which is used to: construct a first reinforcement learning model corresponding to the path planning task training based on an initial policy function and an initial value function; train the first reinforcement learning model to obtain a trained first reinforcement learning model; construct a second reinforcement learning model corresponding to the gait control task based on the trained first reinforcement learning model; train the second reinforcement learning model to obtain a trained second reinforcement learning model; and determine the motion control model based on the trained second reinforcement learning model.

[0166] In some embodiments, the training module is used to update the value function based on the following expression:

[0167] ,

[0168] Where Q is the value function, is the current state, For the state of the next moment, is the action at the current moment, is the learning rate, is the discount factor, Is the action that executes the current moment After the reward, Indicates the action at the next moment.

[0169] In some embodiments, the training module is used to perform policy updates based on the following expression:

[0170]

[0171] in, are the parameters of the policy network, is a strategic network, is the advantage function, , is the action-value function, is the state-value function.

[0172] In some embodiments, the first reinforcement learning model includes a first reward function; the first reward function is related to whether a collision occurs with an obstacle and whether the distance to the end point is shortened; the second reinforcement learning model includes a second reward function; the second reward function is related to the uniformity of walking stride and the uniformity of step frequency.

[0173] In some embodiments, the training module is specifically used to: construct a third reinforcement learning model corresponding to the balance control task training based on the trained second reinforcement learning model; the third reinforcement learning model includes a third reward function; the third reward function is related to whether to fall when subjected to external force, and the speed of restoring balance when subjected to external force; train the third reinforcement learning model to obtain the trained third reinforcement learning model; determine the motion control model based on the trained third reinforcement learning model.

[0174] In some embodiments, the training module is specifically used to: construct a fourth reinforcement learning model based on the trained third reinforcement learning model; the fourth reward function corresponding to the fourth reinforcement learning model is determined based on the first reward function, the second reward function and the third reward function; train the fourth reinforcement learning model to obtain the motion control model.

[0175] In some embodiments, the training module is specifically used to: model the environment in which the robot is located to obtain an environment model; the environment model includes a starting position and a target position; the robot is placed at the starting position to obtain an initial environment state; for each time step, according to the current state, the current action to be performed is determined, and after executing the current action, the next state and the reward value are obtained; the reward value is determined according to the fourth reward function; the current state, the current action, the next state and the reward value are stored in an experience replay buffer; a preset number of training data are randomly extracted from the experience replay buffer; according to the training data, the fourth reinforcement learning model is gradient descent trained to obtain the motion control model.

[0176] In some embodiments, the training module is specifically used to: generate a random number M for each time step, where M is greater than or equal to 0 and M is less than or equal to 1; if M is less than , then randomly select an action from the action space to execute; if M is greater than or equal to , then select the action corresponding to the maximum Q value among the Q values ​​output by the Q network in the current state as the target action, and execute the target action; is a probability value between 0 and 1, indicating the degree of random exploration; The previous iteration round The attenuation factor is greater than 0 and less than 1.

[0177] Figure 5 A schematic diagram of the hardware structure of the robot motion control device provided in an embodiment of the present application.

[0178] Device 50 may include one or more of the following components: a processing component 501 , a memory 502 , a power component 503 , a multimedia component 504 , an audio component 505 , an input / output (I / O) interface 506 , a sensor component 507 , and a communication component 508 .

[0179] The processing component 501 generally controls the overall operation of the device 50, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 501 may include one or more processors 509 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 501 may include one or more modules to facilitate the interaction between the processing component 501 and other components. For example, the processing component 501 may include a multimedia module to facilitate the interaction between the multimedia component 504 and the processing component 501.

[0180] The memory 502 is configured to store various types of data to support operations on the device 50. Examples of such data include instructions for any application or method operating on the device 50, contact data, phone book data, messages, pictures, videos, etc. The memory 502 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0181] The power supply component 503 provides power to the various components of the device 50. The power supply component 503 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device 50.

[0182] The multimedia component 504 includes a screen that provides an output interface between the device 50 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 504 includes a front camera and / or a rear camera. When the device 50 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0183] The audio component 505 is configured to output and / or input audio signals. For example, the audio component 505 includes a microphone (MIC), and when the device 50 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 502 or sent via the communication component 508. In some embodiments, the audio component 505 also includes a speaker for outputting audio signals.

[0184] I / O interface 506 provides an interface between processing component 501 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0185] The sensor assembly 507 includes one or more sensors for providing various aspects of status assessment for the device 50. For example, the sensor assembly 507 can detect the open / closed state of the device 50, the relative positioning of components, such as the display and keypad of the device 50, and the sensor assembly 507 can also detect the position change of the device 50 or a component of the device 50, the presence or absence of user contact with the device 50, the orientation or acceleration / deceleration of the device 50, and the temperature change of the device 50. The sensor assembly 507 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 507 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 507 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0186] The communication component 508 is configured to facilitate wired or wireless communication between the device 50 and other devices. The device 50 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 508 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 508 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0187] In an exemplary embodiment, the device 50 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0188] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 502 including instructions, and the instructions can be executed by a processor 509 of the device 50 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0189] The computer-readable storage medium mentioned above may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium may be any available medium that can be accessed by a general or special-purpose computer.

[0190] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0191] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0192] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the robot motion control method performed by the robot motion control device as described above.

[0193] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A robot motion control method, characterized in that: include: Get the current state of the robot; The current state includes the current position and target position of the robot; The current state is input into a motion control model to obtain a current action that the robot needs to perform; the motion control model is obtained by performing multi-task training on a reinforcement learning model; the multi-task training includes path planning task training, gait control task training and balance control task training; Constructing a first reinforcement learning model corresponding to the path planning task training based on the initial strategy function and the initial value function; After obtaining the trained first reinforcement learning model, using the strategy function and the value function of the trained first reinforcement learning model as the initial strategy function and the initial value function of the second reinforcement learning model corresponding to the gait control task training respectively; After obtaining the trained second reinforcement learning model, the policy function and the value function of the trained second reinforcement learning model are respectively used as the initial policy function and the initial value function of the third reinforcement learning model corresponding to the balance control task training; After obtaining the trained third reinforcement learning model, using the policy function and the value function of the trained third reinforcement learning model as the initial policy function and the initial value function of the fourth reinforcement learning model, respectively, and determining the fourth reward function corresponding to the fourth reinforcement learning model according to the first reward function of the first reinforcement learning model, the second reward function of the second reinforcement learning model, and the third reward function of the third reinforcement learning model; Training the fourth reinforcement learning model to obtain the motion control model; The robot is controlled to execute the current action. If the robot reaches the target position after completing the current action, the current motion task is completed.

2. The method according to claim 1, characterized in that The current state also includes: grid attributes of the grid to which the current position belongs, joint angles, joint angular velocities, stride length, step frequency, external force, body inclination angle, body angular velocity, and sole pressure value.

3. The method according to claim 1 or 2, characterized in that: The first reward function is related to whether a collision with an obstacle occurs and whether the distance to the end point is shortened; Training the first reinforcement learning model to obtain a trained first reinforcement learning model; The second reward function is related to the uniformity of walking stride length and the uniformity of step frequency; The second reinforcement learning model is trained to obtain a trained second reinforcement learning model.

4. The method according to claim 3, characterized in that The training of the first reinforcement learning model includes: The value function is updated based on the following expression: ， Where Q is the value function, is the current state, For the state of the next moment, is the action at the current moment, is the learning rate, is the discount factor, Is the action that executes the current moment After the reward, Indicates the action at the next moment.

5. The method according to claim 3, characterized in that: The training of the second reinforcement learning model comprises: The policy is updated based on the following expression of the gradient update formula: in, are the parameters of the policy network, is a strategic network, is the advantage function, , is the action-value function, is the state-value function.

6. The method according to claim 1 or 2, characterized in that: The third reward function is related to whether the user falls down when subjected to an external force, and the speed of restoring balance when subjected to an external force; The third reinforcement learning model is trained to obtain a trained third reinforcement learning model.

7. The method according to claim 1 or 2, characterized in that: The step of training the fourth reinforcement learning model to obtain the motion control model includes: Modeling the environment in which the robot is located to obtain an environment model; the environment model includes a starting position and a target position; Placing the robot at the starting position to obtain an initial environment state; For each time step, according to the current state, determine the current action to be performed, and after performing the current action, obtain the next state and reward value; the reward value is determined according to the fourth reward function; Storing the current state, the current action, the next state, and the reward value in an experience replay buffer; Randomly extracting a preset amount of training data from the experience replay buffer; According to the training data, gradient descent training is performed on the fourth reinforcement learning model to obtain the motion control model.

8. The method according to claim 1 or 2, characterized in that: The fourth reinforcement learning model is a deep Q network; and the training of the fourth reinforcement learning model includes: For each time step, generate a random number M, M is greater than or equal to 0 and M is less than or equal to 1; If M is less than , then randomly select an action from the action space to execute; If M is greater than or equal to , then select the action corresponding to the maximum Q value among the Q values ​​output by the Q network in the current state as the target action, and execute the target action; is a probability value between 0 and 1, indicating the degree of random exploration; The previous iteration round The attenuation factor is greater than 0 and less than 1.

9. A robot motion control device, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the robot motion control method according to any one of claims 1 to 8.

Images