Robot motion control method and equipment

Abstract

The embodiment of the invention provides a robot motion control method and device.The method comprises the steps that the current state of a robot is obtained, the current state comprises the current position and the target position of the robot, the current state is input into a motion control model, and the current action needing to be executed by the robot is obtained; the motion control model is obtained by performing multi-task training on the reinforcement learning model, the multi-task training comprises path planning task training and at least one of gait control task training and balance control task training, and the robot is controlled to execute the current action. And if the robot reaches the target position after completing the current action, completing the current motion task. According to the method provided by the embodiment of the invention, the robot can improve the stability on the basis of efficiently and reasonably planning the path.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of humanoid robots, and in particular to a robot motion control method and device. Background Art

[0002] With the rapid development of science and technology, humanoid robots have shown great application potential in many fields. In order to improve the performance of humanoid robots, it is necessary to focus on improving the motion control level of humanoid robots.

[0003] In related technologies, during the movement of a humanoid robot, path planning is usually performed using a method based on a geometric algorithm or a search algorithm.

[0004] However, in the process of realizing this application, the inventors found that there are at least the following problems in the prior art: the effectiveness of the path planning method described above is poor, and the stability of the robot during movement cannot be guaranteed. Summary of the Invention

[0005] The embodiments of the present application provide a robot motion control method and device to improve the efficiency and stability of motion control.

[0006] In a first aspect, an embodiment of the present application provides a robot motion control method, comprising:

[0007] Obtaining the current state of the robot; the current state includes the current position and target position of the robot;

[0008] Inputting the current state into a motion control model to obtain a current action that the robot needs to perform; the motion control model is obtained by performing multi-task training on a reinforcement learning model; the multi-task training includes path planning task training, and at least one of gait control task training and balance control task training; the gait control task training and the balance control task training are performed on the basis of the path planning task training;

[0009] The robot is controlled to execute the current action. If the robot reaches the target position after completing the current action, the current motion task is completed.

[0010] In one possible design, the current state also includes: grid properties of the grid to which the current position belongs, joint angles, joint angular velocities, stride length, step frequency, external force, body inclination angle, body angular velocity, and plantar pressure value.

[0011] In one possible design, the method further includes:

[0012] Constructing a first reinforcement learning model corresponding to the path planning task training based on the initial policy function and the initial value function; the first reinforcement learning model includes a first reward function; the first reward function is related to whether a collision with an obstacle occurs and whether the distance to the end point is shortened;

[0013] Training the first reinforcement learning model to obtain a trained first reinforcement learning model;

[0014] Constructing a second reinforcement learning model corresponding to the gait control task based on the trained first reinforcement learning model; the second reinforcement learning model includes a second reward function; the second reward function is related to the uniformity of walking stride length and the uniformity of cadence;

[0015] Training the second reinforcement learning model to obtain a trained second reinforcement learning model;

[0016] The motion control model is determined according to the trained second reinforcement learning model.

[0017] In one possible design, training the first reinforcement learning model includes:

[0018] The value function is updated based on the following expression:

[0019] ,

[0020] Where Q is the value function, is the current state, The state of the next moment, is the action at the current moment, is the learning rate, is the discount factor, Is the action that executes the current moment After receiving the reward, Indicates the action at the next moment.

[0021] In one possible design, training the second reinforcement learning model includes:

[0022] The policy update is based on the following expression:

[0023]

[0024] in, are the parameters of the policy network, It is a strategic network. is the advantage function, , is the action-value function, is the state-value function.

[0025] In one possible design, determining the motion control model based on the trained second reinforcement learning model includes:

[0026] Constructing a third reinforcement learning model corresponding to the balance control task training based on the trained second reinforcement learning model; the third reinforcement learning model includes a third reward function; the third reward function is related to whether the person falls when subjected to an external force and the speed of recovering balance when subjected to the external force;

[0027] Training the third reinforcement learning model to obtain a trained third reinforcement learning model;

[0028] The motion control model is determined according to the trained third reinforcement learning model.

[0029] In one possible design, determining the motion control model according to the balance strategy function and the balance value function includes:

[0030] Constructing a fourth reinforcement learning model based on the trained third reinforcement learning model; a fourth reward function corresponding to the fourth reinforcement learning model is determined based on the first reward function, the second reward function, and the third reward function;

[0031] The fourth reinforcement learning model is trained to obtain the motion control model.

[0032] In one possible design, training the fourth reinforcement learning model to obtain the motion control model includes:

[0033] Modeling the environment in which the robot is located to obtain an environment model; the environment model includes a starting position and a target position;

[0034] Placing the robot at the starting position to obtain an initial environmental state;

[0035] For each time step, determining a current action to be performed based on the current state, and obtaining a next state and a reward value after performing the current action; the reward value is determined according to the first reward function;

[0036] Storing the current state, the current action, the next state, and the reward value in an experience replay buffer;

[0037] Randomly extracting a preset amount of training data from the experience replay buffer;

[0038] Performing gradient descent training on the first reinforcement learning model according to the training data to obtain the motion control model.

[0039] In one possible design, the fourth reinforcement learning model is a deep Q network; and training the fourth reinforcement learning model to obtain the motion control model includes:

[0040] For each time step, generate a random number M, M greater than or equal to 0 and M less than or equal to 1;

[0041] If M is less than , then randomly select an action from the action space to execute;

[0042] If M is greater than or equal to , then select the action corresponding to the maximum Q value among the Q values ​​output by the Q network in the current state as the target action, and execute the target action; Is a probability value between 0 and 1, indicating the degree of random exploration; the current iteration round The previous iteration round The attenuation factor is greater than 0 and less than 1.

[0043] In a second aspect, an embodiment of the present application provides a robot motion control device, comprising:

[0044] An acquisition module is used to acquire the current state of the robot; the current state includes the current position and target position of the robot;

[0045] an input module, configured to input the current state into a motion control model to obtain a current action to be performed by the robot; the motion control model is obtained by performing multi-task training on a reinforcement learning model; the multi-task training includes path planning task training, and at least one of gait control task training and balance control task training; the gait control task training and the balance control task training are performed on the basis of the path planning task training;

[0046] The control module is used to control the robot to perform the current action. If the robot reaches the target position after completing the current action, the current motion task is completed.

[0047] In a third aspect, an embodiment of the present application provides a robot motion control device, comprising: at least one processor and a memory;

[0048] The memory stores computer-executable instructions;

[0049] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method described in the first aspect and various possible designs of the first aspect.

[0050] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the method described in the first aspect and various possible designs of the first aspect is implemented.

[0051] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the method described in the first aspect and various possible designs of the first aspect.

[0052] The robot motion control method and device provided in this embodiment include obtaining the current state of the robot, the current state including the current position and target position of the robot, inputting the current state into the motion control model, obtaining the current action that the robot needs to perform, the motion control model is obtained by multi-task training of the reinforcement learning model, the multi-task training includes path planning task training, and at least one of gait control task training and balance control task training, controlling the robot to perform the current action, and completing the current motion task if the robot reaches the target position after completing the current action. The robot motion control method provided in the embodiment of the present application combines gait control training or balance control training with path planning training based on a reinforcement learning algorithm to obtain a motion control model through training, so that the robot can improve stability on the basis of being able to plan paths efficiently and reasonably. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0054] Figure 1 A schematic diagram of a flow chart of a robot motion control method provided in an embodiment of the present application;

[0055] Figure 2 Schematic diagram of the training process of the motion control model provided in the embodiment of the present application Figure 1 ;

[0056] Figure 3 Schematic diagram of the training process of the motion control model provided in the embodiment of the present application Figure 2 ;

[0057] Figure 4 A schematic diagram of the structure of a robot motion control device provided in an embodiment of the present application;

[0058] Figure 5 Schematic diagram of the hardware structure of the robot motion control device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0060] It should be noted that the robot motion control method provided in this application can be used in the field of humanoid robot technology, and can also be used in any field other than the field of humanoid robots. The application field of the robot motion control method provided in this application is not limited.

[0061] With the rapid development of science and technology, humanoid robots have shown great application potential in many fields, such as industrial manufacturing, medical care, home services, and rescue. However, achieving efficient, stable, and intelligent motion control of humanoid robots in complex environments has always been a major challenge in the field of robotics.

[0062] In related technologies, path planning usually adopts a method based on geometric algorithms or search algorithms. For example, The algorithm finds a path from a starting point to a destination by calculating the heuristic cost of each node in a known map. However, while this approach can be effective in simple, static environments, it has significant limitations in complex and dynamic real-world scenarios. When dynamic obstacles or inaccurate map information appear, traditional path planning algorithms struggle to quickly and effectively adjust the path, which can easily lead to the robot getting stuck or causing collisions, and can also affect the robot's stability and balance during walking.

[0063] In response to the above technical problems, the inventors of this application have discovered that a reinforcement learning algorithm can be used to enable the robot to continuously trial and error in the environment and learn the optimal behavior strategy based on the reward signal fed back from the environment to deal with obstacles in the environment or inaccurate map information. In addition, by combining gait control training or balance control training with path planning training based on the reinforcement learning algorithm, the robot can improve stability on the basis of being able to plan paths efficiently and reasonably.

[0064] The following specific embodiments are used to describe the technical solution of the present application in detail. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0065] Figure 1 Schematic diagram of the flow of the robot motion control method provided in the embodiment of the present application. Figure 1 As shown, the method includes:

[0066] 101. Obtain a current state of the robot; the current state includes a current position and a target position of the robot.

[0067] The execution subject of this embodiment may be a humanoid robot, a control module in the humanoid robot, etc. The control module may be hardware, software, or a combination of hardware and software.

[0068] In some embodiments, the current state may further include at least one of the following: grid properties of the grid to which the current position belongs, joint angles, joint angular velocities, stride length, step frequency, external force, body tilt angle, body angular velocity, and plantar pressure value.

[0069] Specifically, during the robot's movement, the robot's position, posture, joint status, sensor data and other information are obtained in real time, and this information is fed back as the current state to the robot's motion control module (for path planning, and at least one of gait control and balance control).

[0070] 102. Input the current state into a motion control model to obtain a current action that the robot needs to perform; the motion control model is obtained by performing multi-task training on a reinforcement learning model; the multi-task training includes path planning task training, and at least one of gait control task training and balance control task training; the gait control task training and the balance control task training are performed on the basis of the path planning task training.

[0071] Specifically, the motion control model is used to determine the current action that needs to be performed corresponding to the current state based on the trained strategy function according to the current state.

[0072] In this embodiment, during the training process for the motion control model, only two tasks can be trained. Specifically, path planning training can be performed first, followed by gait control training or balance control training, to complete the training. Alternatively, three tasks can be trained. Specifically, considering the varying degrees of impact of the three tasks on the robot's stability, the training can be performed sequentially, from highest to lowest impact. For example, path planning training can be performed first, followed by gait control training, and finally balance control training.

[0073] 103. Control the robot to perform the current action. If the robot reaches the target position after completing the current action, the current motion task is completed.

[0074] Specifically, after executing the current action, the robot can determine whether it has reached the target position. If it has reached the target position, the task is completed. If not, the robot returns to repeat steps 101 to 103 until it reaches the target position or receives a stop command.

[0075] The robot motion control method provided in this embodiment combines gait control training or balance control training with path planning training based on a reinforcement learning algorithm, and obtains a motion control model through training, so that the robot can improve stability while being able to plan paths efficiently and reasonably.

[0076] Figure 2 Schematic diagram of the training process of the motion control model provided in the embodiment of the present application Figure 1 .like Figure 2 As shown, the method includes:

[0077] 201. Construct a first reinforcement learning model corresponding to the path planning task training based on an initial policy function and an initial value function; the first reinforcement learning model includes a first reward function; the first reward function is related to whether a collision with an obstacle occurs and whether the distance to the end point is shortened.

[0078] 202. Train the first reinforcement learning model to obtain a trained first reinforcement learning model.

[0079] Specifically, first define parameters related to robot motion control, including but not limited to the learning rate, discount factor, and environmental parameters (such as the grid size for path planning, the number of joints and related physical parameters for gait control, and the number of sensors and related thresholds for balance control). Initialize the value function and policy function in the first reinforcement learning model to obtain the initial policy function and initial value function.

[0080] For path planning training, the first step is to set up the environment: discretize or parameterize the robot's environment. For example, divide its activity space into small grid regions, each with different properties (such as the presence of obstacles and the smoothness of the surface). This serves as the reinforcement learning environment. The robot's position within this environment and the target destination position constitute the state space. Next, define actions: The robot's actions can include moving a certain number of steps in different directions (such as forward, backward, left, or right for a short distance). Multiple steps constitute the action space. Finally, design rewards: a certain positive reward is given when the robot moves toward the destination without colliding with an obstacle; a larger negative reward is given if it collides with an obstacle; and a high positive reward is given when it successfully reaches the destination. By repeatedly performing trial and error in this environment (i.e., repeatedly trying different action sequences), the robot can learn the optimal action strategy from the starting point to the destination, thereby achieving more efficient and safe path planning. This is based on the principle of continuous sampling (trial and error) to obtain the optimal control strategy.

[0081] Exemplarily, the training process of the first reinforcement learning model corresponding to the path planning task training may include the following steps:

[0082] Step 2021: Define the state representation of the first reinforcement learning model: Assume the position of the robot in the two-dimensional plane is , the target position is , the environment is discretized into The grid, with Represents the state, where Indicates the grid where the robot is located Attributes (such as whether there are obstacles, 0 means no obstacles, 1 means there are obstacles).

[0083] Step 2022: Define the action space of the first reinforcement learning model: define the action set ,in, Indicates the distance to move forward one grid unit , Indicates the distance to move one grid unit backward , Indicates the distance to move one grid unit to the left , Indicates the distance to move one grid unit to the right .

[0084] Step 2023: Calculate the reward function of the first reinforcement learning model: Assume the current position of the robot is , the target position is , the distance function is .

[0085] If the robot moves without hitting any obstacles and is closer to the target, it will be rewarded .in, is the distance after moving, is the distance before moving, Is a positive reward less than the preset value, which can be less than or equal to 0.2. It can be 0.1.

[0086] If you hit an obstacle, you will be rewarded =-10.

[0087] If you reach the target location, you will be rewarded =100.

[0088] Step 2024: Calculate the state transition probability of the first reinforcement learning model: Assuming that the environment is deterministic (the robot's actions can be executed accurately), the state transition probability for:

[0089] If the action is legal (will not cause the robot to exceed the boundaries of the environment or hit an obstacle), then It is based on action The status after updating the location, .

[0090] If the action is illegal, then , .

[0091] Step 2025: Calculate the value function update process of the first reinforcement learning model: Use the Q-learning algorithm to update the value function , the update formula is:

[0092] (1)

[0093] Where Q is the value function, is the current state, The state of the next moment, is the action at the current moment, is the learning rate, is the discount factor, Is the action that executes the current moment After receiving the reward, Indicates the action at the next moment.

[0094] Repeat steps 2023 to 2025 until the robot reaches the target position or reaches the maximum number of iterations, obtain the optimal path planning strategy corresponding to the maximum reward value from the starting point to the end point, and obtain the first reinforcement learning model after training.

[0095] 203. Construct a second reinforcement learning model corresponding to the gait control task based on the trained first reinforcement learning model; the second reinforcement learning model includes a second reward function; the second reward function is related to the uniformity of walking stride and the uniformity of step frequency.

[0096] Specifically, after obtaining the trained first reinforcement learning model, the policy function and value function of the trained first reinforcement learning model can be used as the initial policy function and initial value function of the second reinforcement learning model, and a reward function specifically for the gait control task can be designed for the second reinforcement learning model.

[0097] 204. Train the second reinforcement learning model to obtain a trained second reinforcement learning model.

[0098] Specifically, for gait control tasks, we first define the environment and state representation: The robot's current motion state is described using parameters such as the angle, angular velocity, and torque of each joint, as well as its body posture (e.g., tilt and center of gravity). The entire motion scene (e.g., terrain conditions and the presence of external interference) serves as environmental information. Next, we define actions: actions can include adjusting the torque applied to each joint or changing joint angles, which affect the robot's gait. Finally, we design a reward mechanism: positive rewards are given if the robot's gait remains stable and conforms to the expected motion pattern (e.g., even stride length and steady cadence when walking); negative rewards are given if the gait is unstable, such as when the robot loses balance or falls. Using reinforcement learning algorithms, the robot can learn the optimal action strategy for achieving a stable gait in different environments (e.g., flat ground, rough terrain, etc.) through continuous experimentation. This optimization, based on the dynamic model, optimizes gait control by continuously sampling (experimenting with the effects of different actions on gait).

[0099] Exemplarily, the training process of the second reinforcement learning model corresponding to the gait control task training may include the following steps:

[0100] Step 2041, define state representation: suppose the robot has joints, use X=(X1,X2,…,X n ) represents the joint angle, represents the joint angular velocity, F=(F1, F2,…, F j) represents the external force acting on the robot body (such as gravity, friction, etc.), n and j are both positive integers greater than 1, Indicates status.

[0101] Step 2042: Define action space: Action represents the torque adjustment applied to n joints, is the torque applied to the i-th joint.

[0102] Step 2043: Calculate the reward value based on the reward function: It is a function that measures the gait stability. For example, it can be defined based on the stride deviation, body shaking, etc. when the robot walks.

[0103] award , if the robot does not fall within a time step and its gait is more stable (the value of G is smaller), it will get a higher reward.

[0104] Step 2044, define the dynamic model (state transition): According to the dynamic equation of the robot ,in It is a function based on the physical structure and mechanical principles of the robot, describing the change of state over time. For example, for a simple single-joint robot, the following expression can be used:

[0105] (2)

[0106] in, is the moment of inertia, It's quality, is the acceleration due to gravity, is the connecting rod length, is the damping coefficient, represents the joint angular velocity, for The first derivative of , which represents the joint angular acceleration.

[0107] Step 2045, policy update (policy gradient method): the policy gradient algorithm can be used to update the policy , the gradient update formula is:

[0108] (3)

[0109] in, are the parameters of the policy network, It is a strategic network. is the advantage function, , is the action-value function, is the state-value function.

[0110] Repeat steps 2043 to 2045 until the stopping condition is met (for example, the iteration number threshold is reached). While the robot moves along the planned path, the gait control strategy is continuously adjusted so that the robot can walk stably.

[0111] 205. Construct a third reinforcement learning model corresponding to the balance control task training based on the trained second reinforcement learning model; the third reinforcement learning model includes a third reward function. The third reward function is related to whether the user falls when subjected to an external force and the speed at which the user recovers balance when subjected to the external force.

[0112] Specifically, after obtaining the trained second reinforcement learning model, the policy function and value function of the trained second reinforcement learning model can be used as the initial policy function and initial value function of the third reinforcement learning model, and a reward function specifically for the balance control task can be designed for the third reinforcement learning model.

[0113] 206. Train the third reinforcement learning model to obtain a trained third reinforcement learning model.

[0114] Specifically, for balance control task training, the environment and state are first defined: factors such as the robot's posture (e.g., body tilt angle, angular velocity), foot contact state (e.g., pressure distribution), and external disturbance forces (if any) are combined as state information to describe the robot's current balance. The surrounding terrain conditions and the presence of unexpected external forces constitute the environment. Next, actions are defined: actions can involve fine-tuning different parts of the robot's body (e.g., leg joints, waist), adjusting its center of gravity, changing its support force, and so on, to maintain balance. Finally, rewards can be set: positive rewards are given when the robot maintains balance despite external disturbances (e.g., remaining stable after a slight push); negative rewards are given if it loses balance and falls. Using a reinforcement learning algorithm, the robot is trained through trial and error in this environment (trying different balance adjustment maneuvers) to learn the optimal action strategy for maintaining balance under various circumstances. This transition is achieved from continuous trial and error learning in a simulated environment to effective balance control in real-world scenarios (i.e., using a sim-to-real approach, first training a balance control strategy in a simulated environment and then applying it to real-world scenarios).

[0115] Exemplarily, the training process of the third reinforcement learning model corresponding to the balance control task may include the following steps:

[0116] Step 2061, define state representation: suppose the tilt angle of the robot body is , the angular velocity is , the foot pressure sensor measured value is Assume the robot has Foot pressure sensor), status is .

[0117] Step 2062: Define action space: action Indicates the robot body Fine-tune the key parts of the body, such as adjusting the leg joint angle, changing the waist torque, etc., to adjust the center of gravity and support force.

[0118] Step 2063: Calculate the reward value based on the reward function: is a function that measures the state of balance. For example, it can be defined based on whether the tilt angle is within the stable range, whether the pressure distribution on the sole of the foot is uniform, etc. is the tilt angle of the robot body, is the angular velocity of the robot, and P is the measurement value of the robot's foot pressure sensor.

[0119] Rewards , if the robot can recover its balance faster after being disturbed by external disturbance (i.e. The smaller the value of ), the higher the reward.

[0120] Step 2064, calculate the state transition probability (in the presence of external interference): Assume that the external interference is a random variable, the state transition probability Depends on the robot's physical structure, control actions and external disturbance d. For example, according to the rigid body kinematics and dynamics equations of the robot, it can be established , where α t is the tilt angle of the robot body at the current moment, α t+1 is the predicted value of the robot's tilt angle at the next moment, ω t is the angular velocity of the robot at the current moment, is the time interval, It is a function obtained according to physical principles.

[0121] Step 2065: Update the policy: The Asynchronous Advantage Actor-Critic (A3C) algorithm can be used to update the policy and value functions in multiple parallel threads. For the policy update part, the policy gradient can be updated according to the advantage function, and the value function can be updated to better estimate the state value. The specific formula involves the policy network and value network Gradient update of , such as the policy network gradient update formula: (in, is the parameter of the policy network), r is the reward, is the discount factor, V() is the state-value function, S t is the current state, S t+1 The state at the next moment.

[0122] Repeat steps 2063 to 2065 until the stopping condition is met (for example, the iteration number threshold is reached). As the robot moves along the planned path, it continuously monitors external interference and its own state and adjusts its actions according to the balance control strategy to cope with various terrain changes and external interference and maintain balance.

[0123] 207. Determine the motion control model according to the trained third reinforcement learning model.

[0124] The robot motion control method provided in this embodiment combines gait control training or balance control training with path planning training based on a reinforcement learning algorithm, and obtains a motion control model through training, so that the robot can improve stability while being able to plan paths efficiently and reasonably.

[0125] In some embodiments, as Figure 3 As shown, based on the above embodiment, for example, Figure 2 Based on the illustrated embodiment, step 207 may include:

[0126] 2071. Construct a fourth reinforcement learning model based on the trained third reinforcement learning model; a fourth reward function corresponding to the fourth reinforcement learning model is determined based on the first reward function, the second reward function, and the third reward function.

[0127] Specifically, after obtaining the trained third reinforcement learning model, the policy function and value function of the trained third reinforcement learning model can be used as the initial policy function and initial value function of the fourth reinforcement learning model, and the fourth reward function corresponding to the fourth reinforcement learning model can be determined based on the first reward function, the second reward function and the third reward function.

[0128] 2072. Train the fourth reinforcement learning model to obtain the motion control model.

[0129] In some embodiments, the training of the fourth reinforcement learning model to obtain the motion control model may include: modeling the environment in which the robot is located to obtain an environment model; the environment model includes a starting position and a target position; placing the robot at the starting position to obtain an initial environment state; for each time step, determining the current action to be performed based on the current state, and after performing the current action, obtaining the next state and reward value; the reward value is determined according to the first reward function; storing the current state, the current action, the next state and the reward value in an experience replay buffer; randomly extracting a preset number of training data from the experience replay buffer; and performing gradient descent training on the first reinforcement learning model based on the training data to obtain the motion control model.

[0130] Specifically, after obtaining the trained third reinforcement learning model, considering the mutual influence among path planning, gait control and balance control, in order to further balance the three, a comprehensive reward function, namely the fourth reward function, can be designed to take into account the reward factors of path planning, gait control and balance control.

[0131] For example, the design of the fourth reward function can be defined as . Among them, R{path} is the reward related to path planning. For example, every time the robot approaches the target position by one meter, R{path} increases by 10 points. R{gait} is the reward related to gait control. A normal and stable gait is rewarded with 2 points per step, and points are deducted if there is a gait abnormality. R{balance} is the balance control reward. The robot is rewarded with 3 points per second for maintaining an upright balance during movement, and points are deducted if the tilt angle exceeds a certain threshold. w1, w2, and w3 are weights, which are assigned according to the focus of the task, for example, w1=0.4, w2=0.3, and w3=0.3.

[0132] During reinforcement learning training, the robot takes actions based on the state of the environment. After each action, the reward value is calculated based on this reward function. The agent will learn how to obtain high path planning rewards while also maintaining good gait and balance. Specifically, the following steps may be involved:

[0133] Environment construction: Construct a simulation environment for the humanoid robot, including terrain such as flat ground, slopes, obstacles, etc., starting position and target position.

[0134] Agent Setup: Define a humanoid robot as an agent. Its action space includes discrete or continuous actions such as forward and backward, left and right turns, stride length adjustment, and cadence change. The state space includes the robot's position (x, y coordinates), velocity, attitude angles (pitch, yaw, and roll), sensor-sensed environmental information (such as distance to obstacles), and current gait parameters (stride length, cadence, etc.).

[0135] Reinforcement Learning Algorithm Selection: The fourth reinforcement learning model is determined based on the third reinforcement learning model. Therefore, the fourth reinforcement learning model can use the same algorithms as the third reinforcement learning model, such as Deep Q-Network (DQN) or Proximal Policy Optimization (PPO). Taking DQN as an example, the parameters of the Q-network, including the weights and biases of the neural network, are initialized based on the trained third reinforcement learning model.

[0136] The training cycle process can specifically include the following steps:

[0137] For each training round:

[0138] Reset the environment: Place the robot at the starting position and initialize the environment state s0.

[0139] For each time step t:

[0140] According to the current status , the agent selects an action via the Q network (in DQN) or the policy network (in PPO) For example, actions are selected based on the size of the Q value, or actions are sampled based on the action probability distribution output by the policy network.

[0141] Execute an action After that, the robot generates a new state in the environment , and receive a reward , by the fourth reward function For example, if the robot moves a certain distance toward the goal while maintaining balance and good gait, it will receive a positive reward; if the robot falls or deviates from the planned path, it will receive a negative reward.

[0142] The data Stored in the experience replay buffer. In DQN, the experience replay buffer is used to store a certain amount of historical experience for subsequent random sampling for network training.

[0143] Randomly sample a batch of data from the experience replay buffer , for the DQN algorithm, calculate the target Q value: ,in is a discount factor that weighs the importance of current rewards and future rewards, are the parameters of the target Q network (DQN uses the target network to stabilize the training process). Then according to the loss function Perform gradient descent training on the Q network and update the network parameters θ1. For policy gradient-based algorithms such as PPO, the gradient is calculated according to the policy gradient theorem and the parameters of the policy network are updated to maximize the expectation of the cumulative reward.

[0144] A training round ends when the robot reaches the target position or exceeds the maximum time step.

[0145] To improve model performance, training evaluation and optimization can be performed. Specifically, the agent's performance can be evaluated every certain number of training rounds. For example, metrics such as the average number of steps the robot takes to reach the target, the average reward, and the number of falls can be calculated over multiple test training rounds. Based on the evaluation results, training parameters such as the learning rate, discount factor, and the weights w1, w2, and w3 of the reward function can be adjusted. If the robot is found to be overly focused on path planning and neglecting balance, resulting in frequent falls, the weight of the balance reward, w3, can be appropriately increased.

[0146] The above training cycle is repeated until the performance of the agent reaches a satisfactory level, that is, the robot can efficiently and stably reach the target position from the starting position in various complex environments while maintaining good gait and balance control.

[0147] To clearly illustrate the training process, the following example illustrates the training process of the Q network:

[0148] First, collect interaction data. Let the robot act in the environment according to the current policy, which selects actions based on the output of the Q network. For example, the robot is in the initial state Next, according to the Q network's estimation of the Q value of different actions, the action with the highest Q value is selected. Execution, such as moving forward a certain distance in a straight line.

[0149] Record the status of each interaction ,action ,award and the next state , and store these experience data in the experience playback buffer.

[0150] Sampling from experience replay: Randomly sample a batch of experience data from the experience replay buffer, denoted as , where N is the number of samples.

[0151] Secondly, the target Q value is calculated. For each sampled empirical data, the target Q value is calculated using the target Q network: The target value is used to guide the training of the Q network. is a discount factor that weighs the importance of current rewards and future rewards, is the weight parameter of the target Q network, S i+1 is the next state, a' is the next state S i+1 The best action among all possible actions. If the current state is a terminal state, then The target Q network has the same structure as the trained Q network and is used to provide a relatively stable target Q value estimate. The trained Q network is used to collect empirical data.

[0152] Again, calculate the loss function and update the Q network. Use the mean square error loss function to calculate the error between the predicted Q value of the current Q network and the target Q value: .in, is the target Q value, is the predicted Q value.

[0153] Calculate the gradient of the loss function with respect to the Q network parameters through the back propagation algorithm , and use an optimization algorithm (such as Adam) to update the parameters , the update formula is: in is the learning rate. During the update process, the weight parameters in the target Q network Remain unchanged.

[0154] Then, the strategy is updated. According to the updated Q network, Greedy strategy or other strategy improvement methods update the strategy. For example, in In the greedy strategy, with probability 1- Select the action with the largest Q value, with probability Randomly select an action. As training progresses, gradually reduce The value of makes the robot choose the optimal action based more on the estimation of the Q network. It should be noted that The greedy strategy is a simple and commonly used method to balance between exploring new actions and using known optimal actions. is a probability value between 0 and 1, indicating the degree of random exploration.

[0155] The specific process of strategy update may include: Action selection: At each time step, when the robot needs to select an action, it first generates a random number M uniformly distributed in the interval [0,1]. Then the random number M is compared with Compare: If M is less than , indicating that during exploration, the robot will randomly select an action from the action space to perform. For example, if the robot's action space contains 5 actions (move forward, turn left, turn right, adjust stride, adjust balance torque), then it will randomly select one of the 5 actions with equal probability, regardless of the current Q network's Q value evaluation of these actions. If M is greater than or equal to , which means that the robot will choose the action with the largest Q value output by the Q network in the current state to execute. That is, by inputting the current state s into the updated Q network In the example, we get the corresponding different actions The Q value of each action is then selected and the action with the largest Q value is selected as the action to be executed.

[0156] In some embodiments, the Value: In the early stages of training, a relatively large Values, such as = 0.5 or even higher to ensure that the robot has a greater probability of exploring different actions in the environment, trying various possible path planning methods, gait control adjustments, and balance control operations, so as to collect more data from different situations and help the Q network better learn the value of different state-action pairs. As training continues, that is, after multiple rounds of interaction with the environment and updates to the Q network, it will gradually decrease. The value of, for example, the current iteration round The previous iteration round The decay factor is greater than 0 and less than 1. For example, a certain decay rule can be used, such as after a certain number of training rounds, Multiply it by a decay factor less than 1, such as 0.99, to make it gradually smaller. When it becomes very small, such as close to 0.05 or smaller, the robot will rely more on the optimal action estimated by the Q network to make decisions, that is, it will make more use of it. Because at this time the Q network has undergone a large amount of data learning and parameter updates, its estimation of the action value in each state is relatively more accurate, so the robot is more likely to choose the optimal action to achieve path planning, maintain a good gait and balance control.

[0157] Finally, continuous training and optimization can be achieved. By repeating the above steps, the robot continuously interacts with the environment, collects empirical data, and updates the Q network and policy. After multiple iterations, the robot's Q network will gradually learn the value of each action in different states, and the policy will be continuously optimized, resulting in increasingly better performance in path planning, gait control, and balance control.

[0158] Figure 4This is a schematic diagram of the structure of the robot motion control device provided in the embodiment of the present application. Figure 4 As shown, the robot motion control device 40 includes: an acquisition module 401 and an input module 402 .

[0159] The acquisition module 401 is used to acquire the current state of the robot; the current state includes the current position and target position of the robot.

[0160] An input module 402 is used to input the current state into a motion control model to obtain the current action that the robot needs to perform; the motion control model is obtained by performing multi-task training on a reinforcement learning model; the multi-task training includes path planning task training, and at least one of gait control task training and balance control task training; the gait control task training and the balance control task training are performed on the basis of the path planning task training.

[0161] The control module 403 is used to control the robot to perform the current action. If the robot reaches the target position after completing the current action, the current motion task is completed.

[0162] The robot motion control device provided in the embodiment of the present application combines gait control training or balance control training with path planning training based on a reinforcement learning algorithm, and obtains a motion control model through training, so that the robot can improve stability on the basis of being able to plan paths efficiently and reasonably.

[0163] The robot motion control device provided in the embodiment of the present application can be used to execute the above-mentioned method embodiment. Its implementation principle and technical effects are similar, and this embodiment will not be repeated here.

[0164] In some embodiments, the current state further includes: grid attributes of the grid to which the current position belongs, joint angles, joint angular velocities, stride length, step frequency, external force, body tilt angle, body angular velocity, and sole pressure value.

[0165] In some embodiments, the device 40 also includes a training module (not shown), which is used to: construct a first reinforcement learning model corresponding to the path planning task training based on the initial policy function and the initial value function; train the first reinforcement learning model to obtain a trained first reinforcement learning model; construct a second reinforcement learning model corresponding to the gait control task based on the trained first reinforcement learning model; train the second reinforcement learning model to obtain a trained second reinforcement learning model; and determine the motion control model based on the trained second reinforcement learning model.

[0166] In some embodiments, the training module is configured to update the value function based on the following expression:

[0167] ,

[0168] Where Q is the value function, is the current state, The state of the next moment, is the action at the current moment, is the learning rate, is the discount factor, Is the action that executes the current moment After receiving the reward, Indicates the action at the next moment.

[0169] In some embodiments, the training module is configured to perform policy updates based on the following expression:

[0170]

[0171] in, are the parameters of the policy network, It is a strategic network. is the advantage function, , is the action-value function, is the state-value function.

[0172] In some embodiments, the first reinforcement learning model includes a first reward function; the first reward function is related to whether a collision occurs with an obstacle and whether the distance to the end point is shortened; the second reinforcement learning model includes a second reward function; the second reward function is related to the uniformity of walking stride and the uniformity of step frequency.

[0173] In some embodiments, the training module is specifically used to: construct a third reinforcement learning model corresponding to the balance control task training based on the trained second reinforcement learning model; the third reinforcement learning model includes a third reward function; the third reward function is related to whether to fall when subjected to external force, and the speed of restoring balance when subjected to external force; train the third reinforcement learning model to obtain the trained third reinforcement learning model; determine the motion control model based on the trained third reinforcement learning model.

[0174] In some embodiments, the training module is specifically used to: construct a fourth reinforcement learning model based on the trained third reinforcement learning model; the fourth reward function corresponding to the fourth reinforcement learning model is determined based on the first reward function, the second reward function and the third reward function; and train the fourth reinforcement learning model to obtain the motion control model.

[0175] In some embodiments, the training module is specifically used to: model the environment in which the robot is located to obtain an environment model; the environment model includes a starting position and a target position; the robot is placed at the starting position to obtain an initial environment state; for each time step, according to the current state, the current action to be performed is determined, and after executing the current action, the next state and reward value are obtained; the reward value is determined according to the first reward function; the current state, the current action, the next state and the reward value are stored in an experience replay buffer; a preset number of training data are randomly extracted from the experience replay buffer; based on the training data, the first reinforcement learning model is gradient descent trained to obtain the motion control model.

[0176] In some embodiments, the training module is specifically configured to: generate a random number M for each time step, where M is greater than or equal to 0 and M is less than or equal to 1; if M is less than , then randomly select an action from the action space to execute; if M is greater than or equal to , then select the action corresponding to the maximum Q value among the Q values ​​output by the Q network in the current state as the target action, and execute the target action; is a probability value between 0 and 1, indicating the degree of random exploration; the current iteration round The previous iteration round The attenuation factor is greater than 0 and less than 1.

[0177] Figure 5 Schematic diagram of the hardware structure of the robot motion control device provided in an embodiment of the present application.

[0178] Device 50 may include one or more of the following components: a processing component 501 , a memory 502 , a power component 503 , a multimedia component 504 , an audio component 505 , an input / output (I / O) interface 506 , a sensor component 507 , and a communication component 508 .

[0179] The processing component 501 generally controls the overall operation of the device 50, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. The processing component 501 may include one or more processors 509 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 501 may include one or more modules to facilitate interaction between the processing component 501 and other components. For example, the processing component 501 may include a multimedia module to facilitate interaction between the multimedia component 504 and the processing component 501.

[0180] The memory 502 is configured to store various types of data to support operations on the device 50. Examples of such data include instructions for any application or method operating on the device 50, contact data, phone book data, messages, pictures, videos, etc. The memory 502 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0181] Power supply component 503 provides power to various components of device 50. Power supply component 503 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 50.

[0182] The multimedia component 504 includes a screen that provides an output interface between the device 50 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, it may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensors can not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide action. In some embodiments, the multimedia component 504 includes a front-facing camera and / or a rear-facing camera. When the device 50 is in an operating mode, such as a capture mode or a video mode, the front-facing camera and / or the rear-facing camera can receive external multimedia data. Each front-facing camera and the rear-facing camera can have a fixed optical lens system or have focal length and optical zoom capabilities.

[0183] The audio component 505 is configured to output and / or input audio signals. For example, the audio component 505 includes a microphone (MIC) that is configured to receive external audio signals when the device 50 is in an operating mode, such as a call mode, a recording mode, or a voice recognition mode. The received audio signals may be further stored in the memory 502 or transmitted via the communication component 508. In some embodiments, the audio component 505 also includes a speaker for outputting audio signals.

[0184] The I / O interface 506 provides an interface between the processing component 501 and peripheral interface modules, such as a keyboard, a click wheel, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0185] The sensor assembly 507 includes one or more sensors for providing various aspects of the status assessment of the device 50. For example, the sensor assembly 507 can detect the open / closed state of the device 50, the relative positioning of components, such as the display and keypad of the device 50. The sensor assembly 507 can also detect changes in the position of the device 50 or a component of the device 50, the presence or absence of user contact with the device 50, the orientation or acceleration / deceleration of the device 50, and temperature changes of the device 50. The sensor assembly 507 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 507 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 507 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0186] The communication component 508 is configured to facilitate wired or wireless communication between the device 50 and other devices. The device 50 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 508 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 508 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0187] In an exemplary embodiment, the device 50 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-described methods.

[0188] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 502 including instructions, which can be executed by the processor 509 of the device 50 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0189] The computer-readable storage medium mentioned above can be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The computer-readable storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0190] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0191] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0192] An embodiment of the present application further provides a computer program product, including a computer program. When the computer program is executed by a processor, the robot motion control method performed by the robot motion control device as described above is implemented.

[0193] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A robot motion control method, characterized in that: include: Get the current state of the robot; The current state includes the current position and target position of the robot; The current state is input into a motion control model to obtain a current action that the robot needs to perform; the motion control model is obtained by performing multi-task training on a reinforcement learning model; the multi-task training includes path planning task training, and at least one of gait control task training and balance control task training; the gait control task training and the balance control task training are performed on the basis of the path planning task training; The robot is controlled to execute the current action. If the robot reaches the target position after completing the current action, the current motion task is completed.

2. The method according to claim 1, characterized in that The current state also includes: grid attributes of the grid to which the current position belongs, joint angles, joint angular velocities, stride length, step frequency, external force, body inclination angle, body angular velocity, and sole pressure value.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: A first reinforcement learning model corresponding to the path planning task training is constructed based on the initial strategy function and the initial value function; the first reinforcement learning model includes a first reward function; the first reward function is related to whether a collision with an obstacle occurs and whether the distance to the end point is shortened; Training the first reinforcement learning model to obtain a trained first reinforcement learning model; Constructing a second reinforcement learning model corresponding to the gait control task based on the trained first reinforcement learning model; the second reinforcement learning model includes a second reward function; the second reward function is related to the uniformity of walking stride and the uniformity of step frequency; Training the second reinforcement learning model to obtain a trained second reinforcement learning model; The motion control model is determined according to the trained second reinforcement learning model.

4. The method according to claim 3, characterized in that The training of the first reinforcement learning model includes: The value function is updated based on the following expression: ， Where Q is the value function, is the current state, For the state of the next moment, is the action at the current moment, is the learning rate, is the discount factor, Is the action that executes the current moment After the reward, Indicates the action at the next moment.

5. The method according to claim 3, characterized in that: The training of the second reinforcement learning model comprises: The policy update is based on the following expression: in, are the parameters of the policy network, is a strategic network, is the advantage function, , is the action-value function, is the state-value function.

6. The method according to claim 3, characterized in that Determining the motion control model according to the trained second reinforcement learning model includes: Constructing a third reinforcement learning model corresponding to the balance control task training based on the trained second reinforcement learning model; the third reinforcement learning model includes a third reward function; the third reward function is related to whether to fall down when subjected to external force, and the speed of restoring balance when subjected to external force; Training the third reinforcement learning model to obtain a trained third reinforcement learning model; The motion control model is determined according to the trained third reinforcement learning model.

7. The method according to claim 6, characterized in that Determining the motion control model according to the trained third reinforcement learning model includes: Constructing a fourth reinforcement learning model based on the trained third reinforcement learning model; a fourth reward function corresponding to the fourth reinforcement learning model is determined according to the first reward function, the second reward function and the third reward function; The fourth reinforcement learning model is trained to obtain the motion control model.

8. The method according to claim 7, characterized in that The step of training the fourth reinforcement learning model to obtain the motion control model includes: Modeling the environment in which the robot is located to obtain an environment model; the environment model includes a starting position and a target position; Placing the robot at the starting position to obtain an initial environment state; For each time step, according to the current state, a current action to be performed is determined, and after performing the current action, a next state and a reward value are obtained; the reward value is determined according to the first reward function; Storing the current state, the current action, the next state, and the reward value in an experience replay buffer; Randomly extracting a preset amount of training data from the experience replay buffer; According to the training data, gradient descent training is performed on the first reinforcement learning model to obtain the motion control model.

9. The method according to claim 8, characterized in that The fourth reinforcement learning model is a deep Q network; and the training of the fourth reinforcement learning model includes: For each time step, generate a random number M, M is greater than or equal to 0 and M is less than or equal to 1; If M is less than , then randomly select an action from the action space to execute; If M is greater than or equal to , then select the action corresponding to the maximum Q value among the Q values ​​output by the Q network in the current state as the target action, and execute the target action; is a probability value between 0 and 1, indicating the degree of random exploration; The previous iteration round The attenuation factor is greater than 0 and less than 1.

10. A robot motion control device, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the robot motion control method according to any one of claims 1 to 9.

Images