A biped robot motion control method based on imitation learning and reinforcement learning

Abstract

The application discloses a biped robot motion control method based on imitation learning and reinforcement learning, which comprises the following steps: inputting the state observation value of a biped robot in a simulation environment, the expected action of the last period and the command code into a neural network B, outputting and recording the expected action of the next period, splicing the expected action of the next period into a track, calculating the tolerance value of the spliced track and an extracted track, determining whether the comprehensive reward for quantitatively evaluating the output of the neural network B is gain or loss, and then using the comprehensive reward for optimization of the neural network B until convergence, and using the control method at this time to control the biped robot in an actual environment to complete a specified motion; wherein the neural network B inherits the weight parameters of a neural network A, the neural network A is trained by a heuristic track and a reference track for stable motion of the biped robot, and the heuristic track is obtained by heuristic processing of the reference track. The application can enable the biped robot to fully explore the motion within the range of reasonable motion.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of robots, and particularly relates to a biped robot motion control method based on imitative learning and reinforcement learning. BACKGROUND

[0002] The existing biped robot motion planning method relies on a simplified mathematical model to generate an artificial trajectory, and in the simplification process, a plurality of assumptions and constraints (for example, ground flatness, ground angle, ideal center of mass position and speed, ideal moment of inertia and mass, etc.) are preposed, which inevitably causes errors in actual robot motion, so that the planned trajectory cannot achieve the expected effect when executed. In addition, artificial planning and mathematical models cannot cover enough working conditions, which reduces the upper limit of the motion of the biped robot.

[0003] As one of the core tools in the field of machine learning, the deep neural network has strong nonlinear feature learning ability. The robot's own sensor system (including joint code disc, foot six-axis force sensor, IMU attitude sensor, etc.) can observe and characterize the real-time state of the robot. In a certain motion environment, the real-time state S t There is an optimal action A corresponding to it o-t In a continuous mathematical space, the neural network strategy can adjust the motion of the robot in real time and respond to changes in the state of the robot or disturbances in real time. Through evaluation of the performance, the neural network can be continuously iteratively optimized, so as to have the ability to continuously learn and improve, thereby breaking through the limitations of artificial planning and mathematical models.

[0004] The current method based on machine learning emphasizes free exploration learning or trajectory feature imitation. The former is prone to poor convergence of the neural network, resulting in undesirable motion other than the expected motion. The latter is prone to single motion of the neural network, sacrificing the exploratory nature of the machine learning method. SUMMARY

[0005] In view of the deficiencies in the prior art, the application provides a biped robot motion control method based on imitative learning and reinforcement learning, which fully explores the motion within the range of reasonable motion.

[0006] The application achieves the above technical purpose through the following technical means.

[0007] A biped robot motion control method based on imitative learning and reinforcement learning:

[0008] The joint trajectory of the biped robot is planned to obtain a plurality of joint space reference trajectories, then the joint space reference trajectories are processed heuristically to obtain heuristic trajectories, the tolerance value tolerance1 of the heuristic trajectory and the joint space reference trajectory is calculated and compared with the threshold value E1, if tolerance1≤E1, the biped robot in the simulation environment moves based on the heuristic trajectory, if tolerance1>E1, the robot in the simulation environment moves based on the joint space reference trajectory; the heuristic trajectory and the joint space reference trajectory capable of enabling the biped robot to complete movement without overturning form a coarse sample pair for training the neural network A;

[0009] The state observation value of the biped robot in the simulation environment is obtained, the vector of the expected action of the last period and the command code is merged, and the neural network B is input, and the expected action of the next period of the biped robot is output and recorded; the expected action of the next period is spliced into a trajectory, the tolerance value tolerance2 of the trajectory and the trajectory extracted from the coarse sample pair is calculated, and compared with the set threshold value E2, if tolerance2≤E2, the comprehensive reward is increased, if tolerance2>E2, the comprehensive reward R is impaired; then the comprehensive reward is used for neural network B optimization in this period, until the neural network B converges, and the control method at this time is used for the biped robot in the actual environment, to control the biped robot to complete the specified movement;

[0010] The neural network B inherits the weight parameters of the neural network A.

[0011] In the above technical solution, the joint space reference trajectory is processed heuristically, that is, the joint space reference trajectory value is taken as the mean value, and Gaussian distribution sampling with a standard deviation of 1 is performed:

[0012]

[0013] Wherein: N is Gaussian distribution, CAT represents joint space reference trajectory vector splicing operation, t represents the tth trajectory point in the nth trajectory, T represents the length of the nth trajectory, represents the coordinates corresponding to the tth trajectory point in the nth trajectory, represents the trajectory obtained by Gaussian distribution sampling.

[0014] In the above technical solution, the calculation formula of the tolerance value tolerance1 is:

[0015]

[0016] Wherein: represents the coordinates corresponding to the tth trajectory point in the trajectory obtained by Gaussian distribution sampling, Indicates the coordinates corresponding to the t-th trajectory point in the planned trajectory.

[0017] In the above technical solution, the state observation values ​​include the joint positions and trunk posture of the biped robot in the current state and the joint positions and trunk posture of the previous cycle of the current state.

[0018] In the above technical solution, the calculation formula of the tolerance value tolerance2 is:

[0019]

[0020] Among them, traj t record Represents the joint coordinate value corresponding to the t-th trajectory point in the continuously recorded expected joint trajectory, traj t raw Represents the joint coordinate value corresponding to the t-th trajectory point in the trajectory extracted from the coarse sample pair.

[0021] In the above technical solution, the comprehensive reward satisfies the following formula:

[0022] Comprehensive reward R = regional reward R area +Balance Rewards R balace .

[0023] In the above technical solution, the regional reward R area is one of the following:

[0024] R 期望 =-αK(d c ,2)

[0025] R 过渡 =0

[0026] R 临界 =βK(d b ,2)

[0027] Among them: α, β are constants, K represents the kernel function, d c is the distance between the center of mass projection and the center of the foot, d b R is the shortest distance between the center of mass projection and the foot boundary, 期望 represents the reward of the desired area, R 过渡 represents the reward in the transition region, R 临界 Represents the reward in the critical region.

[0028] In the above technical solution, the balance reward R balace Satisfy the following formula:

[0029]

[0030] wherein: η1, η2, η3, η4, η5 and η6 represent weights, K represents a kernel function, is the centroid linear velocity of the x-y plane desired by the user, v xy is the current actual centroid linear velocity of the x-y plane of the robot, v z represents the current actual z-axis linear velocity of the robot, represents the joint velocity, represents the joint acceleration, τ is the joint torque, orien is the roll, pitch and yaw array returned by the IMU, action t is the desired joint trajectory coordinate output by the neural network B at time t.

[0031] The beneficial effects of the present application are: the present application determines the trajectory of the biped robot motion in the simulation environment by calculating the tolerance value of the heuristic trajectory and the joint space reference trajectory and comparing it with the threshold value, and the heuristic trajectory and the joint space reference trajectory that can enable the biped robot to complete the motion without overturning form a coarse sample pair, and the neural network A is trained, and the weight parameters of the neural network A are inherited by the neural network B; the present application also vector-merges the state observation value of the biped robot in the simulation environment with the expected action and the command code of the last period, inputs the neural network B, outputs and records the expected action of the biped robot in the next period, splices the expected action in the next period into a trajectory, calculates the tolerance value of the trajectory extracted from the coarse sample pair, and compares it with the set threshold value to determine whether the comprehensive reward for quantitatively evaluating the output of the neural network B is gain or loss, and then used for optimization of the neural network B until the neural network B converges, and the control method at this time is used for the biped robot in the actual environment to control the biped robot to complete the specified motion. The method of the present application enables the biped robot motion strategy to have both basic stability and sufficient exploration; compared with the traditional artificial planning method, the motion strategy based on machine learning can understand and adapt to complex environments and tasks. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 is the biped robot motion control block diagram based on imitation learning and reinforcement learning according to the present application;

[0033] Fig. 2(a) is a single foot support period schematic diagram according to the present application;

[0034] Fig. 2(b) is a double foot support period schematic diagram according to the present application;

[0035] Figure 3 is the kernel function diagram according to the present application. DETAILED DESCRIPTION

[0036] The present application will be further described below in conjunction with the drawings and specific embodiments, but the scope of protection of the present application is not limited thereto.

[0037] As Figure 1 shown, the application is a biped robot motion control method based on imitation learning and reinforcement learning, specifically comprising the following steps:

[0038] Step (1), the joint trajectory of the biped robot is planned by using kinematics and dynamics (the planning process is prior art), obtaining several joint space reference trajectories, and the joint space reference trajectory is processed by formula (1) for heuristic, that is, the joint space reference trajectory value is the mean value, and the Gaussian distribution sampling with a standard deviation of 1 is performed:

[0039]

[0040] Where: N is the Gaussian distribution, CAT represents the joint space reference trajectory vector splicing operation, t represents the tth trajectory point in the nth trajectory, T represents the length of the nth trajectory, represents the coordinates corresponding to the tth trajectory point in the nth trajectory, represents the trajectory obtained by Gaussian distribution sampling;

[0041] The sampled trajectory deviates from the original trajectory appropriately and has stronger exploration than the original trajectory; all joint space reference trajectories are sampled multiple times and saved, which will generate multiple sampled trajectories around the original trajectory, denoted as "heuristic trajectory".

[0042] The tolerance state machine is a "new trajectory evaluator", which calculates the tolerance value of the new and old trajectories (i.e. heuristic trajectory and joint space reference trajectory) by formula (2):

[0043]

[0044] Where: represents the coordinates corresponding to the tth trajectory point in the trajectory obtained by Gaussian distribution sampling, represents the coordinates corresponding to the tth trajectory point in the planned trajectory;

[0045] The calculated tolerance value tolerance1 is compared with the set threshold value E1, if tolerance1≤E1, the robot in the physical simulation environment moves based on the sampled trajectory for exploration; if tolerance1>E1, the robot in the physical simulation environment moves based on the planned trajectory; wherein the threshold value E1 is determined by the specific type of the robot.

[0046] The sampled trajectory or the planned trajectory of the exploratory motion constitutes a task sample, and is verified by the robot in the physical simulation environment (the specific verification process is prior art). The trajectory that can be completed without causing the robot to overturn constitutes a coarse sample pair, which is used to train the neural network A. After training, the neural network A has the basic ability of biped motion, such as alternating support of the two legs, reciprocating movement of the center of gravity, and alternating swinging of the two hands, etc., which provides a basis for subsequent reinforcement learning and greatly simplifies the reward design scheme of reinforcement learning. Most importantly, the use of “heuristic processing” and “tolerance state machine” ensures that overfitting is caused by single data characteristics during the training process, and the basic skills of biped walking are learned.

[0047] In step (2), the state observation value of the moving biped robot in the physical simulation environment is obtained by the body sensor, and the state observation value includes the joint position and torso posture of the current state of the biped robot and the joint position and torso posture of the previous period of the current state. The motion of the biped robot in this process is irrelevant to the reference trajectory in the imitation learning; the state observation value is part of the input vector of the neural network B, and is combined with the expected action of the previous period and the command code to form a vector, and the command code includes the linear velocity vector and the angular velocity vector expected by the user; the neural network B outputs the expected action of the next period of the biped robot (i.e. the expected joint trajectory coordinates), which is recorded and transmitted to the biped robot in the physical simulation environment. The neural network B inherits the weight parameters of the neural network A.

[0048] The recorded expected joint trajectory coordinates are spliced, and when the length of the recorded expected joint trajectory reaches T for the first time, a trajectory is continuously extracted from the coarse sample pair, and the tolerance value of the extracted trajectory and the spliced trajectory is calculated: that is, the tolerance value tolerance2 is determined by using the tolerance state machine B represented by formula (3), wherein, traj t record represents the joint coordinate value corresponding to the tth trajectory point in the continuously recorded expected joint trajectory, traj t raw represents the joint coordinate value corresponding to the tth trajectory point in the extracted trajectory in the coarse sample pair (the length of the extracted trajectory is also T).

[0049]

[0050] The calculated tolerance value tolerance2 is compared with the set threshold value E2, if tolerance2≤E2, the comprehensive reward is increased, if tolerance2>E2, the comprehensive reward R is reduced; wherein the threshold value E2 is an empirical value, and the proportion of the increase or the reduction is an empirical value and is related to the type of the robot.

[0051] The above comprehensive rewards include regional rewards and balance rewards, specifically:

[0052] 1) Regional rewards

[0053] Since the stability of biped robots mainly comes from the support of flat feet, the position of the projection of the robot's center of mass in the direction of gravity relative to the foot area plays a decisive role in the robot's balance. Therefore, according to FIGS. 2(a), (b), the single-foot support period and the double-foot support period of the biped robot motion are modeled respectively, and according to the influence of different regions on stability, they are divided into expected regions, transition regions, and critical regions; the expected region refers to the projection of the center of mass in the safe region of the foot, which can not only maintain the stable support of the robot, but also provide a certain safety margin for additional interference or terrain changes; the significance of adding rewards to the expected region is that it can enable the robot to actively adjust the center of mass to project more in the expected region under different environments to ensure the smoothness of biped motion; the transition region refers to the projection of the center of mass in the transition region of the foot, which can also maintain the stable support of the robot and provide compensation actions when unbalanced, but the transition region cannot provide sufficient safety margin, and the risk of robot imbalance increases if external interference is received; the critical region refers to the fact that due to the large structural rigidity of the biped robot, if the projection of the center of mass is in the critical region, there is a high risk of imbalance and overturning, and unpredictable limb rotation may occur, and the critical region is a region that should be avoided as much as possible during motion.

[0054] Different regions correspond to different reward values, and the calculation of rewards corresponding to different regions is shown in formula (4), which is the core of the rewards in the entire biped motion reinforcement learning phase:

[0055]

[0056] wherein: a, β are constants, and are empirical values; d c is the distance between the projection position of the center of mass and the center of the foot, d b is the nearest distance between the projection position of the center of mass and the boundary of the foot, and bonus is the adjustable reward amplitude, when bonus = 2, the kernel function image is as shown in Figure 3 It is defined that in each motion period, only one of the above three exists in the support region, so the support reward R area also only takes one corresponding operation method in formula (4).

[0057] 2) Balance rewards

[0058] In the reinforcement learning process, the balance reward determines the balance of biped robot movement and evaluates the coordination of joint movement and IMU; the balance reward specifically includes the reward items shown in Table 1, the reward items are calculated through state observation values, and the specific calculation process is prior art;

[0059] Table 1

[0060]

[0061] wherein, is the user's desired x-y plane centroid linear velocity, v xy is the current actual x-y plane centroid linear velocity of the robot, q j is the joint position, tau is the joint torque, action t is the desired joint trajectory coordinate output by the neural network B at time t, orien is the roll pitch yaw array returned by the IMU (roll, pitch and yaw, reflecting the body posture), K represents the kernel function in formula (4), v z represents the current actual z-axis velocity of the robot; in this embodiment, the weights are taken as 1, 3, 0.002, 0.0007, 1.5 and 0.01 in turn.

[0062] By combining the above calculation methods, the comprehensive reward R = regional reward R area + balance reward R balace is obtained. The comprehensive reward represents the quantitative evaluation of the expected action output by the neural network B in the current period, and the comprehensive reward will be used for the neural network B optimization process in the period. The optimization process of the neural network B is achieved by calculating the policy gradient (prior art), and the purpose of calculating the policy gradient is to make the comprehensive reward rise. By analogy, after continuous iteration, when the comprehensive reward no longer rises, it is determined that the neural network B converges. The control method at this time is used for the biped robot in the actual environment, and the biped robot is controlled to complete the specified movement.

[0063] The embodiment is a preferred embodiment of the present application, but the present application is not limited to the above-mentioned embodiments, and any obvious improvement, replacement or modification made by those skilled in the art without departing from the essential content of the present application shall fall within the protection scope of the present application.

Claims

1. A method for motion control of a biped robot based on imitation learning and reinforcement learning, characterized in that: joint trajectory of the biped robot is planned to obtain a plurality of joint space reference trajectories, then the joint space reference trajectories are processed heuristically to obtain heuristic trajectories, tolerance1 of the heuristic trajectories and the joint space reference trajectories is calculated and compared with threshold E1, if tolerance1≤E1, the biped robot in the simulation environment moves based on the heuristic trajectories, if tolerance1>E1, the robot in the simulation environment moves based on the joint space reference trajectories; the heuristic trajectories and the joint space reference trajectories that can make the biped robot complete the motion without overturning form a coarse sample pair, which is used to train neural network A; state observation value of the biped robot in the simulation environment is obtained, and the expected action and command code of the last period are vector-merged to input neural network B, and the expected action of the next period of the biped robot is output and recorded; the expected action of the next period is spliced into a trajectory, tolerance2 of the trajectory and the extracted trajectory in the coarse sample pair is calculated, and is compared with the set threshold E2, if tolerance2≤E2, the comprehensive reward is increased, if tolerance2>E2, the comprehensive reward R is impaired; then the comprehensive reward is used for neural network B optimization in the period until neural network B converges, and the control method at this time is used for the biped robot in the actual environment to control the biped robot to complete the specified motion; the neural network B inherits the weight parameters of the neural network A. The joint space reference trajectories are processed heuristically, that is, the joint space reference trajectory values are sampled with a mean value and a standard deviation of 1. The formula for calculating tolerance1 is: The state observation value includes the joint position and torso posture of the current state of the biped robot and the joint position and torso posture of the last period of the current state.

2. The biped robot motion control method based on imitation learning and reinforcement learning according to claim 1, wherein, The formula for calculating tolerance2 is: wherein: N is a Gaussian distribution, CAT represents a joint space reference trajectory vector concatenation operation, t represents the tth trajectory point in the nth trajectory, T represents the length of the nth trajectory, represents the coordinates corresponding to the tth trajectory point in the nth trajectory, represents a trajectory obtained by Gaussian distribution sampling.

3. The biped robot motion control method based on imitation learning and reinforcement learning according to claim 2, characterized in that, The comprehensive reward satisfies the following formula: wherein: denotes the coordinate corresponding to the t-th trajectory point in the trajectory sampled by Gaussian distribution, denotes the coordinate corresponding to the t-th trajectory point in the planned trajectory.

4. The biped robot motion control method based on imitation learning and reinforcement learning according to claim 1, wherein, ​ 5. The biped robot motion control method based on imitation learning and reinforcement learning according to claim 2, wherein, ​ wherein, traj t record represents the joint coordinate value corresponding to the tth trajectory point in the desired joint trajectory, traj t raw represents the joint coordinate value corresponding to the tth trajectory point in the trajectory.

6. The biped robot motion control method based on imitation learning and reinforcement learning according to claim 1, wherein, ​ R = R + R area + R balace .

7. The biped robot motion control method based on imitation learning and reinforcement learning according to claim 6, characterized in that, The region reward R area is one of the following: R 期望 = -aK(d c ,2) R 过渡 ＝0 R 临界 = βK(d b ,2) where: a, b are constants, K represents a kernel function, d c is the distance of the center of mass projection position from the foot center, d b is the nearest distance of the center of mass projection position from the foot boundary, R 期望 represents the reward of the desired region, R 过渡 represents the reward of the transition region, R 临界 represents the reward of the critical region.

8. The biped robot motion control method based on imitation learning and reinforcement learning according to claim 6, characterized in that, The balance reward R balace satisfies the following equation: wherein: η1, η2, η3, η4, η5, and η6 represent weights, K represents a kernel function, is the centroid linear velocity of the x-y plane desired by the user, v xy is the current actual centroid linear velocity of the x-y plane of the robot, v z represents the current actual z-axis linear velocity of the robot, represents the joint velocity, represents the joint acceleration, τ is the joint torque, orien is the roll, pitch, and yaw array returned by the IMU, action t is the desired joint trajectory coordinate output by the neural network B at time t.

Images