Quadruped robot motion control method combining expert learning and reinforcement learning

Abstract

The invention discloses a quadruped robot motion control method combining expert learning and reinforcement learning, which is characterized by comprising the following steps: in a simulation environment, setting a plurality of simulation quadruped robots, and controlling the simulation quadruped robots to carry out reinforcement learning algorithm iterative training by a first intelligent agent, acquiring a strategy parameter of the first intelligent agent as a first optimal parameter; controlling the test quadruped robot to move, and acquiring test parameters of the test quadruped robot; and the second intelligent body controls the simulation quadruped robot to carry out reinforcement learning algorithm iterative training, the obtained strategy parameter of the second intelligent body is a second optimal parameter, and the second optimal parameter is used for guiding the entity quadruped robot motion controller. The mode of combining reinforcement learning and expert learning is adopted, self-iteration training of the whole process is achieved, some prejudices possibly generated by human beings are removed, and therefore the training iteration process is accelerated, and the fairness and reliability of control decision making of the quadruped robot are improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of quadruped robot motion control and relates to a quadruped robot motion control method combining expert learning and reinforcement learning. BACKGROUND

[0002] In recent years, quadruped robots have been increasingly widely applied due to their strong terrain adaptability. There are mainly three kinds of current quadruped robot motion control methods: traditional control method, reinforcement learning algorithm method and imitation learning method. The traditional control method often needs to accurately model kinematics and dynamics of a robot model, designs a reasonable motion trajectory through a trajectory optimization algorithm, and controls through a carefully designed control strategy and a hand-adjusted parameter. The reinforcement learning algorithm shows great potential in solving complex robot control problems, and a robot autonomously learns a motion control strategy through random interaction with an environment and continuously strengthens correct motion control logic through a reward function. The imitation learning method learns skills contained in demonstration through demonstration data, thereby reducing the need to design a reward function.

[0003] However, the traditional control method is difficult for a robot to exhibit strong robustness in a complex environment due to inevitable approximation and linearization in the modeling process. The reinforcement learning algorithm usually needs complex reward weight adjustment and reward function design. The imitation learning mainly depends on demonstration data, which may cause the robot to lack generalization ability and be difficult to make appropriate responses when facing unknown situations.

[0004] Based on the above background, to solve the problem of quadruped robot motion control, a new control method needs to be explored, which combines the advantages of existing motion control methods to finally form a motion control strategy with strong generalization ability and high stability. SUMMARY

[0005] To achieve the above purpose, the application adopts the following technical solutions:

[0006] In a first aspect, a quadruped robot motion control method combining expert learning and reinforcement learning comprises:

[0007] A first agent is created, and a reward function of the first agent is set;

[0008] In a simulation environment, a plurality of simulation quadruped robots are set, the first agent controls the simulation quadruped robots to perform reinforcement learning algorithm iterative training, acquires strategy parameters of the first agent, and when a round error amount before and after iteration converges to a set threshold, the iteration is stopped, and the strategy parameters of the first agent are acquired as first optimal parameters;

[0009] establish a test terrain and a test quadruped robot, import the first optimal parameter into the test environment, the first agent generates a motion trajectory according to the test terrain, controls the test quadruped robot to move, and obtains test parameters of the test quadruped robot;

[0010] create a second agent, import the test parameters into the second agent, and output the weight and hyper function of the reward function of the first agent; the second agent controls the simulation quadruped robot to perform iterative training of the reinforcement learning algorithm, obtains the strategy parameters of the second agent, and when the error amount before and after iteration converges to a set threshold, stops iteration, and the strategy parameters of the second agent are the second optimal parameters, which are used to guide the motion controller of the entity quadruped robot.

[0011] Further, the second optimal parameters are set as the reward function of the first agent, and the following steps are repeated:

[0012] In the simulation environment, a plurality of simulation quadruped robots are set, the first agent controls the simulation quadruped robots to perform iterative training of the reinforcement learning algorithm, obtains the strategy parameters of the first agent, and when the error amount before and after iteration converges to a set threshold, stops iteration, and the strategy parameters of the first agent are the first optimal parameters;

[0013] establish a test terrain and a test quadruped robot, import the first optimal parameter into the test environment, the first agent generates a motion trajectory according to the test terrain, controls the test quadruped robot to move, and obtains test parameters of the test quadruped robot;

[0014] create a second agent, import the test parameters into the second agent, and output the weight and hyper function of the reward function of the first agent; the second agent controls the simulation quadruped robot to perform iterative training of the reinforcement learning algorithm, obtains the strategy parameters of the second agent, and when the error amount before and after iteration converges to a set threshold, stops iteration, and the strategy parameters of the second agent are the second optimal parameters, which are used to guide the motion controller of the entity quadruped robot.

[0015] Further, the reward function of the first agent includes the sum of squares of the body posture deviation of the simulation quadruped robot and the weighted sum of the body speed and the distance of crossing obstacles.

[0016] Further, in the simulation environment, a plurality of simulation quadruped robots are set, the first agent controls the simulation quadruped robots to perform iterative training of the reinforcement learning algorithm, obtains the strategy parameters of the first agent, and when the error amount before and after iteration converges to a set threshold, stops iteration, and the strategy parameters of the first agent are the first optimal parameters, including:

[0017] In the simulation environment, a physical model of the simulation quadruped robot is established;

[0018] establishing an external environment physical model in the simulation environment, the types of the physical model including stairs, concave-convex road surface and steps;

[0019] establishing a state observer in the simulation environment, the state observer being used to acquire relevant parameters of the simulation quadruped robot in the external environment physical model, the relevant parameters including joint angle values, joint torque values, joint position values and body IMU data of the quadruped robot;

[0020] based on the relevant parameters, acquiring, from the first agent, expected torque values of each joint of the simulation quadruped robot, based on the expected torque values, the first agent controlling the quadruped robot to perform iterative training of the reinforcement learning algorithm, and acquiring policy parameters of the first agent;

[0021] when the error amount before and after iteration converges to a set threshold, stopping iteration, and acquiring the policy parameters of the first agent as first optimal parameters.

[0022] Further, the acquiring, from the first agent, expected torque values of each joint of the simulation quadruped robot based on the relevant parameters, and the acquiring, by the first agent, policy parameters of the first agent based on the expected torque values and the iterative training of the quadruped robot by the first agent through the reinforcement learning algorithm, include:

[0023] the first agent calculating and outputting expected torque instructions of each joint according to a reward function and the relevant parameters obtained from the state observer;

[0024] after the simulation quadruped robot receives the torque instructions, calculating, by a physics engine, a dynamic response thereof and performing corresponding motion;

[0025] the state observer collecting new relevant parameters;

[0026] based on the new relevant parameters, the agent acquiring policy parameters of the first agent through the reinforcement learning algorithm;

[0027] the policy parameters of the first agent in the nth round being brought into a reward function in the (n+1)th round, and being iterated in a loop in the training process until the policy parameters converge.

[0028] Further, the establishing of the test terrain and the test quadruped robot, the importing of the first optimal parameters into the test environment, the generation of a motion trajectory by the first agent according to the test terrain, the control of the motion of the test quadruped robot, and the acquisition of test parameters of the test quadruped robot, include:

[0029] A simulation quadruped robot test environment is established in a simulation environment, and a test terrain and a test quadruped robot are established in the test environment, the test terrain including but not limited to: flat road surface, stairs, concave-convex road surface, steps, low-attached road surface;

[0030] The first optimal parameter is introduced into the test environment, the first agent generates a motion trajectory according to the test terrain, controls the motion of the test quadruped robot, and obtains test parameters of the test quadruped robot, the test parameters being used to indicate a body motion state of the quadruped robot, and the test parameters including positions, velocities, torques of respective joints, three-direction accelerations and three-direction angular velocities of the body, and a time of crossing an obstacle.

[0031] Further, the first agent controls the simulation quadruped robot to perform iterative training of the reinforcement learning algorithm or the second agent controls the simulation quadruped robot to perform iterative training of the reinforcement learning algorithm, including: respectively performing motion simulation of a plurality of simulation quadruped robots under different initial positions and different load conditions.

[0032] Further, the number of times of iterative training of the reinforcement learning algorithm of the first agent controlling the simulation quadruped robot is not less than 1000, and the number of times of iterative training of the reinforcement learning algorithm of the second agent controlling the simulation quadruped robot is not less than 500.

[0033] In a second aspect, a training device applied to a quadruped robot includes,

[0034] A first obtaining module is configured to create a first agent and set a reward function of the first agent.

[0035] A first test module is configured to, in a simulation environment, set a plurality of simulation quadruped robots, control the simulation quadruped robots to perform iterative training of a reinforcement learning algorithm by the first agent, obtain a policy parameter of the first agent, stop the iteration when a round error amount before and after iteration converges to a set threshold, and obtain the policy parameter of the first agent as a first optimal parameter.

[0036] A second test module is configured to establish a test terrain and a test quadruped robot, introduce the first optimal parameter into a test environment, generate a motion trajectory according to the test terrain by the first agent, control the motion of the test quadruped robot, and obtain test parameters of the test quadruped robot.

[0037] The output module is used to create a second agent, import test parameters into the second agent, output weights and hyperfunctions of a reward function of the first agent, control the simulation quadruped robot to perform iterative training of the reinforcement learning algorithm, obtain policy parameters of the second agent, stop iteration when a round error amount before and after iteration converges to a set threshold, and obtain the second optimal parameters as the policy parameters of the second agent, which are used to guide the entity quadruped robot motion controller.

[0038] The updating module is used to set the second optimal parameters as the reward function of the first agent and perform multiple rounds of training.

[0039] In a third aspect, a computer device comprises:

[0040] A memory and a processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the quadruped robot motion control method combining expert learning and reinforcement learning.

[0041] The present application has the following advantages:

[0042] A quadruped robot motion control method combining expert learning and reinforcement learning, which uses an expert learning method to make up for the shortcomings of reinforcement learning. The method combines reinforcement learning and expert learning, trains a first agent using a reinforcement learning method to obtain first optimal parameters, establishes a second agent for expert learning, feeds the first optimal parameters of the first agent back to a test quadruped robot in a test environment, and outputs test parameters from the test quadruped robot. The test parameters and a reward function are input into the second agent to obtain second optimal parameters, which are used to guide an entity quadruped robot motion controller. Expert learning solves the process of human inspection and adjustment after reinforcement learning, realizes self-iterative training of the whole process, removes some biases that may be generated by humans, thereby speeding up the training iteration process and improving the fairness and reliability of quadruped robot control decisions. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 FIG. 1 is a flowchart of a quadruped robot motion control method combining expert learning and reinforcement learning;

[0044] Figure 2 FIG. 2 is a logic diagram of a reinforcement learning method in the prior art;

[0045] Figure 3 FIG. 3 is a logic diagram of a quadruped robot motion control method combining expert learning and reinforcement learning;

[0046] Figure 4is another flowchart of a four-legged robot motion control method combining expert learning and reinforcement learning;

[0047] Figure 5 is a flowchart of acquiring the first optimal parameter in the embodiment of the application;

[0048] Figure 6 is a flowchart of acquiring the strategy parameter of the first agent in the embodiment of the application;

[0049] Figure 7 is a logic diagram of acquiring the strategy parameter of the first agent in the embodiment of the application;

[0050] Figure 8 is a flowchart of acquiring the test parameter in the embodiment of the application. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. It should be understood that the present application is not limited by the example embodiments disclosed herein. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0052] In the description of the present application, the terms "first", "second" are only used for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise specifically limited.

[0053] The technical terms appearing in the present embodiment are explained as follows:

[0054] An agent is a software or entity that can perceive its environment through sensors or data inputs and autonomously take actions through actuators or outputs based on these perceptions to achieve specific goals or complete specific tasks.

[0055] A simulated quadruped robot is a virtual model existing in a computer simulation environment. It simulates the morphology of a physical robot, such as geometric structure, joints, physical properties such as mass, friction, simulated output of sensors such as cameras, lidar, force sensors, etc., and simulated behavior of actuators such as motors, etc.

[0056] The agent receives data from the "sensors" of the simulated robot, according to its decision logic, sends control commands to the "actuators" of the simulated robot, so that the simulated robot moves, grabs, interacts, etc. in the simulated environment.

[0057] The reward function is a parameter in the reinforcement learning algorithm that needs to balance the task goal, motion stability and physical constraints. The reward function is the "evaluation standard" for the agent to learn, which provides quantitative feedback for the agent's behavior, positive reward indicating "correct behavior" and negative reward indicating "incorrect behavior". Such feedback directly determines the adjustment direction and amplitude of the agent's parameters.

[0058] Embodiment 1

[0059] As shown in Figure 1 A four-legged robot motion control method combining expert learning and reinforcement learning, comprising:

[0060] S1: Create a first agent and set the reward function of the first agent;

[0061] S2: In the simulation environment, set a plurality of simulated four-legged robots, the first agent controls the simulated four-legged robots to perform iterative training of the reinforcement learning algorithm, obtains the policy parameters of the first agent, and when the iteration error quantity converges to a set threshold before and after iteration, stops iteration, and obtains the policy parameters of the first agent as the first best parameters;

[0062] S3: Establish a test terrain and a test four-legged robot, import the first best parameters into the test environment, the first agent generates a motion trajectory according to the test terrain, controls the test four-legged robot to move, and obtains test parameters of the test four-legged robot;

[0063] S4: Create a second agent, import the test parameters into the second agent, and output the weight and super function of the reward function of the first agent, the second agent controls the simulated four-legged robot to perform iterative training of the reinforcement learning algorithm, obtains the policy parameters of the second agent, and when the iteration error quantity converges to a set threshold before and after iteration, stops iteration, and obtains the policy parameters of the second agent as the second best parameters, which are used to guide the motion controller of the real four-legged robot.

[0064] The traditional control method needs to model the controlled object. In order to ensure the understandability of the model, approximation and linearization processing is inevitable in the modeling process. Even so, the traditional control method still has a large number of parameters, which need to be adjusted according to different environments, and it is difficult to adapt to the responsible environment. If the number of modeling parameters is increased, too many parameters will lead to a too slow optimization process.

[0065] Reinforcement learning can well solve this problem. As Figure 2As shown, it can be regarded as a control model containing a large number of parameters, and the parameters are adjusted through training iterations and finally converge to an optimal solution. Since the training process can simulate various complex environments through a large number of simulations, the optimal solution obtained has strong environmental adaptability.

[0066] However, after the reinforcement learning training is completed, because the training environment is concurrent in multiple machines and multiple environments, there may be a situation of converging to a local optimal solution or failing to converge at all. At this time, it is necessary to rely on human algorithm engineers to redeploy the trained network parameters to the test environment, that is, the simulation or real machine for verification, and adjust the training reward function of the reinforcement learning according to the test results. The test needs to be carried out in the test environment, and the reward function weight is set and modified according to the test results, so as to obtain more ideal results.

[0067] Therefore, as Figure 3 As shown, the four-legged robot motion control method combining expert learning and reinforcement learning of the embodiment uses the expert learning method to make up for the shortcomings of reinforcement learning. In steps S1 and S2, the first agent is trained using the reinforcement learning method to obtain the first optimal parameter. In steps S3-S4, the second agent is established for expert learning, the first optimal parameter of the first agent is fed back to the test four-legged robot in the test environment, the test four-legged robot outputs the test parameter, and the test parameter and the reward function are input into the second agent to obtain the second optimal parameter. The second optimal parameter is used to guide the entity four-legged robot motion controller. Expert learning solves the process of human inspection and adjustment after reinforcement learning in the past, realizes self-iterative training of the whole process, removes some biases that may be produced by humans, thereby speeding up the training iteration process and improving the fairness and reliability of the four-legged robot control decision.

[0068] In practice, the test parameter includes the stability of the machine body, and the root mean square (rms) value of the roll and pitch angles of the machine body attitude angle is used to judge. If the rms value of the pitch angle exceeds the expectation, the second agent needs to adjust the weight of the reward function to increase the weight of the pitch angle stability in the reward function, train a network with more stable machine body, and output the relationship of the reward function at this moment as the second optimal parameter. The second optimal parameter is used to adjust the stability of the entity four-legged robot machine body.

[0069] Embodiment 2

[0070] As shown in the figure, a four-legged robot motion control method combining expert learning and reinforcement learning includes: Figure 4

[0071] S1: Create a first agent and set a reward function for the first agent; ​

[0072] S2: In the simulation environment, a plurality of simulation quadruped robots are set, a first agent controls the simulation quadruped robots to perform iterative training of the reinforcement learning algorithm, and the policy parameters of the first agent are obtained. When the iteration error amount converges to a set threshold before and after iteration, the iteration is stopped, and the policy parameters of the first agent are obtained as the first optimal parameters;

[0073] S3: A test terrain and a test quadruped robot are established, the first optimal parameters are imported into the test environment, the first agent generates a motion trajectory according to the test terrain, controls the motion of the test quadruped robot, and obtains test parameters of the test quadruped robot;

[0074] S4: A second agent is created, the test parameters are imported into the second agent, and the output is set as the weight and super function of the reward function of the first agent. The second agent controls the simulation quadruped robot to perform iterative training of the reinforcement learning algorithm, and the policy parameters of the second agent are obtained. When the iteration error amount converges to a set threshold before and after iteration, the iteration is stopped, and the policy parameters of the second agent are obtained as the second optimal parameters, which are used to guide the motion controller of the entity quadruped robot.

[0075] S5: The second optimal parameters are set as the reward function of the first agent, and steps S1-S5 are repeated.

[0076] Specifically, first, in step S2, the first agent is independently trained multiple times, and each iteration includes complete forward propagation, loss calculation, back propagation and parameter update. Step S2 makes the first agent reach a relatively stable state. This helps to establish a solid foundation and avoid unstable interference of the global part in subsequent complex training.

[0077] Secondly, on the basis of the training results in step S2, the test parameters of the simulation robot are obtained from the test environment. In step S4, the test parameters are input to the second agent, and the second agent is independently trained multiple times. The second agent relies on the foundation established by the first agent and adds new test results, and can adjust independently and discard the subjective interference that human thinking may bring, so as to execute the task in a more precise and objective manner. When training the second agent, other new components, data or tasks can also be introduced.

[0078] Finally, the whole system steps S1-S4 are iteratively trained as a whole, and the model learns to correct itself, adjust its state or make more accurate predictions using feedback information. The whole model is fine-tuned and optimized, and each subpart achieves an optimal cooperative working state to maximize the overall performance. The overall iterative model can learn more fine-grained and global information-based feature representations and decision boundaries.

[0079] In Embodiment 1 and Embodiment 2, the reward function of the first intelligent agent includes the square sum of the body posture of the simulated quadruped robot deviating from stability, and the weighted sum of the body speed and the distance across the obstacle, and the reward function expression is:

[0080] R total =-k1∑ i (θ i -θ pi ) 2 +k2*v2+k3*do bstacle ;

[0081] Wherein, the body posture deviates from stability, and the degree of the robot trunk pitch angle (θ1), roll angle (θ2) and the like deviating from the target posture (θ p ) is punished; the body speed encourages the robot to reach the target speed in the forward direction, for example, (x-axis) corresponds to the target speed (v x ); the distance across the obstacle distance, rewards the robot for successfully crossing the vertical height or horizontal distance d obstacle of the obstacle.

[0082] Under the premise of ensuring not to fall (body posture deviating from stability reward), the faster (body speed reward) the more complex the terrain (distance across the obstacle reward) is crossed, which is the ultimate goal of the motion control of the quadruped robot.

[0083] Specifically, the body posture deviates from stability, which is used to optimize the balance of the quadruped robot, and the pitch angle affects the front and rear balance, and the roll angle affects the lateral stability. The quadratic penalty function is more sensitive to large angle deviation, such as 30° deviation, which is 36 times of 5° deviation, forcing the robot to preferentially avoid the risk of overturning. If k1 is too large, it will lead to an over-cautious "creeping" movement, and it is necessary to reduce k1 and increase the speed reward k2.

[0084] Specifically, the body speed is used to improve the motion efficiency, and when k2 is too large, the robot may exhibit a "sprint-fall-rise" oscillation behavior, and a fall penalty term can be added to adjust the k2 / k1 ratio.

[0085] Specifically, the distance across the obstacle is used to enhance the environmental adaptability, and if k3 is too large, it will lead to sacrificing stability for crossing obstacles, such as high leg lifting, and if k3 is too small, it will appear the behavior of bypassing the obstacle instead of crossing it, therefore, it is necessary to increase k3 in steps and slowly debug.

[0086] In specific embodiments, for the quadruped robot, the reward function can also add the following:

[0087] Basic motion reward, including forward speed reward, which is achieved by collecting the speed close to the target; direction alignment reward, which is achieved by checking the consistency of the body orientation and the motion direction.

[0088] Stability reward, including body posture reward, achieved by suppressing pitch and roll angle;

[0089] Body height reward, achieved by maintaining target height; Center of mass acceleration smooth line reward, achieved by detecting the number of body shaking.

[0090] Gait symmetry reward, achieved by detecting the phase angle of the diagonal leg in motion.

[0091] Energy efficiency reward, achieved by detecting joint power consumption.

[0092] Obstacle and terrain adaptability reward, including obstacle crossing reward, achieved by detecting the height of the foot end lifting to surpass the obstacle; Foot bottom stability reward, achieved by detecting the horizontal speed of the foot end.

[0093] Safety constraint reward, including joint limit reward, achieved by detecting joint angle; Anti-falling reward, achieved by detecting whether the knee contacts the ground.

[0094] When setting the reward function, the weights of different reward functions are designed, and the basic stability reward is prioritized, and then other task targets are gradually added.

[0095] In embodiments 1 and 2, as shown in Figure 5 In the simulation environment, a plurality of simulation quadruped robots are set, a first agent controls the simulation quadruped robot to perform iterative training of the reinforcement learning algorithm, obtains the policy parameter of the first agent, when the iteration error amount converges to a set threshold before and after iteration, the iteration is stopped, and the policy parameter of the first agent is obtained as the first best parameter, including:

[0096] S210: A physical model of a simulation quadruped robot is established in a simulation environment;

[0097] S220: An external environment physical model is established in the simulation environment, and the types of the physical model include stairs, concave-convex road surface and steps;

[0098] S230: A state observer is established in the simulation environment, the state observer is used to obtain related parameters of the simulation quadruped robot in the external environment physical model, and the related parameters include joint angle value, joint torque value, joint position value and body IMU data of the quadruped robot;

[0099] S240: Based on the related parameters, the expected torque value of each joint of the simulation quadruped robot is obtained from the first agent, and based on the expected torque value, the first agent controls the simulation quadruped robot to perform iterative training of the reinforcement learning algorithm, and obtains the policy parameter of the first agent;

[0100] S250: when the iteration of the front and rear wheel error amount converges to a set threshold, stop iteration, and obtain the policy parameter of the first agent as the first optimal parameter.

[0101] In embodiments 1 and 2, as shown in Figure 6 and 7 Based on the relevant parameters, the expected torque value of each joint of the simulated quadruped robot is obtained from the first agent; based on the expected torque value, the first agent controls the quadruped robot to perform iterative training of the reinforcement learning algorithm, and obtains the policy parameter of the first agent, including:

[0102] S241: the first agent calculates and outputs the expected torque command of each joint according to the reward function and the relevant parameters obtained from the state observer;

[0103] S242: after receiving the torque command, the simulated quadruped robot calculates its dynamic response through the physics engine and executes the corresponding motion;

[0104] S243: the state observer tries to collect new relevant parameters;

[0105] S244: based on the new relevant parameters, the agent obtains the policy parameter of the first agent through the reinforcement learning algorithm;

[0106] S245: the policy parameter of the first agent in the nth round is brought into the reward function in the n+1 round, and the training process is iterated repeatedly until the policy parameter converges.

[0107] Specifically, the reinforcement learning first agent receives real-time data provided by the state observer based on the current policy parameter, including joint angle value, joint torque value, joint position value and body IMU data, generates and outputs the expected torque command of each joint motor. These torque values constitute the action space output of the first agent, which directly determines the motion intention of the robot. The simulated quadruped robot motion receives the expected torque command, and after solving, the simulated quadruped robot makes corresponding motion. In this process, the simulated quadruped robot converts the torque into actual joint motion, and the physical model of the robot body and the external environment interacts physically, and makes motion response on stairs, concave-convex road and steps. In the process of motion, the state observer synchronously captures new generation of state data, generates relevant parameters, and the relevant parameters affect the next round of decision. The updated policy parameter will output improved torque command in the next iteration, gradually approaching the optimal policy parameter.

[0108] In embodiments 1 and 2, as shown in Figure 8 A test terrain and a test quadruped robot are established, the first optimal parameter is imported into the test environment, the first agent generates a motion trajectory according to the test terrain, controls the test quadruped robot to move, and obtains test parameters of the test quadruped robot; including:

[0109] S310: A simulation quadruped robot test environment is established in a simulation environment, a test terrain and a test quadruped robot are established in the test environment, and the test terrain includes but is not limited to: flat road, stairs, concave-convex road, steps, low-attached road;

[0110] S320: The first optimal parameter is introduced into the test environment, the first intelligent agent generates a motion trajectory according to the test terrain, controls the motion of the test quadruped robot, and obtains a test parameter of the test quadruped robot, the test parameter is used to indicate the body motion state of the quadruped robot, and the test parameter includes: the position, speed, torque of each joint, three-axis acceleration and three-axis angular velocity of the body, and the time of crossing obstacles.

[0111] Specifically, in step S3, the simulated physical quadruped robot is moved, the first optimal parameter is input, the first optimal parameter controls the motion of the simulated quadruped robot, and the motion parameter of the simulated quadruped robot is collected as the test parameter. The subsequent step S4 is equivalent to "brain" calculation, analyzing the test parameter, further optimizing the first optimal parameter, and obtaining the second optimal parameter.

[0112] In embodiment 1 and embodiment 2, the first intelligent agent controls the simulated quadruped robot to perform iterative training of the reinforcement learning algorithm or the second intelligent agent controls the simulated quadruped robot to perform iterative training of the reinforcement learning algorithm, which includes: respectively simulating the motion of a plurality of simulated quadruped robots in different initial positions and under different load conditions.

[0113] Specifically, in the simulation environment, a plurality of simulated robots are set, and by simulating the motion of quadruped robots in different initial positions, the adaptability of the quadruped robot to different starting environments can be evaluated. For example, it is found that when the quadruped robot starts on a slope, it needs to adjust the strength and motion trajectory of the leg joints to maintain body balance; and when starting in a narrow space, the quadruped robot needs to adjust the gait more flexibly to avoid collision with surrounding obstacles. Different initial positions will affect the path planning of the quadruped robot. The simulation results show that the quadruped robot can automatically plan the optimal motion path according to the initial position and the target position. In complex terrain, the quadruped robot will choose a relatively flat area to walk to improve the stability and efficiency of the motion. Simulating the cooperative motion of multiple quadruped robots, different initial positions will make the cooperative strategy between the quadruped robots more complex.

[0114] By simulating the motion of the quadruped robot under different loads, the motion stability of the quadruped robot can be evaluated. The simulation results show that as the load increases, the center of gravity of the quadruped robot will change, and the support force and motion rhythm of the leg joints need to be adjusted to prevent falling. For example, when the load is large, the quadruped robot will adopt a lower gait to increase the contact time of the foot with the ground and improve stability. The load has a significant impact on the motion speed and energy consumption of the quadruped robot. Generally speaking, the greater the load, the slower the motion speed of the quadruped robot, and the higher the energy consumption. By simulating the motion under different load conditions, the structural design of the quadruped robot can be optimized, and appropriate drivers and energy systems can be selected to improve the motion speed and reduce energy consumption under the premise of meeting the load requirements. Under different load conditions, the force on each joint of the quadruped robot will change significantly. Simulation analysis can help us understand the force distribution of the joints and evaluate the strength and durability of the joints. For example, it is found that the torque of the leg joints will increase significantly when the load increases.

[0115] Through the relationship between multiple variables, the interference of accidental factors is excluded. The motion data of the simulated quadruped robot has higher accuracy and reliability, and the output of the first best parameter or the second best parameter can be closer to the actual scene, providing strong theoretical support for controlling the physical quadruped robot.

[0116] In embodiment 1 and embodiment 2, the number of iterations of the reinforcement learning algorithm of the first agent controlling the simulated quadruped robot is not less than 1000 times, and the number of iterations of the reinforcement learning algorithm of the second agent controlling the simulated quadruped robot is not less than 500 times.

[0117] Specifically, the reinforcement training of the first agent is the basis for the subsequent steps, and requires a large number of iterations to achieve the purpose of conditional convergence.

[0118] Embodiment 3

[0119] A training device applied to a quadruped robot, comprising,

[0120] The first acquisition module is configured to create a first agent and set a reward function of the first agent.

[0121] The first test module is configured to set a plurality of simulated quadruped robots in a simulation environment, control the simulated quadruped robots by the first agent to perform reinforcement learning algorithm iteration training, obtain a strategy parameter of the first agent, and stop iteration when an iteration error amount converges to a set threshold, and obtain the strategy parameter of the first agent as a first best parameter.

[0122] The first test module comprises:

[0123] The first test unit: a physical model of the simulated quadruped robot is established in the simulation environment.

[0124] The second test unit: an external environment physical model is established in the simulation environment, and the types of the physical model include stairs, concave-convex road surfaces and steps;

[0125] The third test unit: a state observer is established in the simulation environment, and the state observer is used to obtain relevant parameters of the simulation quadruped robot in the external environment physical model, and the relevant parameters include joint angle values, joint torque values, joint position values and body IMU data of the quadruped robot;

[0126] The fourth test unit: based on the relevant parameters, the expected torque values of each joint of the simulation quadruped robot are obtained from the first agent, and based on the expected torque values, the first agent controls the quadruped robot to perform iterative training of the reinforcement learning algorithm to obtain the policy parameters of the first agent;

[0127] The fifth test unit: when the iteration error amount converges to a set threshold before and after iterations, the iteration is stopped, and the policy parameters of the first agent are obtained as the first optimal parameters.

[0128] The fourth test unit includes:

[0129] The first layer: the first agent calculates and outputs the expected torque instructions of each joint according to the reward function and the relevant parameters obtained from the state observer;

[0130] The second layer: after the simulation quadruped robot receives the torque instructions, the dynamics response thereof is calculated through a physical engine, and corresponding movements are performed;

[0131] The third layer: the state observer collects new relevant parameters;

[0132] The fourth layer: based on the new relevant parameters, the agent obtains the policy parameters of the first agent through the reinforcement learning algorithm;

[0133] The fifth layer: the policy parameters of the first agent in the nth round are brought into the reward function in the n+1th round, and the training process is iterated repeatedly until the policy parameters converge.

[0134] The second test module is used to establish a test terrain and a test quadruped robot, import the first optimal parameters into the test environment, generate a movement trajectory according to the test terrain, control the test quadruped robot to move, and obtain test parameters of the test quadruped robot.

[0135] The second test module includes:

[0136] The sixth test unit: a simulation quadruped robot test environment is established in the simulation environment, a test terrain and a test quadruped robot are established in the test environment, and the test terrain includes but is not limited to flat road surfaces, stairs, concave-convex road surfaces, steps and low-attached road surfaces.

[0137] The seventh test unit: the first optimal parameter is introduced into the test environment, the first agent generates a motion trajectory according to the test terrain, controls the motion of the test quadruped robot, and obtains the test parameters of the test quadruped robot, the test parameters are used to indicate the body motion state of the quadruped robot, and the test parameters include the position, speed, torque of each joint, three-axis acceleration and three-axis angular velocity of the body, and the time of crossing the obstacle.

[0138] The output module is used to create a second agent, introduce the test parameters into the second agent, output the weight and super function of the reward function of the first agent, control the simulation quadruped robot to perform iterative training of the reinforcement learning algorithm, obtain the policy parameters of the second agent, stop iteration when the iteration error amount converges to a set threshold, and obtain the policy parameters of the second agent as the second optimal parameters, and the second optimal parameters are used to guide the motion controller of the entity quadruped robot.

[0139] The update module is used to set the second optimal parameters as the reward function of the first agent and perform multiple rounds of training.

[0140] Since the further function description of the module of the training device applied to the quadruped robot is the same as the above-mentioned corresponding embodiments, the above-mentioned technical effects are also possessed, and will not be repeated here.

[0141] The training device applied to the quadruped robot in the embodiment is presented in the form of a functional unit, and the unit here refers to an ASIC (Application Specific Integrated Circuit, Application Specific Integrated Circuit) circuit, a processor and a memory executing one or more software or fixed programs, and / or other devices that can provide the above-mentioned functions.

[0142] Embodiment 4

[0143] A computer device, comprising: a memory and a processor, the memory and the processor are connected with each other in communication, the memory stores computer instructions, and the processor executes the computer instructions to perform the expert learning and reinforcement learning combined quadruped robot motion control method of any one of the above.

[0144] Since the expert learning and reinforcement learning combined quadruped robot motion control method has the above-mentioned technical effects, the expert learning and reinforcement learning combined quadruped robot motion control method should also have corresponding technical effects, which will not be repeated here.

[0145] In the embodiments, all or some of the steps can be implemented by using software, hardware, firmware or any combination thereof. When implemented by using software, all or some of the steps can be implemented by using one or more computer programs. The computer programs can be stored in one or more computer readable storage media. When the computer programs are loaded into the computer, the computer can execute the computer programs. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatuses. The computer programs can be stored in one or more computer readable storage media, or transmitted from one computer readable storage medium to another computer readable storage medium, for example, from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center through a wired (for example, a coaxial cable, an optical fiber, a digital subscriber line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.) manner. The computer readable storage medium can be any available medium or a set of medium including one or more of the available medium that is accessible by the computer. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a Solid State Disk (SSD)), etc.

[0146] It should also be noted that the terms "first", "second" and the like in this text can only be used to distinguish one entity or action from another, and do not necessarily require or imply any actual such relationship or order between the entities or actions. Moreover, the terms "comprise", "comprising", or any other variant thereof are intended to cover non-exclusive inclusions, such that processes, methods, articles, or apparatuses that comprise a list of elements do not only include those elements, but can also include other elements not expressly listed, or also include elements inherent in such processes, methods, articles, or apparatuses. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0147] The above description of disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A quadruped robot motion control method combining expert learning and reinforcement learning, characterized in that: include: Create the first agent and set the reward function of the first agent; In a simulation environment, several simulated quadruped robots are set up. A first agent controls the simulated quadruped robots to perform iterative training with a reinforcement learning algorithm to obtain the strategy parameters of the first agent. When the error between the previous and next iterations converges to a set threshold, the iteration is stopped and the strategy parameters of the first agent are obtained as the first optimal parameters. Establishing a test terrain and a test quadruped robot, importing the first optimal parameters into the test environment, generating a motion trajectory according to the test terrain by the first intelligent agent, controlling the motion of the test quadruped robot, and obtaining test parameters of the test quadruped robot; Create a second intelligent agent, import the test parameters into the second intelligent agent, set the output as the weight and hyperfunction of the reward function of the first intelligent agent, and use the second intelligent agent to control the simulated quadruped robot to perform iterative training of the reinforcement learning algorithm to obtain the strategy parameters of the second intelligent agent. When the error amount of the rounds before and after the iteration converges to the set threshold, stop the iteration and obtain the strategy parameters of the second intelligent agent as the second optimal parameters. The second optimal parameters are used to guide the motion controller of the physical quadruped robot.

2. The quadruped robot motion control method combining expert learning and reinforcement learning according to claim 1 is characterized in that: Also includes: Set the second optimal parameter as the reward function of the first agent and repeat the following steps: In a simulation environment, several simulated quadruped robots are set up. A first agent controls the simulated quadruped robots to perform iterative training with a reinforcement learning algorithm to obtain the strategy parameters of the first agent. When the error between the previous and next iterations converges to a set threshold, the iteration is stopped and the strategy parameters of the first agent are obtained as the first optimal parameters. Establishing a test terrain and a test quadruped robot, importing the first optimal parameters into the test environment, generating a motion trajectory according to the test terrain by the first intelligent agent, controlling the motion of the test quadruped robot, and obtaining test parameters of the test quadruped robot; Create a second intelligent agent, import the test parameters into the second intelligent agent, set the output as the weight and hyperfunction of the reward function of the first intelligent agent, and use the second intelligent agent to control the simulated quadruped robot to perform iterative training of the reinforcement learning algorithm to obtain the strategy parameters of the second intelligent agent. When the error amount of the rounds before and after the iteration converges to the set threshold, stop the iteration and obtain the strategy parameters of the second intelligent agent as the second optimal parameters. The second optimal parameters are used to guide the motion controller of the physical quadruped robot.

3. The quadruped robot motion control method combining expert learning and reinforcement learning according to claim 2 is characterized in that: The reward function of the first intelligent agent includes the sum of squares of the deviation of the simulated quadruped robot's body posture from stability, and the weighted sum of the body speed and the obstacle crossing distance.

4. The quadruped robot motion control method combining expert learning and reinforcement learning according to claim 2 is characterized in that: In a simulation environment, several simulated quadruped robots are set up. A first agent controls the simulated quadruped robots to perform iterative training of a reinforcement learning algorithm to obtain the strategy parameters of the first agent. When the error between the previous and next iterations converges to a set threshold, the iteration is stopped and the strategy parameters of the first agent are obtained as the first optimal parameters, including: Establish a physical model of a simulated quadruped robot in a simulation environment; Establishing an external environment physical model in the simulation environment, wherein the types of the physical model include stairs, uneven roads and steps; Establishing a state observer in the simulation environment, wherein the state observer is used to obtain relevant parameters of the physical model of the simulated quadruped robot in the external environment, wherein the relevant parameters include joint angle values, joint torque values, joint position values ​​and body IMU data of the quadruped robot; Based on the relevant parameters, obtaining expected torque values ​​for each joint of the simulated quadruped robot from the first agent, and based on the expected torque values, controlling the simulated quadruped robot with the first agent to perform iterative training using a reinforcement learning algorithm to obtain strategy parameters of the first agent; When the error between the previous and next rounds of iteration converges to the set threshold, the iteration is stopped and the strategy parameters of the first agent are obtained as the first optimal parameters.

5. The quadruped robot motion control method combining expert learning and reinforcement learning according to claim 4 is characterized in that: acquiring, from the first agent, expected torque values ​​of the joints of the simulated quadruped robot based on the relevant parameters; Based on the expected torque value, the first agent controls the quadruped robot to perform iterative training of a reinforcement learning algorithm to obtain strategy parameters of the first agent, including: The first agent calculates and outputs the expected torque command of each joint based on the reward function and the relevant parameters obtained from the state observer; After receiving the torque command, the simulated quadruped robot calculates its dynamic response through the physics engine and performs the corresponding movement; The state observer tries to collect new relevant parameters; Based on the new relevant parameters, the agent obtains the strategy parameters of the first agent through the reinforcement learning algorithm; The strategy parameters of the first agent in round n are introduced into the reward function of round n+1, and the training process is iterated continuously until the strategy parameters converge.

6. The quadruped robot motion control method combining expert learning and reinforcement learning according to claim 1 is characterized in that: The method of establishing a test terrain and a test quadruped robot, importing the first optimal parameters into the test environment, generating a motion trajectory according to the test terrain by the first intelligent agent, controlling the motion of the test quadruped robot, and obtaining the test parameters of the test quadruped robot comprises: Establishing a simulated quadruped robot test environment in a simulation environment, establishing a test terrain and testing the quadruped robot in the test environment, wherein the test terrain includes but is not limited to: a flat road surface, stairs, an uneven road surface, steps, and a low-adhesion road surface; The first optimal parameters are imported into the test environment, the first intelligent agent generates a motion trajectory according to the test terrain, controls the movement of the test quadruped robot, and obtains the test parameters of the test quadruped robot. The test parameters are used to indicate the motion state of the quadruped robot's body. The test parameters include: the position, speed, torque of each joint, the three-dimensional acceleration and three-dimensional angular velocity of the body, and the time to cross the obstacle.

7. The quadruped robot motion control method combining expert learning and reinforcement learning according to any one of claims 1 to 6, characterized in that: The first intelligent agent controls the simulated quadruped robot to perform iterative training of the reinforcement learning algorithm or the second intelligent agent controls the simulated quadruped robot to perform iterative training of the reinforcement learning algorithm, including: performing motion simulation on several simulated quadruped robots at different initial positions and different load conditions.

8. The quadruped robot motion control method combining expert learning and reinforcement learning according to any one of claim 7, characterized in that: The first intelligent agent controls the simulated quadruped robot to perform reinforcement learning algorithm iterative training for no less than 1,000 times, and the second intelligent agent controls the simulated quadruped robot to perform reinforcement learning algorithm iterative training for no less than 500 times.

9. A training device for a quadruped robot, characterized in that: include, A first acquisition module is used to create a first agent and set a reward function for the first agent; The first testing module is used to set up several simulated quadruped robots in a simulation environment, and the first intelligent agent controls the simulated quadruped robots to perform iterative training of the reinforcement learning algorithm to obtain the strategy parameters of the first intelligent agent. When the error amount of the rounds before and after the iteration converges to a set threshold, the iteration is stopped and the strategy parameters of the first intelligent agent are obtained as the first optimal parameters; The second testing module is used to establish a test terrain and a test quadruped robot, import the first optimal parameters into the test environment, generate a motion trajectory according to the test terrain by the first intelligent agent, control the motion of the test quadruped robot, and obtain the test parameters of the test quadruped robot; Output module: used to create a second agent, import test parameters into the second agent, and set the output as the weight and hyperfunction of the reward function of the first agent. The second agent controls the simulated quadruped robot to perform iterative training of the reinforcement learning algorithm to obtain the policy parameters of the second agent. When the error between the previous and next rounds converges to the set threshold, the iteration is stopped and the policy parameters of the second agent are obtained as the second optimal parameters. The second optimal parameters are used to guide the motion controller of the physical quadruped robot. Update module: used to set the second best parameter as the reward function of the first agent and perform multiple rounds of training.

10. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the quadruped robot motion control method combining expert learning and reinforcement learning as described in any one of claims 1 to 8 by executing the computer instructions.

Images