A parameter control method for a virtual model controller of a quadruped robot based on reinforcement learning

Abstract

The present invention discloses a parameter control method of a virtual model controller of a quadruped robot based on reinforcement learning. The specific control method is as follows: step (1): establishing a mathematical model of the quadruped robot: calculating the positive kinematic model of the quadruped robot; step (2): gait generation: optimizing the cycloid trajectory to obtain the cycloid trajectory after constraint optimization; step (3): designing a virtual model controller: establishing a virtual model control of the quadruped robot; step (4): combining a deep reinforcement learning algorithm: selecting and designing a state space, an action space and a reward function; step (5): training simulation of a virtual prototype model; the present invention utilizes the powerful exploration ability of deep reinforcement learning to find the best controller parameters and reduces the difficulty of setting the controller parameters; the present invention improves the control accuracy of a traditional virtual model controller and has a certain anti-interference performance, and the controller can still accurately control the movement of the quadruped robot.

Description

Technical Field

[0001] The present invention relates to the technical field of robot control methods, and in particular to a parameter control method of a quadruped robot virtual model controller based on reinforcement learning. Background Art

[0002] With the development of robot bionic technology, the application field of multi-legged robots is becoming more and more extensive. Compared with bipedal robots, quadrupedal robots have better stability and ground passing performance. Compared with hexapod and above robots, they have better balance and trade-offs in the complexity of control algorithms, and have stronger adaptability than traditional wheeled and tracked mobile robots. In nature, almost all large mammals move by moving on four legs. According to the principles of bionics, the movement strategy of quadrupedal robots can be constructed. Virtual model control is a control method widely used in the field of quadrupedal robots. This method does not require accurate and complex dynamics and inverse kinematics modeling of quadrupedal robots. The robot can be made to move accordingly according to the designed gait trajectory planning, but the algorithm sacrifices control accuracy and has large errors. It is difficult to achieve accurate control of quadrupedal robots. Based on this, the present invention proposes an adaptive control method of a virtual model controller of a quadrupedal robot based on reinforcement learning based on the adaptive learning ability of deep reinforcement learning. Summary of the invention

[0003] The main purpose of the present invention is to provide a parameter control method of a quadruped robot virtual model controller based on reinforcement learning, aiming to solve the existing technical problems.

[0004] To achieve the above object, the present invention provides a parameter control method of a quadruped robot virtual model controller based on reinforcement learning, which is characterized in that the specific control method is as follows:

[0005] Step (1): Establish a mathematical model of a quadruped robot: Calculate the forward kinematic model of the quadruped robot. The quadruped robot has 12 joints in total. Each leg consists of three leg parts: upper, middle and lower. The upper joint is L1 meters long, and the middle and lower joints are L2 and L3 meters long respectively. Obtain the mapping between the rotation angle of the robot leg joint and the position relationship between the end of the leg and foot.

[0006] Step (2): gait generation: optimizing the cycloid trajectory to obtain the constrained optimized cycloid trajectory;

[0007] Step (3): Design of virtual model controller: Using the mathematical model obtained in step (1), establish the virtual model control of the quadruped robot and initialize the parameters of the virtual model controller;

[0008] Step (4): Integrate deep reinforcement learning algorithm: Select and design the state space, action space and reward function, and combine the virtual model controller designed in step (3) to design the reinforcement learning agent and select the algorithm;

[0009] Step (5): Training simulation of the virtual prototype model: Train the constructed virtual model to obtain the optimal control strategy.

[0010] Furthermore, the specific steps of step (1) are as follows:

[0011] ① Quadruped robot body structure design

[0012] Quadrupeds can move in six dimensions, including x, y, z, and movement around the three axes of x, y, and z. Therefore, according to its movement requirements, the quadruped robot has a total of 12 degrees of freedom. Each leg is composed of three leg-foot parts: upper, middle, and lower. The upper joint is L1 meter long, and the middle and lower joints are L2 and L3 meters long respectively. The full elbow movement mode is selected.

[0013] ②Robot forward kinematics modeling

[0014] The world coordinate system {W} is established with the ground origin, the body coordinate system {B} is established at the center of mass of the robot body, and the coordinate system is established at the origin of the hip joint of the robot leg, which are respectively left front {FL}, right front {RL}, left back {BL}, and right back {BR}. The hip joint, shoulder joint, knee joint and foot end are represented by {0} to {3} respectively. According to the parameters of the quadruped robot described in step (1), the DH method is used to establish the positive kinematic equation. The structural parameters of the four legs are consistent. The positive kinematic equation is as follows:

[0015]

[0016] The transformation matrix for rotation around the z-axis is:

[0017]

[0018] The translation transformation matrix along the y direction is:

[0019]

[0020] The translation transformation matrix along the z direction is:

[0021]

[0022] From equations (2), (3), and (4), the transformation matrix of the robot foot end coordinate system {3} relative to the hip joint coordinate system {0} is as follows:

[0023]

[0024] In formula (5), c ij =cos(θ i +θ j ), c i =cos(θ i ), s i = sin(θ i ).

[0025] From formula (5), the forward kinematic equation can be obtained as follows:

[0026]

[0027] Furthermore, the specific steps of step (2) are as follows:

[0028] During the movement of the quadruped robot, each leg has two states: swing phase and support phase. To solve the problem of sliding and dragging between the end of the robot's legs and the ground during walking, the position, velocity and acceleration in the horizontal x direction and the vertical y direction are constrained. The constraint equations are as follows:

[0029] Horizontal x-direction constraint:

[0030]

[0031] Constraints in the vertical y direction:

[0032]

[0033] In formulas (7) to (12), T is a gait cycle, S is the stride of a gait cycle, and H is the lifting height of the end of the leg. Through the above constraints, the expected trajectory of the robot is obtained as shown in formula (13):

[0034]

[0035] Furthermore, in step (3), the specific steps of extracting motion characteristic parameters are as follows: the virtual model controller is used as the basic controller

[0036] ④ Calculation of Jacobian matrix:

[0037] The virtual model controller requires the Jacobian matrix of the controlled object, which can be derived by the sitting position (x, y, z) of the robot leg relative to the joint variables. The joint variables are recorded as q1, q2, and q3, and their physical meanings are the rotation angles of the motors of the hip joint, shoulder joint, and knee joint. According to the forward kinematics equation (6) of the robot leg, its Jacobian matrix can be obtained as:

[0038]

[0039] ⑤ Calculation of virtual force:

[0040] The virtual model control assumes that there is a spring damping component, and the current position (x, y, z) and velocity of the control object are input. And the expected reference trajectory (x bd ,y bd , z bd ), the virtual force f of the swing phase 摆 The calculation formula is:

[0041]

[0042] Where: (k x , k y , k z ) is the elastic coefficient of the fictitious force, (b x , b y , b z ) is the damping coefficient of the virtual force.

[0043] The virtual force calculation methods of the support phase control and the swing phase are different. In the swing phase, the virtual force is applied to the end of the leg and foot. In the support phase, the position of the end of the leg and foot relative to the ground does not change much. The fuselage moves relative to the ground contact point. The virtual force is applied to the hip joint position, which is equivalent to applying -f at the end of the leg and foot. 支 It should be noted that during the robot's uniform forward motion, only the forward speed The height of the aircraft from the ground needs to have a value, and the rest are all set to 0. Therefore, the control law of the support phase can be obtained:

[0044]

[0045] ⑥Calculation of joint torque

[0046] After obtaining the virtual force applied to the controlled object, according to the calculated Jacobian matrix, the torque of the leg joint motor in the swing phase and the support phase can be obtained as follows:

[0047]

[0048] Furthermore, the specific steps of step (4) are as follows:

[0049] ① Design of state space:

[0050] Reinforcement learning is based on the Markov decision process. During the interaction between the agent and the environment, a series of state vectors {s1, s2, ...s n}, the selected state is the position coordinates of the four leg ends relative to the hip joint, the flip angles of the robot body are rollover, pitch and lateral rotation, the error sum between the leg end position and the reference expected trajectory, and the previous action output vector k has a total of 48 states:

[0051] ②Design of action space:

[0052] The action space design output is the spring damping coefficients k and b of the designed virtual model controller. Each leg has 3 degrees of freedom, each leg requires 6 action inputs, and a total of 24 action outputs. In order to reduce the learning difficulty of the reinforcement learning agent, it is necessary to pre-test the approximate range of parameters. For convenience, the elastic damping coefficient of each leg can be set to be consistent. On this basis, the action space output range of the reinforcement learning agent is limited to avoid the reinforcement learning agent's optimization space being too large and the algorithm failing to converge.

[0053] ③ Design of reward function:

[0054] The main purpose of the quadruped robot is to move forward as smoothly as possible, maintain a certain height, and actively return to the pre-set trajectory when disturbed by external forces. x is the forward speed of the robot, y is the lateral displacement of the robot, u is the output of the robot action, θ i is the flip angle of the robot body in three dimensions, T f is the maximum duration of the simulation training process, T s is the minimum step unit in the simulation process, T s / T f In order to allow the robot to run as many steps as possible during training and get more rewards, the reward function is set as follows:

[0055]

[0056] ④Setting of termination function

[0057] In each training round, the reinforcement learning agent sets a termination function to terminate the training in time when the robot body deviates from a certain threshold and proceed to the next round of training, which can reduce the training time. The thresholds are set as the sum of the foot trajectory errors E sum , the robot body flip angles α, β, γ, the robot's lateral offset y and the height of the body from the ground z. The termination function is set as follows:

[0058] S isD =(E sum ≥0.57)|(α, β, γ≥0.36)|(|y|≥0.4)|(z≤0.20) (15)

[0059] The round ends when any of the conditions in formula (19) is met.

[0060] ⑤Reinforcement learning algorithm design

[0061] The state space and action space of the multi-legged robot are both multi-dimensional continuous space vectors. The present invention selects the deep deterministic policy gradient (DDPG) method to optimize the parameters of the virtual model controller of the quadruped robot. The method includes two neural networks, namely an actor network and a critic network. The actor network actor inputs the state s of the current environment and outputs the intelligent body action a. The critic network critor evaluates the output of the actor network according to the action of the actor network and the reaction of the current environment. The corresponding actor network improves the strategy output according to the evaluation of the critor network. As the number of rounds increases, the output of the actor network is more and more in line with the current environmental status, and the evaluation level of the critor network on the actor network is more and more accurate, and finally the optimal strategy is reached.

[0062] Furthermore, the specific steps of step (5) are as follows.

[0063] ① Quadruped robot body modeling

[0064] Use SolidWorks to build the robot body model, and export the built physical model into a urdf file through the sm2urdf plug-in.

[0065] ②Build a joint simulation model of matlab and simulink

[0066] According to the urdf file generated in step ①, use the smimport function of MATLAB to import the model into Simulink and build the corresponding algorithm model.

[0067] The beneficial effects of the present invention are embodied in:

[0068] (1) The present invention utilizes the powerful exploration capability of deep reinforcement learning to find the optimal controller parameters, thus reducing the difficulty of setting the controller parameters.

[0069] (2) The present invention improves the control accuracy of the traditional virtual model controller and has a certain anti-interference performance, so that when the sensor is interfered by the outside world, the controller can still accurately control the movement of the quadruped robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 is a flow chart of the method of the present invention;

[0071] Figure 2 This is the topological diagram of the quadruped robot structure;

[0072] Figure 3 It is a control structure block diagram of the present invention;

[0073] Figure 4 This is the update flow chart of the deep deterministic policy gradient algorithm;

[0074] Figure 5 The composite cycloid trajectory of the foot end of the quadruped robot of the present invention;

[0075] Figure 6 is the reward function curve of the reinforcement learning controller of the present invention;

[0076] Figure 7 is the actual foot end trajectory of the robot in the XZ space of the present invention;

[0077] Figure 8 is the position curve of the robot of the present invention in the world coordinate system;

[0078] Fig. 9 is a transformation curve of the spring coefficient of a single leg of the robot of the present invention;

[0079] Fig.10 is a transformation curve of the damping coefficient of a single leg of the robot of the present invention;

[0080] Fig.11 It is a comparison curve of the error curves of the present invention and the existing control method;

[0081] Fig.12 is the lateral position change curve of the quadruped robot of the present invention

[0082] Fig.13 The present invention is a motor torque output curve under the control of the existing control method;

[0083] Fig.14 This is the torque output curve of the joint motor of the present invention. DETAILED DESCRIPTION

[0084] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. In the absence of conflict, the embodiments in this application and the features in the embodiments can be combined with each other. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0085] It should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0086] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the meaning of "and / or" appearing in the full text includes three parallel schemes. Taking "A and / or B" as an example, it includes scheme A, or scheme B, or schemes that A and B meet at the same time. In addition, "multiple" refers to more than two. In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the ability of ordinary technicians in this field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist.

[0087] See also Figure 1 and 2 The present invention provides a parameter control method of a quadruped robot virtual model controller based on reinforcement learning, and the specific control method is as follows:

[0088] Step (1): Establish a mathematical model of a quadruped robot: Calculate the forward kinematic model of the quadruped robot. The quadruped robot has 12 joints in total. Each leg consists of three leg parts: upper, middle and lower. The upper joint is L1 meters long, and the middle and lower joints are L2 and L3 meters long respectively. Obtain the mapping between the rotation angle of the robot leg joint and the position relationship between the end of the leg and foot.

[0089] Step (2): gait generation: optimizing the cycloid trajectory to obtain the constrained optimized cycloid trajectory;

[0090] Step (3): Design of virtual model controller: Using the mathematical model obtained in step (1), establish the virtual model control of the quadruped robot and initialize the parameters of the virtual model controller;

[0091] Step (4): Integrate deep reinforcement learning algorithm: Select and design the state space, action space and reward function, and combine the virtual model controller designed in step (3) to design the reinforcement learning agent and select the algorithm;

[0092] Step (5): Training simulation of the virtual prototype model: Train the constructed virtual model to obtain the optimal control strategy.

[0093] Traditional public method: The traditional VMC controller parameter estimation adopts the empirical method, and the damping coefficient and elastic coefficient of each joint of the mechanical leg are estimated by random empirical estimation. Once the parameters are determined, the parameters will not change in subsequent experiments. In reality, there is no guarantee that the ground environment where the robot walks is not uniform and flat, but uneven. Therefore, the fixed parameter selection leads to poor dynamic control performance of the system, and the corresponding parameters cannot be reasonably set according to the specific road conditions, so the error is large. When the robot body is subject to external interference forces, such as lateral or longitudinal force interference, it is easy to flip and overturn.

[0094] The method of the present invention: The present invention mainly improves the parameter selection method of the VMC controller for the disclosed method. The original fixed parameter selection method is replaced by the corresponding parameters output by the deep reinforcement learning agent in real time decision, so as to realize the adaptive control of the quadruped robot and improve the anti-interference ability of the disclosed method. The specific effects are as follows:

[0095] Among them, the parameter adaptation curve of the quadruped robot during walking is shown in Fig. 9 and Fig.10 .

[0096] The present invention utilizes the powerful exploration capability of deep reinforcement learning to find the optimal controller parameters, thereby reducing the difficulty of setting the controller parameters.

[0097] The present invention improves the control accuracy of the traditional virtual model controller and has a certain anti-interference performance, so that when the sensor is interfered by the outside world, the controller can still accurately control the movement of the quadruped robot.

[0098] In one embodiment, the specific steps of step (1) are as follows:

[0099] ① Quadruped robot body structure design

[0100] Quadrupeds can move in six dimensions, including x, y, z, and movement around the three axes of x, y, and z. Therefore, according to its movement requirements, the quadruped robot has a total of 12 degrees of freedom. Each leg is composed of three leg-foot parts: upper, middle, and lower. The upper joint is L1 meter long, and the middle and lower joints are L2 and L3 meters long respectively. The full elbow movement mode is selected.

[0101]

[0102] Table 1 Physical parameters of the quadruped robot

[0103] ②Robot forward kinematics modeling

[0104] The world coordinate system {W} is established with the ground origin, the body coordinate system {B} is established at the center of mass of the robot body, and the coordinate system is established at the origin of the hip joint of the robot leg, which are respectively the left front {FL}, the right front {RL}, the left back {BL}, and the right back {BR}. The hip joint, shoulder joint, knee joint and foot end are represented by {0} to {3} respectively. According to the parameters of the quadruped robot described in step (1), the DH method is used to establish the positive kinematic equation. The structural parameters of the four legs are consistent.

[0105]

[0106] Table 2 Robot DH parameter table The positive kinematic equation is as follows:

[0107]

[0108] The transformation matrix for rotation around the z-axis is:

[0109]

[0110] The translation transformation matrix along the y direction is:

[0111]

[0112] The translation transformation matrix along the z direction is:

[0113]

[0114] From equations (2), (3), and (4), the transformation matrix of the robot foot end coordinate system {3} relative to the hip joint coordinate system {0} is as follows:

[0115]

[0116] In formula (5), c ij =cos(θ i +θ j ), c i =cos(θ i ), s i = sin(θ i ).

[0117] From formula (5), the forward kinematic equation can be obtained as follows:

[0118]

[0119] In one embodiment, the specific steps of step (2) are as follows:

[0120] During the movement of the quadruped robot, each leg has two states: swing phase and support phase. To solve the problem of sliding and dragging between the end of the robot's legs and the ground during walking, the position, velocity and acceleration in the horizontal x direction and the vertical y direction are constrained. The constraint equations are as follows:

[0121] Horizontal x-direction constraint:

[0122]

[0123] Constraints in the vertical y direction:

[0124]

[0125]

[0126] In formulas (7) to (12), T is a gait cycle, S is the stride of a gait cycle, and H is the lifting height of the end of the leg. Through the above constraints, the expected trajectory of the robot is obtained as shown in formula (13):

[0127]

[0128] In one embodiment, in step (3), the specific steps of extracting motion characteristic parameters are as follows: the virtual model controller is used as the basic controller

[0129] ⑦ Calculation of Jacobian matrix:

[0130] The virtual model controller requires the Jacobian matrix of the controlled object, which can be derived by the sitting position (x, y, z) of the robot leg relative to the joint variables. The joint variables are recorded as q1, q2, and q3, and their physical meanings are the rotation angles of the motors of the hip joint, shoulder joint, and knee joint. According to the forward kinematics equation (6) of the robot leg, its Jacobian matrix can be obtained as:

[0131]

[0132] ⑧ Calculation of virtual force:

[0133] The virtual model control assumes that there is a spring damping component, and the current position (x, y, z) and velocity of the control object are input. And you need to enter the expected reference trajectory ( x bd ,y bd , z bd) , the virtual force f of the swing phase 摆 The calculation formula is:

[0134]

[0135] Where: (k x , ky , k z ) is the elastic coefficient of the fictitious force, (b x , b y , b z) is the damping coefficient of the virtual force.

[0136] The virtual force calculation methods of the support phase control and the swing phase are different. In the swing phase, the virtual force is applied to the end of the leg and foot. In the support phase, the position of the end of the leg and foot relative to the ground does not change much. The fuselage moves relative to the ground contact point. The virtual force is applied to the hip joint position, which is equivalent to applying -f at the end of the leg and foot. 支 It should be noted that during the robot's uniform forward motion, only the forward speed The height of the aircraft from the ground needs to have a value, and the rest are all set to 0. Therefore, the control law of the support phase can be obtained:

[0137]

[0138] ⑨Calculation of joint torque

[0139] After obtaining the virtual force applied to the controlled object, according to the calculated Jacobian matrix, the torque of the leg joint motor in the swing phase and the support phase can be obtained as follows:

[0140]

[0141] The present invention improves the tracking performance of the legs of the quadruped robot by introducing a foot-end trajectory tracking performance index C;

[0142]

[0143] Where: x, y, z are the actual foot end trajectory positions, x d ,y d 、z d is the desired three-axis trajectory.

[0144] like Fig.11 As shown, the dotted line is the trajectory error curve of the robot foot under the public method, and the red solid line is the trajectory error curve of the foot after adding deep reinforcement learning. It can be seen that compared with the public method, the maximum error peak is 0.15, while the error peak of the method of the present invention is about 0.05, and the effect is obvious. However, due to its dynamic update parameters, its error changes are irregular, but it does not affect its overall stability.

[0145] In one embodiment, the specific steps of step (4) are as follows:

[0146] ① Design of state space:

[0147] Reinforcement learning is based on the Markov decision process. During the interaction between the agent and the environment, a series of state vectors {s1, s2, ...s n}, the selected state is the position coordinates of the four leg ends relative to the hip joint, the flip angles of the robot body are rollover, pitch and lateral rotation, the error sum between the leg end position and the reference expected trajectory, and the previous action output vector k has a total of 48 states:

[0148] ②Design of action space:

[0149] The action space design output is the spring damping coefficients k and b of the designed virtual model controller. Each leg has 3 degrees of freedom, each leg requires 6 action inputs, and a total of 24 action outputs. In order to reduce the learning difficulty of the reinforcement learning agent, it is necessary to pre-test the approximate range of parameters. For convenience, the elastic damping coefficient of each leg can be set to be consistent. On this basis, the action space output range of the reinforcement learning agent is limited to avoid the reinforcement learning agent's optimization space being too large and the algorithm failing to converge.

[0150] ③ Design of reward function:

[0151] The main purpose of the quadruped robot is to move forward as smoothly as possible, maintain a certain height, and actively return to the pre-set trajectory when disturbed by external forces. x is the forward speed of the robot, y is the lateral displacement of the robot, u is the output of the robot action, θ i is the flip angle of the robot body in three dimensions, T f is the maximum duration of the simulation training process, T s is the minimum step unit in the simulation process, T s / T f In order to allow the robot to run as many steps as possible during training and get more rewards, the reward function is set as follows:

[0152]

[0153] ④Setting of termination function

[0154] In each training round, the reinforcement learning agent sets a termination function to terminate the training in time when the robot body deviates from a certain threshold and proceed to the next round of training, which can reduce the training time. The thresholds are set as the sum of the foot trajectory errors E sum , the robot body flip angles α, β, γ, the robot's lateral offset y and the height of the body from the ground z. The termination function is set as follows:

[0155] S isD =(E sum≥0.57)|(α, β, γ≥0.36)|(|y|≥0.4)|(z≤0.20) (15)

[0156] The round ends when any of the conditions in formula (19) is met.

[0157] ⑤Reinforcement learning algorithm design

[0158] The state space and action space of the multi-legged robot are both multi-dimensional continuous space vectors. The present invention selects the deep deterministic policy gradient (DDPG) method to optimize the parameters of the virtual model controller of the quadruped robot. The method includes two neural networks, namely an actor network and a critic network. The actor network actor inputs the state s of the current environment and outputs the intelligent body action a. The critic network critor evaluates the output of the actor network according to the action of the actor network and the reaction of the current environment. The corresponding actor network improves the strategy output according to the evaluation of the critor network. As the number of rounds increases, the output of the actor network is more and more in line with the current environmental status, and the evaluation level of the critor network on the actor network is more and more accurate, and finally the optimal strategy is reached.

[0159]

[0160] Table 3 DDPG algorithm training parameters

[0161] Comparison of the anti-interference performance of the quadruped robot when it is subjected to lateral interference in the y direction:

[0162] Fig.12 The figure shows the change of the lateral position of the quadruped robot after the robot body is subjected to the interference force in the lateral position under the disclosed method and the method of the present invention, respectively. The interference force F applied is: a +10N interference force is applied at 1 to 2 seconds, and a reverse -10N interference force is applied at 4 to 5 seconds. It can be seen from the above figure that the disclosed method cannot adapt to the changes in the external environment in real time. After being subjected to the lateral interference force, the quadruped robot will deflect in one direction uncontrollably, causing it to deviate from the preset trajectory (the motion trajectory in the present invention is a positive linear motion, that is, motion in the lateral direction y=0). After 4 to 5 seconds, a reverse interference force is applied. After the interference force is applied, the robot's movement direction is deflected. After the interference force is removed, the robot still moves in the previous direction, as shown by the dotted line in the figure. Finally, the lateral displacement slope under this method is not 0 overall, indicating that the robot offset will become larger and larger. In the present invention, because the deep reinforcement learning method is adopted, not only the accurate tracking of the foot end trajectory is considered, but also the robot's movement trajectory is considered as an important indicator. For details, see the setting rules of the reinforcement learning agent reward function in the invention patent text (18). Therefore, when the robot body is disturbed in the lateral direction, it can have a certain ability to self-correct its route, such as Fig.12 As shown by the solid line in .

[0163] Comparison of output torque of robot leg joints:

[0164] See also Fig.13 In the disclosed methods, there is no further explanation on the robot leg joint torque. Its output size is closely related to the set elastic coefficient and damping coefficient, as well as the current robot foot position and speed. The greater the difference between the foot state and the actual expected state, the greater the actual output torque. The robot leg has two states, namely swing and support. When in the swing phase, if the output exceeds the current required torque, the predetermined state can be achieved, but on the other hand, it also causes energy waste. In the real robot, it will increase energy consumption and reduce endurance. Similarly, when in the support phase, if the torque is too large, it is easy to cause force imbalance, resulting in slipping on the ground or even tipping over. When the torque is insufficient, the robot cannot maintain the preset height, resulting in slow speed and other phenomena.

[0165] The present invention adopts the method of deep reinforcement learning dynamic parameter adjustment, taking into account the output of joint torque. The general principle is to make the output torque as small as possible to achieve the purpose of reducing energy consumption and protecting the body. Sensors are used to obtain the three-axis position of the body, the foot end speed, and the output torque of the joint motor. In the process of continuous interaction between the reinforcement learning agent and the environment, the most suitable output torque is obtained to achieve stable operation of the robot. Fig.14 shown.

[0166] Fig.13 It can be seen that the motor will produce a large torque mutation every time it switches from the swing phase to the support phase, and this phenomenon will occur in each gait cycle, increasing the instability of the body.

[0167] Fig.14 It can be seen that its torque peak is roughly the same as the effective part of the public method, but the output torque is smoother, which effectively suppresses the torque mutation of the motor and achieves the expected control goal.

[0168] In one embodiment, the specific steps of step (5) are as follows.

[0169] ① Quadruped robot body modeling

[0170] Use SolidWorks to build the robot body model, and export the built physical model into a urdf file through the sm2urdf plug-in.

[0171] ②Build a joint simulation model of matlab and simulink

[0172] According to the urdf file generated in step ①, use the smimport function of MATLAB to import the model into Simulink and build the corresponding algorithm model.

[0173] The experimental platform used in this experiment is MATLAB R2021a, the operating system is win10 system, the central processor is Intel 10600KF, the image display card is RTX3060, and the simscape multibody toolbox and Deep reforecement toolbox of simulink are used to obtain the reward function curve during the training process. The position and speed relationship between the quadruped robot body and the world coordinate system are obtained through the Transform Sensor in simulink, and Simulink / sinks / scope are added to output the final simulation results. Figure 7 is the actual motion trajectory of the foot end of the quadruped robot of the present invention, Figure 8 is the position change curve of the robot after training, where Figure 8 (a) is the position change curve of the present invention in the forward direction of the x-axis, Figure 8 (b) is the position change curve of the present invention in the y-axis side deviation direction, Figure 8 (c) is the position change curve in the z vertical direction.

[0174] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A parameter control method for a quadruped robot virtual model controller based on reinforcement learning, characterized in that ,The specific control methods are as follows: Step (1): Establish a mathematical model of a quadruped robot: Calculate the forward kinematic model of the quadruped robot. The quadruped robot has 12 joints in total. Each leg consists of three leg parts: upper, middle and lower. The upper joint is L1 meters long, and the middle and lower joints are L2 and L3 meters long respectively. Obtain the mapping between the rotation angle of the robot leg joint and the position relationship between the end of the leg and foot. Step (2): gait generation: optimizing the cycloid trajectory to obtain the constrained optimized cycloid trajectory; Step (3): Design of virtual model controller: Using the mathematical model obtained in step (1), establish the virtual model control of the quadruped robot and initialize the parameters of the virtual model controller; Step (4): Integrate deep reinforcement learning algorithm: Select and design the state space, action space and reward function, and combine the virtual model controller designed in step (3) to design the reinforcement learning agent and select the algorithm; Step (5): Training simulation of the virtual prototype model: training the constructed virtual model to obtain the optimal control strategy; In step (3), the specific steps of extracting motion feature parameters are as follows: Virtual Model Controller as Base Controller ① Calculation of Jacobian matrix: The virtual model controller requires the Jacobian matrix of the controlled object, which can be derived by the sitting position (x, y, z) of the robot leg relative to the joint variables. The joint variables are denoted as q1, q2, and q3, and their physical meanings are the rotation angles of the motors of the hip joint, shoulder joint, and knee joint. According to the obtained forward kinematics equation (6) of the robot leg, its Jacobian matrix can be obtained as: ② Calculation of virtual force: The virtual model control assumes that there is a spring damping component, and the current position (x, y, z) and velocity of the control object are input. And the expected reference trajectory (x bd ,y bd , z bd ), the virtual force f of the swing phase 摆 The calculation formula is: Where: (k x , k y , k z ) is the elastic coefficient of the fictitious force, (b x , b y , b z ) is the damping coefficient of the virtual force; The virtual force calculation methods of the support phase control and the swing phase are different. In the swing phase, the virtual force is applied to the end of the leg and foot. In the support phase, the position of the end of the leg and foot relative to the ground does not change much. The fuselage moves relative to the ground contact point. The virtual force is applied to the hip joint position, which is equivalent to applying -f at the end of the leg and foot. 支 It should be noted that during the robot's uniform forward motion, only the forward speed The height of the aircraft from the ground needs to have a value, and the rest are all set to 0; therefore, the control law of the support phase can be obtained: ③Calculation of joint torque After obtaining the virtual force applied to the controlled object, according to the calculated Jacobian matrix, the torque of the leg joint motor in the swing phase and the support phase can be obtained as follows: Step (4) The specific steps are as follows: ① Design of state space: Reinforcement learning is based on the Markov decision process. During the interaction between the agent and the environment, a series of state vectors {s1, s2, ...s n }, the selected state is the position coordinates of the four leg ends relative to the hip joint, the flip angles of the robot body are rollover, pitch and lateral rotation, the error sum between the leg end position and the reference expected trajectory, and the previous action output vector k has a total of 48 states: ②Design of action space: The action space design output is the spring damping coefficients k and b of the designed virtual model controller. Each leg has 3 degrees of freedom, each leg requires 6 action inputs, and a total of 24 action outputs are required. In order to reduce the learning difficulty of the reinforcement learning agent, it is necessary to pre-test the approximate range of the parameters. For convenience, the elastic damping coefficient of each leg can be set to be consistent. On this basis, the action space output range of the reinforcement learning agent is limited to avoid the reinforcement learning agent's optimization space being too large and the algorithm failing to converge. ③ Design of reward function: The main purpose of the quadruped robot is to move forward as smoothly as possible, maintain a certain height, and actively return to the pre-set trajectory when disturbed by external forces. x is the forward speed of the robot, y is the lateral displacement of the robot, u is the output of the robot action, θ i is the flip angle of the robot body in three dimensions, T f is the maximum duration of the simulation training process, T s is the minimum step unit in the simulation process, T s / T f In order to allow the robot to run as many steps as possible during training and get more rewards, the reward function is set as follows: ④Setting of termination function In each training round, the reinforcement learning agent sets a termination function to terminate the training in time when the robot body deviates from a certain threshold and proceed to the next round of training, which can reduce the training time. The thresholds are set as the sum of the foot trajectory errors E sum , the robot body flip angle α, β, γ, the robot's lateral offset y and the height of the body from the ground z; the termination function is set as follows: S is D=(E sum ≥0.57)|(α、β、γ≥0.36)|(|y|≥0.4)|(z≤0.20) (15) If any of the conditions in formula (19) is met, the round ends; ⑤Reinforcement learning algorithm design The state space and action space of the multi-legged robot are both multi-dimensional continuous space vectors. The present invention selects the deep deterministic policy gradient (DDPG) method to optimize the parameters of the virtual model controller of the quadruped robot. The method includes two neural networks, namely an actor network and a critic network. The actor network actor inputs the state s of the current environment and outputs the intelligent body action a. The critic network critor evaluates the output of the actor network according to the action of the actor network and the reaction of the current environment. The corresponding actor network improves the strategy output according to the evaluation of the critor network. As the number of rounds increases, the output of the actor network is more and more in line with the current environmental status, and the evaluation level of the critor network on the actor network is more and more accurate, and finally the optimal strategy is reached.

2. The parameter control method of a quadruped robot virtual model controller based on reinforcement learning as claimed in claim 1, characterized in that: Step (1) The specific steps are as follows: ① Quadruped robot body structure design Quadrupeds can move in six dimensions, including x, y, z and around the three axes of x, y, and z. Therefore, according to its movement requirements, the quadruped robot has 12 degrees of freedom. Each leg is composed of three leg parts: upper, middle, and lower. The upper joint is L1 meters long, and the middle and lower joints are L2 and L3 meters long respectively. The full elbow movement mode is selected; ②Robot forward kinematics modeling The world coordinate system {W} is established with the ground origin, the body coordinate system {B} is established at the center of mass of the robot body, and the coordinate system is established at the origin of the hip joint of the robot leg, which are respectively left front {FL}, right front {RL}, left back {BL}, and right back {BR}. The hip joint, shoulder joint, knee joint and foot end are represented by {0} to {3} respectively. According to the parameters of the quadruped robot described in step (1), the DH method is used to establish the positive kinematic equation. The structural parameters of the four legs are consistent. The positive kinematic equation is as follows: The transformation matrix for rotation around the z-axis is: The translation transformation matrix along the y direction is: The translation transformation matrix along the z direction is: From equations (2), (3), and (4), the transformation matrix of the robot foot end coordinate system {3} relative to the hip joint coordinate system {0} is as follows: c in Equation (5) ij = cos(θ i + θ j ), c i = cos(θ i ), s i = sin(θ i ); From formula (5), the forward kinematic equation can be obtained as follows:

3. The parameter control method of a quadruped robot virtual model controller based on reinforcement learning as claimed in claim 2, characterized in that: The specific steps of step (2) are as follows: During the movement of the quadruped robot, each leg has two states: swing phase and support phase. To solve the problem of sliding and dragging between the end of the robot's legs and the ground during walking, the position, velocity and acceleration in the horizontal x direction and the vertical y direction are constrained. The constraint equations are as follows: Horizontal x-direction constraint: Constraints in the vertical y direction: In formulas (7) to (12), T is a gait cycle, S is the stride of a gait cycle, and H is the lifting height of the end of the leg. Through the above constraints, the expected trajectory of the robot is obtained as shown in formula (13):

4. The parameter control method of a quadruped robot virtual model controller based on reinforcement learning as claimed in claim 1, characterized in that: Step (5) The specific steps are as follows: ① Quadruped robot body modeling Use SolidWorks to build the robot body model, and export the built physical model into a urdf file through the sm2urdf plug-in; ②Build a joint simulation model of matlab and simulink According to the urdf file generated in step ①, use the smimport function of MATLAB to import the model into Simulink and build the corresponding algorithm model.

Images