A quasi-passive seven-link biped robot gait control method
By designing a gait control strategy based on deep reinforcement learning method, combining dynamic model and multi-gait characteristics, the problem of insufficient gait stability and disturbance suppression ability of quasi-passive seven-link bipedal robot is solved, and a higher Walking stability and disturbance suppression effect.
Patent Information
- Application Number
- CN202411068151.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-06
- Publication Date
- 2024-12-03
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing quasi-passive seven-link bipedal robot gait control method is difficult to improve walking stability and disturbance suppression while ensuring gait flexibility, and its multi-stage dynamic characteristics are not fully utilized.
Using a method based on deep reinforcement learning, combining the Euler-Lagrangian equation and the conservation theorem of angular momentum to establish a dynamic model of the oscillation and collision stages, combining the Newton-Raphson iteration method and variable perturbation method to quickly search for the state of the fixed point, and A multi-gait stage characteristic design gait control strategy is introduced, and the depth deterministic strategy gradient algorithm is used to estimate joint torque values.
It significantly improves the walking stability and external disturbance suppression ability of the quasi-passive seven-link bipedal robot, and has strong environmental generalization ability.
Smart Images

Figure CN118584820B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of robot gait control, and in particular to a gait control method for a quasi-passive seven-link biped robot. Background Art
[0002] Bionic humanoid bipedal robots have become the focus of researchers due to their development potential in education, services, and human-computer interaction. However, they still have problems such as stiff gait, poor walking stability, and weak disturbance suppression ability, making them difficult to apply in real life. How to improve the walking stability and disturbance suppression ability of bionic humanoid bipedal robots while ensuring gait flexibility has become a key issue that needs to be solved urgently. Using gait control methods to apply active torque control based on the natural dynamics of the robot will help improve the walking stability and disturbance suppression ability of the quasi-passive seven-link bipedal robot.
[0003] Most of the existing gait control methods for quasi-passive seven-link biped robots require obtaining information such as the change in the target motion angle and calculating the corresponding torque through dynamics, but do not fully utilize the multi-stage dynamic characteristics unique to quasi-passive seven-link biped robots to control the robot's gait. Although this avoids the complex dynamic calculations of the robot and makes the control problem easier to handle, it ignores the advantages of the seven-link biped robot's dynamic model, such as its strong adaptability and ability to achieve more precise gait control. Therefore, under the action of this type of control method, the seven-link biped robot still has certain limitations in terms of its ability to adapt to the walking environment. How to improve the walking stability and external disturbance suppression ability of the quasi-passive seven-link biped robot has become a difficult point. Therefore, the study of gait control for quasi-passive seven-link biped robots urgently needs a gait control method with strong environmental generalization ability. Summary of the invention
[0004] In view of the above-mentioned technical problem that the existing gait control methods are difficult to apply to bionic humanoid bipedal robots with complex structures, the present invention provides a gait control method for a quasi-passive seven-linkage bipedal robot based on deep reinforcement learning. A gait control strategy is designed with the help of the deep reinforcement learning method and the introduction of the multi-gait stage characteristics unique to the quasi-passive seven-linkage bipedal robot, thereby achieving improvement in the walking stability and external terrain disturbance suppression capability of the quasi-passive seven-linkage bipedal robot.
[0005] The technical means adopted by the present invention are as follows:
[0006] A quasi-passive seven-link biped robot gait control method, comprising:
[0007] S1. Establish the swing phase dynamics model and collision phase dynamics model of the quasi-passive seven-link biped robot, including:
[0008] S11. A quasi-passive seven-link biped robot with additional ankle joint characteristics was used as the research object. According to the real walking process images of the subjects and the contact state change characteristics between the swinging foot and the ground, the single-cycle gait of the quasi-passive seven-link biped robot was divided into multiple continuous swinging stages and discrete collision stages.
[0009] S12, assuming that the swinging foot and the ground collide and separate instantly, and ignoring the influence of the contact force between the foot and the ground on the motion state of the quasi-passive seven-link biped robot;
[0010] S13. The swing phase dynamics model of the quasi-passive seven-link biped robot is established using the Euler-Lagrange equation and the theorem of conservation of angular momentum, as follows:
[0011]
[0012]
[0013]
[0014]
[0015] In the above formula, represents the Lagrangian of the system, which is composed of the kinetic energy of the system Subtract potential energy get; The generalized coordinates of the system , represents the degree of freedom of the robot at the current stage; The time derivative is denoted by ; express Derivative of time; represents the supporting leg angle, represents angular velocity; Indicates the swing leg thigh angle, is the angular velocity of the swinging leg thigh, Indicates the swing leg calf angle, represents the angular velocity of the swing leg calf; Indicates the robot i The generalized external moment of each degree of freedom; represents the generalized force on the system; Indicates the system i The generalized forces on each degree of freedom; represents the inertia matrix of the system, where - express System inertia matrix The specific sub-items are as follows:
[0016]
[0017]
[0018]
[0019] S14. The dynamic model of the collision phase of the quasi-passive seven-link biped robot is established using the Euler-Lagrange equation and the theorem of conservation of angular momentum, as follows:
[0020]
[0021]
[0022]
[0023] In the above formula, Represents the radius vector from the angular momentum conservation point to the joint center of mass in the current collision stage; It represents the joint center of mass velocity immediately before the collision; Indicates the joint center of mass velocity immediately after the collision; Indicates the joint rotation speed immediately before the collision; Indicates the rotation speed of the joint immediately after the collision; Indicates the angle between the robot's supporting leg and the vertical direction; Indicates the angle between the robot's swinging leg and the vertical direction; represents the angle between the robot's swinging legs and the ground; The coefficient matrix representing the angular velocity of each joint of the robot before collision; The coefficient matrix representing the angular velocity of each joint of the robot after the collision; - Representation coefficient matrix ; - Representation coefficient matrix The specific sub-items are as follows:
[0024]
[0025]
[0026]
[0027]
[0028]
[0029]
[0030]
[0031]
[0032]
[0033]
[0034]
[0035]
[0036]
[0037]
[0038]
[0039] In the above formula, Represents body mass, represents the hip mass, represents the leg mass, Indicates the quality of the thigh, Indicates the mass of the calf, Indicates sufficient quality, represents the body moment of inertia, is the moment of inertia of the legs, is the moment of inertia of the thigh, is the moment of inertia of the lower leg, represents the moment of inertia, represents the distance from the body center of mass to the hip joint, Indicates the length of the leg, represents the distance from the leg center of mass to the hip joint, Indicates the length of the thigh, represents the distance from the thigh mass center to the hip joint, Indicates the length of the calf. represents the distance from the calf mass center to the hip joint, Indicates foot length, The distance from heel to ankle joint. represents the distance from the ankle joint to the foot center of mass, It represents the distance from the foot center to the toe;
[0040] S2, combining the Newton-Raphson iteration method with the variable perturbation method, quickly search for the fixed point state of the quasi-passive seven-link biped robot;
[0041] S3. Combine the unique multi-stage gait characteristics of the quasi-passive seven-link biped robot with the deep reinforcement learning algorithm to obtain the gait control strategy of the quasi-passive seven-link biped robot based on deep reinforcement learning, and estimate the torque value required for each joint of the quasi-passive seven-link biped robot in different states.
[0042] Furthermore, step S2 specifically includes:
[0043] S21, initialization iteration number threshold n , Convergence tolerance , small disturbance And the kinetic equation , ,in, = , Indicates the current state of the robot. Indicates the next state of the robot;
[0044] S22. Randomly set the initial state of the quasi-passive seven-link biped robot at the Poincare section ;
[0045] S23, the initial state of the quasi-passive seven-link biped robot at the Poincare section set in step S22 Apply a small disturbance , and the state after the disturbance is obtained ;
[0046] S24, according to the kinetic equation , calculate the initial state and the disturbed state Single cycle gait The state at the Poincare section after and ;
[0047] S25, based on the initial state calculated in step S24 and the disturbed state Single cycle gait The state at the Poincare section after and , calculate the Jacobian matrix , the calculation formula is as follows:
[0048]
[0049] S26. Judgment Is it established? If Greater than the convergence tolerance , then execute step S27 and step S28 in sequence; if Less than the convergence tolerance , then the fixed point state of the quasi-passive seven-link biped robot is searched;
[0050] S27, calculate the step size of the iteration , ,in, Represents the identity matrix; and updates the number of iterations k ;
[0051] S28, Judgment Is it established? If , then update the initial state of the robot , , and return to step S23; if , then return to step S22.
[0052] Furthermore, step S3 includes: introducing multi-gait stage characteristics to split the complete gait cycle, establishing a gait control strategy for a single swing stage, using the Poincare section to intercept the fixed point end state of each swing stage to design a reward function, adjusting the amount of active torque applied and the joint position to adapt to the differences between different stages, improving the gait control effect of a single swing stage, and reintegrating the control strategies of each swing stage according to the order of the gait stages.
[0053] Further, step S3 specifically includes:
[0054] S31. In the dual-delay deep deterministic policy gradient algorithm for deep reinforcement learning, the main network and the target network are used to estimate the main network action and the target network action respectively. The estimation formula is as follows:
[0055]
[0056]
[0057] in, Represents the main network action, that is, the current output joint torque value; represents the target network action; Indicates the main network strategy; Indicates the target network policy; Indicates the current state of the robot;
[0058] S32. Action on target network Smoothing regularization is performed to reduce the fluctuation of the agent strategy during training. The mathematical description of smoothing regularization is as follows:
[0059]
[0060] in, represents normal distribution; represents variance; represents the noise clipping threshold, and They represent the upper and lower bounds of action clipping, respectively, to ensure that the final estimated action value is within the specified range and adapts to the requirements of different task action spaces. After estimating the current joint torque, the joint torque is applied to the corresponding joint of the robot, and the robot's motion state will also change accordingly.
[0061] S33. In order to introduce multi-stage gait characteristics, the Poincare section is set to intercept the final state of each swing stage of the robot with the fixed point state and the random initial state as the initial value, and the two norms are calculated as the immediate reward. The two norm calculation formula is as follows:
[0062]
[0063] S34, estimate the main network through the neural network The estimated formula is as follows:
[0064]
[0065] in, Indicates i The main network value;
[0066] S35. Calculate the target network The value is calculated as follows:
[0067]
[0068] in, Indicates immediate reward; express j The state of the robot at the time of the step; Indicates status Robot movements; represents the discount factor; express j +1 step target network strategy;
[0069] S36. Calculate strategy network loss , by minimizing the policy network loss To update the main network parameters, the policy network loss The calculation formula is as follows:
[0070]
[0071] in, express expectations; NIndicates the amount of experience drawn from the experience replay pool at one time; Indicates the current robot status s The action value taken next;
[0072] S37, based on the main network Value, calculate the deterministic policy gradient , and update the main network parameters by gradient ascent to implement the TD3 algorithm to update the estimated torque of each joint of the robot, deterministic policy gradient The calculation formula is as follows:
[0073]
[0074] S38. The parameters of the main network are gradually updated to the target network by means of soft update. The formula of soft update is as follows:
[0075]
[0076] in, Indicates the update speed of the control network parameters;
[0077] S39. After each swing phase is trained to convergence, it is combined again in the order of the gait cycle and a discrete collision phase is introduced to form a complete gait control strategy.
[0078] Compared with the prior art, the present invention has the following advantages:
[0079] 1. The present invention provides a quasi-passive seven-link biped robot gait control method, which is based on the double-delay deep deterministic policy gradient algorithm in deep reinforcement learning, introduces the multi-gait stage characteristics unique to the quasi-passive seven-link biped robot, and constructs a gait control strategy based on deep reinforcement learning, thereby improving the walking stability and terrain disturbance suppression ability of the quasi-passive seven-link biped robot.
[0080] 2. The present invention is of great significance to the fields of high-precision dynamic model construction, humanoid bipedal robot development and rehabilitation equipment design.
[0081] Based on the above reasons, the present invention can be widely promoted in the fields of robot gait control and the like. BRIEF DESCRIPTION OF THE DRAWINGS
[0082] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0083] Figure 1 The figure is a flow chart of the method of the present invention.
[0084] Figure 2 Schematic diagram of a simplified model of a quasi-passive seven-link biped robot provided in an embodiment of the present invention.
[0085] Figure 3 Schematic diagram of gait phase division of a quasi-passive seven-link biped robot provided in an embodiment of the present invention.
[0086] Figure 4 A flow chart of a stable fixed point state search algorithm for a quasi-passive seven-link biped robot provided in an embodiment of the present invention.
[0087] Figure 5 A stick figure of a quasi-passive seven-link biped robot provided by an embodiment of the present invention walking stably on a horizontal ground.
[0088] Figure 6 A schematic diagram of the structure of a double-delayed deep deterministic policy gradient (TD3) algorithm provided in an embodiment of the present invention.
[0089] Figure 7 A Poincare cross-section diagram is provided at the end state of the swing phase of the quasi-passive seven-link biped robot provided in an embodiment of the present invention.
[0090] Figure 8 Schematic diagram of the interaction between a quasi-passive seven-link biped robot in a single swing phase and the TD3 algorithm provided by an embodiment of the present invention.
[0091] Fig. 9 A schematic diagram of adjusting the motion state of a robot in a complete gait cycle by the trained gait control strategy provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0092] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0093] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0094] like Figure 1 As shown, the present invention provides a quasi-passive seven-link biped robot gait control method based on deep reinforcement learning, comprising:
[0095] S1. Establish the swing phase dynamics model and collision phase dynamics model of the quasi-passive seven-link biped robot;
[0096] S2, combining the Newton-Raphson iteration method with the variable perturbation method, quickly search for the fixed point state of the quasi-passive seven-link biped robot;
[0097] S3. Combine the unique multi-stage gait characteristics of the quasi-passive seven-link biped robot with the deep reinforcement learning algorithm to obtain the gait control strategy of the quasi-passive seven-link biped robot based on deep reinforcement learning, and estimate the torque value required for each joint of the quasi-passive seven-link biped robot in different states.
[0098] In specific implementation, as a preferred embodiment of the present invention, step S1 specifically includes:
[0099] S11. A quasi-passive seven-link biped robot with additional ankle joint characteristics is taken as the research object. According to the images of the subjects' actual walking process and the contact state change characteristics between the swinging feet and the ground, the single-cycle gait of the quasi-passive seven-link biped robot is divided into multiple continuous swinging stages and discrete collision stages. In this embodiment, in order to establish a simplified model of a humanoid biped robot that can better reflect the walking motion characteristics of humans, the present invention additionally introduces the characteristic of adaptively adjusting the stiffness of the ankle joint on the basis of the existing simplified model of a biped robot with a torso and knee joints, which constitutes the research object of the present invention, namely, the simplified model of a quasi-passive seven-link biped robot, such as Figure 2 Referring to the change of the contact state of the swinging foot during human walking, the robot's single-cycle gait is divided to simplify the complexity of dynamic modeling. The gait stage division results are shown in Figure 3 shown.
[0100] S12, assuming that the swinging foot and the ground collide and separate instantly, and ignoring the influence of the contact force between the foot and the ground on the motion state of the quasi-passive seven-link biped robot;
[0101] S13. The swing phase dynamics model of the quasi-passive seven-link biped robot is established using the Euler-Lagrange equation and the theorem of conservation of angular momentum, as follows:
[0102]
[0103]
[0104]
[0105]
[0106] In the above formula, represents the Lagrangian of the system, which is composed of the kinetic energy of the system Subtract potential energy get; The generalized coordinates of the system , represents the degree of freedom of the robot at the current stage; The time derivative is denoted by ; express Derivative of time; represents the supporting leg angle, represents angular velocity; Indicates the swing leg thigh angle, is the angular velocity of the swinging leg thigh, Indicates the swing leg calf angle, represents the angular velocity of the swing leg calf; Indicates the robot i The generalized external moment of each degree of freedom; represents the generalized force on the system; Indicates the system i The generalized forces on each degree of freedom; represents the inertia matrix of the system, where - express System inertia matrix The specific sub-items are as follows:
[0107]
[0108]
[0109]
[0110] S14. The dynamic model of the collision phase of the quasi-passive seven-link biped robot is established using the Euler-Lagrange equation and the theorem of conservation of angular momentum, as follows:
[0111]
[0112]
[0113]
[0114] In the above formula, Represents the radius vector from the angular momentum conservation point to the joint center of mass in the current collision stage; It represents the joint center of mass velocity immediately before the collision; Indicates the joint center of mass velocity immediately after the collision; Indicates the joint rotation speed immediately before the collision; Indicates the rotation speed of the joint immediately after the collision; Indicates the angle between the robot's supporting leg and the vertical direction; Indicates the angle between the robot's swinging leg and the vertical direction; represents the angle between the robot's swinging legs and the ground; The coefficient matrix representing the angular velocity of each joint of the robot before collision; The coefficient matrix representing the angular velocity of each joint of the robot after the collision; - Representation coefficient matrix ; - Representation coefficient matrix The specific sub-items are as follows:
[0115]
[0116]
[0117]
[0118]
[0119]
[0120]
[0121]
[0122]
[0123]
[0124]
[0125]
[0126]
[0127]
[0128]
[0129]
[0130] In the above formula, Represents body mass, represents the hip mass, represents the leg mass, Indicates the quality of the thigh, Indicates the mass of the calf, Indicates sufficient quality, represents the body moment of inertia, is the moment of inertia of the legs, is the moment of inertia of the thigh, is the moment of inertia of the lower leg, represents the moment of inertia, represents the distance from the body center of mass to the hip joint, Indicates the length of the leg, represents the distance from the leg center of mass to the hip joint, Indicates the length of the thigh, represents the distance from the thigh mass center to the hip joint, Indicates the length of the calf. represents the distance from the calf mass center to the hip joint, Indicates foot length, The distance from heel to ankle joint. represents the distance from the ankle joint to the foot center of mass, It indicates the distance from the center of mass of the foot to the toe.
[0131] When specifically implemented, as a preferred embodiment of the present invention, Figure 4 As shown in FIG. 1 , it is a flow chart of the stable fixed point state search algorithm of the quasi-passive seven-link biped robot, namely step S2, which specifically includes:
[0132] S21, initialization iteration number threshold n , Convergence tolerance , small disturbance And the kinetic equation , ,in, = , Indicates the current state of the robot. Indicates the next state of the robot;
[0133] S22. Randomly set the initial state of the quasi-passive seven-link biped robot at the Poincare section ;
[0134] S23, the initial state of the quasi-passive seven-link biped robot at the Poincare section set in step S22 Apply a small disturbance , and the state after the disturbance is obtained ;
[0135] S24, according to the kinetic equation , calculate the initial state and the disturbed state Single cycle gait The state at the Poincare section after and ;
[0136] S25, based on the initial state calculated in step S24 and the disturbed state Single cycle gait The state at the Poincare section after and , calculate the Jacobian matrix , the calculation formula is as follows:
[0137]
[0138] S26. Judgment Is it established? If Greater than the convergence tolerance , then execute step S27 and step S28 in sequence; if Less than the convergence tolerance , then the fixed point state of the quasi-passive seven-link biped robot is searched;
[0139] S27, calculate the step size of the iteration , ,in, Represents the identity matrix; and updates the number of iterations k ;
[0140] S28, Judgment Is it established? If , then update the initial state of the robot , , and return to step S23; if , then return to step S22.
[0141] In this embodiment, after the robot's fixed point state is searched, the robot starts to walk stably with the fixed point state as the initial value. The corresponding stick diagram is shown in the attached figure. Figure 5 As shown, its stick graph shows regular periodic changes.
[0142] In specific implementation, as a preferred embodiment of the present invention, step S3 includes: introducing multi-gait stage characteristics to split the complete gait cycle, establishing a gait control strategy for a single swing stage, using the Poincare section to intercept the fixed point end state of each swing stage to design a reward function, adjusting the amount of active torque applied and the joint position to adapt to the differences between different stages, improving the gait control effect of a single swing stage, and reintegrating the control strategies of each swing stage according to the order of the gait stages.
[0143] In specific implementation, as a preferred embodiment of the present invention, step S3 specifically includes:
[0144] The structure of S31 and TD3 algorithms is as follows Figure 6 As shown in the Twin Delayed Deterministic Policy Gradient (TD3) algorithm for deep reinforcement learning, the main network and the target network are used to estimate the main network action and the target network action respectively. The estimation formula is as follows:
[0145]
[0146]
[0147] in, Represents the main network action, that is, the current output joint torque value; represents the target network action; Indicates the main network strategy; Indicates the target network policy; Indicates the current state of the robot;
[0148] S32. Action on target network Smoothing regularization is performed to reduce the fluctuation of the agent strategy during training. The mathematical description of smoothing regularization is as follows:
[0149]
[0150] in, represents normal distribution; represents variance; represents the noise clipping threshold, and They represent the upper and lower bounds of action clipping, respectively, to ensure that the final estimated action value is within the specified range and adapts to the requirements of different task action spaces. After estimating the current joint torque, the joint torque is applied to the corresponding joint of the robot, and the robot's motion state will also change accordingly.
[0151] S33. In order to introduce multi-stage gait characteristics, the Poincare section is set to intercept the final state of each swing stage of the robot with the fixed point state and the random initial state as the initial value, and the two norms are calculated as the immediate reward. The two norm calculation formula is as follows: Set the Poincare section at the final state of each swing stage as shown in the attached figure. Figure 7 As shown:
[0152]
[0153] Among them, the sparse reward function is designed as follows:
[0154]
[0155] S34, estimate the main network through the neural network The estimated formula is as follows:
[0156]
[0157] in, Indicates i The main network value;
[0158] S35. Calculate the target network The value is calculated as follows:
[0159]
[0160] in, Indicates immediate reward; express j The state of the robot at the time of the step; Indicates status Robot movements; represents the discount factor; express j +1 step target network strategy;
[0161] S36. Calculate strategy network loss , by minimizing the policy network loss To update the main network parameters, the policy network loss The calculation formula is as follows:
[0162]
[0163] in, express expectations; N Indicates the amount of experience drawn from the experience replay pool at one time; Indicates the current robot status s The value of the action taken next.
[0164] S37, based on the main network Value, calculate the deterministic policy gradient , and update the main network parameters by gradient ascent to implement the TD3 algorithm to update the estimated torque of each joint of the robot, deterministic policy gradient The calculation formula is as follows:
[0165]
[0166] S38. The parameters of the main network are gradually updated to the target network by means of soft update. The formula of soft update is as follows:
[0167]
[0168] in, Indicates the control network parameter update speed.
[0169] S39. After each swing phase is trained to convergence, it is combined again in the order of the gait cycle and a discrete collision phase is introduced to form a complete gait control strategy.
[0170] Example:
[0171] The symbolic computing toolbox is used in Matlab to establish the dynamic model of a quasi-passive seven-link biped robot. The TD3 algorithm is built in Python, and the weights of the main strategy network in each swinging stage after training are imported. The real-time interaction between the robot dynamic model and the TD3 algorithm in Matlab is realized through the Matlab engine to achieve numerical simulation verification of the gait control method. After the numerical simulation verification, in order to solve the problem that the idealized assumptions such as completely inelastic collision in the numerical simulation lead to a large difference from the actual environment, the present invention further builds a virtual simulation model based on Matlab and Simscape Multibody, switches different swinging stages by setting the handles of the angle constraint and the foot-ground state contact constraint, and finally realizes the verification of the gait control effect in the virtual simulation scene.
[0172] Finally, numerical simulation and virtual simulation were carried out in horizontal ground scenes, fixed slope scenes and terrain disturbance scenes, respectively, to improve the walking stability and disturbance suppression ability of the quasi-passive seven-link biped robot. The interaction process between the robot and the TD3 algorithm in each swing phase is shown in the figure. Figure 8 As shown in Figure 2, the complete gait control strategy is as follows: Fig. 9shown.
[0173] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A quasi-passive seven-link biped robot gait control method, characterized in that: include: S1. Establish the swing phase dynamics model and collision phase dynamics model of the quasi-passive seven-link biped robot, including: S11. A quasi-passive seven-link biped robot with additional ankle joint characteristics was used as the research object. According to the real walking process images of the subjects and the contact state change characteristics between the swinging foot and the ground, the single-cycle gait of the quasi-passive seven-link biped robot was divided into multiple continuous swinging stages and discrete collision stages. S12, assuming that the swinging foot and the ground collide and separate instantly, and ignoring the influence of the contact force between the foot and the ground on the motion state of the quasi-passive seven-link biped robot; S13. The swing phase dynamics model of the quasi-passive seven-link biped robot is established using the Euler-Lagrange equation and the theorem of conservation of angular momentum, as follows: In the above formula, represents the Lagrangian of the system, which is composed of the kinetic energy of the system Subtract potential energy get; The generalized coordinates of the system , represents the degree of freedom of the robot at the current stage; The time derivative is denoted by ; express Derivative of time; represents the supporting leg angle, represents angular velocity; Indicates the swing leg thigh angle, is the angular velocity of the swinging leg thigh, Indicates the swing leg calf angle, represents the angular velocity of the swing leg calf; Indicates the robot i The generalized external moment of each degree of freedom; represents the generalized force on the system; Indicates the system i The generalized forces on each degree of freedom; represents the inertia matrix of the system, where - express System inertia matrix The specific sub-items are as follows: S14. The dynamic model of the collision phase of the quasi-passive seven-link biped robot is established using the Euler-Lagrange equation and the theorem of conservation of angular momentum, as follows: In the above formula, Represents the radius vector from the angular momentum conservation point to the joint center of mass in the current collision stage; It represents the joint center of mass velocity immediately before the collision; Indicates the joint center of mass velocity immediately after the collision; Indicates the joint rotation speed immediately before the collision; Indicates the rotation speed of the joint immediately after the collision; Indicates the angle between the robot's supporting leg and the vertical direction; Indicates the angle between the robot's swinging leg and the vertical direction; represents the angle between the robot's swinging legs and the ground; The coefficient matrix representing the angular velocity of each joint of the robot before collision; The coefficient matrix representing the angular velocity of each joint of the robot after the collision; - Representation coefficient matrix ; - Representation coefficient matrix The specific sub-items are as follows: In the above formula, Represents body mass, represents the hip mass, represents the leg mass, Indicates the quality of the thigh, Indicates the mass of the calf, Indicates sufficient quality, represents the body moment of inertia, is the moment of inertia of the legs, is the moment of inertia of the thigh, is the moment of inertia of the lower leg, represents the moment of inertia, represents the distance from the body center of mass to the hip joint, Indicates the length of the leg, represents the distance from the leg center of mass to the hip joint, Indicates the length of the thigh, represents the distance from the thigh mass center to the hip joint, Indicates the length of the calf. represents the distance from the calf mass center to the hip joint, Indicates foot length, The distance from heel to ankle joint. represents the distance from the ankle joint to the foot center of mass, It represents the distance from the foot center to the toe; S2, combining the Newton-Raphson iteration method with the variable perturbation method, quickly search for the fixed point state of the quasi-passive seven-link biped robot; S3. Combine the unique multi-stage gait characteristics of the quasi-passive seven-link biped robot with the deep reinforcement learning algorithm to obtain the gait control strategy of the quasi-passive seven-link biped robot based on deep reinforcement learning, and estimate the torque value required for each joint of the quasi-passive seven-link biped robot in different states.
2. The quasi-passive seven-link biped robot gait control method according to claim 1, characterized in that: Step S2 specifically includes: S21, initialization iteration number threshold n , Convergence tolerance , small disturbance And the kinetic equation , ,in, = , Indicates the current state of the robot. Indicates the next state of the robot; S22. Randomly set the initial state of the quasi-passive seven-link biped robot at the Poincare section ; S23, applying a small disturbance to the initial state of the quasi-passive seven-link biped robot at the Poincare section set in step S22 to obtain a disturbed state; S24, according to the kinetic equation , calculate the initial state and the disturbed state Single cycle gait The state at the Poincare section after and ; S25, based on the initial state calculated in step S24 and the disturbed state Single cycle gait The state at the Poincare section after and , calculate the Jacobian matrix , the calculation formula is as follows: S26. Judgment Is it established? If Greater than the convergence tolerance , then execute step S27 and step S28 in sequence; if Less than the convergence tolerance , then the fixed point state of the quasi-passive seven-link biped robot is searched; S27, calculate the step size of the iteration , ,in, Represents the identity matrix; and updates the number of iterations k ; S28, Judgment Is it established? If , then update the initial state of the robot , , and return to step S23; if , then return to step S22.
3. The quasi-passive seven-link biped robot gait control method according to claim 1, characterized in that: Step S3 includes: introducing multi-gait stage characteristics to split the complete gait cycle, establishing a gait control strategy for a single swing stage, using the Poincare section to intercept the fixed point end state of each swing stage to design a reward function, adjusting the amount of active torque applied and the joint position to adapt to the differences between different stages, improving the gait control effect of a single swing stage, and reintegrating the control strategies of each swing stage according to the gait stage sequence.
4. The quasi-passive seven-link biped robot gait control method according to claim 3, characterized in that: Step S3 specifically includes: S31. In the dual-delay deep deterministic policy gradient algorithm for deep reinforcement learning, the main network and the target network are used to estimate the main network action and the target network action respectively. The estimation formula is as follows: in, Represents the main network action, that is, the current output joint torque value; represents the target network action; Indicates the main network strategy; Indicates the target network policy; Indicates the current state of the robot; S32. Action on target network Smoothing regularization is performed to reduce the fluctuation of the agent strategy during training. The mathematical description of smoothing regularization is as follows: in, represents normal distribution; represents variance; represents the noise clipping threshold, and They represent the upper and lower bounds of action clipping, respectively, to ensure that the final estimated action value is within the specified range and adapts to the requirements of different task action spaces. After estimating the current joint torque, the joint torque is applied to the corresponding joint of the robot, and the robot's motion state will also change accordingly. S33. In order to introduce multi-stage gait characteristics, the Poincare section is set to intercept the final state of each swing stage of the robot with the fixed point state and the random initial state as the initial value, and the two norms are calculated as the immediate reward. The two norm calculation formula is as follows: S34, estimate the main network through the neural network The estimated formula is as follows: in, Indicates i The main network value; S35. Calculate the target network The value is calculated as follows: in, Indicates immediate reward; express j The state of the robot at the time of the step; Indicates status Robot movements; represents the discount factor; express j +1 step target network strategy; S36. Calculate strategy network loss , by minimizing the policy network loss To update the main network parameters, the policy network loss The calculation formula is as follows: in, express expectations; N Indicates the amount of experience drawn from the experience replay pool at one time; Indicates the current robot status s The action value taken next; S37, based on the main network Value, calculate the deterministic policy gradient , and update the main network parameters by gradient ascent to implement the TD3 algorithm to update the estimated torque of each joint of the robot, deterministic policy gradient The calculation formula is as follows: S38. The parameters of the main network are gradually updated to the target network by means of soft update. The formula of soft update is as follows: in, Indicates the update speed of the control network parameters; S39. After each swing phase is trained to convergence, it is combined again in the order of the gait cycle and a discrete collision phase is introduced to form a complete gait control strategy.
