Seven-degree-of-freedom biped robot based on heavy-load direct-drive joint and control method

Abstract

This invention belongs to the field of robotics technology, specifically a seven-DOF bipedal robot based on heavy-duty direct-drive joints and its control method. The bipedal robot includes a pelvis and two sets of leg structures symmetrically arranged below the pelvis. Each set of leg structures includes a hip joint mechanism, a knee joint mechanism, and an ankle joint mechanism connected sequentially from top to bottom. The control method controls the robot's joint angles through model-driven and data-driven approaches. The hip joint mechanism of this invention can respectively realize roll, pitch, and rotation movements, achieving three-DOF control of the hip joint, thus making the bipedal robot's movement smoother and more natural, and more suitable for heavy-duty working conditions.

Description

Technical Field

[0001] The present invention belongs to the technical field of robotics, and specifically relates to a 7-degree-of-freedom bipedal robot based on heavy-duty direct-drive joints and a control method thereof. Background Art

[0002] With the rapid development of robotics technology, its application fields continue to expand, functional requirements are increasingly diversified, and more and more functions need to be realized by robots. The primary function that a robot must satisfy is its own movement. Especially for legged robots, how to realize that the robot can simulate the walking state of human legs is an important research direction.

[0003] In the mechanical configuration design of humanoid bipedal robots, the single-leg mechanism needs to completely reproduce the joint system of the human lower limb, which usually includes the collaborative configuration of the hip joint, knee joint and ankle joint. These joints can realize human-like natural gait movement through precise motion coupling and coordinated control. In addition, among the traditional control methods for bipedal robots, the traditional ZMP control has the problems of relying on an accurate mathematical model and poor adaptability, while the pure PPO learning control is completely data-driven and may be unstable. Summary of the Invention

[0004] In view of the above problems, the purpose of the present invention is to provide a 7-degree-of-freedom bipedal robot based on heavy-duty direct-drive joints and a control method thereof.

[0005] The purpose of the present invention is achieved through the following technical solution: A 7-degree-of-freedom bipedal robot based on heavy-duty direct-drive joints comprises a pelvis and two sets of leg structures arranged symmetrically on the left and right below the pelvis, and each set of the leg structures includes a hip joint mechanism, a knee joint mechanism and an ankle joint mechanism which are connected in sequence from top to bottom;

[0006] The hip joint mechanism of each set of the leg structures has a total of three degrees of freedom, which are sequentially a degree of freedom of rotating around an axis parallel to the front-rear direction of the bipedal robot, a degree of freedom of rotating around an axis parallel to the left-right direction of the bipedal robot, and a degree of freedom of rotating around an axis perpendicular to the left-right direction of the bipedal robot;

[0007] The knee joint mechanism of each set of the leg structures has a total of one degree of freedom, which is a degree of freedom of rotating around an axis parallel to the left-right direction of the bipedal robot;

[0008] The ankle joint mechanism of each set of the leg structures has a total of three degrees of freedom, which are sequentially a degree of freedom of rotating around an axis perpendicular to the left-right direction of the bipedal robot, a degree of freedom of rotating around an axis parallel to the left-right direction of the bipedal robot, and a degree of freedom of rotating around an axis parallel to the front-rear direction of the bipedal robot.

[0009] Each hip joint mechanism of the leg structure includes a first hip joint drive module, a first hip joint frame, a first hip joint drive component, a first hip joint drive connector, a second hip joint passive connector, a second hip joint passive bearing, a second hip joint passive mounting bracket, a second hip joint drive module, a second hip joint drive module frame, a second hip joint drive component, a second hip joint drive connector, a third hip joint mounting component, a third hip joint drive module, a third hip joint frame, a third hip joint drive component, a first hip joint fixation component, a first hip joint passive bearing, and a first hip joint passive mounting bracket.

[0010] The first hip joint frame and the first hip joint passive mounting bracket together serve as the fixed ends of the entire hip joint mechanism and are respectively fixed to the lower side of the pelvis. The positions of the first hip joint frame and the first hip joint passive mounting bracket are corresponding front to back, and the first hip joint frame is located behind the first hip joint passive mounting bracket. The housing of the first hip joint drive module is installed behind the first hip joint frame. The drive end of the first hip joint drive module is connected to the first hip joint drive component. The first hip joint drive connector, the second hip joint passive mounting bracket, the second hip joint drive module frame, and the first hip joint... The joint fixing members are respectively located between the first hip joint frame and the first hip joint passive mounting frame. The rear end of the first hip joint drive connector is connected to the first hip joint drive member. The rear ends of the second hip joint passive mounting frame and the second hip joint drive module frame are respectively connected to the front ends of the first hip joint drive connector. The first hip joint fixing member is fixedly connected to the front ends of the second hip joint passive mounting frame and the second hip joint drive module frame. The first hip joint fixing member is rotatably connected to the first hip joint passive mounting frame through a first hip joint passive bearing. The housing of the moving module is mounted between the passive mounting bracket of the second hip joint and the drive module bracket of the second hip joint. The drive component of the second hip joint is located on the side of the drive module bracket away from the passive mounting bracket of the second hip joint. One end of the drive component is connected to the drive end of the drive module. The passive connector of the second hip joint is located on the side of the passive mounting bracket away from the drive module bracket of the second hip joint. The passive connector is rotatably connected to the passive mounting bracket of the second hip joint via a passive bearing. The module has a hip joint third joint mounting component and a hip joint third joint frame respectively arranged from top to bottom below. The hip joint third joint mounting component is connected to the hip joint second joint passive connector. The hip joint third joint frame is connected to the other end of the hip joint second joint drive component through the hip joint second joint drive connector. The outer shell of the hip joint third joint drive module is installed between the hip joint third joint mounting component and the hip joint third joint frame. The drive end of the hip joint third joint drive module faces downward and is connected to the hip joint third joint drive component. The hip joint third joint drive component serves as the drive end of the entire hip joint mechanism and is connected to the fixed end of the knee joint mechanism.

[0011] The axis of the drive end of the first hip joint drive module is parallel to the front-back direction of the bipedal robot, the axis of the drive end of the second hip joint drive module is parallel to the left-right direction of the bipedal robot, and the axis of the drive end of the third hip joint drive module is perpendicular to the left-right direction of the bipedal robot.

[0012] The axes of the drive ends of the first hip joint drive module, the second hip joint drive module, and the third hip joint drive module intersect at the same point.

[0013] Each knee joint mechanism of the leg structure includes a hip joint third joint drive connector, a knee joint drive module, a knee joint drive module mounting component, a knee joint drive component, a knee joint drive connector, a knee joint drive module passive mounting bracket, a knee joint passive bearing, a knee joint passive mounting bracket, an ankle joint first joint drive module mounting component, and an ankle joint first joint drive mounting component.

[0014] The upper end of the hip joint third joint drive connector serves as the fixed end of the knee joint mechanism as a whole and is connected to the drive end of the hip joint mechanism as a whole. The upper end of the knee joint drive module mounting piece and the upper end of the knee joint drive module passive mounting frame are respectively connected to the lower end of the hip joint third joint drive connector on the left and right sides. The outer shell of the knee joint drive module is installed between the knee joint drive module mounting piece and the knee joint drive module passive mounting frame. The knee joint drive component is located on the side of the knee joint drive module mounting piece away from the knee joint drive module passive mounting frame. One end of the knee joint drive component is fixedly connected to the drive end of the knee joint drive module. The knee joint passive mounting frame is located on the passive end of the knee joint drive module. On the side of the mounting bracket away from the knee joint drive module mounting component, the knee joint passive mounting bracket is rotatably connected to the knee joint drive module passive mounting bracket via the knee joint passive bearing. Below the knee joint drive module, from top to bottom, are respectively provided an ankle joint first joint drive module mounting component and an ankle joint first joint drive mounting component. The ankle joint first joint drive module mounting component is connected to the knee joint passive mounting bracket. The ankle joint first joint drive mounting component is fixedly connected to the other end of the knee joint drive component via the knee joint drive connector. The ankle joint first joint drive module mounting component and the ankle joint first joint drive mounting component together serve as the drive end of the entire knee joint mechanism and are connected to the fixed end of the ankle joint mechanism.

[0015] The axis of the drive end of the knee joint drive module is parallel to the left-right direction of the bipedal robot.

[0016] The degree of freedom of rotation of the knee joint mechanism of each leg structure about an axis parallel to the left and right direction of the bipedal robot lies in the same plane as the degree of freedom of rotation of the hip joint mechanism of the same leg structure about an axis parallel to the left and right direction of the bipedal robot.

[0017] Each ankle joint mechanism of the leg structure includes an ankle joint first joint drive module, an ankle joint first joint drive component, an ankle joint first joint drive connector, an ankle joint second joint drive module, an ankle joint second joint drive module mounting component, an ankle joint second joint drive component, an ankle joint second joint drive connector, an ankle joint second joint drive module passive mounting bracket, an ankle joint second joint passive bearing, an ankle joint second joint passive connector, an ankle joint third joint passive connector, an ankle joint third joint passive bearing, an ankle joint third joint drive module passive mounting bracket, an ankle joint third joint drive module, an ankle joint third joint drive module passive mounting bracket, a foot, and an ankle joint third joint drive component;

[0018] The outer shell of the ankle joint first joint drive module serves as the fixed end of the ankle joint mechanism as a whole and is connected to the drive end of the knee joint mechanism as a whole. The drive end of the ankle joint first joint drive module faces downward and is connected to the ankle joint first joint drive component. The ankle joint first joint drive connector is located below the ankle joint first joint drive module, and its upper end is connected to the ankle joint first joint drive component. The upper end of the ankle joint second joint drive module mounting component and the ankle joint second joint drive module passive mounting bracket are respectively connected to the lower end of the ankle joint first joint drive connector on the left and right sides, respectively. The housing of the assembly is mounted between the ankle joint second joint drive module mounting component and the ankle joint second joint drive module passive mounting bracket. The ankle joint second joint drive component is located on the side of the ankle joint second joint drive module mounting component away from the ankle joint second joint drive module passive mounting bracket. One end of the ankle joint second joint drive component is connected to the drive end of the ankle joint second joint drive module. The ankle joint second joint passive connector is located on the side of the ankle joint second joint drive module passive mounting bracket away from the ankle joint second joint drive module mounting component. The ankle joint second joint passive connector is connected to the ankle joint second joint drive module via an ankle joint second joint passive bearing. The foot is rotatably connected to the ankle joint second joint drive module. The upper side of the foot is respectively provided with the ankle joint third joint passive connector and the ankle joint third joint drive component. The ankle joint third joint passive connector is located in front of the ankle joint second joint drive module, and the ankle joint third joint drive component is located in front of the ankle joint third joint passive connector. The ankle joint third joint drive module passive mounting bracket, the ankle joint third joint drive module, and the ankle joint third joint drive module passive mounting bracket are sequentially arranged from back to front between the ankle joint third joint passive connector and the ankle joint third joint drive component. The passive mounting bracket of the moving module is rotatably connected to the passive connector of the third joint of the ankle joint via the passive bearing of the third joint of the ankle joint. The passive mounting bracket of the driving module of the third joint of the ankle joint is connected to the passive connector of the second joint of the ankle joint. The passive mounting bracket of the driving module of the third joint of the ankle joint is connected to the other end of the driving component of the second joint of the ankle joint via the driving connector of the second joint of the ankle joint. The outer shell of the driving module of the third joint of the ankle joint is fixedly connected to the passive mounting bracket of the driving module of the third joint of the ankle joint and the passive mounting bracket of the driving module of the third joint of the ankle joint. The driving end of the driving module of the third joint of the ankle joint is fixedly connected to the passive mounting bracket of the driving module of the third joint of the ankle joint.

[0019] The axis of the drive end of the first ankle joint drive module is perpendicular to the left-right direction of the bipedal robot, the axis of the drive end of the second ankle joint drive module is parallel to the left-right direction of the bipedal robot, and the axis of the drive end of the third ankle joint drive module is parallel to the front-back direction of the bipedal robot; the axes of the drive ends of the first, second, and third ankle joint drive modules intersect at the same point.

[0020] A control method for a seven-DOF bipedal robot based on heavy-duty direct-drive joints, which controls the joint angles of the robot through model-driven and data-driven approaches, includes the following steps:

[0021] 1) Initialize the PPO reinforcement learning algorithm and ZMP method to construct the robot state. And initialize, where q is the joint angle, For joint velocity, P COM p is the location of the center of mass. ZMP At the zero torque point, p support The supporting polygon is the bottom surface of the foot;

[0022] 2) Based on the center of mass position and robot mass distribution, the position of the zero torque point is calculated using the ZMP simplified model, and its deviation from the center of the supporting polygon is determined to generate the ZMP compensation torque for balance correction.

[0023] 3) Use the PPO reinforcement learning algorithm for policy optimization, collecting data including state s. t Action a t Rewards r t and the next state s t+1 The complete trajectory data is combined with multiple reward designs such as ZMP stability, energy consumption, trajectory tracking and survival. The advantage function is calculated using generalized advantage estimation, which guides the next step of the policy network to continuously adjust its parameters through loss function optimization.

[0024] 4) Using the advantage function generated in step 3), the PPO reinforcement learning algorithm stabilizes policy updates by constructing a pruning target loss function, improves the accuracy of future reward estimation by constructing a value function loss, and encourages policy exploration by constructing an entropy loss. The three are then combined into a total loss function according to set weights. The total loss function is optimized to obtain the PPO original action.

[0025] 5) Based on the original PPO motion, a hybrid control method is used to determine whether the zero-torque point deviates from the center of the support polygon. When the deviation exceeds the threshold, the original PPO motion is corrected using the ZMP compensation torque calculated in step 2), resulting in hybrid control motion a. finalOtherwise, the action output by the PPO reinforcement learning algorithm is directly used as the final control command, and the amplitude of the final action is limited by a function to obtain the hybrid control action a. final ;

[0026] 6) Combine the mixed control action a from step 5). final The signal is sent to the actuators to control the torque τ of each actuator.

[0027] 7) Update the PPO parameters and update the status s. t+1 Simultaneously, update the policy network parameters and value function network parameters, and use the updated network for the next round of environmental interaction;

[0028] 8) Repeat steps 2) to 7) until the total loss function is less than the threshold, thus achieving the training objective and obtaining the final policy; use the final policy to control the robot.

[0029] Step 2) Based on the center of mass position and robot mass distribution, calculate the zero-torque point position using the ZMP simplified model, determine its deviation from the center of the supporting polygon, and generate a compensation torque for balance correction, including the following steps:

[0030] (1) Location of zero torque point x ZMP y ZMP It is obtained through the following formula;

[0031]

[0032] Among them, the acceleration of the center of mass Obtained through measurement, Let x be the acceleration of the center of mass in the x-axis direction. Let x be the acceleration of the center of mass in the y-axis direction. Let p be the acceleration of the center of mass in the z-axis direction; when p ZMP When ∈P, it indicates that the bipedal robot is stable, where P is the region formed by the vertices {v1,v2,v3,v4} of the current supporting polygon; g is the acceleration due to gravity, P COM (x COM ,y COM ,z COM ) represents the position of the centroid, p ZMP (x ZMP ,y ZMP () indicates the location of the zero torque point;

[0033] Location of the center of mass P COM Based on the mass of each link and the position of its center of mass, we obtain:

[0034]

[0035] Where, m i p represents the mass of the i-th link.i This indicates the position of the centroid of the i-th link;

[0036] (2) Calculate p ZMP The deviation relative to the supporting polygon, i.e., p ZMP To the center point p of the supporting polygon ref Distance: Δp ZMP =p ZMP -p ref ;

[0037] When p ZMP If the robot moves beyond the supporting polygon, it indicates that the robot's balance is threatened. The robot's movement is adjusted by calculating compensating torque. Where, Δτ ZMP It is ZMP-compensated torque. It is the transpose of the Jacobian matrix from the center of mass to the joint space, Δp ZMP It is p ZMP The offset.

[0038] Step 3) Use the PPO reinforcement learning algorithm to optimize the policy and collect data including state s. t Action a t Rewards r t and the next state s t+1 The complete trajectory data is used, and combined with multiple reward designs such as ZMP stability, energy consumption, trajectory tracking, and survival, the advantage function is calculated using generalized advantage estimation. This guides the next step of the policy network to continuously adjust its parameters through loss function optimization, including the following steps:

[0039] Value function network for the current state s t Value estimation:

[0040]

[0041] Dominance function calculation:

[0042] δ t =r t +γV(s t+1 )-V(s t )

[0043] Where γ is the discount factor, λ is the smoothing parameter for the generalized advantage estimation, and V(s) t ) is the value function network for the current state s t Value estimation, V(s) t+1 ) represents the value estimate of the next state by the value function, δ t This represents the current TD error, k is the offset of the future time step, T is the total time steps of the trajectory, and t is the time.

[0044] Reward function: r t =ω1r ZMP +ω2r energy +ω3r track +ω4r alive

[0045] Where, r t+k The reward function r at time t+k t ;

[0046] r ZMP ZMP stability bonus:

[0047]

[0048] Where, Δp ZMP It is the offset of ZMP;

[0049] Energy consumption reward τ i Let be the torque of the i-th joint;

[0050] Gait tracking reward r track =-γ||qq ref || 2 q ref It is the expected joint angle trajectory;

[0051] Survival reward r alive =+1.0.

[0052] Step 4) Using the advantage function generated in Step 3), the PPO reinforcement learning algorithm stabilizes policy updates by constructing a pruning target loss function, improves the accuracy of future reward estimation by constructing a value function loss, and encourages policy exploration by constructing an entropy loss. These three are then combined with set weights into a total loss function. Optimizing the total loss function yields the original PPO action, including the following steps:

[0053] Construct the pruning target loss function:

[0054] L CLIP (θ)=E t [min(r t (θ)A t ,clip(r t (θ), 1-ε, 1+ε)A t )];

[0055] Among them, E t This represents the expectation, where θ is the policy parameter and r is the expected value. t (θ) is the importance sampling ratio, clip(x,a,b) is the clipping function that restricts x to [a,b], and r t (θ) represents the probability ratio between the old and new strategies. ε is the cutting factor;

[0056] Value function loss L value (θ)=(V θ (s t )-R t ) 2 ;

[0057] Among them, V θ (s t ) represents the current policy pair for state s t Value estimate, R t It is the actual reward at time step t, that is, the weighted sum of all future rewards accumulated from that moment onwards;

[0058] Entropy loss (π) θ )=-∑π θ logπ θ ;

[0059] Total loss function L Total Combinatorial optimization combines the three weighted factors into a total loss:

[0060] L Total =L CLIP -c1L value (θ)+c2Entropy(π θ )

[0061] Where c1 and c2 are weight hyperparameters;

[0062] Optimize the total loss function, update the policy network parameters and value function network parameters using gradient descent, and obtain the original PPO action 'a'. PPO .

[0063] Step 5) After the strategy generates the action, a hybrid control method is used to determine whether the zero-moment point deviates from the center of the supporting polygon. If the deviation exceeds the threshold, the ZMP compensation torque calculated in Step 2) is used to correct the original PPO action; otherwise, the action output by the PPO reinforcement learning algorithm is directly used as the final control command, and the amplitude of the final action is limited by a function to ensure that the joint torque is within a safe and controllable range. This includes the following steps:

[0064] If the deviation of ZMP relative to the supporting polygon is greater than 0.1m, meaning the zero-moment point is not located within the supporting polygon, ZMP compensation is applied, resulting in the following action:

[0065] a final =saturate(a PPO +Δτ ZMP ,a max )

[0066] Among them, a PPO For the original action of PPO, a max Δτ represents the maximum value of the joint movement. ZMP It is ZMP-compensated torque;

[0067] If the deviation of ZMP relative to the supporting polygon is less than 0.1m, that is, the zero moment point is located within the supporting polygon, the original PPO action a can be used directly. final =saturate(a PPO ,a max );

[0068] The saturate() function is used to limit the magnitude of motion or torque.

[0069] The advantages and positive effects of this invention are as follows:

[0070] 1. The hip joint mechanism of the present invention can realize roll, pitch and rotation movements respectively, realizing three degrees of freedom control of the hip joint; the knee joint mechanism can realize single degree of freedom control of the rotation movement of the lower leg; the ankle joint mechanism can realize rotation, pitch and roll movements respectively, realizing three degrees of freedom control of the ankle joint; the overall single leg structure can realize seven degrees of freedom of movement, which is more similar to human walking. At the same time, the movement is directly driven by each drive module, reducing torque loss, thereby making the bipedal robot movement more stable and natural, and more suitable for heavy-duty working conditions.

[0071] 2. The various parts of this invention can be manufactured extensively using 3D printing intelligent manufacturing, which reduces the overall weight of the robot and lowers the manufacturing cost of the bipedal robot while ensuring the strength of the structural components.

[0072] 3. This invention combines the Proximal Policy Optimization (PPO) reinforcement learning algorithm with the Zero Moment Point (ZMP) method, fully integrating the advantages of data-driven and model-driven control. This significantly improves the performance of a seven-DOF bipedal robot in dynamic gait planning and stability control. The PPO algorithm achieves adaptive optimization of joint movements and improves energy efficiency, while ZMP is introduced as a stability constraint to guide the design of the reward function and the construction of the state space during policy learning, thereby effectively enhancing the robot's dynamic balance and anti-interference capabilities. Compared to traditional pure model-driven or pure data-driven methods, this invention not only improves the robustness and adaptability of gait generation but also reduces instability and policy degradation risks during training. It possesses good generalization and practicality, making it suitable for dexterous motion control tasks of bipedal robots in complex environments. Attached Figure Description

[0073] Figure 1 This is a schematic diagram of the front axonal structure of the present invention;

[0074] Figure 2 This is a schematic diagram of the rear axial side structure of the present invention;

[0075] Figure 3 This is an exploded view of the hip joint mechanism of the present invention;

[0076] Figure 4 This is an exploded view of the knee joint mechanism and the first joint portion of the ankle joint of the present invention;

[0077] Figure 5 This is an exploded view of the second joint portion and the third joint portion of the ankle joint of the present invention;

[0078] Figure 6 This is the control flowchart of the present invention;

[0079] In the image: 1 represents the pelvis;

[0080] 2 is the hip joint mechanism; 2-1 is the first hip joint drive module; 2-2 is the first hip joint frame; 2-3 is the first hip joint drive component; 2-4 is the first hip joint drive connector; 2-5 is the second hip joint passive connector; 2-6 is the second hip joint passive bearing; 2-7 is the second hip joint passive mounting bracket; 2-8 is the second hip joint drive module; 2-9 is the second hip joint drive module frame; 2-10 is the second hip joint drive component; 2-11 is the second hip joint drive connector; 2-12 is the third hip joint mounting component; 2-13 is the third hip joint drive module; 2-14 is the third hip joint frame; 2-15 is the third hip joint drive component; 2-16 is the first hip joint fixation component; 2-17 is the first hip joint passive bearing; 2-18 is the first hip joint passive mounting bracket.

[0081] 3 is the knee joint mechanism, 3-1 is the third hip joint drive connector, 3-2 is the knee joint drive module, 3-3 is the knee joint drive module mounting component, 3-4 is the knee joint drive component, 3-5 is the knee joint drive connector, 3-6 is the knee joint drive module passive mounting bracket, 3-7 is the knee joint passive bearing, 3-8 is the knee joint passive mounting bracket, 3-9 is the ankle joint first joint drive module mounting component, and 3-10 is the ankle joint first joint drive mounting component.

[0082] 4 represents the ankle joint mechanism; 4-1 represents the first ankle joint drive module; 4-2 represents the first ankle joint drive component; 4-3 represents the first ankle joint drive connector; 4-4 represents the second ankle joint drive module; 4-5 represents the second ankle joint drive module mounting component; 4-6 represents the second ankle joint drive component; 4-7 represents the second ankle joint drive connector; 4-8 represents the second ankle joint drive module passive mounting bracket; 4-9 represents the second ankle joint passive bearing; 4-10 represents the second ankle joint passive connector; 4-11 represents the third ankle joint passive connector; 4-12 represents the third ankle joint passive bearing; 4-13 represents the third ankle joint drive module passive mounting bracket; 4-14 represents the third ankle joint drive module; 4-15 represents the third ankle joint drive module passive mounting bracket; 4-16 represents the foot; and 4-17 represents the third ankle joint drive component. Detailed Implementation

[0083] The following is in conjunction with the appendix Figure 1-6 The present invention will be described in further detail below.

[0084] A seven-DOF bipedal robot based on heavy-duty direct-drive joints, such as Figure 1-5 As shown, this embodiment includes a pelvis 1 and two sets of leg structures arranged symmetrically on the left and right sides below the pelvis 1. Each set of leg structures includes a hip joint mechanism 2, a knee joint mechanism 3, and an ankle joint mechanism 4 connected sequentially from top to bottom.

[0085] Each leg structure's hip joint mechanism 2 has three degrees of freedom arranged sequentially: a degree of freedom to rotate about an axis parallel to the bipedal robot's front-back direction, a degree of freedom to rotate about an axis parallel to the bipedal robot's left-right direction, and a degree of freedom to rotate about an axis perpendicular to the bipedal robot's left-right direction.

[0086] Each leg structure's knee joint mechanism 3 has one degree of freedom to rotate about an axis parallel to the left and right directions of the bipedal robot.

[0087] Each leg structure's ankle joint mechanism 4 has three degrees of freedom: a rotational degree about an axis perpendicular to the left-right direction of the bipedal robot, a rotational degree about an axis parallel to the left-right direction of the bipedal robot, and a rotational degree about an axis parallel to the front-back direction of the bipedal robot.

[0088] Specifically, such as Figure 3As shown, in this embodiment, each hip joint mechanism 2 of the leg structure includes a hip joint first joint drive module 2-1, a hip joint first joint frame 2-2, a hip joint first drive component 2-3, a hip joint first drive connector 2-4, a hip joint second joint passive connector 2-5, a hip joint second joint passive bearing 2-6, a hip joint second joint passive mounting frame 2-7, a hip joint second joint drive module 2-8, a hip joint second joint drive module frame 2-9, a hip joint second joint drive component 2-10, a hip joint second joint drive connector 2-11, a hip joint third joint mounting component 2-12, a hip joint third joint drive module 2-13, a hip joint third joint frame 2-14, a hip joint third joint drive component 2-15, a hip joint first joint fixation component 2-16, a hip joint first joint passive bearing 2-17, and a hip joint first joint passive mounting frame 2-18.

[0089] The first hip joint bracket 2-2 and the first hip joint passive mounting bracket 2-18 together serve as the fixed ends of the entire hip joint mechanism 2 and are respectively fixed to the lower side of the pelvis 1. The positions of the first hip joint bracket 2-2 and the first hip joint passive mounting bracket 2-18 are corresponding front and rear, and the first hip joint bracket 2-2 is located behind the first hip joint passive mounting bracket 2-18. The outer shell of the first hip joint drive module 2-1 is installed behind the first hip joint bracket 2-2. The drive end of the first hip joint drive module 2-1 is connected to the first hip joint drive component 2-3. The first hip joint drive connector 2-4, the second hip joint passive mounting bracket 2-7, the second hip joint drive module bracket 2-9, and the hip joint drive connector 2-4 are also connected. The first joint fixation component 2-16 is located between the first hip joint frame 2-2 and the first hip joint passive mounting frame 2-18. The rear end of the first hip joint drive connector 2-4 is connected to the first hip joint drive component 2-3. The rear ends of the second hip joint passive mounting frame 2-7 and the second hip joint drive module frame 2-9 are respectively connected to the front ends of the first hip joint drive connector 2-4. The first hip joint fixation component 2-16 is fixedly connected to the front ends of the second hip joint passive mounting frame 2-7 and the second hip joint drive module frame 2-9. The first hip joint fixation component 2-16 is connected to the first hip joint passive mounting frame 2-4 via the first hip joint passive bearing 2-17. -18 Rotary Connection: The housing of the second hip joint drive module 2-8 is mounted between the second hip joint passive mounting bracket 2-7 and the second hip joint drive module frame 2-9. The second hip joint drive component 2-10 is located on the side of the second hip joint drive module frame 2-9 away from the second hip joint passive mounting bracket 2-7. One end of the second hip joint drive component 2-10 is connected to the drive end of the second hip joint drive module 2-8. The second hip joint passive connector 2-5 is located on the side of the second hip joint passive mounting bracket 2-7 away from the second hip joint drive module frame 2-9. The second hip joint passive connector 2-5 is connected to the second hip joint via the second hip joint passive bearing 2-6. The passive mounting bracket 2-7 is rotatably connected. Below the second hip joint drive module 2-8, from top to bottom, are a third hip joint mounting component 2-12 and a third hip joint bracket 2-14. The third hip joint mounting component 2-12 is connected to the second hip joint passive connector 2-5. The third hip joint bracket 2-14 is connected to the other end of the second hip joint drive component 2-10 via the second hip joint drive connector 2-11. The outer shell of the third hip joint drive module 2-13 is installed between the third hip joint mounting component 2-12 and the third hip joint bracket 2-14. The drive end of the third hip joint drive module 2-13 faces downwards and is connected to the third hip joint drive component 2-15.The third joint drive component 2-15 of the hip joint is connected to the fixed end of the hip joint mechanism 3 as the drive end of the whole hip joint mechanism 2.

[0090] In this embodiment, the first hip joint of the hip joint mechanism 2 is responsible for lateral rolling, the second hip joint is responsible for pitching, and the third hip joint is responsible for rotation. The axis of the drive end of the first hip joint drive module 2-1 is parallel to the front-back direction of the bipedal robot, the axis of the drive end of the second hip joint drive module 2-8 is parallel to the left-right direction of the bipedal robot, and the axis of the drive end of the third hip joint drive module 2-13 is perpendicular to the left-right direction of the bipedal robot. The axes of the drive ends of the first hip joint drive module 2-1, the second hip joint drive module 2-8, and the third hip joint drive module 2-13 intersect at the same point, which is more similar to the human body structure, making it easier to control humanoid walking and also easier to perform forward and inverse kinematics calculations.

[0091] Specifically, such as Figure 4 As shown, in this embodiment, each knee joint mechanism 3 of the leg structure includes a hip joint third joint drive connector 3-1, a knee joint drive module 3-2, a knee joint drive module mounting component 3-3, a knee joint drive component 3-4, a knee joint drive connector 3-5, a knee joint drive module passive mounting bracket 3-6, a knee joint passive bearing 3-7, a knee joint passive mounting bracket 3-8, an ankle joint first joint drive module mounting component 3-9, and an ankle joint first joint drive mounting component 3-10.

[0092] The upper end of the hip joint third joint drive connector 3-1 serves as the fixed end of the knee joint mechanism 3 as well as the drive end of the hip joint mechanism 2 as well, i.e., it is connected to the hip joint third joint drive component 2-15. The upper ends of the knee joint drive module mounting component 3-3 and the upper ends of the knee joint drive module passive mounting bracket 3-6 are respectively connected to the lower ends of the hip joint third joint drive connector 3-1. The outer shell of the knee joint drive module 3-2 is installed between the knee joint drive module mounting component 3-3 and the knee joint drive module passive mounting bracket 3-6. The knee joint drive component 3-4 is located on the side of the knee joint drive module mounting component 3-3 away from the knee joint drive module passive mounting bracket 3-6. One end of the knee joint drive component 3-4 is fixedly connected to the drive end of the knee joint drive module 3-2. The knee joint passive mounting bracket 3-8 is located on the knee joint drive component 3-2. On the side of the passive mounting bracket 3-6 of the moving module away from the knee joint drive module mounting component 3-3, the passive mounting bracket 3-8 of the knee joint is rotatably connected to the passive mounting bracket 3-6 of the knee joint drive module via the passive bearing 3-7 of the knee joint. Below the knee joint drive module 3-2, from top to bottom, there are respectively the ankle joint first joint drive module mounting component 3-9 and the ankle joint first joint drive mounting component 3-10. The ankle joint first joint drive module mounting component 3-9 is connected to the passive mounting bracket 3-8 of the knee joint. The ankle joint first joint drive mounting component 3-10 is fixedly connected to the other end of the knee joint drive component 3-4 via the knee joint drive connector 3-5. The ankle joint first joint drive module mounting component 3-9 and the ankle joint first joint drive mounting component 3-10 together serve as the drive end of the entire knee joint mechanism 3 and are connected to the fixed end of the ankle joint mechanism 4.

[0093] In this embodiment, the knee joint mechanism 3 is used to control the rotation of the lower leg of the bipedal robot; the axis of the drive end of the knee joint drive module 3-2 is parallel to the left and right direction of the bipedal robot; and the degree of freedom of rotation of the knee joint mechanism 3 of each leg structure about the axis parallel to the left and right direction of the bipedal robot and the degree of freedom of rotation of the hip joint mechanism 2 of the same leg structure about the axis parallel to the left and right direction of the bipedal robot are in the same plane, that is, the axis of the drive end of the knee joint drive module 3-2 of each leg structure and the axis of the drive end of the second joint drive module 2-8 of the hip joint of the same leg structure are in the same plane.

[0094] Specifically, such as Figure 4 and Figure 5As shown, in this embodiment, each ankle joint mechanism 4 of the leg structure includes an ankle joint first joint drive module 4-1, an ankle joint first joint drive component 4-2, an ankle joint first joint drive connector 4-3, an ankle joint second joint drive module 4-4, an ankle joint second joint drive module mounting component 4-5, an ankle joint second joint drive component 4-6, an ankle joint second joint drive connector 4-7, an ankle joint second joint drive module passive mounting frame 4-8, an ankle joint second joint passive bearing 4-9, an ankle joint second joint passive connector 4-10, an ankle joint third joint passive connector 4-11, an ankle joint third joint passive bearing 4-12, an ankle joint third joint drive module passive mounting frame 4-13, an ankle joint third joint drive module 4-14, an ankle joint third joint drive module passive mounting frame 4-15, a foot 4-16, and an ankle joint third joint drive component 4-17.

[0095] The outer shell of the ankle joint first joint drive module 4-1 serves as the fixed end of the ankle joint mechanism 4 as well as the drive end of the knee joint mechanism 3 as well, and is connected to the ankle joint first joint drive module mounting parts 3-9 and 3-10 respectively. The outer shell of the ankle joint first joint drive module 4-1 is located between the ankle joint first joint drive module mounting parts 3-9 and 3-10.The driving end of the first ankle joint drive module 4-1 faces downward and is connected to the first ankle joint drive component 4-2. The first ankle joint drive connector 4-3 is located below the first ankle joint drive module 4-1, and its upper end is connected to the first ankle joint drive component 4-2. The upper end of the second ankle joint drive module mounting component 4-5 and the passive mounting bracket 4-8 are respectively connected to the lower end of the first ankle joint drive connector 4-3. The outer shell of the second ankle joint drive module 4-4 is installed between the second ankle joint drive module mounting component 4-5 and the passive mounting bracket 4-8. The second joint drive component 4-6 is located on the side of the ankle joint second joint drive module mounting component 4-5 away from the ankle joint second joint drive module passive mounting bracket 4-8. One end of the ankle joint second joint drive component 4-6 is connected to the drive end of the ankle joint second joint drive module 4-4. The ankle joint second joint passive connector 4-10 is located on the side of the ankle joint second joint drive module passive mounting bracket 4-8 away from the ankle joint second joint drive module mounting component 4-5. The ankle joint second joint passive connector 4-10 is rotatably connected to the ankle joint second joint drive module passive mounting bracket 4-8 through the ankle joint second joint passive bearing 4-9. The foot part 4-16 is located below the ankle joint second joint drive module 4-4. The upper side of component 16 is provided with a passive connector 4-11 and a drive component 4-17 for the third ankle joint. The passive connector 4-11 is located in front of the drive module 4-4 for the second ankle joint, and the drive component 4-17 is located in front of the passive connector 4-11. Passive mounting brackets 4-13, 4-14, and 4-15 for the third ankle joint drive module are arranged sequentially from back to front between the passive connector 4-11 and the drive component 4-17. The passive connectors 4-11 are rotatably connected via the passive bearing 4-12 of the third ankle joint. The passive mounting bracket 4-13 of the third ankle joint drive module is connected to the passive connector 4-10 of the second ankle joint. The passive mounting bracket 4-15 of the third ankle joint drive module is connected to the other end of the second ankle joint drive component 4-6 via the drive connector 4-7 of the second ankle joint. The outer shell of the third ankle joint drive module 4-14 is fixedly connected to the passive mounting brackets 4-13 and 4-15 of the third ankle joint drive module. The drive end of the third ankle joint drive module 4-14 is fixedly connected to the passive mounting bracket 4-15 of the third ankle joint drive module.

[0096] In this embodiment, the ankle joint mechanism 4 forms the first, second, and third joint portions of the ankle joint, which can respectively achieve vertical rotation, pitch, and roll, realizing three degrees of freedom of ankle joint movement. The axis of the drive end of the first ankle joint drive module 4-1 is perpendicular to the left-right direction of the bipedal robot, the axis of the drive end of the second ankle joint drive module 4-4 is parallel to the left-right direction of the bipedal robot, and the axis of the drive end of the third ankle joint drive module 4-14 is parallel to the front-back direction of the bipedal robot. Furthermore, the axes of the drive ends of the first, second, and third ankle joint drive modules 4-1, 4-4, and 4-14 intersect at the same point, thereby better mimicking human ankle joint movement and making the bipedal robot's movement more human-like.

[0097] In this embodiment, the specific configuration structure of the hip joint first joint drive module 2-1, hip joint second joint drive module 2-8, hip joint third joint drive module 2-13, knee joint drive module 3-2, ankle joint first joint drive module 4-1, ankle joint second joint drive module 4-4, and ankle joint third joint drive module 4-14 all adopt existing technologies, such as a component structure consisting of a reducer, a driver, and a brushless motor encapsulated in the same housing.

[0098] This invention uses a combination of model-driven (zero moment point) and data-driven (proximal policy optimization reinforcement learning algorithm) approaches to design a control algorithm, mainly including the following steps:

[0099] Step 1: Initialize the PPO (Proximal Policy Optimization) reinforcement learning algorithm and the ZMP (Zero Moment Point) method to initialize the robot state.

[0100] Step 2: If step one has already been initialized, this step is skipped; collect the robot's state s. t To obtain the dynamic parameters of the bipedal robot, including mass distribution and support polygon; among which, q is the joint angle. For joint velocity, P COM p is the location of the center of mass. ZMP At the zero torque point, p support To support polygons;

[0101] Step 3: Based on the center of mass acceleration and robot mass distribution, use the ZMP simplified model to calculate the position of the zero torque point, determine its deviation from the center of the supporting polygon, and generate the compensation torque for balance correction.

[0102] Step 4: Use PPO for policy optimization. Collect complete trajectory data including state, action, reward and next state. Combine ZMP stability, energy consumption, trajectory tracking and survival and other reward designs. Calculate the advantage function using generalized advantage estimation. This will guide the policy network to continuously adjust its parameters through loss function optimization in the next step.

[0103] Step 5: Using the advantage function generated in Step 4, the PPO algorithm stabilizes policy updates by constructing a pruning target loss function, improves the accuracy of future reward estimation by constructing a value function loss, and encourages policy exploration by constructing an entropy loss. The three are then combined into a total loss function according to set weights. The total loss function is optimized to obtain the PPO optimized original action.

[0104] Step 6: After the strategy generates the action, to ensure the stability of the robot's movement, a hybrid control method is used to determine whether the zero torque point (ZMP) deviates from the center of the supporting polygon. When the deviation exceeds 0.1m, the ZMP compensation torque calculated in Step 3 is used to correct the original PPO action; otherwise, the action output by PPO is directly used as the final control command, and the amplitude of the final action is limited by a function to ensure that the joint torque is within a safe and controllable range, avoiding system instability or equipment damage due to excessive control.

[0105] Step 7: Combine the mixed control action a from step 6 final The signal is sent to the actuators to control the torque τ of each actuator.

[0106] Step 8: Update the PPO parameters and update the status s. t+1 This includes joint angles, velocities, centroids, ZMP positions, etc., while updating the policy network parameters and value function network parameters, and using the updated network for the next round of environmental interaction;

[0107] Step 9: Repeat steps 2 to 8 until the total loss function is small enough to achieve the training objective, and until the final policy exhibits stable and efficient movement capabilities during walking.

[0108] The specific steps in this embodiment are as follows:

[0109] Step 1: Initialize the parameters of the Proximal Policy Optimization (PPO) reinforcement learning algorithm and the Zero Moment Point (ZMP) method. The ZMP method is a zero moment point method. Initialize the robot state s. t When time t=0

[0110] Where q0 is the initial joint angle. The initial joint velocities are all zero, PCOM0 Initial centroid position: P COM0 =x COM0 ,y COM0 ,z COM0 p ZMP0 p is the initial zero torque point position. ZMP0 =(x ZMP0 ,y ZMP0 ), p support To support polygons:

[0111] state s t The state space is a description of the robot's current environment and internal state, typically containing...

[0112] The following are some key parameters:

[0113] Joint angle q: The angles of the robot's joints (e.g., the joint angles of a bipedal robot with 7 degrees of freedom). These joint angles determine the robot's posture at a given moment.

[0114] Joint speed The angular velocities of each joint of a robot describe the robot's motion state and directly affect its dynamic behavior.

[0115] Location of the center of mass P COM The position of the center of mass reflects the robot's center of gravity. The position of the center of mass affects stability during movement, especially when performing gait.

[0116] Zero torque point position p ZMP Zero Moment Point (ZMP) position is used to evaluate the stability of a robot. The relative position of the ZMP plays a crucial role in whether the robot can maintain balance.

[0117] Before training begins, the parameters of the PPO policy and ZMP method need to be initialized: the PPO policy is responsible for learning and optimizing the robot's joint control methods. The ZMP method is responsible for ensuring the robot's stability during walking. Step one is the preparatory work for the entire control method, providing the necessary dynamics and stability constraints for subsequent policy sampling, optimization, and control.

[0118] Step 2: If Step 1 has already been initialized, this step is skipped; collect the robot's state s. t To obtain the dynamic parameters of the bipedal robot, including mass distribution and support polygon; among which, q is the joint angle. For joint velocity, P COM p is the location of the center of mass. ZMP At the zero torque point, p ZMP =(xZMP ,y ZMP ), p support To support polygons.

[0119] This step involves collecting training data for PPO optimization. Feedback data (state, action, reward) obtained after the robot executes the current policy will be used for subsequent policy optimization and adjustment.

[0120] Step 3: Perform ZMP stability calculations using a simplified single rigid body model to calculate the zero-moment point position x. ZMP and y ZMP :

[0121]

[0122] in Let x be the acceleration of the center of mass in the x-axis direction. Let x be the acceleration of the center of mass in the y-axis direction. Let p be the acceleration of the center of mass in the z-axis direction. ZMP The bipedal robot is stable under the influence of P, where P is the vertex of the current supporting foot, {v1,v2,v3,v4}.

[0123] Location of the center of mass P COM Calculations using the robot's kinematic model require knowledge of the mass of each link and the position of its center of mass.

[0124]

[0125] Center of mass acceleration Alternatively, the torso acceleration can be measured directly using an inertial measurement unit (IMU) and then converted to the world coordinate system.

[0126] Calculate the deviation of ZMP relative to the supporting polygon, i.e., the distance from ZMP to the center point of the supporting polygon: Δp ZMP =p ZMP -p ref If the ZMP exceeds the supporting polygon, it indicates that the robot's balance is threatened. ZMP This is the location of the currently calculated zero torque point, p. ref It is a reference position, usually p. support The center point of the supporting polygon.

[0127] The robot's motion is adjusted by calculating compensation torque, a process that typically uses a Jacobian matrix to map torque compensation in the robot's joint space.

[0128] The ZMP compensation torque can be calculated using the following formula:

[0129] Where Δτ ZMPIt is ZMP-compensated torque; It is the transpose of the Jacobian matrix from the center of mass to the joint space; Δp ZMP It is the offset of ZMP.

[0130] The compensation torque Δτ is obtained ZMP Used for subsequent steps.

[0131] Step 4: Use PPO for policy optimization. First, let the robot use the current randomization policy π. θ Interact with the simulation environment, perform motion simulation using initial torque, and collect trajectory data. The collected data includes: state... Action a t Rewards r t Next state s t+1 .

[0132] Action a t Action is the decision output by the robot based on the current strategy, which directly affects the robot's movement. Joint torque is the driving torque of the robot joints, used to control the joint angles.

[0133] Reward r t The reward serves as a feedback signal to the robot, guiding the optimization of the strategy. In this method, the reward function is typically a weighted sum of multiple factors, including:

[0134] ZMP stability bonus r ZMP If the ZMP exceeds the supporting polygon, a negative reward is given, forcing the robot to improve its stability.

[0135] Energy reward r energy This encourages robots to minimize energy consumption when performing tasks. For example, energy consumption can be calculated using the sum of squares of joint torques.

[0136] Tracking reward r track The penalty is the difference between the joint angle and the reference trajectory to ensure that the robot moves along the predetermined trajectory.

[0137] Survival reward r alive The robot is rewarded for continuing to run without falling, preventing the robot from being interrupted during the simulation.

[0138] Strategy π θA policy is a rule or mapping by which an agent (in this case, a robot) takes a certain action given a state. The policy is continuously optimized through a learning process. The goal of optimization is to adjust the policy so that the agent can make better decisions in the environment, thereby maximizing cumulative rewards. Given a state, the policy does not always choose a specific action, but rather selects actions based on a probability distribution, where π represents the policy function; θ represents all parameters of the neural network, such as the weights and biases in a multilayer perceptron.

[0139] The purpose of collecting this data is to estimate the performance of the policy, train the value function, and optimize the policy itself.

[0140] Estimate the potential reward from the current state by calculating the value function network's response to the current state s. t Value estimation:

[0141] Dominance function calculation:

[0142] Where δ t =r t +γV(s t+1 )-V(s t )

[0143] Where γ is the discount factor, λ is the smoothing parameter for generalized advantage estimation, and V(s) t ) is the value function network for the current state s t Value estimation, V(s) t+1 δ: Value function's estimate of the value of the next state. t The current TD error is represented by k, the offset of the future time step is k, and the total time steps of the trajectory are T: the total number of steps of a complete trajectory (from start to end).

[0144] The TD error is the temporal difference error at the current time step t. It is an important measure in reinforcement learning to measure the difference between the estimated state value of the current policy and the actual reward obtained.

[0145] Where r t+k The reward function r at time t+k t .

[0146] Calculate the reward function r t =ω1r ZMP +ω2r energy +ω3r track +ω4r alive

[0147] Where r ZMPZMP stability reward (core constraint):

[0148]

[0149] The reward function is given if ZMP falls within the support region, i.e., Δp. ZMP 2 <0.01m: Reward is 0 (stable);

[0150] If ZMP deviates: a negative reward is given based on the deviation distance (the greater the deviation, the greater the penalty);

[0151] This mechanism forces the network to learn to keep the ZMP within the supporting polygon; otherwise, it cannot obtain positive rewards.

[0152] Energy reward r energy Punishment for large actions consumes energy. To punish large joint torques, the control output of the motion should be "small and stable"; τ i : Torque of the i-th joint; Energy reward helps learn a more energy-efficient gait.

[0153] Gait tracking reward r track Fitted reference trajectory: r track =-γ||qq ref || 2 Meaning: Encourage the current joint angle q to be as close as possible to the reference trajectory q. ref γ: Weights in the tracking loss. This ensures the movements are not erratic and form a regular gait. q ref This refers to the ideal joint angle trajectory that a robot should follow, which is a sinusoidal trajectory function for periodic walking.

[0154] Survival bonus: r alive Maintain an upright posture and prevent falls: r alive =+1.0;

[0155] Meaning: A positive reward is given for each successful step; the reward stops if the robot falls or a termination condition is triggered. This simple yet effective reward strongly encourages the strategy to keep the robot "keeping moving".

[0156] Step 5: Utilize the advantage function A generated in Step 4 t Three loss functions are constructed, and the current policy π is evaluated through these loss functions. θ And value function V(s) t Whether it's good or bad, the loss function outputs a "loss value". The larger the value, the worse the current policy is. The smaller the value, the more stable the policy is and the better it can control the robot. The neural network can "learn" a better policy or prediction by optimizing the loss function.

[0157] I. Constructing the pruning target loss function LCLIP (θ) ensures that the policy update magnitude is not too large, thus making training more stable.

[0158] L CLIP (θ)=E t [min(r t (θ)A t ,clip(r t (θ), 1-ε, 1+ε)A t )]

[0159] Where: θ is the policy parameter, r t (θ) is the importance sampling ratio, clip(x,a,b) is the clipping function that restricts x to [a,b], and r t (θ) represents the probability ratio between the old and new strategies. ε is the clipping factor, which is usually taken as 0.1 to 0.3 to limit the range of strategy variation.

[0160] II. Value Function Loss (Auxiliary Optimization) L value (θ)=(V θ (s t )-R t ) 2

[0161] The loss function of the value function affects the accuracy of the neural network's prediction of the future total reward, thus impacting the accuracy of the advantage function estimation. Where V... θ (s t ) represents the current policy pair for state s t The value estimate is generated by the output of a neural network. t R is the actual reward at time step t, which is the weighted sum of all future rewards accumulated from that moment onwards. t =r t +γr t+1 +γ 2 r t+2 +…+γ n r t+n .

[0162] III. Entropy Loss (π) θ )=-∑π θ logπ θ

[0163] A higher entropy loss indicates a more diversified strategy and more varied action exploration, preventing the agent from falling into a fixed strategy too quickly.

[0164] Total loss function L Total Combinatorial optimization combines the three weighted factors into a total loss:

[0165] L Total =L CLIP-c1L value (θ)+c2Entropy(π θ )

[0166] Where c1 and c2 are weight hyperparameters;

[0167] The next step is to optimize the total loss function by using gradient descent to update the parameters of the policy network and the value function network. The goal of this policy network is to output the probability distribution of the optimal action a given state s: π θ (a|s)

[0168] Where: π θ : Represents the policy function; θ: Represents all parameters of the neural network (e.g., weights and biases in a multilayer perceptron); Input: state s; Output: action a.

[0169] Obtain the original action for PPO optimization: a PPO =π θ (a|s), where a PPO Generate the original action for PPO.

[0170] Step 6: Generate hybrid control actions and perform ZMP stability testing, using the torque Δτ compensated by ZMP generated in Step 3. ZMP And perform motion detection.

[0171] If the deviation of ZMP relative to the supporting polygon is greater than 0.1m, meaning the zero-moment point is not located within the supporting polygon, ZMP compensation is applied, resulting in the following action: a final =saturate(a PPO +Δτ ZMP ,a max ).

[0172] If the deviation of ZMP relative to the supporting polygon is less than 0.1m, that is, the zero moment point is located within the supporting polygon, the original PPO action a can be used directly. final =saturate(a PPO ,a max )

[0173] The `saturate()` function is used to limit the amplitude of an action or torque. Its core purpose is to ensure that the output value is within a physically feasible range, avoiding hardware damage or control instability caused by excessively large values. max This represents the maximum value of each joint's movement.

[0174]

[0175] Step 7: Combine the mixed control action a from step 6 finalThe signal is sent to the actuators to control the torque τ of each actuator.

[0176] Step 8: Update the PPO parameters and update the status s. t+1 This includes joint angles, velocities, centroids, ZMP positions, etc., while updating the policy network parameters and value function network parameters, and using the updated network for the next round of environmental interaction.

[0177] Step 9: Repeat steps 2 through 8 until the total loss function is sufficiently small to achieve the training objective, and until the final policy exhibits stable and efficient motion capabilities during walking. The objective includes three joint target values: some tasks may require the robot to reach specific joint angles at a given time step; centroid target value: in balance tasks, it is expected that the robot maintains a stable centroid position; ZMP target value: it is expected that the ZMP remains within the supporting polygon. Refer to [link to detailed process] for further information. Figure 6 .

[0178] Figure 6 This paper demonstrates a robot control training process that integrates PPO reinforcement learning and ZMP stability constraints. From state initialization, environmental interaction, action generation, ZMP stability judgment and compensation, to the closed-loop structure of hybrid control execution and policy network update, it reflects the collaborative optimization mechanism of model-driven and data-driven approaches. This ensures that the robot always meets the dynamic stability requirements while learning the control strategy, and finally improves walking performance through iterative loops to achieve a stable, energy-saving, and controllable intelligent gait.

[0179] Initialization phase (top left)

[0180] "Initialize the near-end policy optimization algorithm and zero moment point method": Corresponding to step 1, initialize the PPO policy network parameters and the dynamic information such as mass distribution and supporting polygons required by the ZMP method.

[0181] Status acquisition phase (top right)

[0182] "Collect robot status and obtain robot update parameters": Also step 1, obtain the robot's initial or new joint angles, speed, center of mass position and support polygons after the last iteration.

[0183] Environmental observation (Central region, first line)

[0184] The two inputs are merged to read the current environmental state, which is to summarize the robot's own state with physical quantities such as ground reaction force and zero torque point position.

[0185] Action and physical quantity branches

[0186] Left branch:

[0187] "Motion: joint position, joint velocity, and center of mass position" → provides input to the PPO Actor network.

[0188] “Perform PPO strategy optimization” → In step 4, calculate the advantage function A based on the sampled state-action-reward-next state. t And construct three losses to obtain the original action a. PPO .

[0189] Right branch:

[0190] "Ground reaction force, zero moment point location" → used for ZMP stability calculation in step 3 to determine whether the current ZMP falls within the supporting polygon.

[0191] "Does ZMP stability calculation meet?" → If not, proceed to the "Compensation Action" branch, utilizing the pre-calculated compensation torque Δτ. ZMP If so, skip the compensation. Motion composition (middle section, bottom).

[0192] "Final Hybrid Motion Synthesis": Based on the ZMP detection results, select a hybrid motion or only the original PPO motion and limit its amplitude to obtain a safe and feasible control command. Execution and feedback.

[0193] “Execute action a_final to drive joint motor movement”: Outputs the final action to the simulation or real robot, generating new environmental feedback.

[0194] PPO Parameter Update (Part 2)

[0195] "PPO parameter update, state update, policy network and value function network parameters update": corresponding to steps 5 to 9, recalculate the reward, TD error and advantage function based on the latest sampled data, and update the policy network parameters and value function network parameters by minimizing the total loss through gradient descent.

[0196] Iterative loop: The last "loop" arrow points back to environmental observation until the policy converges and the overall control objective is achieved.

[0197] In summary, the hip joint uses three drive motors to achieve roll, pitch, and rotation, respectively, realizing three degrees of freedom control of the hip joint. The knee joint uses a single degree of freedom to control the rotation of the lower leg. The ankle joint uses three drive motors to achieve rotation, pitch, and roll, respectively, realizing three degrees of freedom control of the ankle joint. Overall, each leg achieves seven degrees of freedom, one more degree of freedom than existing bipedal robots, making it more closely resemble human walking. Furthermore, direct drive control by joint motors reduces torque loss, resulting in smoother and more natural bipedal robot movement. Extensive use of 3D printing intelligent manufacturing ensures structural strength while reducing the overall weight of the robot. Adaptive gait control is achieved, balancing stability, energy consumption, and motion efficiency, ultimately demonstrating high robustness in complex dynamic environments.

[0198] The proposed seven-DOF bipedal robot control method combines the PPO reinforcement learning algorithm with ZMP stability analysis, and achieves stable gait control of the robot through a rigorous process. First, the PPO strategy and ZMP method parameters are initialized (Step 1), and the robot's current state is collected (Step 2), providing an environmental and dynamic basis for subsequent decision-making. Then, the ZMP position and offset are calculated using a single rigid body model (Step 3), the robot's balance is evaluated in real time, and raw joint torque actions are generated through PPO during the strategy sampling phase (Step 4). Simultaneously, a multi-objective reward function (including ZMP stability, energy consumption, trajectory tracking, and survival reward) is constructed, and the strategy optimization is guided by the dominance function estimation and triple loss functions (pruning loss, value function loss, and entropy loss) (Step 5). Next, the raw PPO actions and ZMP compensated actions are dynamically mixed according to the ZMP offset to generate the final joint actions after amplitude constraints (Steps 6-7), driving the robot motors to complete gait control. Finally, complete trajectory data is collected, and the strategy and value function network are iteratively optimized repeatedly (Steps 8-9), ultimately achieving the goal of enabling the robot to learn energy-saving, stable, and precise walking control while maintaining ZMP stability. The entire process, through a combination of data-driven and model-driven approaches, ensures the stability, safety, and adaptability of the robot control process.

Claims

1. A seven-DOF bipedal robot based on heavy-duty direct-drive joints, characterized in that: It includes a pelvis (1) and two sets of leg structures arranged symmetrically on the left and right sides below the pelvis (1). Each set of leg structures includes a hip joint mechanism (2), a knee joint mechanism (3), and an ankle joint mechanism (4) connected from top to bottom. Each set of the hip joint mechanism (2) of the leg structure has three degrees of freedom arranged in sequence: a degree of freedom to rotate about an axis parallel to the front-back direction of the bipedal robot, a degree of freedom to rotate about an axis parallel to the left-right direction of the bipedal robot, and a degree of freedom to rotate about an axis perpendicular to the left-right direction of the bipedal robot. Each set of leg structures has a knee joint mechanism (3) with one degree of freedom to rotate about an axis parallel to the left and right direction of the bipedal robot. Each ankle joint mechanism (4) of the leg structure has three degrees of freedom: a degree of freedom to rotate about an axis perpendicular to the left and right direction of the bipedal robot, a degree of freedom to rotate about an axis parallel to the left and right direction of the bipedal robot, and a degree of freedom to rotate about an axis parallel to the front and back direction of the bipedal robot.

2. The seven-DOF bipedal robot based on heavy-duty direct-drive joints according to claim 1, characterized in that: Each hip joint mechanism (2) of the leg structure includes a first hip joint drive module (2-1), a first hip joint frame (2-2), a first hip joint drive component (2-3), a first hip joint drive connector (2-4), a second hip joint passive connector (2-5), a second hip joint passive bearing (2-6), a second hip joint passive mounting frame (2-7), a second hip joint drive module (2-8), a second hip joint drive module frame (2-9), a second hip joint drive component (2-10), a second hip joint drive connector (2-11), a third hip joint mounting component (2-12), a third hip joint drive module (2-13), a third hip joint frame (2-14), a third hip joint drive component (2-15), a first hip joint fixation component (2-16), a first hip joint passive bearing (2-17), and a first hip joint passive mounting frame (2-18). The first hip joint frame (2-2) and the first hip joint passive mounting frame (2-18) together serve as the fixed ends of the entire hip joint mechanism (2) and are respectively fixed to the lower side of the pelvis (1). The positions of the first hip joint frame (2-2) and the first hip joint passive mounting frame (2-18) are corresponding front to back, and the first hip joint frame (2-2) is located behind the first hip joint passive mounting frame (2-18). The outer shell of the first hip joint drive module (2-1) is installed behind the first hip joint frame (2-2). The drive end of the first hip joint drive module (2-1) is connected to the first hip joint drive member (2-3). The drive connector (2-4), the passive mounting bracket for the second hip joint (2-7), the drive module frame for the second hip joint (2-9), and the fixation component for the first hip joint (2-16) are respectively located between the first hip joint frame (2-2) and the passive mounting bracket for the first hip joint (2-18). The rear end of the first drive connector (2-4) is connected to the first drive component (2-3). The rear ends of the passive mounting bracket for the second hip joint (2-7) and the rear ends of the drive module frame for the second hip joint (2-9) are respectively connected to the front ends of the first drive connector (2-4). The fixation component for the first hip joint (2-16) is connected to the passive mounting bracket for the second hip joint. The front end of the frame (2-7) and the front end of the hip joint second joint drive module frame (2-9) are fixedly connected. The hip joint first joint fixing member (2-16) is rotatably connected to the hip joint first joint passive mounting frame (2-18) through the hip joint first joint passive bearing (2-17). The outer shell of the hip joint second joint drive module (2-8) is installed between the hip joint second joint passive mounting frame (2-7) and the hip joint second joint drive module frame (2-9). The hip joint second joint drive member (2-10) is located on the side of the hip joint second joint drive module frame (2-9) away from the hip joint second joint passive mounting frame (2-7). One end of the hip joint second joint drive member (2-10) is connected to the hip joint second joint drive module frame (2-9). The drive end of the second joint drive module (2-8) is connected to the hip joint second joint passive connector (2-5), which is located on the side of the hip joint second joint passive mounting bracket (2-7) away from the hip joint second joint drive module frame (2-9). The hip joint second joint passive connector (2-5) is rotatably connected to the hip joint second joint passive mounting bracket (2-7) via a hip joint second joint passive bearing (2-6). Below the hip joint second joint drive module (2-8), from top to bottom, are respectively provided a hip joint third joint mounting component (2-12) and a hip joint third joint frame (2-14). The hip joint third joint mounting component (2-12) is connected to the hip joint second joint passive connector (2-5).The third hip joint frame (2-14) is connected to the other end of the second hip joint drive component (2-10) via the second hip joint drive connector (2-11). The outer shell of the third hip joint drive module (2-13) is installed between the third hip joint mounting component (2-12) and the third hip joint frame (2-14). The drive end of the third hip joint drive module (2-13) faces downward and is connected to the third hip joint drive component (2-15). The third hip joint drive component (2-15) serves as the drive end of the entire hip joint mechanism (2) and is connected to the fixed end of the knee joint mechanism (3). The axis of the drive end of the first hip joint drive module (2-1) is parallel to the front-back direction of the bipedal robot, the axis of the drive end of the second hip joint drive module (2-8) is parallel to the left-right direction of the bipedal robot, and the axis of the drive end of the third hip joint drive module (2-13) is perpendicular to the left-right direction of the bipedal robot. The axis of the drive end of the first hip joint drive module (2-1), the axis of the drive end of the second hip joint drive module (2-8), and the axis of the drive end of the third hip joint drive module (2-13) intersect at the same point.

3. A seven-DOF bipedal robot based on heavy-duty direct-drive joints according to claim 1, characterized in that: Each knee joint mechanism (3) of the leg structure includes a hip joint third joint drive connector (3-1), a knee joint drive module (3-2), a knee joint drive module mounting component (3-3), a knee joint drive component (3-4), a knee joint drive connector (3-5), a knee joint drive module passive mounting bracket (3-6), a knee joint passive bearing (3-7), a knee joint passive mounting bracket (3-8), an ankle joint first joint drive module mounting component (3-9), and an ankle joint first joint drive mounting component (3-10). The upper end of the hip joint third joint drive connector (3-1) serves as the fixed end of the entire knee joint mechanism (3) and is connected to the drive end of the entire hip joint mechanism (2). The upper end of the knee joint drive module mounting piece (3-3) and the upper end of the knee joint drive module passive mounting frame (3-6) are respectively connected to the lower end of the hip joint third joint drive connector (3-1). The outer shell of the knee joint drive module (3-2) is installed between the knee joint drive module mounting piece (3-3) and the knee joint drive module passive mounting frame (3-6). The knee joint drive piece (3-4) is located on the side of the knee joint drive module mounting piece (3-3) away from the knee joint drive module passive mounting frame (3-6). One end of the knee joint drive piece (3-4) is fixedly connected to the drive end of the knee joint drive module (3-2). The knee joint passive mounting frame (3-8) is located on the knee joint drive module passive mounting frame (3-1). On the side away from the knee joint drive module mounting component (3-3) of 6), the knee joint passive mounting bracket (3-8) is rotatably connected to the knee joint drive module passive mounting bracket (3-6) through the knee joint passive bearing (3-7). The knee joint drive module (3-2) is provided with ankle joint first joint drive module mounting component (3-9) and ankle joint first joint drive mounting component (3-10) from top to bottom. The ankle joint first joint drive module mounting component (3-9) is connected to the knee joint passive mounting bracket (3-8). The ankle joint first joint drive mounting component (3-10) is fixed to the other end of the knee joint drive component (3-4) through the knee joint drive connector (3-5). The ankle joint first joint drive module mounting component (3-9) and the ankle joint first joint drive mounting component (3-10) together serve as the drive end of the entire knee joint mechanism (3) and are connected to the fixed end of the ankle joint mechanism (4). The axis of the drive end of the knee joint drive module (3-2) is parallel to the left-right direction of the bipedal robot.

4. A seven-DOF bipedal robot based on heavy-duty direct-drive joints according to claim 1, characterized in that: The degree of freedom of rotation of the knee joint mechanism (3) of each leg structure about an axis parallel to the left and right direction of the bipedal robot is in the same plane as the degree of freedom of rotation of the hip joint mechanism (2) of the same leg structure about an axis parallel to the left and right direction of the bipedal robot.

5. A seven-DOF bipedal robot based on heavy-duty direct-drive joints according to claim 1, characterized in that: Each ankle joint mechanism (4) of the leg structure includes an ankle joint first joint drive module (4-1), an ankle joint first joint drive component (4-2), an ankle joint first joint drive connector (4-3), an ankle joint second joint drive module (4-4), an ankle joint second joint drive module mounting component (4-5), an ankle joint second joint drive component (4-6), an ankle joint second joint drive connector (4-7), an ankle joint second joint drive module passive mounting frame (4-8), an ankle joint second joint passive bearing (4-9), an ankle joint second joint passive connector (4-10), an ankle joint third joint passive connector (4-11), an ankle joint third joint passive bearing (4-12), an ankle joint third joint drive module passive mounting frame (4-13), an ankle joint third joint drive module (4-14), an ankle joint third joint drive module passive mounting frame (4-15), a foot (4-16), and an ankle joint third joint drive component (4-17). The outer shell of the first ankle joint drive module (4-1) serves as the fixed end of the entire ankle joint mechanism (4) and is connected to the drive end of the entire knee joint mechanism (3). The drive end of the first ankle joint drive module (4-1) faces downward and is connected to the first ankle joint drive component (4-2). The first ankle joint drive connector (4-3) is located below the first ankle joint drive module (4-1). The upper end of the first ankle joint drive connector (4-3) is connected to the first ankle joint drive component (4-2). The upper end of the second ankle joint drive module mounting component (4-5) and the passive mounting bracket (4-8) of the second ankle joint drive module are connected to each other. The ankle joint second joint drive module (4-4) is connected to the lower end of the ankle joint first joint drive connector (4-3) on the left and right respectively. The outer shell of the ankle joint second joint drive module (4-4) is installed between the ankle joint second joint drive module mounting component (4-5) and the ankle joint second joint drive module passive mounting bracket (4-8). The ankle joint second joint drive component (4-6) is located on the side of the ankle joint second joint drive module mounting component (4-5) away from the ankle joint second joint drive module passive mounting bracket (4-8). One end of the ankle joint second joint drive component (4-6) is connected to the drive end of the ankle joint second joint drive module (4-4). The ankle joint second joint passive connector (4-10) is located at the lower end of the ankle joint first joint drive connector (4-3). The passive mounting bracket (4-8) of the ankle joint second joint drive module is located on the side away from the ankle joint second joint drive module mounting member (4-5). The ankle joint second joint passive connector (4-10) is rotatably connected to the ankle joint second joint drive module passive mounting bracket (4-8) via the ankle joint second joint passive bearing (4-9). The foot (4-16) is located below the ankle joint second joint drive module (4-4). The upper side of the foot (4-16) is respectively provided with the ankle joint third joint passive connector (4-11) and the ankle joint third joint drive member (4-17). The ankle joint third joint passive connector (4-11) is located on the ankle joint second joint drive module. The ankle joint third joint drive component (4-17) is located on the front side of the ankle joint third joint passive connector (4-11). The ankle joint third joint drive module passive mounting bracket (4-13), the ankle joint third joint drive module (4-14), and the ankle joint third joint drive module passive mounting bracket (4-15) are arranged sequentially from back to front between the ankle joint third joint passive connector (4-11) and the ankle joint third joint drive component (4-17). The ankle joint third joint drive module passive mounting bracket (4-13) and the ankle joint third joint passive connector (4-11) are rotatably connected by the ankle joint third joint passive bearing (4-12).The passive mounting bracket (4-13) of the ankle joint third joint drive module is connected to the passive connector (4-10) of the ankle joint second joint. The passive mounting bracket (4-15) of the ankle joint third joint drive module is connected to the other end of the ankle joint second joint drive component (4-6) through the ankle joint second joint drive connector (4-7). The outer shell of the ankle joint third joint drive module (4-14) is fixedly connected to both the ankle joint third joint drive module passive mounting bracket (4-13) and the ankle joint third joint drive module passive mounting bracket (4-15). The drive end of the ankle joint third joint drive module (4-14) is fixedly connected to the ankle joint third joint drive module passive mounting bracket (4-15). The axis of the driving end of the first ankle joint driving module (4-1) is perpendicular to the left-right direction of the bipedal robot, the axis of the driving end of the second ankle joint driving module (4-4) is parallel to the left-right direction of the bipedal robot, and the axis of the driving end of the third ankle joint driving module (4-14) is parallel to the front-back direction of the bipedal robot; the axes of the driving ends of the first ankle joint driving module (4-1), the second ankle joint driving module (4-4), and the third ankle joint driving module (4-14) intersect at the same point.

6. A control method for a seven-DOF bipedal robot based on heavy-duty direct-drive joints, characterized in that, Controlling the joint angles of a robot using model-driven and data-driven methods includes the following steps: 1) Initialize the PPO reinforcement learning algorithm and ZMP method to construct the robot state. And initialize, where q is the joint angle, For joint velocity, P COM p is the location of the center of mass. ZMP At the zero torque point, p support The supporting polygon is the bottom surface of the foot; 2) Based on the center of mass position and robot mass distribution, the position of the zero torque point is calculated using the ZMP simplified model, and its deviation from the center of the supporting polygon is determined to generate the ZMP compensation torque for balance correction. 3) Use the PPO reinforcement learning algorithm for policy optimization, collecting data including state s. t Action a t Rewards r t and the next state s t+1 The complete trajectory data is combined with multiple reward designs such as ZMP stability, energy consumption, trajectory tracking and survival. The advantage function is calculated using generalized advantage estimation, which guides the next step of the policy network to continuously adjust its parameters through loss function optimization. 4) Using the advantage function generated in step 3), the PPO reinforcement learning algorithm stabilizes policy updates by constructing a pruning target loss function, improves the accuracy of future reward estimation by constructing a value function loss, and encourages policy exploration by constructing an entropy loss. The three are then combined into a total loss function according to set weights. The total loss function is optimized to obtain the PPO original action. 5) Based on the original PPO motion, a hybrid control method is used to determine whether the zero torque point deviates from the center of the support polygon. When the deviation exceeds the threshold, the original PPO motion is corrected using the ZMP compensation torque calculated in step 2), resulting in hybrid control motion a. final Otherwise, the action output by the PPO reinforcement learning algorithm is directly used as the final control command, and the amplitude of the final action is limited by a function to obtain the hybrid control action a. final ; 6) Combine the mixed control action a from step 5). final The signal is sent to the actuators to control the torque τ of each actuator. 7) Update the PPO parameters and update the status s. t+1 Simultaneously, update the policy network parameters and value function network parameters, and use the updated network for the next round of environmental interaction; 8) Repeat steps 2) to 7) until the total loss function is less than the threshold, thus achieving the training objective and obtaining the final policy; use the final policy to control the robot.

7. The control method for a seven-DOF bipedal robot based on heavy-duty direct-drive joints according to claim 6, characterized in that, Step 2) Based on the center of mass position and robot mass distribution, calculate the zero-torque point position using the ZMP simplified model, determine its deviation from the center of the supporting polygon, and generate a compensation torque for balance correction, including the following steps: (1) Location of zero torque point x ZMP y ZMP It is obtained through the following formula; Among them, the acceleration of the center of mass Obtained through measurement, Let x be the acceleration of the center of mass in the x-axis direction. Let x be the acceleration of the center of mass in the y-axis direction. Let p be the acceleration of the center of mass in the z-axis direction; when p ZMP When ∈P, it indicates that the bipedal robot is stable, where P is the region formed by the vertices {v1,v2,v3,v4} of the current supporting polygon; g is the acceleration due to gravity, P COM (x COM ,y COM ,z COM ) represents the position of the centroid, p ZMP (x ZMP ,y ZMP () indicates the location of the zero torque point; Location of the center of mass P COM Based on the mass of each link and the position of its center of mass, we obtain: Where, m i p represents the mass of the i-th link. i This indicates the position of the centroid of the i-th link; (2) Calculate p ZMP The deviation relative to the supporting polygon, i.e., p ZMP To the center point p of the supporting polygon ref Distance: Δp ZMP =p ZMP -p ref ; When p ZMP If the robot moves beyond the supporting polygon, it indicates that the robot's balance is threatened. The robot's movement is adjusted by calculating compensating torque. Where, Δτ ZMP It is ZMP-compensated torque. It is the transpose of the Jacobian matrix from the center of mass to the joint space, Δp ZMP It is p ZMP The offset.

8. The control method for a seven-DOF bipedal robot based on heavy-duty direct-drive joints according to claim 6, characterized in that, Step 3) Use the PPO reinforcement learning algorithm to optimize the policy and collect data including state s. t Action a t Rewards r t and the next state s t+1 The complete trajectory data is used, and combined with multiple reward designs such as ZMP stability, energy consumption, trajectory tracking, and survival, the advantage function is calculated using generalized advantage estimation. This guides the next step of the policy network to continuously adjust its parameters through loss function optimization, including the following steps: Value function network for the current state s t Value estimation: Dominance function calculation: δ t =r t +γV(s t+1 )-V(s t ) Where γ is the discount factor, λ is the smoothing parameter for the generalized advantage estimation, and V(s) t ) is the value function network for the current state s t Value estimation, V(s) t+1 ) represents the value estimate of the next state by the value function, δ t This represents the current TD error, k is the offset of the future time step, T is the total time steps of the trajectory, and t is the time. Reward function: r t =ω1r ZMP +ω2r energy +ω3r track +ω4r alive Where, r t+k The reward function r at time t+k t ; r ZMP ZMP stability bonus: Where, Δp ZMP It is the offset of ZMP; Energy consumption reward τ i Let be the torque of the i-th joint; Gait tracking reward r track =-γ||qq ref || 2 q ref It is the expected joint angle trajectory; Survival reward r alive =+1.

0.

9. A control method for a seven-DOF bipedal robot based on heavy-duty direct-drive joints according to claim 6, characterized in that, Step 4) Using the advantage function generated in Step 3), the PPO reinforcement learning algorithm stabilizes policy updates by constructing a pruning target loss function, improves the accuracy of future reward estimation by constructing a value function loss, and encourages policy exploration by constructing an entropy loss. These three are then combined with set weights into a total loss function. Optimizing the total loss function yields the original PPO action, including the following steps: Construct the pruning target loss function: L CLIP (θ)=E t [min(r t (i)A t ,clip(r t (θ),1-ε,1+ε)A t )]; Among them, E t This represents the expectation, where θ is the policy parameter and r is the expected value. t (θ) is the importance sampling ratio, clip(x,a,b) is the clipping function that restricts x to [a,b], and r t (θ) represents the probability ratio between the old and new strategies. ε is the cutting factor; Value function loss L value (θ)=(V θ (s t )-R t ) 2 ; Among them, V θ (s t ) represents the current policy pair for state s t Value estimate, R t It is the actual reward at time step t, that is, the weighted sum of all future rewards accumulated from that moment onwards; Entropy loss (π) θ )=-∑π θ logπ θ ; Total loss function L Total Combinatorial optimization combines the three weighted factors into a total loss: L Total =L CLIP -c1L value (θ)+c2Entropy(π θ ) Where c1 and c2 are weight hyperparameters; Optimize the total loss function, update the policy network parameters and value function network parameters using gradient descent, and obtain the original PPO action 'a'. PPO .

10. A control method for a seven-DOF bipedal robot based on heavy-duty direct-drive joints according to claim 6, characterized in that, Step 5) After the strategy generates the action, the hybrid control method is used to determine whether the zero torque point deviates from the center of the support polygon. When the deviation exceeds the threshold, the ZMP compensation torque calculated in step 2) is used to correct the original PPO action. Otherwise, the action output by the PPO reinforcement learning algorithm is directly used as the final control command, and the amplitude of the final action is limited by a function to ensure that the joint torque is within a safe and controllable range, including the following steps: If the deviation of ZMP relative to the supporting polygon is greater than 0.1m, meaning the zero-moment point is not located within the supporting polygon, ZMP compensation is applied, resulting in the following action: a final =saturate(a PPO +Δt ZMP ,a max ) Among them, a PPO For the original action of PPO, a max Δτ represents the maximum value of the joint movement. ZMP It is ZMP-compensated torque; If the deviation of ZMP relative to the supporting polygon is less than 0.1m, meaning the zero moment point is located within the supporting polygon, then the original PPO action a can be used directly. final =saturate(a PPO ,a max ); The saturate() function is used to limit the magnitude of motion or torque.

Images