Biped robot complex terrain adaptive gait planning method and biped robot

Abstract

The invention relates to a biped robot complex terrain self-adaptive gait planning method and a biped robot, and belongs to the technical field of robotics.The biped robot complex terrain self-adaptive gait planning method comprises the steps that a fractional order PID controller is constructed, and the fractional order PID controller comprises a multi-sensor network module, a self-adaptive module and a fractional order calculus control module; constructing a reinforcement learning model for biped robot gait planning, setting a state, an action, a decision and a reinforcement learning reward function of reinforcement learning, performing continuous iteration and optimization adjustment on the reinforcement learning model by using a strategy gradient method, updating strategy parameters, and outputting control parameters of the biped robot; and performing decision fusion on control parameters output by the fractional order PID controller and the reinforcement learning model through a comprehensive loss function to obtain globally optimal biped robot control parameters. According to the method, more accurate gait planning can be provided for the biped robot in complex terrains, and better walking flexibility and adaptability are achieved.

Description

Technical Field

[0001] The present invention relates to the field of robot technology, and in particular to a complex terrain adaptive gait planning method for a biped robot and the biped robot. Background Art

[0002] For model-based control of bipedal robots, motion of bipedal robots has traditionally been investigated through concepts of gait stability, such as the ZMP criterion, which can be implemented using a reduced-order model of the robot. While these approaches can effectively control flat-footed humanoid robots, their walking style tends to be conservative. Alternatively, techniques based on Hybrid Zero Dynamics (HZD) can also utilize a full-order model of the robot and then produce stable periodic walks based on input-output linearization. However, HZD-based 3D robot controllers usually require extensive parameter tuning in both simulation and the real world and have difficulty adapting to environmental changes.

[0003] Existing gait planning methods for bipedal robots can achieve good results on flat ground, but on inclined and undulating roads, problems such as unstable center of gravity and inability to adapt to tilt are very likely to occur, causing the robot to become unstable or even fall. In complex terrain environments, the robot's gait planning needs to take into account a variety of factors (terrain features, obstacles, the robot's own state, etc.), resulting in a significant increase in the amount of algorithm calculation. Existing algorithms cannot respond to environmental changes in real time and cannot adapt to complex terrain. At present, a variety of advanced gait planning algorithms represented by deep reinforcement learning have problems such as excessive computational complexity, especially in optimizing gait strategies. There is a problem of low computational efficiency, especially in complex terrain environments, where robots must make correct choices within a limited time. The shortcomings of existing gait planning algorithms are even more obvious.

[0004] At the same time, in existing technologies, the inaccuracy of bipedal robot sensor data will affect the accuracy of gait planning, especially in complex terrain or low visibility conditions, which can easily lead to wrong decisions. In addition, existing gait planning methods lack automatic adjustment capabilities and cannot dynamically adjust gait parameters according to environmental changes. On complex roads, bipedal robots find it difficult to maintain a stable gait plan, which may cause falls or slips. Summary of the invention

[0005] In order to solve at least one of the above technical problems, the present invention provides a complex terrain adaptive gait planning method for a bipedal robot and a bipedal robot. The present invention utilizes a fractional-order PID controller with higher degrees of freedom and better dynamic performance for bipedal robot control of complex and nonlinear systems, combines a deep reinforcement algorithm optimization strategy, and integrates the fractional-order calculus control and reinforcement learning algorithm at the decision level based on the loss function, so that the bipedal robot can generate stable walking behaviors suitable for various complex terrains.

[0006] The technical solution adopted by the present invention is as follows:

[0007] The complex terrain adaptive gait planning method of a biped robot includes the following steps:

[0008] Step S1: construct a fractional-order PID controller, which includes a multi-sensor network module, an adaptive module and a fractional-order calculus control module. The multi-sensor network module fuses and encodes the external environment data collected by multiple sensor nodes on the biped robot, generates an external vector and transmits it to the adaptive module. The adaptive module estimates the current external vector online according to the historical state sequence and historical action sequence sensed from the body, and outputs the corresponding control component. The fractional-order calculus control module finely adjusts the control component obtained by the adaptive module, and finally outputs the control parameters of the biped robot.

[0009] Step S2: Construct a reinforcement learning model for bipedal robot gait planning, set the state, action, decision and reinforcement learning reward function of reinforcement learning, and use the policy gradient method to continuously iterate and optimize the reinforcement learning model, update the policy parameters, and output the control parameters of the bipedal robot. The expression of the reinforcement learning reward function is as follows:

[0010] R=R stability +R energy +R speed +R smoothness +R fall (18)

[0011] Among them, R stability is the walking stability reward function, R energy is the energy efficiency reward function, R speed is the walking speed reward function, R smoothness is the gait smoothness reward function, R fall is the fall penalty function;

[0012] Step S3: weighted fusion of the biped robot control parameters output by the fractional-order PID controller and the reinforcement learning model. The fused control signal is:

[0013] u(t)=β1·u fractional (t)+β2·u RL (t) (22)

[0014] Among them, u fractional (t) is the control parameter output by the fractional-order PID controller, u RL (t) is the control parameter output by the reinforcement learning model, β1 and β2 are weight parameters;

[0015] A comprehensive loss function L is set, and the control parameters output by the fractional-order PID controller and the reinforcement learning model are fused by the comprehensive loss function to obtain the globally optimal control parameters of the biped robot. The expression of the comprehensive loss function L is as follows:

[0016] L=α1·u fractional (t)+α2·u RL (t)+α3·u(t) (23)

[0017] Among them, α1, α2 and α3 are weight coefficients.

[0018] The beneficial effects of the present invention are as follows: the present invention transmits the external environment data obtained by the external sensor node to the fractional-order PID controller, utilizes the characteristics of fractional-order calculus that can better handle complex and nonlinear data, accurately processes the external environment data, and utilizes the loss function to fuse the parameters obtained by reinforcement learning and the parameters obtained by the fractional-order PID controller for decision-making, thereby providing a more accurate gait planning for the bipedal robot in complex terrain and achieving better walking flexibility and adaptability.

[0019] Compared with existing methods, the adaptive module of the external vector can better, faster and more accurately monitor the external environment and input it into the controller, so as to better process nonlinear data and better adjust the gait of the bipedal robot. The loss function integrates reinforcement learning and fractional calculus for decision-making, which can make the bipedal robot select the appropriate gait in different terrain environments and achieve more accurate gait control. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flow chart of the complex terrain adaptive gait planning method for a biped robot according to an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The technical solution of the present invention will be described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0022] See also Figure 1This embodiment provides a complex terrain adaptive gait planning method for a biped robot based on fractional calculus and reinforcement learning. The method mainly includes the following steps:

[0023] Step S1: construct a fractional-order PID controller;

[0024] Step S2: constructing a reinforcement learning model for biped robot gait planning;

[0025] Step S3: Design a comprehensive loss function to integrate fractional-order calculus control and reinforcement learning for decision-making, and achieve a more natural human gait. It should be pointed out that the above step numbers do not necessarily represent the order in which the steps are implemented. The order of step S1 and step S2 can be changed, and the changed technical solution still belongs to the protection scope of the present invention.

[0026] In this embodiment, step S1 first constructs a fractional-order proportional-integral-differential (PID) controller, which includes a multi-sensor network module for generating basic control components of target joints for the biped robot, an adaptive module that uses proprioception to continuously estimate external vectors, and a fractional-order calculus control module. First, the external environment data is collected through the multi-sensor nodes on the biped robot, and the external environment data is fused and encoded and compressed through the multi-sensor network module. After the external vector is generated, the external vector is passed to the evaluation model in the adaptive module. The adaptive module estimates the current external vector online based on the historical state sequence and historical action sequence sensed from the proprioception, and outputs the corresponding control component. However, a single evaluation model will cause the resolution of the control component to fail to meet expectations, and can only adjust integers. Therefore, the fractional-order calculus control module is introduced to enable the control component to adjust a larger range such as fractions.

[0027] In a complex environment, the robot will receive a series of special information, the so-called "external" information. This information includes environmental parameters such as terrain friction coefficient, mass distribution, inertia, etc., which have an important impact on the robot's walking. On this basis, this embodiment adopts a multi-sensor network module, which is based on a convolutional neural network model to fuse the external environment data collected by multiple sensor nodes on the bipedal robot. These fused external information are encoded into a compressed external vector, and then the obtained external vector is passed to the adaptive module. The adaptive module infers the specific information of the external environment by analyzing the difference between the robot instructions and the actual joint position, and uses the robot's action and state information over the past period of time to estimate the current external vector.

[0028] However, since some external information may not be completely restored under observation, the accuracy of the evaluation model of the adaptive module is not high enough. On this basis, a fractional calculus control module is introduced to finely adjust the control component.

[0029] The control component π takes two parameters as input, namely the current state x t and the current external vector z t ∈R 3 , where R 3 is a 3-dimensional real vector that contains specific information about the environment and is used to help the robot quickly adapt to different walking environments. Then predict the next action a t , as shown in formula (2). The predicted action a t are the target positions of the 10 driven robot joints, which are converted into torques using a fractional-order PID controller. The environment vector e t Including friction, inertia and center of mass of the robot, etc. But the external vector z t Only the control component required for adaptation from the environment vector e is retained t information.

[0030] z t =μ(e t ) (1)

[0031] a t =π(x t ,a t-1 ,g t ,z t ) (2)

[0032] Among them, a t-1 For the previous action, g t is the ground truth tilt angle, and μ is the rule-based control strategy.

[0033] Multilayer Perceptrons (MLPs) consist of multiple layers, including an input layer, multiple hidden layers, and an output layer. Each layer consists of multiple neurons, which are connected by weights. μ and the control component π are passed into MLPs and trained end-to-end jointly using reinforcement learning to maximize the following expected return of the control component π:

[0034]

[0035] Among them, τ={(x0,a0,r0),(x1,a1,r1)...} represents the trajectory of the robot when executing the control component π, x0 is the specific situation of the environment at that moment or the sensory input of the robot, a0 is the action behavior selected in this state, r0 is the feedback obtained from the environment after executing the action, x1 is the state of the next time step, a1 is the action behavior of the next time step, and r1 is the reward feedback of the next time step; p(τ|π) represents the possibility of the trajectory under the control component π; is the expected value symbol, which represents the long-term average value of a random variable; γ is a time-related discount factor, which is used in reinforcement learning to measure the importance of future rewards relative to current rewards; r t is the instant reward obtained at time step t; T is the perception input of the bipedal robot and the total number of time steps in the training cycle.

[0036] Adaptive module for estimating external environment According to the history of proprioception (x t-k:t-1 ,a t-k:t-1 ) Online estimation of the external vector Estimated external vector Given by:

[0037]

[0038] Among them, x t-k:t-1 is the state sequence from time step tk to time step t-1, a t-k:t-1 is the action sequence from time step tk to time step t-1, and φ() is the adaptive module function.

[0039] The adaptive module can be trained via supervised learning in simulation to minimize:

[0040]

[0041] Among them, z t =μ(e t ), the adaptive module φ is modeled as a 1-DCNN to capture temporal correlations.

[0042] Fractional calculus is an extension of traditional integer calculus, which can better describe the dynamic behavior of the system. Compared with integer-order controllers, fractional-order controllers have more degrees of freedom and better dynamic characteristics, and are particularly suitable for modeling and controlling complex nonlinear systems. Fractional-order calculus control is divided into the following steps:

[0043] 1.1. Dynamic modeling of bipedal robot:

[0044] The dynamic equation of a biped robot with n joints is expressed by the Lagrange equation:

[0045]

[0046] Where q is the n-dimensional joint position vector, and are velocity vector and acceleration vector respectively, M(q) is the n×n mass matrix, are the Coriolis and centrifugal force matrices, G(q) is the gravity vector, is the control input torque, F ext is the external disturbance torque.

[0047] 1.2.Introduction of fractional calculus into dynamic systems:

[0048] In traditional dynamics equations, velocity and acceleration are usually integer derivatives. In order to better capture the complexity and memory effect of the system, fractional derivatives are introduced here. The definition of Caputo fractional derivatives is used here:

[0049]

[0050] Where α is the order of the derivative, That is, α is a non-negative real number, n is the smallest positive integer such that n-1<α≤n, which is used to determine the integer-order differentials required for fractional derivatives, Γ(·) is the Gamma function, which is a generalization of the factorial function, and f (n) (τ) is the nth derivative of the function f(τ) with respect to τ, τ is the integral variable, integrated from 0 to t.

[0051] 1.3. Establish the fractional order kinetic equation:

[0052] Introducing fractional calculus into the dynamic model of the biped robot, assuming that the velocity and acceleration parts are in fractional order form, the fractional order dynamic equation of the biped robot can be expressed as:

[0053]

[0054] in, is the fractional derivative of the joint position vector q(t), with order α, It is the fractional derivative of the joint acceleration vector, with an order of β, generally β=α+1, α>0.

[0055] 1.4. Fractional calculus control module:

[0056] The fractional-order calculus control module is mainly divided into the proportional (P) control module, the fractional-order integral (I λ ) control module, fractional order differential (D μ )The control module consists of three parts.

[0057] 1.4.1. Proportional (P) control module:

[0058] The proportional control module generates a control signal in response to the current error value (the difference between the set value and the actual value) so that the system condition can be corrected in time. For bipedal robots, the proportional control module is used to make real-time adjustments to the errors generated during walking, such as gait deviation from the predetermined track. The transfer function of the proportional control module is:

[0059] C P (s) = K p ·e(t) (9)

[0060] Among them, K p is the proportional gain and e(t) is the error signal.

[0061] 1.4.2. Fractional integral (I λ )Control module:

[0062] The fractional order integral control module integrates the error at each moment, thereby eliminating the steady-state error of the system. The use of fractional order integral control makes the system more flexible. The fractional order integral control module can better solve the continuous errors existing in the walking process of the bipedal robot, such as forward movement and turning. The adjustment of the fractional order λ provides more precise control over the error accumulation speed. The transfer function of the fractional order integral control module is:

[0063]

[0064] Where λ is the fractional order of integration, K i is the integral gain, and s is the complex variable in the Laplace transform, which is usually used to describe the frequency domain behavior of linear time-invariant systems.

[0065] 1.4.3. Fractional Differentiation (D μ )Control module:

[0066] The fractional order differential control module is used to control the rate of change of the error to predict the dynamic changes of the system and reduce overshoot or oscillation. At the same time, the fractional order differential control method is used to improve the dynamic characteristics of the system. For bipedal robots, the fractional order differential control method can better capture the gait and posture of the robot and make more stable adjustments, avoiding shaking and falling during walking. The transfer function of the fractional order differential control module is:

[0067] C D (s) = K d ·s u (11)

[0068] Where u is the fractional order of the differential, K d is the differential gain.

[0069] The overall transfer function of the fractional-order calculus control module can be expressed as:

[0070]

[0071] Among them, K p , K i , K d Also called control gain.

[0072] The structure of the fractional-order calculus control module consists of three core parts: proportion, fractional-order integration and fractional-order differentiation. By introducing fractional-order integration and differentiation operations, the control parameters of the bipedal robot can be fine-tuned more accurately, and finally the control parameters of the bipedal robot are output.

[0073] In step S2, a reinforcement learning model for bipedal robot gait planning is constructed, and the state, action, decision and reinforcement learning reward function of the reinforcement learning are set. The reinforcement learning model is continuously iterated and optimized using the policy gradient method, the policy parameters are updated, and finally the control parameters of the bipedal robot are output.

[0074] Reinforcement learning enables bipedal robots to learn multiple gaits in complex terrain and environments. By interacting with the environment, the robot can optimize its gait strategy to maintain balance and stability. Step S2 mainly includes the following steps:

[0075] 2.1. Set up the state and action space of the reinforcement learning model:

[0076] Let the state space be S and the action space be A. At each time step t, the agent observes state s t , and take action a t , to maximize the cumulative reward R.

[0077] 2.2. Establish a reinforcement learning reward function for bipedal robot gait planning. The reinforcement learning reward function needs to take into account multiple factors to ensure that the robot can learn a stable and efficient gait. The reinforcement learning reward function mainly consists of the following key components:

[0078] 2.2.1. Walking stability reward

[0079] Ensuring that the robot can maintain balance and stability while walking is the basic goal of gait planning. The walking stability reward function is:

[0080] R stability = -λ1×CoM deviation -λ2×angulardeviation (13)

[0081] Among them, CoM deviation The distance that the center of gravity deviates from the body's central axis. Keeping the center of gravity stable can reduce the risk of falling; angular deviation It is the angle between the robot's body axis and the ground. Maintaining an upright posture can help the bipedal robot walk more smoothly. λ1 and λ2 are weight coefficients.

[0082] 2.2.2. Energy Efficiency Rewards

[0083] The robot is encouraged to reduce energy consumption as much as possible when walking and improve the economy of walking. The energy efficiency reward function is:

[0084] R energy =-λ3×energy consumption (14)

[0085] Among them, energy consumption is the energy consumption of the robot at each moment, which can be calculated based on the joint torque and walking distance completed by the robot; λ3 is the weight coefficient.

[0086] 2.2.3. Walking speed bonus

[0087] Encourage the robot to maintain the correct pace and not walk too slowly or too fast. The walking speed reward function is:

[0088]

[0089] Among them, v actual is the current walking speed of the robot, v target is the predetermined target walking speed, λ4 is the weight coefficient. The robot is rewarded for walking close to the target speed.

[0090] 2.2.4. Gait Smoothness Reward

[0091] Reward the robot for producing smooth, natural gaits and avoiding sudden or jerky movements. The gait smoothness reward function is:

[0092] R smoothness =-λ5×∑ i (Δu i ) 2 (16)

[0093] Among them, u i It refers to the change of control signals such as torque and angle of each kinematic pair in adjacent time steps. Slight changes mean smoother walking. λ5 is the weight coefficient.

[0094] 2.2.5. Fall Penalty

[0095] A penalty is imposed on the robot when it falls or exhibits obvious unstable behavior, forcing it to avoid unstable gaits. The fall penalty function is:

[0096]

[0097] Among them, λ6 is the weight coefficient.

[0098] Combining the above functions into a comprehensive reward function, we can get the reinforcement learning reward function:

[0099] R=R stability +R energy +R speed +R smoothness +R fall (18)

[0100] The weight coefficients λ1, λ2, …, λ6 can be adjusted according to the priorities of different tasks.

[0101] 2.3. Strategy optimization:

[0102] Use the policy gradient method to optimize the reinforcement learning model: First, initialize the policy function (π θ (a|s)), and then run the simulation under the current policy to collect state-action pairs (s t ,a t ) and the corresponding reward R t , and use the collected samples to calculate the policy gradient and update the policy parameters according to the following formula:

[0103]

[0104] Among them, α is the learning rate, θ is the weight of the control policy model, which is the core of optimization, and J(θ) is the goal of policy optimization, which is usually the expected cumulative reward.

[0105] Furthermore, based on the Actor-Critic network architecture, we first use the Critic network to estimate each state S t The value of V φ (s t ), and calculate the advantage function through the TD error:

[0106] A(s t ,a t )=r t +γV φ (s t+1 )-V φ (s t ) (20)

[0107] Among them, A(s t ,a t ) means in state s t Take action a t The degree of superiority of the strategy relative to the average strategy, r t is the immediate reward obtained by the robot from the environment at time step t, and γ is a value between 0 and 1, indicating the discount rate of future rewards.

[0108] Furthermore, the Proximal Policy Optimization (PPO) algorithm is used to stabilize the policy update. PPO introduces a clip to limit the amplitude of each update to prevent the policy parameters from being updated too much, which would lead to instability. The objective function of the PPO algorithm is:

[0109]

[0110] in, is the estimated advantage function; ∈ is the shear range; It represents the entropy value of the current strategy at time step t, which measures the strategy π θ (a t |s t ) in state s t Next select action a t The randomness or uncertainty of θ (a t |s t ) is the robot according to the current state s t Select action a t The probability distribution of is controlled by the parameter θ.

[0111] Based on the policy gradient method, especially the reinforcement learning algorithm based on the Actor-Critic framework, the gait of the bipedal robot is optimized. By setting the reinforcement learning reward function, selecting the appropriate gradient policy algorithm, and continuously iterating and optimizing the reinforcement learning model, the control parameters of the bipedal robot are finally output, laying the foundation for realizing a stable and efficient walking gait strategy for the robot in a complex environment.

[0112] In step S3, the decision-making of fractional-order calculus control and reinforcement learning is integrated through the loss function, which provides an effective strategy combining classical control theory with modern intelligent methods. Fractional-order differential control has advantages in dealing with complex dynamics and memory, while reinforcement learning realizes autonomous learning of the optimal strategy based on interaction with the outside world. Combining these two methods, the loss function can be used to comprehensively evaluate them, making them more intelligent. The control parameters output by the fractional-order PID controller and the reinforcement learning model are integrated through the comprehensive loss function, and finally the globally optimal control parameters of the bipedal robot are obtained, realizing high-efficiency, high-stability dynamic adaptive gait planning of the bipedal robot under complex terrain. The following are the specific steps and methods:

[0113] 3.1. Integration of fractional-order calculus control and reinforcement learning:

[0114] The parallel fusion method is used to realize the fusion of fractional-order calculus control and reinforcement learning. First, the fractional-order PID controller and reinforcement learning run independently, and each calculates the control signal and action strategy. Then, the control parameters output by the fractional-order PID controller and the control parameters output by the reinforcement learning are weighted fused. The fused control signal can be expressed by the following formula:

[0115] u(t)=β1·u fractional (t)+β2·u RL (t) (22)

[0116] Among them, u fractional (t) is the control parameter output by the fractional-order PID controller, u RL (t) is the control parameter output by the reinforcement learning model, and β1 and β2 are weight parameters.

[0117] 3.2. Set the comprehensive loss function:

[0118] The loss function is an important indicator to measure the performance of the system. Combining fractional calculus with reinforcement learning makes it have better global optimality. Combining the above objectives, we can define a comprehensive loss function, as shown in the following formula:

[0119] L=α1·u fractional (t)+α2·u RL (t)+α3·u(t) (23)

[0120] Among them, α1, α2 and α3 are weight coefficients.

[0121] By integrating the loss function, the output of fractional-order differential control and reinforcement learning is fused to achieve the respective advantages of the two methods, and improve the control effect and intelligence level of the system. By designing a reasonable loss function, combining parallel fusion strategy, and combining simulation with actual operation, efficient control of complex systems can be achieved.

[0122] The present invention transmits the external environment data obtained by the external sensor node into the fractional-order PID controller, utilizes the characteristics of fractional-order calculus that can better handle complex and nonlinear data, accurately processes the external environment data, and utilizes the loss function to fuse the parameters obtained by reinforcement learning and the parameters obtained by the fractional-order PID controller for decision-making, thereby providing more accurate gait planning for the bipedal robot in complex terrain and achieving better walking flexibility and adaptability.

[0123] Compared with existing methods, the adaptive module of the external vector can better, faster and more accurately monitor the external environment and input it into the controller, so as to better process nonlinear data and better adjust the gait of the bipedal robot. The loss function integrates reinforcement learning and fractional calculus for decision-making, which can make the bipedal robot select the appropriate gait in different terrain environments and achieve more accurate gait control.

[0124] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0125] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A complex terrain adaptive gait planning method for a bipedal robot, characterized in that: The following steps are involved: Step S1: construct a fractional-order PID controller, which includes a multi-sensor network module, an adaptive module and a fractional-order calculus control module. The multi-sensor network module fuses and encodes the external environment data collected by multiple sensor nodes on the biped robot, generates an external vector and transmits it to the adaptive module. The adaptive module estimates the current external vector online according to the historical state sequence and historical action sequence sensed from the body, and outputs the corresponding control component. The fractional-order calculus control module finely adjusts the control component obtained by the adaptive module, and finally outputs the control parameters of the biped robot. Step S2: Construct a reinforcement learning model for bipedal robot gait planning, set the state, action, decision and reinforcement learning reward function of reinforcement learning, and use the policy gradient method to continuously iterate and optimize the reinforcement learning model, update the policy parameters, and output the control parameters of the bipedal robot. The expression of the reinforcement learning reward function is as follows: R=R stability +R energy +R speed +R smoothness +R fall (18) Among them, R stability is the walking stability reward function, R energy is the energy efficiency reward function, R speed is the walking speed reward function, R smoothness is the gait smoothness reward function, R fall is the fall penalty function; Step S3: weighted fusion of the biped robot control parameters output by the fractional-order PID controller and the reinforcement learning model. The fused control signal is: u(t)=β1·u fractional (t)+β2·u RL (t) (22) Among them, u fractional (t) is the control parameter output by the fractional-order PID controller, u RL (t) is the control parameter output by the reinforcement learning model, β1 and β2 are weight parameters; A comprehensive loss function L is set, and the control parameters output by the fractional-order PID controller and the reinforcement learning model are fused by the comprehensive loss function to obtain the globally optimal control parameters of the biped robot. The expression of the comprehensive loss function L is as follows: L=α1·u fractional (t)+α2·u RL (t)+α3·u(t) (23) Among them, α1, α2 and α3 are weight coefficients.

2. The complex terrain adaptive gait planning method for a bipedal robot according to claim 1, characterized in that: The control component is based on the current state x t and the current external vector z t Two parameters are used as input to predict the next action a t , expressed as: With t =μ(e t ) (1) a t =π(x t ,a t-1 ,g t ,z t ) (2) Among them, a t-1 For the previous action, g t is the true tilt angle of the ground, e t is the environment vector, and μ is the rule-based control strategy.

3. The complex terrain adaptive gait planning method for a bipedal robot according to claim 2, characterized in that: The predicted next action a t is the target position of the 10 driven biped robot joints, and the environment vector e t These include friction, the robot's inertia, and its center of mass.

4. The complex terrain adaptive gait planning method for a bipedal robot according to claim 2 or 3, characterized in that: Training the adaptive module via supervised learning minimizes: in, is the estimated current external vector, x t-k：t-1 is the state sequence from time step tk to time step t-1, a t-k：t-1 is the action sequence from time step tk to time step t-1, and φ() is the adaptive module function.

5. The complex terrain adaptive gait planning method for a bipedal robot according to claim 1, characterized in that: The fractional-order calculus control module includes a proportional control module for real-time adjustment of the error generated when the biped robot walks, a fractional-order integral control module for integrating the error at each moment, and a fractional-order differential control module for controlling the rate of change of the error.

6. The complex terrain adaptive gait planning method for a bipedal robot according to claim 5, characterized in that: The transfer function of the proportional control module is: C P (s)=K p ·e(t) (9) The transfer function of the fractional-order integral control module is: The transfer function of the fractional-order differential control module is: C D (s)=K d ·s u (11) The overall transfer function of the fractional-order calculus control module is: Among them, K p , K i , K d are proportional gain, integral gain, and differential gain respectively, e(t) is the error signal, s is the complex variable in Laplace transform, λ is the fractional order of integration, and u is the fractional order of differentiation.

7. The complex terrain adaptive gait planning method for a bipedal robot according to claim 1, characterized in that: The policy gradient method uses the Actor-Critic network architecture for optimization.

8. The complex terrain adaptive gait planning method for a biped robot according to claim 1, characterized in that: The policy gradient method uses a proximal policy optimization algorithm to stabilize policy updates. The objective function of the proximal policy optimization algorithm is: in, is the estimated advantage function; ∈ is the shear range; It represents the entropy value of the current strategy at time step t, which measures the strategy π θ (a t |s t ) in state s t Next select action a t The randomness or uncertainty of θ (a t |s t ) is the robot according to the current state s t Select action a t The probability distribution of is controlled by the parameter θ.

9. The complex terrain adaptive gait planning method for a biped robot according to claim 1, characterized in that: R stability =-λ1×COM deviation -λ2×angular deviation (13) Among them, CoM deviation is the distance that the center of gravity deviates from the body's central axis, angular deviation is the angle between the robot's body axis and the ground, λ1 and λ2 are weight coefficients; R energy =-λ3×energy consumption (14) Among them, energy consumption is the energy consumption of the robot at each moment, λ3 is the weight coefficient; Among them, v actual is the current walking speed of the robot, v target is the predetermined target walking speed, λ4 is the weight coefficient; R smoothness =-λ5×∑ i (D) i ) 2 (16) Among them, u i It refers to the change of the control signal of each kinematic pair in adjacent time steps, and λ5 is the weight coefficient; Among them, λ6 is the weight coefficient.

10. A bipedal robot, characterized in that: include: A memory and a processor, wherein the memory stores a computer program executable by the processor, and when the processor executes the computer program, the complex terrain adaptive gait planning method for a bipedal robot as described in any one of claims 1 to 9 is implemented.

Images