Robot gait planning and reinforcement learning motion control method and system

Abstract

The invention provides a robot gait planning and reinforcement learning motion control method and system, and the method comprises the following steps: employing a Bezier curve to plan the walking path of a robot, and obtaining an initial target foothold; a gait database is constructed, and a sensor is adopted to collect real-time motion data of the robot; training data in the gait database based on a machine learning algorithm to obtain a gait prediction model, and predicting whether a left leg or a right leg is used as an ideal supporting leg in the next step; whether the current leg is an ideal supporting leg or not is judged, if yes, the leg lifting action is executed, if not, the leg falling action is executed, the robot adopts a reinforcement learning algorithm to construct a step posture adjustment model in the process of executing the leg lifting and leg falling actions, and the walking action of the robot is optimized through iterative training and action strategy adjustment. Through combination of gait prediction and reinforcement learning algorithms, accurate control and dynamic adjustment of walking actions of the robot are realized, and walking stability and environment adaptability of the robot are improved.

Description

Technical Field

[0001] The present invention relates to the field of robot control technology, and in particular to a robot gait planning and reinforcement learning motion control method and system. Background Art

[0002] The autonomous stable balance of the humanoid robot's gait movement is achieved through the legs. In addition to providing gravity support, the control of the legs must also make adjustments at any time according to the changes in the body's state to ensure the stability and balance of the body during static or dynamic movement. During the gait movement of the humanoid robot, foothold control plays an important role in maintaining the stability of the body during the gait switching stage.

[0003] A gait planning method, system and equipment for a bipedal humanoid robot with publication number CN116520869A, including obtaining walking parameters of the bipedal humanoid robot; the walking parameters include walking cycle, step length, double-foot support period, ZMP width, foot lift height and number of walking steps; planning a target ZMP trajectory according to the walking parameters, and planning a swing leg ankle joint trajectory through a third-order Bezier curve; based on a preview control algorithm, using the planned target ZMP trajectory as future reference information for preview control, planning a center of mass trajectory; using a three-center of mass model to compensate the planned center of mass trajectory to generate a compensated center of mass trajectory; calculating the joint angles of each leg joint of the bipedal humanoid robot through an inverse kinematics method based on the compensated center of mass trajectory and the swing leg ankle joint trajectory; and controlling the walking of the bipedal humanoid robot according to the joint angles of each leg joint.

[0004] Currently, there are two main types of gait planning for humanoid robots. Traditional model-based methods include inverted pendulum models, zero-point torque, etc. These control methods mainly pre-set the foothold and adjust the position of the foothold in real time according to the posture of the fuselage. However, these control methods are not robust enough, especially on uneven ground. The reinforcement learning method is to give feedback based on certain goals, such as speed in a certain direction, so that the robot can find a posture with a large reward. However, this method has a large search range and slow learning, which will result in unexpectedly unsightly and unsmooth gaits. Summary of the invention

[0005] In view of this, the present invention proposes a robot gait planning and reinforcement learning motion control method and system, which realizes precise control and dynamic adjustment of the robot's walking motion by combining gait prediction and reinforcement learning algorithms, thereby improving the robot's walking stability and environmental adaptability.

[0006] The technical solution of the present invention is implemented as follows: The present invention provides a robot gait planning and reinforcement learning motion control method, comprising the following steps:

[0007] S1, use Bezier curve to plan the robot's walking path and obtain the initial target landing point;

[0008] S2, building a gait database, which contains human walking data corresponding to the robot's walking gait style and scene, robot stride data under ideal conditions, and robot attribute data, and uses sensors to collect real-time robot motion data;

[0009] S3, training the data in the gait database based on the machine learning algorithm to obtain a gait prediction model, using sensors to collect the real-time motion data of the robot, inputting the collected real-time motion data of the robot into the gait prediction model, and predicting and outputting whether the left leg or the right leg will be used as the ideal supporting foot in the next step;

[0010] S4, determines whether the current leg is an ideal supporting leg. If so, it performs the leg-lifting action, otherwise it performs the leg-dropping action. The robot uses a reinforcement learning algorithm to build a gait adjustment model during the leg-lifting and leg-dropping actions. The gait adjustment model selects actions based on the current state and constructs a reward function to evaluate the value of the action. The robot's walking action is optimized through iterative training and adjustment of action strategies.

[0011] On the basis of the above technical solution, preferably, the walking path of the robot is planned using a Bezier curve in step S1 to obtain an initial target landing point, including using a Bezier curve to plan the walking path of the robot to generate a smooth and continuous landing point sequence as the initial walking path of the robot, and each landing point will serve as the initial target landing point when the robot walks.

[0012] On the basis of the above technical scheme, preferably, the gait style categories in step S2 include long steps, short steps, fast steps, slow steps and alternating walking with two feet, the scene categories include flat ground, stairs and slopes, the attribute data of the robot include height, weight and leg length information, and the collected real-time motion data of the robot includes the center of gravity position, linear velocity, angular velocity and acceleration of the robot's overall motion, the front ground height and the position, linear velocity, angular velocity, acceleration and angular acceleration of each joint.

[0013] Based on the above technical solution, preferably, step S3 includes the following sub-steps:

[0014] S31, acquiring data in a gait database and performing data preprocessing to obtain standard training data;

[0015] S32, extracting features from the standard training data to obtain training feature information, where the training feature information is consistent with features of the collected real-time motion data of the robot;

[0016] S33, constructing a gait prediction model based on a machine learning algorithm, inputting the training feature information as input data into the gait prediction model, training the gait prediction model, and predicting and outputting whether the left leg or the right leg will be the ideal supporting foot in the next step.

[0017] On the basis of the above technical scheme, preferably, the gait prediction model includes an input layer, a position encoding layer, a multi-head self-attention layer, a fully connected layer, a normalization layer and a logistic regression layer, the output end of the input layer is connected to the input end of the position encoding layer, the input layer is used to receive real-time motion data of the robot collected from the sensor, the output end of the position encoding layer is connected to the input end of the multi-head self-attention layer, the position encoding layer is used to add position information to the input data, the output end of the multi-head self-attention layer is connected to the input end of the fully connected layer, and the multi-head self-attention layer is residually connected to the fully connected layer, the output end of the fully connected layer is connected to the input end of the normalization layer, and the connection layer is residually connected to the normalization layer, the output end of the normalization layer is connected to the input end of the logistic regression layer, and the logistic regression layer is used to output the prediction result.

[0018] On the basis of the above technical solution, preferably, step S33 includes the following sub-steps:

[0019] S331, the robot operation data in the training data set is continuous time series data, assuming that there are n time points of input data, the expression is: x = (x1, x2, x3...x n );

[0020] The input feature vector at each time point is expressed as: i=1, 2, 3...n; where m0 is the dimension of motion data features;

[0021] The input data also includes the feature vector of the robot's gait style, let it be y2;

[0022] S332, preset a conversion matrix, multiply each element in the input time series data by the conversion matrix, and obtain an update sequence, which is expressed as:

[0023] i=1,2,3...n

[0024] Where c is the update sequence;

[0025] S333, adding position codes to each element in the update sequence to obtain a position sequence, the expression is:

[0026] u=c+PE

[0027] In the formula, u is the position sequence, PE is the position code;

[0028] The positional encoding is calculated as:

[0029]

[0030] In the formula, pos represents the position index in the position sequence;

[0031] S334, input the position sequence into the multi-head self-attention layer to obtain the output weighted result v, which is expressed as:

[0032] v=MHA(u)

[0033] Where MHA() is the function of the multi-head self-attention layer;

[0034] S335, add the output weighted result and the position sequence, and perform regularity change to obtain a regular sequence, the expression is:

[0035] w1=Norm(u+v)

[0036] In the formula, Norm() is the normalization layer operation;

[0037] S336, perform full connection operation on the regular sequence, and add the regular sequence to change the regularity, and obtain the regular output sequence, the expression is:

[0038] w2=Norm(FC(w)+w)

[0039] In the formula, FC() is the fully connected layer operation;

[0040] S337, preset the number of cycles N, repeat the steps S334 to S336 N times, obtain the first output result y0, perform a full connection operation on the first output result y0, and obtain the second output result y1, which is expressed as:

[0041] y1=Wy0+b

[0042] Where W is the weight matrix of the fully connected layer, and b is the bias term;

[0043] Logistically regress the second output result with the feature vector of the specified robot's gait style to obtain the final output result y, which is expressed as:

[0044] y=σ(k1*y1+k2*y2)

[0045] Where k1 and k2 are vectors with the same dimensions as y1 and y2, k1 and k2 are weight vectors corresponding to the corresponding inputs y1 and y2, and σ() is the sigmoid function;

[0046] If y>0.5, then the right leg is output as the ideal supporting foot, otherwise the left leg is output as the ideal supporting foot.

[0047] On the basis of the above technical solution, preferably, step S4 includes the following sub-steps:

[0048] According to the position of each joint, determine whether the leg currently on the ground is the ideal supporting leg. If so, perform the leg-lifting action; otherwise, perform the leg-dropping action;

[0049] The robot uses the PPO algorithm to build a gait adjustment model in the process of performing leg lifting and leg lowering actions. The PPO algorithm includes a policy network and a value function network. The policy network is used to generate the robot's action strategy, and the value function network is used to evaluate the value of the action taken in the current state.

[0050] The position, linear velocity, angular velocity, acceleration and angular acceleration of each joint in the real-time motion data of the robot collected by the sensor are used as input feature data and input into the gait adjustment model;

[0051] Define the reward functions for leg-lifting and leg-dropping actions, and use the input feature data at the current time point as the state space and the angles of each joint as the action space;

[0052] According to the action probability distribution generated by the current policy network, actions are performed in the environment and state transition data is collected. The value function network evaluates the value of the action based on the reward function and selects the action with the greatest value. The value function network is updated according to the reward function given by the environment. The policy network adjusts the policy based on the evaluation of the value function network. Repeat the iteration until the predetermined number of training times is reached, output the ideal angle of each joint, and adjust it through PID.

[0053] On the basis of the above technical solution, preferably, the reward function during the leg-lifting action includes:

[0054] Let the center of gravity of the robot be (x, y, z), the position of the supporting foot be (px0, py0, pz0), and the position of the non-supporting foot be (px1, py1, pz1), where the supporting foot is on the ground;

[0055] The reward function expression for the leg-lifting action is defined as:

[0056] r up =a 1-1 *r 1-1 +a 2-1 *r 2-1 +a 3-1 *r 3-1 +a 4-1 *r 4-1 +a 5-1 *r 5-1

[0057] In the formula, r upis the total evaluation action value of the leg-lifting action, r 1-1 is the forward speed reward, r 1-1 =V x , V x is the speed of the robot in the x-axis direction, a 1-1 is the weight coefficient of the forward speed reward, when a 1-1 >0 means the robot moves forward in the x-axis direction;

[0058] r 2-1 is the center of gravity height reward, r 2-1 =(zh) 2 , h is the target height of leg lifting, a 2-1 is the weight coefficient of the center of gravity to maintain the target height, when a 2-1 When <0, it means the robot's center of gravity remains at the target height;

[0059] r 3-1 Reward for the support foot to keep the ground, r 3-1 =pz0 2 , a 3-1 is the weight coefficient of the supporting foot remaining on the ground, when a 3-1 <0 means the robot’s supporting feet remain on the ground;

[0060] r 4-1 Gradually raise the reward for the air foot, r 4-1 = pz1(t+dt)-pz1(t), pz1(t) is the height of the foot in the air at time t, pz1(t+dt) is the height of the foot in the air at time t+dt, a 4-1 is the reward coefficient for gradually raising the foot in the air. 4-1 >0 means the robot keeps the foot height increasing gradually when lifting the foot;

[0061] r 5-1 Reward for gradually moving your feet forward in the air, r 5-1 = px1(t+dt)-px1(t), px1(t) is the relative x-axis position of the air foot at time point t, px1(t+dt) is the relative x-axis position of the air foot at time point t+dt, a 5-1 is the bonus coefficient for the air foot to gradually move forward, a 5-1 >0, indicating that the robot keeps the foot moving forward gradually when lifting the foot.

[0062] On the basis of the above technical solution, preferably, the reward function during the leg-dropping action includes:

[0063] Get the target landing point, let the coordinates of the initial landing point be (x0, y0, z0), the center of gravity of the robot be (x1, y1, z1), the position of the supporting foot is (px2, py2, pz2) and the position of the non-supporting foot is (px3, py3, pz3). At this time, the position of the supporting foot is in the air and the position of the non-supporting foot is on the ground. The reward function expression of the landing action is:

[0064] r down =a 1-2 *r 1-2 +a 2-2 *r 2-2 +a 3-2 *r 3-2 +a 4-2 *r 4-2 +a 5-2 *r 5-2 +a 6-2 *r 6-2

[0065] In the formula, r down is the total evaluation value of the leg-dropping action, r 1-2 is the forward speed reward, r 1-2 =V x , V x is the speed of the robot in the x-axis direction, a 1-2 is the weight coefficient of the forward speed reward, a 1-2 >0, indicating that the robot moves forward in the x-axis direction;

[0066] r 2-2 is the center of gravity height reward, r 2-2 =(z1-h) 2 , h is the target height of the leg drop, a 2-2 is the weight coefficient for keeping the center of gravity at the target height, a 2-2 <0, indicating that the robot's center of gravity remains at the target height;

[0067] r 3-2 Reward for the support foot to keep the ground, r 3-2 =pz3 2 , a 3-2 is the weight factor of the non-supporting foot staying on the ground, a 3-2 <0, indicating that the robot's non-supporting feet remain on the ground;

[0068] r 4-2 The reward for the air foot gradually descending, r 4-2 = pz2(t+dt)-pz2(t), pz2(t) is the height of the foot in the air at time t, pz2(t+dt) is the height of the foot in the air at time t+dt, a 4-2is the bonus coefficient for the gradual descent of the aerial foot, a 4-2 <0, indicating that the robot keeps the height of the foot gradually decreasing when lifting the foot;

[0069] r 5-2 Reward for gradually moving your feet forward in the air, r 5-2 = px2(t+dt)-px2(t), px2(t) is the relative x-axis position of the foot in the air at time point t, px2(t+dt) is the relative x-axis position of the foot in the air at time point t+dt, a 5-2 is the bonus coefficient for the air foot to gradually move forward, a 5-2 >0, indicating that the robot's feet in the air gradually move forward;

[0070] r 6-2 Reward for the air foot gradually approaching the target landing point, r 6-2 =(px2-x0) 2 +(py2-y0) 2 , a 5-2 <0, indicating that the robot's feet in the air are gradually approaching the target landing point.

[0071] In a second aspect, the present invention further provides a robot gait planning and reinforcement learning motion control system, which is implemented by the robot gait planning and reinforcement learning motion control method as described above, and the system comprises:

[0072] The path planning module is used to plan the robot's walking path using Bezier curves to obtain the initial target landing point;

[0073] The data storage module is used to construct a gait database, which stores the walking data of humans corresponding to the robot's walking gait style and scene, the robot's stride data under ideal conditions, and the robot's attribute data, and uses sensors to collect the robot's real-time motion data;

[0074] The supporting foot prediction module is used to train the data in the gait database based on the machine learning algorithm to obtain the gait prediction model, use sensors to collect the real-time motion data of the robot, input the collected real-time motion data of the robot into the gait prediction model, and predict and output whether the left leg or the right leg will be used as the ideal supporting foot in the next step;

[0075] The posture adjustment optimization module is used to determine whether the current leg is an ideal supporting leg. If so, it performs the leg-lifting action, otherwise it performs the leg-dropping action. The robot uses a reinforcement learning algorithm to build a gait adjustment model during the leg-lifting and leg-dropping actions. The gait adjustment model selects actions based on the current state and constructs a reward function to evaluate the value of the action. The robot's walking action is optimized through iterative training and adjustment of action strategies.

[0076] The robot gait planning and reinforcement learning motion control method and system of the present invention have the following beneficial effects compared with the prior art:

[0077] (1) By using Bezier curves to plan walking paths and combining gait databases and machine learning algorithms to predict gait, accurate planning and prediction of the robot's walking path and gait are achieved. A reinforcement learning algorithm is used to build a gait adjustment model, which selects actions based on the current state and evaluates the value of actions through reward functions. The walking actions are continuously optimized, which improves the robot's walking stability, fluency, and accuracy.

[0078] (2) The data stored in the gait database takes into account the different gait styles and scene adaptability of the robot, as well as the collection and processing of real-time motion data, so that the robot can better adapt to complex and changing walking environments, improving its overall performance and applicability;

[0079] (3) The gait prediction model includes an input layer, a position encoding layer, a multi-head self-attention layer, a fully connected layer, a normalization layer, and a logistic regression layer. This not only improves the accuracy of the robot's walking gait prediction, but also enhances its ability to adapt to different gait styles and scenes, providing strong support for the robot's walking control and stability improvement.

[0080] (4) By using reinforcement learning, the robot can keep its center of gravity as stable as possible and replace the broad forward walking task with the specific task of lifting and landing the foot. This makes the search range small, the calculation fast, and the learned gait smooth and fluent, avoiding unsightly gaits such as stomping the feet. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0082] Figure 1 A flowchart of the steps of the robot gait planning and reinforcement learning motion control method of the present invention;

[0083] Figure 2 A schematic diagram of the flow of the robot gait planning and reinforcement learning motion control method of the present invention;

[0084] Figure 3 This is an example flow chart of the robot gait planning and reinforcement learning motion control method of the present invention. DETAILED DESCRIPTION

[0085] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0086] like Figure 1 As shown, a robot gait planning and reinforcement learning motion control method of the present invention comprises the following steps:

[0087] S1, use Bezier curve to plan the robot's walking path and obtain the initial target landing point.

[0088] Among them, step S1 includes planning the walking path of the robot using a Bezier curve to generate a smooth and continuous sequence of footholds as the initial walking path of the robot, and each foothold will serve as the initial target foothold when the robot walks.

[0089] It can be understood that the control points of the Bezier curve need to be determined first. The control points will determine the overall shape and direction of the curve. When selecting the control points, the robot's starting position, target position, and any intermediate obstacles or areas that need to be bypassed should be considered. According to the control points, the Bezier curve formula can be used to generate the curve. Since the robot needs to move at discrete time points, the Bezier curve needs to be discretized into a series of points, namely, landing points. This is achieved by selecting a series of values ​​within the range of parameter t and substituting these values ​​into the Bezier curve formula for calculation. Finally, the landing point sequence generated by the above steps will serve as the robot's initial walking path, and each landing point will become the initial target landing point when the robot walks. During the actual walking process of the robot, it will move to these landing points in turn until it reaches the final target position.

[0090] S2, construct a gait database, which contains human walking data corresponding to the robot's walking gait style and scene, robot stride data under ideal conditions and robot attribute data, and uses sensors to collect real-time robot motion data.

[0091] Among them, the gait style categories described in step S2 include long steps, short steps, fast steps, slow steps and alternating walking with two feet, the scene categories include flat ground, stairs and slopes, the robot's attribute data include height, weight and leg length information, and the collected robot real-time motion data includes the center of gravity position, linear velocity, angular velocity and acceleration of the robot's overall motion, the front ground height and the position, linear velocity, angular velocity, acceleration and angular acceleration of each joint.

[0092] It should be noted that the position of the center of gravity of the overall movement reflects the stability and balance state of the robot. The linear velocity and angular velocity of the robot provide information on the current movement speed and direction of the robot. The acceleration of the robot reflects the robot's ability to accelerate or decelerate. The height of the ground in front is used to adjust the leg movements in advance to avoid collisions or falls, especially in complex terrains such as stairs and slopes. The position, linear velocity, angular velocity, acceleration and angular acceleration of each joint provide detailed information on the robot's limb movements, which is the basis for accurately controlling gait and realizing complex movements.

[0093] S3, based on the machine learning algorithm, trains the data in the gait database to obtain a gait prediction model, uses sensors to collect the real-time motion data of the robot, inputs the collected real-time motion data of the robot into the gait prediction model, and predicts and outputs whether the left leg or the right leg will be used as the ideal supporting foot in the next step.

[0094] Step S3 includes the following sub-steps:

[0095] S31, acquiring data in a gait database and performing data preprocessing to obtain standard training data;

[0096] S32, extracting features from the standard training data to obtain training feature information, where the training feature information is consistent with features of the collected real-time motion data of the robot;

[0097] S33, constructing a gait prediction model based on a machine learning algorithm, inputting the training feature information as input data into the gait prediction model, training the gait prediction model, and predicting and outputting whether the left leg or the right leg will be the ideal supporting foot in the next step.

[0098] It should be noted that by accurately predicting the ideal supporting foot for the robot's next step, the robot's stability during walking can be improved, and the robot can be prevented from falling or losing balance due to uncoordinated gait or prediction errors. Compared with the ZMP model, it can pay attention to the ground conditions in front in a timely manner, reduce the stride in time in scenarios such as climbing stairs, and replace the supporting foot in time when stumbling.

[0099] The gait prediction model in this embodiment includes an input layer, a position encoding layer, a multi-head self-attention layer, a fully connected layer, a normalization layer and a logistic regression layer. The output end of the input layer is connected to the input end of the position encoding layer. The input layer is used to receive real-time motion data of the robot collected from the sensor. The output end of the position encoding layer is connected to the input end of the multi-head self-attention layer. The position encoding layer is used to add position information to the input data. The output end of the multi-head self-attention layer is connected to the input end of the fully connected layer, and the multi-head self-attention layer is residually connected to the fully connected layer. The output end of the fully connected layer is connected to the input end of the normalization layer, and the connection layer is residually connected to the normalization layer. The output end of the normalization layer is connected to the input end of the logistic regression layer, and the logistic regression layer is used to output prediction results.

[0100] It should be noted that the gait prediction model can efficiently capture the key features of the robot's real-time motion data, including position information, motion trends, etc., so as to achieve accurate prediction of gait. The multi-head self-attention mechanism enables the model to focus on multiple aspects of the input data at the same time, effectively improving the comprehensiveness and accuracy of the prediction. By introducing diverse data during the training process, the model can learn the common features between different gait patterns, so as to maintain good prediction performance in new scenarios. The use of residual connections and normalization layers further enhances the robustness of the model, enabling it to cope with various complex environments and conditions.

[0101] Specifically, step S33 in this embodiment includes the following sub-steps:

[0102] S331, the robot operation data in the training data set is continuous time series data, assuming that there are n time points of input data, the expression is: x = (x1, x2, x3...x n );

[0103] The input feature vector at each time point is expressed as: i=1, 2, 3...n; where m0 is the dimension of motion data features;

[0104] The input data also includes the feature vector of the robot's gait style, let it be y2;

[0105] S332, preset a conversion matrix, multiply each element in the input time series data by the conversion matrix, and obtain an update sequence, which is expressed as:

[0106] i=1,2,3...n

[0107] Where c is the update sequence;

[0108] S333, adding position codes to each element in the update sequence to obtain a position sequence, the expression is:

[0109] u=c+PE

[0110] In the formula, u is the position sequence, PE is the position code;

[0111] The positional encoding is calculated as:

[0112]

[0113] In the formula, pos represents the position index in the position sequence;

[0114] S334, input the position sequence into the multi-head self-attention layer to obtain the output weighted result v, which is expressed as:

[0115] v=MHA(u)

[0116] Where MHA() is the function of the multi-head self-attention layer;

[0117] S335, add the output weighted result and the position sequence, and perform regularity change to obtain a regular sequence, the expression is:

[0118] w1=Norm(u+v)

[0119] In the formula, Norm() is the normalization layer operation;

[0120] S336, perform full connection operation on the regular sequence, and add the regular sequence to change the regularity, and obtain the regular output sequence, the expression is:

[0121] w2=Norm(FC(w)+w)

[0122] In the formula, FC() is the fully connected layer operation;

[0123] S337, preset the number of cycles N, repeat the steps S334 to S336 N times, obtain the first output result y0, perform a full connection operation on the first output result y0, and obtain the second output result y1, which is expressed as:

[0124] y1=Wy0+b

[0125] Where W is the weight matrix of the fully connected layer, and b is the bias term;

[0126] Logistically regress the second output result with the feature vector of the specified robot's gait style to obtain the final output result y, which is expressed as:

[0127] y=σ(k1*y1+k2*y2)

[0128] Where k1 and k2 are vectors with the same dimensions as y1 and y2, k1 and k2 are weight vectors corresponding to the corresponding inputs y1 and y2, and σ() is the sigmoid function;

[0129] If y>0.5, then the right leg is output as the ideal supporting foot, otherwise the left leg is output as the ideal supporting foot.

[0130] It should be noted that by organizing the robot operation data into continuous time series data and clarifying the dimension of the input features, a structured input is provided for the model. The introduction of the robot's gait style feature vector enables the model to take individual differences into account and improve the degree of personalization of the prediction. The preset transformation matrix maps the input data to a new dimensional space, which helps the model capture more complex feature relationships. By unifying the dimensions, the consistency and effectiveness of subsequent processing steps are ensured. Position encoding is added to the time series data, so that the model can distinguish data at different time points and capture time order information. The calculation method of position encoding ensures that the model accurately captures time sensitivity. The multi-head self-attention layer can focus on multiple aspects of the input data at the same time, improving the model's ability to capture complex features. By calculating the weighted output results, the model can more accurately predict the next gait. The residual connection helps alleviate the gradient vanishing problem in deep networks and improves the training efficiency and stability of the model. Normalization ensures that the data is at an appropriate scale, which helps the model learn more effective feature representations. By repeating steps S334 to S336 multiple times, the model can gradually extract deeper features and improve the accuracy of prediction. The final output result integrates feature information from different layers, improves the generalization ability of the model, and combines the robot's gait style feature vector for logistic regression to make the prediction result more personalized. The output is mapped to the (0,1) interval through the Sigmoid function, and the ideal supporting foot for the next step is determined based on the threshold, thereby achieving accurate prediction of gait.

[0131] S4, determines whether the current leg is an ideal supporting leg. If so, it performs the leg-lifting action, otherwise it performs the leg-dropping action. The robot uses a reinforcement learning algorithm to build a gait adjustment model during the leg-lifting and leg-dropping actions. The gait adjustment model selects actions based on the current state and constructs a reward function to evaluate the value of the action. The robot's walking action is optimized through iterative training and adjustment of action strategies.

[0132] It should be noted that this step depends on whether the current leg is an ideal supporting foot. If so, the foot-lifting process is performed; if not, the foot-landing process is performed. Reinforcement learning is used to make the robot keep its center of gravity as stable as possible, because the broad forward walking task has been replaced by specific foot-lifting and foot-landing tasks. The search range is small, the calculation is fast, and the learned gait is smooth and smooth, avoiding unsightly gaits such as stomping. On the other hand, compared with the ZMP model, ZMP requires the center of gravity to remain on the same horizontal plane, which is very strict. Using reinforcement learning to transform constraints into rewards can be generalized to scenarios such as slopes. By combining gait prediction and reinforcement learning algorithms, precise control and dynamic adjustment of the robot's walking movements are achieved, thereby improving the robot's walking stability and environmental adaptability.

[0133] Wherein, step S4 includes the following sub-steps:

[0134] According to the position of each joint, determine whether the leg currently on the ground is the ideal supporting leg. If so, perform the leg-lifting action; otherwise, perform the leg-dropping action;

[0135] The robot uses the PPO algorithm to build a gait adjustment model in the process of performing leg lifting and leg lowering actions. The PPO algorithm includes a policy network and a value function network. The policy network is used to generate the robot's action strategy, and the value function network is used to evaluate the value of the action taken in the current state.

[0136] The position, linear velocity, angular velocity, acceleration and angular acceleration of each joint in the real-time motion data of the robot collected by the sensor are used as input feature data and input into the gait adjustment model;

[0137] Define the reward functions for leg-lifting and leg-dropping actions, and use the input feature data at the current time point as the state space and the angles of each joint as the action space;

[0138] According to the action probability distribution generated by the current policy network, actions are performed in the environment and state transition data is collected. The value function network evaluates the value of the action based on the reward function and selects the action with the greatest value. The value function network is updated according to the reward function given by the environment. The policy network adjusts the policy based on the evaluation of the value function network. Repeat the iteration until the predetermined number of training times is reached, output the ideal angle of each joint, and adjust it through PID.

[0139] It should be noted that PPO is an algorithm used to solve decision-making problems in the field of reinforcement learning. It is an improved version of the policy gradient method, especially suitable for problems with continuous action space. PPO adopts an architecture that combines the Actor policy network and the Critic value function network; the Actor is responsible for selecting actions based on the current state, while the Critic evaluates the quality of the action. The specific steps include calculating advantage estimation, optimizing the objective function, and updating the strategy.

[0140] It can be understood that the robot collects the position information of each joint in real time through the built-in sensor, and judges whether the leg currently on the ground is an ideal supporting leg based on this information. If it is judged to be an ideal supporting leg, the leg lifting action is performed so that the other leg can move forward; if it is judged to be a non-ideal supporting leg, the leg dropping action is performed to maintain the body balance. In the robot gait adjustment, the PPO algorithm is used to build a gait adjustment model. The real-time motion data of the robot collected by the sensor is input into the gait adjustment model as input feature data. The model gradually optimizes its parameters and strategies through continuous learning and iterative training to better adapt to different walking environments and needs. In the robot gait adjustment, the reward function is designed to reflect the robot's walking stability, energy consumption, speed and other aspects of the index; in the training process, the action is executed according to the action probability distribution generated by the current policy network, and the state transition data is collected. The value function network evaluates the value of the action according to the reward function and selects the action with the greatest value. The policy network adjusts the strategy according to the evaluation of the value function network to achieve precise adjustment of the gait. After the training, the model outputs the ideal angle of each joint. These angles are the optimal solutions obtained by the model after multiple iterations of training, which can guide the robot to achieve more stable walking. By continuously adjusting the parameters of the PID controller, the robot can control its gait more accurately and achieve smoother and more stable walking.

[0141] The reward function during the leg-lifting action in this embodiment includes:

[0142] Let the center of gravity of the robot be (x, y, z), the position of the supporting foot be (px0, py0, pz0), and the position of the non-supporting foot be (px1, py1, pz1), where the supporting foot is on the ground;

[0143] The reward function expression for the leg-lifting action is defined as:

[0144] r up =a 1-1 *r 1-1 +a 2-1 *r 2-1 +a 3-1 *r 3-1 +a 4-1 *r4-1 +a 5-1 *r 5-1

[0145] In the formula, r up is the total evaluation action value of the leg-lifting action, r 1-1 is the forward speed reward, r 1-1 =V x , V x is the speed of the robot in the x-axis direction, a 1-1 is the weight coefficient of the forward speed reward, when a 1-1 >0 means the robot moves forward in the x-axis direction;

[0146] Among them, this item aims to encourage the robot to move forward in the x-axis direction. By setting the weight coefficient of the forward speed reward, the robot can obtain the corresponding reward when it generates a positive speed in the x-axis direction.

[0147] r 2-1 is the center of gravity height reward, r 2-1 =(zh) 2 , h is the target height of leg lifting, a 2-1 is the weight coefficient of the center of gravity to maintain the target height, when a 2-1 When <0, it means the robot's center of gravity remains at the target height;

[0148] Among them, this item is used to ensure that the robot maintains an appropriate center of gravity height during the leg lifting process. When the center of gravity height approaches the target height h, the reward value approaches zero, thereby guiding the robot to maintain the target center of gravity height.

[0149] r 3-1 Reward for the support foot to keep the ground, r 3-1 =pz0 2 , a 3-1 is the weight coefficient of the supporting foot remaining on the ground, when a 3-1 <0 means the robot’s supporting feet remain on the ground;

[0150] Among them, this item aims to ensure that the robot's supporting foot always keeps in contact with the ground during the leg lifting process. When the z coordinate of the supporting foot remains unchanged, the robot can get a reward, a 3-1 It is a very small number, -9999, which means that the supporting foot is always on the ground.

[0151] r 4-1 Gradually raise the reward for the air foot, r 4-1 = pz1(t+dt)-pz1(t), pz1(t) is the height of the foot in the air at time t, pz1(t+dt) is the height of the foot in the air at time t+dt, a 4-1is the reward coefficient for gradually raising the foot in the air. 4-1 >0 means the robot keeps the foot height increasing gradually when lifting the foot;

[0152] Among them, this reward encourages the robot to gradually increase the height of the non-supporting foot during the leg lifting process. When the height of the foot in the air increases, the robot can get the reward.

[0153] r 5-1 Reward for gradually moving your feet forward in the air, r 5-1 = px1(t+dt)-px1(t), px1(t) is the relative x-axis position of the air foot at time point t, px1(t+dt) is the relative x-axis position of the air foot at time point t+dt, a 5-1 is the bonus coefficient for the air foot to gradually move forward, a 5-1 >0, indicating that the robot keeps the foot moving forward gradually when lifting the foot;

[0154] Among them, this reward is intended to guide the robot's non-supporting foot to gradually move forward in the x-axis direction during the leg lifting process. When the foot in the air moves positively in the x-axis direction, the robot can get the reward.

[0155] It should be noted that the forward speed reward and the reward for gradually moving the foot forward in the air jointly prompt the robot to maintain a certain forward speed during the leg lifting process, thereby improving the walking efficiency. The center of gravity height reward and the reward for keeping the supporting foot on the ground help the robot maintain a stable posture during the leg lifting process to avoid falling or losing balance. The reward for gradually raising the foot in the air and the reward for gradually moving the foot forward in the air guide the robot to perform a smoother and more efficient leg lifting action, which helps to reduce energy consumption and improve the continuity of walking, thereby guiding the robot to perform a more efficient and stable leg lifting action. By continuously optimizing and adjusting the parameters of the reward function, the robot's walking performance and adaptability can be further improved.

[0156] The reward function during the leg drop action in this embodiment includes:

[0157] Get the target landing point, let the coordinates of the initial landing point be (x0, y0, z0), the center of gravity of the robot be (x1, y1, z1), the position of the supporting foot is (px2, py2, pz2) and the position of the non-supporting foot is (px3, py3, pz3). At this time, the position of the supporting foot is in the air and the position of the non-supporting foot is on the ground. The reward function expression of the landing action is:

[0158] r down =a 1-2 *r 1-2 +a 2-2 *r 2-2 +a 3-2 *r 3-2 +a4-2 *r 4-2 +a 5-2 *r 5-2 +a 6-2 *r 6-2

[0159] In the formula, r down is the total evaluation value of the leg-dropping action, r 1-2 is the forward speed reward, r 1-2 =V x , V x is the speed of the robot in the x-axis direction, a 1-2 is the weight coefficient of the forward speed reward, a 1-2 >0, indicating that the robot moves forward in the x-axis direction;

[0160] Among them, this reward is intended to encourage the robot to maintain a certain forward speed in the x-axis direction. When the robot generates a positive speed in the x-axis direction, it can obtain the corresponding reward.

[0161] r 2-2 is the center of gravity height reward, r 2-2 =(z1-h) 2 , h is the target height of the leg drop, a 2-2 is the weight coefficient for keeping the center of gravity at the target height, a 2-2 <0, indicating that the robot's center of gravity remains at the target height;

[0162] The reward is used to ensure that the robot maintains an appropriate center of gravity height during the leg landing process. When the center of gravity height approaches the target height h, the reward value approaches zero, thereby guiding the robot to maintain the target center of gravity height. This helps to improve walking stability.

[0163] r 3-2 Reward for the support foot to keep the ground, r 3-2 =pz3 2 , a 3-2 is the weight factor of the non-supporting foot staying on the ground, a 3-2 <0, indicating that the robot's non-supporting feet remain on the ground;

[0164] Among them, this reward is intended to ensure that the robot's non-supporting foot (that is, the foot currently on the ground) always remains in contact with the ground during the leg landing process. When the z coordinate of the non-supporting foot remains unchanged, the robot can receive a reward to prevent the robot from losing balance during the leg landing process.

[0165] r 4-2 The reward for the air foot gradually descending, r 4-2= pz2(t+dt)-pz2(t), pz2(t) is the height of the foot in the air at time t, pz2(t+dt) is the height of the foot in the air at time t+dt, a 4-2 is the bonus coefficient for the gradual descent of the aerial foot, a 4-2 <0, indicating that the robot keeps the height of the foot gradually decreasing when lifting the foot;

[0166] Among them, this reward encourages the robot to gradually lower its feet in the air during the leg landing process. When the height of the feet in the air gradually decreases, the robot can get the reward, guiding the robot to perform a smoother leg landing action.

[0167] r 5-2 Reward for gradually moving your feet forward in the air, r 5-2 = px2(t+dt)-px2(t), px2(t) is the relative x-axis position of the foot in the air at time point t, px2(t+dt) is the relative x-axis position of the foot in the air at time point t+dt, a 5-2 is the bonus coefficient for the air foot to gradually move forward, a 5-2 >0, indicating that the robot's feet in the air gradually move forward;

[0168] Among them, this reward is intended to guide the robot's feet in the air to gradually move forward in the x-axis direction during the leg landing process. When the feet in the air move positively in the x-axis direction, the robot can get the reward, thereby improving the continuity and efficiency of walking.

[0169] r 6-2 Reward for the air foot gradually approaching the target landing point, r 6-2 =(px2-x0) 2 +(py2-y0) 2 , a 5-2 <0, indicating that the robot's feet in the air are gradually approaching the target landing point.

[0170] Among them, this reward is intended to ensure that the robot's aerial foot gradually approaches and accurately lands on the target landing point. When the aerial foot gradually approaches the target landing point, the reward value gradually decreases to zero, thereby guiding the robot to land accurately.

[0171] It should be noted that, through the design of the above-mentioned reward function, the center of gravity height reward and the non-supporting foot keeping the ground reward jointly ensure that the robot maintains a stable posture during the leg landing process to avoid falling or losing balance. The reward for the gradual descent of the foot in the air and the gradual forward movement of the foot in the air guide the robot to perform a smoother and more efficient leg landing action, which helps to reduce energy consumption and improve the continuity of walking. The reward for the foot in the air gradually approaching the target landing point can ensure that the robot lands accurately. By continuously optimizing and adjusting the parameters of the reward function, the robot's landing accuracy and walking performance are improved.

[0172] In this embodiment, the walking path is planned by using Bezier curves, and the gait is predicted by combining the gait database and machine learning algorithm to achieve accurate planning and prediction of the robot's walking path and gait. At the same time, a gait adjustment model is constructed using a reinforcement learning algorithm, and actions are selected according to the current state. The action value is evaluated through a reward function, and the walking action is continuously optimized, thereby improving the robot's walking stability, fluency and accuracy. In addition, this method also takes into account the robot's different gait styles and scene adaptability, as well as the collection and processing of real-time motion data, so that the robot can better adapt to complex and changeable walking environments, improving its overall performance and applicability.

[0173] like Figure 3 The figure shows an example of a robot moving forward. The boldface is stage 1, the italics are stage 2, and the part framed by a rounded rectangle is a repeated part. In normal walking, it takes a long time (1 to 2 seconds) to keep the same supporting foot, so the steps are relatively large. When the supporting foot has been for a long time, the front foot touches the ground, the ground is rough and the attitude is cautious, the supporting foot is changed until the end; the features such as "supporting for a long time" or "climbing the stairs" in the first stage are the judgments of the supporting foot learned by the gait prediction model, which are well explainable.

[0174] In a second aspect, the present invention further provides a robot gait planning and reinforcement learning motion control system, which is implemented by the robot gait planning and reinforcement learning motion control method as described above, and the system comprises:

[0175] The path planning module is used to plan the robot's walking path using Bezier curves to obtain the initial target landing point;

[0176] The data storage module is used to construct a gait database, which stores the walking data of humans corresponding to the robot's walking gait style and scene, the robot's stride data under ideal conditions, and the robot's attribute data, and uses sensors to collect the robot's real-time motion data;

[0177] The supporting foot prediction module is used to train the data in the gait database based on the machine learning algorithm to obtain the gait prediction model, use sensors to collect the real-time motion data of the robot, input the collected real-time motion data of the robot into the gait prediction model, and predict and output whether the left leg or the right leg will be used as the ideal supporting foot in the next step;

[0178] The posture adjustment optimization module is used to determine whether the current leg is an ideal supporting leg. If so, it performs the leg-lifting action, otherwise it performs the leg-dropping action. The robot uses a reinforcement learning algorithm to build a gait adjustment model during the leg-lifting and leg-dropping actions. The gait adjustment model selects actions based on the current state and constructs a reward function to evaluate the value of the action. The robot's walking action is optimized through iterative training and adjustment of action strategies.

[0179] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A robot gait planning and reinforcement learning motion control method, characterized in that: The following steps are involved: S1, use Bezier curve to plan the robot's walking path and obtain the initial target landing point; S2, building a gait database, which contains human walking data corresponding to the robot's walking gait style and scene, robot stride data under ideal conditions, and robot attribute data, and uses sensors to collect real-time robot motion data; S3, training the data in the gait database based on the machine learning algorithm to obtain a gait prediction model, using sensors to collect the real-time motion data of the robot, inputting the collected real-time motion data of the robot into the gait prediction model, and predicting and outputting whether the left leg or the right leg will be used as the ideal supporting foot in the next step; S4, determines whether the current leg is an ideal supporting leg. If so, it performs the leg-lifting action, otherwise it performs the leg-dropping action. The robot uses a reinforcement learning algorithm to build a gait adjustment model during the leg-lifting and leg-dropping actions. The gait adjustment model selects actions based on the current state and constructs a reward function to evaluate the value of the action. The robot's walking action is optimized through iterative training and adjustment of action strategies.

2. The robot gait planning and reinforcement learning motion control method according to claim 1, characterized in that: The step S1 described in which the walking path of the robot is planned using a Bezier curve to obtain an initial target landing point includes planning the walking path of the robot using a Bezier curve to generate a smooth and continuous landing point sequence as the initial walking path of the robot, and each landing point will serve as the initial target landing point when the robot walks.

3. The robot gait planning and reinforcement learning motion control method according to claim 2, characterized in that: The gait style categories described in step S2 include long steps, short steps, fast steps, slow steps and alternating walking with two feet; the scene categories include flat ground, stairs and slopes; the attribute data of the robot include height, weight and leg length information; the real-time motion data of the robot collected includes the center of gravity position, linear velocity, angular velocity and acceleration of the robot's overall motion, the front ground height and the position, linear velocity, angular velocity, acceleration and angular acceleration of each joint.

4. The robot gait planning and reinforcement learning motion control method according to claim 3, characterized in that: Step S3 includes the following sub-steps: S31, acquiring data in a gait database and performing data preprocessing to obtain standard training data; S32, extracting features from the standard training data to obtain training feature information, where the training feature information is consistent with features of the collected real-time motion data of the robot; S33, constructing a gait prediction model based on a machine learning algorithm, inputting the training feature information as input data into the gait prediction model, training the gait prediction model, and predicting and outputting whether the left leg or the right leg will be the ideal supporting foot in the next step.

5. The robot gait planning and reinforcement learning motion control method according to claim 4, characterized in that: The gait prediction model includes an input layer, a position encoding layer, a multi-head self-attention layer, a fully connected layer, a normalization layer and a logistic regression layer. The output end of the input layer is connected to the input end of the position encoding layer. The input layer is used to receive real-time motion data of the robot collected from a sensor. The output end of the position encoding layer is connected to the input end of the multi-head self-attention layer. The position encoding layer is used to add position information to the input data. The output end of the multi-head self-attention layer is connected to the input end of the fully connected layer, and the multi-head self-attention layer is residually connected to the fully connected layer. The output end of the fully connected layer is connected to the input end of the normalization layer, and the connection layer is residually connected to the normalization layer. The output end of the normalization layer is connected to the input end of the logistic regression layer, and the logistic regression layer is used to output prediction results.

6. The robot gait planning and reinforcement learning motion control method according to claim 5, characterized in that: Step S33 includes the following sub-steps: S331, the robot operation data in the training data set is continuous time series data, assuming that there are n time points of input data, the expression is: x = (x1, x2, x3...x n ); The input feature vector at each time point is expressed as: i=1, 2, 3...n; where m0 is the dimension of motion data features; The input data also includes the feature vector of the robot's gait style, let it be y2; S332, preset a conversion matrix, multiply each element in the input time series data by the conversion matrix, and obtain an update sequence, which is expressed as: i=1,2,3...n Where c is the update sequence; S333, adding position codes to each element in the update sequence to obtain a position sequence, the expression is: u=c+PE In the formula, u is the position sequence, PE is the position code; The positional encoding is calculated as: In the formula, pos represents the position index in the position sequence; S334, input the position sequence into the multi-head self-attention layer to obtain the output weighted result v, which is expressed as: v=MHA(u) Where MHA() is the function of the multi-head self-attention layer; S335, add the output weighted result and the position sequence, and perform regularity change to obtain a regular sequence, the expression is: w1=Norm(u+v) In the formula, Norm() is the normalization layer operation; S336, perform full connection operation on the regular sequence, and add the regular sequence to change the regularity, and obtain the regular output sequence, the expression is: w2=Norm(FC(w)+w) In the formula, FC() is the fully connected layer operation; S337, preset the number of cycles N, repeat the steps S334 to S336 N times, obtain the first output result y0, perform a full connection operation on the first output result y0, and obtain the second output result y1, which is expressed as: y1=Wy0+b Where W is the weight matrix of the fully connected layer, and b is the bias term; Logistically regress the second output result with the feature vector of the specified robot's gait style to obtain the final output result y, which is expressed as: y=σ(k1*y1+k2*y2) Where k1 and k2 are vectors with the same dimensions as y1 and y2, k1 and k2 are weight vectors corresponding to the corresponding inputs y1 and y2, and σ() is the sigmoid function; If y>0.5, then the right leg is output as the ideal supporting foot, otherwise the left leg is output as the ideal supporting foot.

7. The robot gait planning and reinforcement learning motion control method according to claim 6, characterized in that: Step S4 includes the following sub-steps: According to the position of each joint, determine whether the leg currently on the ground is the ideal supporting leg. If so, perform the leg-lifting action; otherwise, perform the leg-dropping action; The robot uses the PPO algorithm to build a gait adjustment model in the process of performing leg lifting and leg lowering actions. The PPO algorithm includes a policy network and a value function network. The policy network is used to generate the robot's action strategy, and the value function network is used to evaluate the value of the action taken in the current state. The position, linear velocity, angular velocity, acceleration and angular acceleration of each joint in the real-time motion data of the robot collected by the sensor are used as input feature data and input into the gait adjustment model; Define the reward functions for leg-lifting and leg-dropping actions, and use the input feature data at the current time point as the state space and the angles of each joint as the action space; According to the action probability distribution generated by the current policy network, actions are performed in the environment and state transition data is collected. The value function network evaluates the value of the action based on the reward function and selects the action with the greatest value. The value function network is updated according to the reward function given by the environment. The policy network adjusts the policy based on the evaluation of the value function network. Repeat the iteration until the predetermined number of training times is reached, output the ideal angle of each joint, and adjust it through PID.

8. The robot gait planning and reinforcement learning motion control method according to claim 7, characterized in that: The reward function during the leg lifting action includes: Let the center of gravity of the robot be (x, y, z), the position of the supporting foot be (px0, py0, pz0), and the position of the non-supporting foot be (px1, py1, pz1), where the supporting foot is on the ground; The reward function expression for the leg-lifting action is defined as: r up =a 1-1 *r 1-1 +a 2-1 *r 2-1 +a 3-1 *r 3-1 +a 4-1 *r 4-1 +a 5-1 *r 5-1 In the formula, r up is the total evaluation action value of the leg-lifting action, r 1-1 is the forward speed reward, r 1-1 =V x , V x is the speed of the robot in the x-axis direction, a 1-1 is the weight coefficient of the forward speed reward, when a 1-1 >0 means the robot moves forward in the x-axis direction; r 2-1 is the center of gravity height reward, r 2-1 =(zh) 2 , h is the target height of leg lifting, a 2-1 is the weight coefficient of the center of gravity to maintain the target height, when a 2-1 When <0, it means the robot's center of gravity remains at the target height; r 3-1 Reward for the support foot to keep the ground, r 3-1 =pz0 2 , a 3-1 is the weight coefficient of the supporting foot remaining on the ground, when a 3-1 <0 means the robot’s supporting feet remain on the ground; r 4-1 Gradually raise the reward for the air foot, r 4-1 = pz1(t+dt)-pz1(t), pz1(t) is the height of the foot in the air at time t, pz1(t+dt) is the height of the foot in the air at time t+dt, a 4-1 is the reward coefficient for gradually raising the foot in the air. 4-1 >0 means the robot keeps the foot height increasing gradually when lifting the foot; r 5-1 Reward for gradually moving your feet forward in the air, r 5-1 = px1(t+dt)-px1(t), px1(t) is the relative x-axis position of the air foot at time point t, px1(t+dt) is the relative x-axis position of the air foot at time point t+dt, a 5-1 is the bonus coefficient for the air foot to gradually move forward, a 5-1 >0, indicating that the robot keeps the foot moving forward gradually when lifting the foot.

9. The robot gait planning and reinforcement learning motion control method according to claim 8, characterized in that: The reward function during the leg drop action includes: Get the target landing point, let the coordinates of the initial landing point be (x0, y0, z0), the center of gravity of the robot be (x1, y1, z1), the position of the supporting foot is (px2, py2, pz2) and the position of the non-supporting foot is (px3, py3, pz3). At this time, the position of the supporting foot is in the air and the position of the non-supporting foot is on the ground. The reward function expression of the landing action is: r down =a 1-2 *r 1-2 +a 2-2 *r 2-2 +a 3-2 *r 3-2 +a 4-2 *r 4-2 +a 5-2 *r 5-2 +a 6-2 *r 6-2 In the formula, r down is the total evaluation value of the leg-dropping action, r 1-2 is the forward speed reward, r 1-2 =V x , V x is the speed of the robot in the x-axis direction, a 1-2 is the weight coefficient of the forward speed reward, a 1-2 >0, indicating that the robot moves forward in the x-axis direction; r 2-2 is the center of gravity height reward, r 2-2 =(z1-h) 2 , h is the target height of the leg drop, a 2-2 is the weight coefficient for keeping the center of gravity at the target height, a 2-2 <0, indicating that the robot's center of gravity remains at the target height; r 3-2 Reward for the support foot to keep the ground, r 3-2 =pz3 2 , a 3-2 is the weight factor of the non-supporting foot staying on the ground, a 3-2 <0, indicating that the robot's non-supporting feet remain on the ground; r 4-2 The reward for the air foot gradually descending, r 4-2 = pz2(t+dt)-pz2(t), pz2(t) is the height of the foot in the air at time t, pz2(t+dt) is the height of the foot in the air at time t+dt, a 4-2 is the bonus coefficient for the gradual descent of the aerial foot, a 4-2 <0, indicating that the robot keeps the height of the foot gradually decreasing when lifting the foot; r 5-2 Reward for gradually moving your feet forward in the air, r 5-2 = px2(t+dt)-px2(t), px2(t) is the relative x-axis position of the foot in the air at time point t, px2(t+dt) is the relative x-axis position of the foot in the air at time point t+dt, a 5-2 is the bonus coefficient for the air foot to gradually move forward, a 5-2 >0, indicating that the robot's feet in the air gradually move forward; r 6-2 Reward for the air foot gradually approaching the target landing point, r 6-2 =(px2-x0) 2 +(py2-y0) 2 , a 5-2 <0, indicating that the robot's feet in the air are gradually approaching the target landing point.

10. A robot gait planning and reinforcement learning motion control system, implemented by the robot gait planning and reinforcement learning motion control method according to any one of claims 1 to 9, characterized in that: The system comprises: The path planning module is used to plan the robot's walking path using Bezier curves to obtain the initial target landing point; The data storage module is used to construct a gait database, which stores the walking data of humans corresponding to the robot's walking gait style and scene, the robot's stride data under ideal conditions, and the robot's attribute data, and uses sensors to collect the robot's real-time motion data; The supporting foot prediction module is used to train the data in the gait database based on the machine learning algorithm to obtain the gait prediction model, use sensors to collect the real-time motion data of the robot, input the collected real-time motion data of the robot into the gait prediction model, and predict and output whether the left leg or the right leg will be used as the ideal supporting foot in the next step; The posture adjustment optimization module is used to determine whether the current leg is an ideal supporting leg. If so, it performs the leg-lifting action, otherwise it performs the leg-dropping action. The robot uses a reinforcement learning algorithm to build a gait adjustment model during the leg-lifting and leg-dropping actions. The gait adjustment model selects actions based on the current state and constructs a reward function to evaluate the value of the action. The robot's walking action is optimized through iterative training and adjustment of action strategies.

Images