Gait control method and system for humanoid robot based on imitation and inverse reinforcement learning

Abstract

The application discloses a kind of humanoid robot gait control method and system based on imitation and inverse reinforcement learning, collect human or simulation robot walking data to construct high-quality training dataset, then through graph convolution structure, the similarity of robot and expert data is calculated.Based on task target design multidimensional linear reward function, four kinds of indexes such as stability, gait periodicity, trajectory similarity and energy efficiency are comprehensively considered, and multi-objective optimization is realized through weight distribution.Neural network is used to construct strategy model, and training is carried out in combination with proximal policy optimization algorithm, and the update amplitude is constrained using strategy ratio clipping technology to improve learning stability.Maximal entropy principle is introduced to dynamically update reward function, and the adaptability to expert strategy is enhanced.Finally, simulation verification is carried out through MuJoCo physical engine, and visual analysis and performance evaluation of high human-like gait are realized.The application solves the problems of robot gait adaptability and robustness in complex dynamic environment.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of intelligent robot control and three-dimensional reconstruction, and particularly relates to reinforcement learning, imitation learning, gait planning, robot control and the like. Specifically, it refers to a humanoid robot gait control method and system based on imitation learning and inverse reinforcement learning. BACKGROUND

[0002] Humanoid robots have wide application value in many fields, including service robots, medical rehabilitation, disaster rescue, industrial manufacturing, and human-computer interaction scenarios. With the rapid development of artificial intelligence and robot technology, humanoid robots are gradually moving from laboratory research to practical application. Their flexible movement ability and intelligent perception system enable them to perform a variety of complex tasks. In the service industry, humanoid robots can be used for intelligent tour guide, home companion, and catering service, providing efficient and convenient human-computer interaction experience; in the field of medical rehabilitation, humanoid robots can assist patients in gait training and provide precise rehabilitation guidance; in disaster rescue scenarios, humanoid robots can perform search and rescue tasks in complex terrain or extreme environments with human-like gait control ability, improving rescue efficiency. In addition, the application of humanoid robots in intelligent manufacturing is also growing, which can be used for precision assembly, quality detection and human-computer collaboration, improving production efficiency and safety.

[0003] Although humanoid robots have great application potential in many fields, their gait control still faces many challenges. Due to the high degree of freedom of the joint structure of humanoid robots, their gait control needs to balance stability, flexibility and energy consumption to ensure that the robot can walk stably in different environments. In addition, complex terrain, dynamic disturbances and changes in external environment will affect the stability of the robot's gait, making it difficult for traditional control methods based on dynamic modeling to meet the needs of practical applications. In order to solve these problems, in recent years, researchers have introduced deep reinforcement learning algorithms, which have greatly improved the adaptability of robots to complex environments. However, existing reinforcement learning methods still have problems such as low gait learning efficiency, strong dependence on expert demonstration data, and insufficient generalization ability of walking gait. Therefore, further research on more efficient, stable and adaptable humanoid robot gait control methods is of great significance to the development of humanoid robot technology. SUMMARY

[0004] The present application discloses a humanoid robot gait control method and system based on imitation learning and inverse reinforcement learning, aiming to solve the problems of insufficient adaptability and robustness of robot gait in complex dynamic environments.

[0005] The humanoid robot gait control method designed by the present application includes the following steps:

[0006] S1, collect human, humanoid robot walking node and inertial data and pretreatment, the node represents the body part or joint of the robot;

[0007] S2, the joint data of the robot and the joint data of the person are respectively represented as a graph structure, in the graph structure, each node represents a body part or joint of the robot, and the edge represents the connection relationship between the joints; the imitation similarity of gait is calculated through the graph convolution network; the robot stability, gait periodicity and energy efficiency are calculated at the same time;

[0008] S3, a linear reward function including walking stability, gait periodicity, imitation similarity and energy efficiency is designed;

[0009] S4, a neural network is used as a strategy model, the robot state is input, and the gait control action is output;

[0010] S5, the reward function is initialized;

[0011] S6, the PPO algorithm is used to train the strategy model, and the clipped probability ratio The optimization target is constructed by combining the strategy loss, the value function loss and the entropy regularization term, the optimization target gradient is calculated and the strategy model parameters are updated;

[0012] S7, the reward function is updated by using the maximum entropy inverse reinforcement learning;

[0013] S8, the effect of the gait control strategy is simulated and verified.

[0014] Further, the gait data is collected, the tester mainly installs reflective marker points on the main joints, and a force measuring platform is arranged on the ground to record the gait contact force and impact force, and an inertial measurement element measures the target acceleration and speed; the main joints include ankle, knee, hip, waist, shoulder and elbow.

[0015] Further, the calculation of the imitation similarity of gait specifically includes:

[0016] S2-1, the joint data of the robot and the joint data of the person are respectively represented as a graph structure, in the graph structure, each node represents a body part or joint of the robot, and the edge represents the connection relationship between the joints;

[0017] S2-2, the IMU collects sensor timing data, clips according to time segments, and splices according to the order of human acceleration data G 人 , robot acceleration data G 机 , human gyroscope data A 人 , robot gyroscope data A 机 ;

[0018] S2-3, the graph structure in S2-1 is input into a convolution layer, and a feature vector is output;

[0019] S2-4, input the feature vector into the Transformer model;

[0020] S2-5, input the S2-2 concatenated data into LSTM;

[0021] S2-6, concatenate the outputs of the Transformer model and the LSTM model, and then input them into the fully connected layer to obtain the concatenated feature vector;

[0022] S2-7, regress the concatenated feature vector to obtain the similarity between the robot and human gait data, which is .

[0023] Furthermore, the stability reward function is designed based on the trunk inclination angle as follows:

[0024]

[0025] in, and are the pitch and roll angles of the trunk;

[0026] The periodic reward function is designed based on gait cycle consistency as follows:

[0027]

[0028] in, is the current gait cycle, is the target gait cycle;

[0029] The energy efficiency reward function is designed based on the sum of squared joint torques:

[0030]

[0031] in, is the torque of the i-th joint;

[0032] The trajectory similarity reward function is output by S2 :

[0033]

[0034] in, The coordinates of the robot's i-th joint node, Represents the coordinates of the i-th joint node of a person, represents the Euclidean distance;

[0035] The overall reward function can be expressed as:

[0036]

[0037] in, It is the robot state, It's action. is the weight.

[0038] Furthermore, the neural network includes an input layer, a hidden layer and an output layer, wherein the hidden layer uses a multi-layer perceptron combined with a long short-term memory network to process data.

[0039] Furthermore, the clipping objective function in step S6 is:

[0040]

[0041] in, is the advantage function, is the cropping range, is the clipping function to ensure the stability of gait learning;

[0042] The optimization goal is:

[0043]

[0044] in , is a hyperparameter.

[0045] Furthermore, S6-1, the current gait state of the robot and its corresponding control instructions Input the experience pool and calculate the reward value according to the reward function designed by S2 , record state-action pairs and reward value In the experience pool;

[0046] S6-2, the state-action pairs in the experience pool Input the policy model designed in S4 , calculate the action probability distribution of the current strategy output, and calculate the strategy ratio based on the PPO algorithm:

[0047]

[0048] in, Is the current strategy in state Select Action probability; Is the strategy in state before the last update Next select action probability;

[0049] S6-3, the strategy ratio calculated in the previous step Input cropping strategy update module;

[0050] S6-4, input the policy ratio after cutting in the last step into the target loss calculation module, combine the policy loss, the entropy regularization term and the value function loss to construct a complete PPO optimization target as:

[0051]

[0052] to improve the exploratory and convergence of the gait policy, wherein , is a hyperparameter;

[0053] S6-5, input the optimization target calculated in the last step into the gradient calculation module, calculate the gradient of the policy model using the back propagation algorithm, and update the parameters of the policy model to make the gait control policy more stable and adaptive to different environments.

[0054] Based on the same inventive concept, the present application also discloses a system for implementing a humanoid robot gait control method based on imitation and inverse reinforcement learning, comprising:

[0055] a data acquisition module for acquiring human or simulated humanoid robot walking gait data and extracting gait features;

[0056] a gait similarity calculation module for representing the gait node data of the robot and the human gait node data as a graph structure, calculating the imitation similarity of the gait by fusing inertial information through a graph convolution network, and simultaneously calculating the robot stability, gait periodicity and energy efficiency;

[0057] a data processing module for designing a reward function, wherein the reward function involves walking stability, gait periodicity, imitation similarity and energy efficiency; using a neural network as a policy model, inputting the robot state and outputting the gait control action; initializing the reward function; training the policy model using the PPO algorithm, and the specific process includes:

[0058] storing interaction data, calculating the probability ratio of the new and old policies , clipping the probability ratio by clipping the target function, combining the clipped probability ratio with the policy loss, the value function loss and the entropy regularization term to construct an optimization target, calculating the gradient of the optimization target and updating the parameters of the policy model;

[0059] updating the reward function using the maximum entropy inverse reinforcement learning;

[0060] a simulation verification module for simulating and verifying the effect of the gait control policy.

[0061] Based on the same inventive concept, the present application also designs an electronic device, comprising:

[0062] one or more processors;

[0063] a storage device storing one or more programs;

[0064] When the one or more programs are executed by the one or more processors, the one or more processors implement the biped robot gait control method based on imitation and inverse reinforcement learning.

[0065] Based on the same inventive concept, the present application also designs a computer readable medium having a computer program stored thereon, which, when executed by a processor, implements the biped robot gait control method based on imitation and inverse reinforcement learning.

[0066] Advantages of the present application:

[0067] 1. The present application innovatively proposes a strategy model training method for robot gait learning, introduces a multi-module collaborative optimization mechanism, stores interaction data through an experience pool module, evaluates the probability ratio of new and old strategies using a strategy ratio calculation module, and combines a pruning strategy update module to limit the strategy update amplitude and prevent violent fluctuations; Unlike the prior art, which usually optimizes the strategy loss or value function loss separately, the strategy loss, value function loss and entropy regularization term are fused in the target loss calculation module to construct a comprehensive optimization target, ensuring the stability and adaptability of gait control, thereby significantly improving the robustness and convergence efficiency of strategy training.

[0068] 2. The present application innovatively proposes an inverse reinforcement learning method for robot gait control, introduces reward regularization and dynamic update mechanism, ensures the reasonable distribution of reward function parameters through normalization constraint, avoids the excessive influence of single reward item on strategy, and combines experience replay module to dynamically update reward value and strategy performance evaluation, ensures the reward model converges to the optimal configuration through iterative optimization, thereby significantly improving the stability and generalization ability of the algorithm.

[0069] 3. The present application innovatively proposes a reward function for robot gait similarity measurement, which calculates the similarity by fusing joint torque data and IMU inertial data, wherein the torque data extracts feature information through a graph convolution network, and the inertial data extracts feature information through LSTM. Compared with traditional single-mode and Euclidean distance measurement methods, the present application can better utilize the topological structure information of machine joints and the inertial information of motion, thereby more accurately comparing the similarity of expert demonstration gait and robot generated gait, and forming a better reward effect. BRIEF DESCRIPTION OF DRAWINGS

[0070] Figure 1 A schematic diagram of the strategy model training process.

[0071] Figure 2 Design a schematic for the reward function.

[0072] Figure 3 Schematic diagram of the graph convolution process. DETAILED DESCRIPTION

[0073] In order to further understand the content of the present invention and make the objectives, technical solutions and advantages of the present invention clearer, the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0074] The present invention discloses a humanoid robot gait control method based on imitation learning and inverse reinforcement learning, the main steps of which are as follows:

[0075] Step S1, data collection and processing, uses the optical motion capture system OptiTrack to collect walking trajectory data of humans or simulated humanoid robots, uses the IMU sensor to collect inertial information, and performs denoising, normalization, and interpolation on the data. The specific implementation process is described as follows:

[0076] S1-1: 12 to 16 infrared cameras are deployed in the laboratory, and reflective markers are installed at the test subject's major joints (ankles, knees, hips, waist, shoulders, elbows, etc.) to enable the camera system to accurately track the human gait. Simultaneously, IMUs (inertial measurement units) are installed at these joints to supplement motion details and compensate for optical occlusions using accelerometer and gyroscope data. An AMT (Advanced Motion Measurement Platform) is deployed on the ground to record key data such as gait contact force and impact force.

[0077] S1-2, all data are denoised by Kalman filtering, and time aligned and normalized to form a gait dataset.

[0078] Step S2: Gait similarity calculation. The robot's gait data and expert demonstration data are represented as a graph structure. Using a graph convolutional network, the gait imitation similarity is calculated. Stability, gait periodicity, and energy efficiency are calculated, and a linear reward function is designed.

[0079] As attached Figure 3 As shown, the specific steps for calculating the similarity are as follows:

[0080] S2-1, the joint data of the robot and the human are represented as graph structures. Assume that the number of joints is n , node information is represented as n×3 The structure of , where 3 is the three dimensions of X, Y, and Z, and the edge information is saved as n x n The adjacency matrix of n× ( n+3) of the robot. In the figure, each node represents a body part or joint of the robot, such as a knee, an ankle, etc., and the edges represent the connection relationship between the joints;

[0081] S2-2, the IMU collected accelerometer, gyroscope and other time series data are cut into 60 3, the time slice is 60, where 3 is the X, Y, Z three dimensions; according to the human acceleration data G 人 , robot acceleration data G 机 , human gyroscope data A 人 , robot gyroscope data A 机 Spliced in order to 120 3 size;

[0082] S2-3, input the graph structure in S2-1 into the convolution layer, output 2 60 1 feature vectors;

[0083] S2-4, input the vector in S2-3 into the Transformer model, output 2 128 1 feature vectors;

[0084] S2-5, input the data in S2-2 into LSTM, output 128 1 feature vectors;

[0085] S2-6, splice the output of S2-4 and S2-5 into 384 1 feature vectors, and input into a fully connected layer to output 512 1 feature vectors;

[0086] S2-7, the feature vectors output by S2-6 are regressed to obtain the similarity of the robot and the human gait data .

[0087] Step S3, reward function design, as shown in the attached Figure 2 figure, the reward function is designed from stability, gait periodicity, imitation similarity and trajectory similarity. The specific steps are:

[0088] S3-1, decompose the control target into multiple sub-targets, including stability, gait cycle, energy efficiency and trajectory similarity.

[0089] S3-2, design stability reward, reward the robot to keep the trunk balance, based on the trunk inclination angle, designed as:

[0090]

[0091] where, and are the pitch and roll angles of the torso.

[0092] S3-3, design periodic rewards to reward the regularity and periodicity of the robot's gait. Based on the gait cycle consistency, the design is:

[0093] in, is the current gait cycle, is the target gait cycle.

[0094] S3-4, design energy efficiency rewards to reward the robot for reducing energy consumption, which is designed based on the sum of squares of joint torques:

[0095]

[0096] in, It is i The torque of the joint.

[0097] S3-5, trajectory similarity reward function is output by S2 gives:

[0098]

[0099] in, The coordinates of the robot's i-th joint node, Represents the coordinates of the i-th joint node of a person, represents the Euclidean distance;

[0100] S3-6, combining S3-2, S3-3, S3-4, and S3-5, designs the reward function as follows:

[0101]

[0102] in, It's a state. It's action. is the weight, which is used to balance the importance of each sub-goal.

[0103] Step S4, strategy model design, as shown in the attached Figure 1 As shown in the figure, a neural network is designed to realize gait control of a humanoid robot. The neural network includes an input layer, a hidden layer, and an output layer, and uses a multi-layer perceptron (MLP) combined with a long short-term memory (LSTM) network. The input of the strategy model is the robot state information, including joint angles, angular velocities, contact force data, etc. The output of the model is a gait control action, which is used to adjust the joint angles or control torque to achieve gait optimization. The specific implementation process is described as follows:

[0104] S4-1, input the robot state data into the normalization module, normalize the joint angle, angular velocity, and contact force information to improve the stability of the data distribution and the convergence speed of the model training, and obtain the normalized gait state features;

[0105] S4-2, input the normalized gait state features of the previous step into the first fully connected layer, set the input dimension to 256, and use the ReLU activation function to perform preliminary feature extraction on the data, obtaining a 256-dimensional gait feature representation;

[0106] S3-3, input the 256-dimensional gait features extracted in the previous step into the second fully connected layer, set the output dimension to 128, and use the ReLU activation function to further extract deep features of the gait, obtaining a 128-dimensional high-order gait feature representation;

[0107] S4-4, input the 128-dimensional gait features extracted in the previous step into the LSTM layer, model the time dependence of the gait state through the memory and forgetting mechanism, and capture the dynamic change trend of the gait, obtaining a 64-dimensional time sequence feature vector;

[0108] S4-5, input the 64-dimensional time sequence features extracted in the previous step into the third fully connected layer, keep the output dimension as 64, and use the ReLU activation function to optimize the gait feature expression ability, while adding Dropout to prevent overfitting, obtaining the optimized 64-dimensional gait control features;

[0109] S4-6, input the 64-dimensional gait control features optimized in the previous step into the fourth fully connected layer, set the output dimension to 128, and use the ReLU activation function to enhance the nonlinear expression ability of the gait control signal, obtaining a 128-dimensional high-dimensional gait control signal;

[0110] S4-7, input the 128-dimensional gait control signal generated in the previous step into the fifth fully connected layer, set the output dimension to , and use the Tanh activation function to map the control signal to a reasonable range to ensure the stability of the robot gait control, obtaining the gait control instruction.

[0111] Step S5, reward function initialization, randomly initialize the weights of the reward function.

[0112] Step S6, policy model training, use PPO to optimize the policy model: first, store the interaction data in the experience pool module; then, use the policy ratio calculation module to calculate the probability ratio of the new and old policies Then, in the policy update module, the policy update amplitude is limited by clipping the objective function to prevent drastic fluctuations. After that, in the target loss calculation module, the policy loss, value function loss, and entropy regularization term are innovatively combined to construct a complete optimization objective. Finally, the gradient calculation module is used to calculate the gradient and update the policy model parameters to ensure the stability and adaptability of gait control. The specific implementation steps are as follows:

[0113] S6-1, the robot's current gait state and its corresponding control instructions Input experience pool, which is used to store information such as state-action pairs, reward values, and next state collected by the agent during its interaction with the environment. The reward value is calculated based on the reward function designed by S2 , record state-action pairs and reward values In the experience pool;

[0114] S6-2, the state-action pairs in the experience pool Input the policy model designed in S4 , calculate the action probability distribution of the current strategy output, and calculate the strategy ratio based on the PPO algorithm:

[0115]

[0116] Used to measure the change between the old and new strategies to ensure that the strategy remains stable during the update. Is the current strategy (new strategy) in state Next select action probability; Is the strategy before the last update (old strategy) in the state Next select action probability.

[0117] S6-3, the strategy ratio calculated in the previous step Enter the clipping policy update module, which limits the policy update range through the PPO clipping objective function to prevent drastic fluctuations in the policy network:

[0118]

[0119] in, is the advantage function, is the cropping range, is the clipping function to ensure the stability of gait learning;

[0120] S6-4, input the policy ratio after the previous step into the target loss calculation module, combined with the policy loss and entropy regularization term and value function loss , construct the complete PPO optimization objective as:

[0121]

[0122] To improve the exploration and convergence of gait strategy, , is a hyperparameter;

[0123] S6-5, input the optimization target calculated in the previous step into the gradient calculation module, use the back propagation algorithm to calculate the gradient of the strategy model, and update the parameters of the strategy model to make the gait control strategy more stable and adaptable to different environments.

[0124] Step S7: Update the reward function. Use the maximum entropy inverse reinforcement learning method to update the reward function by learning from expert samples. The specific implementation steps are as follows:

[0125] S7-1, the reward function parameters of S3 As the variable to be optimized, the maximum entropy inverse reinforcement learning objective function is introduced:

[0126]

[0127] in, is the reward function; Indicates that in state , the probability distribution of the policy over all possible actions; is the policy entropy, α is the entropy coefficient. By maximizing this objective function, the reward function can better fit the expert trajectory while avoiding premature convergence;

[0128] S7-2, input the objective function of the previous step into the gradient optimization module, and calculate the gradient update direction of the reward parameter based on the policy gradient method:

[0129]

[0130] Update the reward function parameters using the gradient ascent method:

[0131]

[0132] Where η is the learning rate, represents the gradient. In this way, the reward function gradually approaches the optimal reward signal implied by the expert strategy;

[0133] S7-3, input the reward parameters optimized in the previous step into the reward regularization module, introduce normalization constraints, and ensure that the relative weights of each reward item remain within a reasonable range to prevent a single reward from having an excessive impact on the policy convergence. Use the projected gradient method to normalize 𝜔 so that all weight parameters meet the above constraints;

[0134] S7-4, input the optimized reward function into the experience replay module, and use the newly sampled state-action trajectory Calculate the dynamically updated reward value and re-evaluate the policy's performance. If the reward update does not lead to an improvement in policy returns, adjust the learning rate and re-optimize to ensure that the reward model converges to the optimal weight configuration.

[0135] Step S8: Simulation verification, using MuJoCo to simulate the gait of the humanoid robot and visualize the training results. The specific implementation process is as follows:

[0136] S8-1 transmits the gait control signal from S7 to the robot model in the MuJoCo simulation environment, drives the robot to perform gait motion, and collects physical feedback data such as joint angle, velocity, and contact force in real time to build a complete gait motion dataset;

[0137] S8-2: Input the gait data collected in the previous step into the gait evaluation module, perform performance analysis based on indicators such as gait periodicity, consistency, and energy consumption, and define the gait performance measurement function:

[0138]

[0139] in, Represent gait stability, periodicity, energy efficiency and imitation degree respectively, is the weight coefficient, which is adjusted according to experimental requirements;

[0140] S8-3, input the gait performance measurement results into the visualization module, and use trajectory rendering, gait heat map and joint angle change curve to intuitively present the simulated gait, so as to evaluate the robot's gait quality and analyze and adjust unstable or abnormal gaits that appear during the simulation process;

[0141] S8-4, compare the simulated gait data with the real human gait data, calculate the gait trajectory similarity, and use the dynamic time warping (DTW) algorithm to calculate the gait trajectory error:

[0142]

[0143] Among them, X is the robot simulation gait trajectory, and is the real human gait trajectory. is the Euclidean distance between the two, and π is the matching path to measure the closeness of the robot gait to the real gait;

[0144] S8-5, input the simulation evaluation results into the optimization module. If the gait performance metric Below the set threshold or gait trajectory error If it is too large, adjust the gait control strategy parameters and re-simulate iterative optimization to ensure that the final gait control strategy can achieve a highly human-like gait in a physical simulation environment.

[0145] Example 2

[0146] Based on the same inventive concept, this embodiment further discloses a system for implementing a humanoid robot gait control method based on imitation and inverse reinforcement learning, comprising:

[0147] Data acquisition module, which collects walking node and inertial data of humans and simulated humanoid robots, and extracts gait features;

[0148] The gait similarity calculation module represents the robot's gait data and expert demonstration data as a graph structure, and calculates the gait imitation similarity through a graph convolutional network. It also calculates the robot's stability, gait periodicity, and energy efficiency.

[0149] The data processing module designs a reward function that considers walking stability, gait periodicity, imitation similarity, and energy efficiency. A neural network is used as the policy model, which inputs the robot state and outputs gait control actions. The reward function is initialized. The policy model is trained using the PPO algorithm. The specific process includes:

[0150] Store interaction data and calculate the probability ratio of new and old strategies , by clipping the objective function to the probability ratio Perform trimming and set the trimmed probability ratio Combine the policy loss, value function loss and entropy regularization term to construct the optimization target, calculate the optimization target gradient and update the policy model parameters;

[0151] Use maximum entropy inverse reinforcement learning to update the reward function;

[0152] Simulation verification module, used to simulate and verify the effect of gait control strategy.

[0153] Since the system introduced in the second embodiment of the present invention is a system adopted for implementing a humanoid robot gait control method according to the embodiment of the present invention, based on the method introduced in the first embodiment of the present invention, technical personnel in this field can understand the specific structure and deformation of the electronic device, so they will not be repeated here.

[0154] Example 3

[0155] Based on the same inventive concept, the present invention also provides an electronic device, comprising one or more processors; a storage device for storing one or more programs; when one or more programs are executed by the one or more processors, the one or more processors implement the method described in Example 1.

[0156] Since the device described in Example 3 of the present invention is an electronic device used to implement the humanoid robot gait control method based on imitation and inverse reinforcement learning in Example 1 of the present invention, those skilled in the art will be able to understand the specific structure and variations of the electronic device based on the method described in Example 1 of the present invention, and therefore will not be described in detail here. All electronic devices used in a method of an embodiment of the present invention fall within the scope of protection of the present invention.

[0157] Example 4

[0158] Based on the same inventive concept, the present invention further provides a computer-readable medium having a computer program stored thereon, which implements the method described in the first embodiment when the program is executed by a processor.

[0159] Since the device described in Example 4 of the present invention is a computer-readable medium used to implement the humanoid robot gait control method based on imitation and inverse reinforcement learning in Example 1 of the present invention, those skilled in the art will be able to understand the specific structure and variations of the electronic device based on the method described in Example 1 of the present invention, and therefore will not be described in detail here. All electronic devices used in a method of an embodiment of the present invention fall within the scope of protection of the present invention.

[0160] The specific embodiments described herein are merely illustrative of the spirit of the present invention. Persons skilled in the art may make various modifications, additions, or substitutions to the described specific embodiments without departing from the spirit of the present invention or exceeding the scope of the present invention.

Claims

1. A humanoid robot gait control method based on imitation and inverse reinforcement learning, characterized in that: The following steps are involved: S1, collecting and preprocessing walking nodes and inertial data of humans and simulated humanoid robots, wherein the nodes represent body parts or joints of the robots; S2: The robot's joint data and the human's joint data are represented as graph structures. In the graph structure, each node represents a body part or joint of the robot or human, and the edge represents the connection between joints. The gait imitation similarity is calculated through the graph convolutional network. Simultaneously calculate the robot's stability, gait periodicity, and energy efficiency; among which, calculating gait imitation similarity includes: S2-1, representing the robot's joint data and the human's joint data as graph structures; S2-2, collect the sensor time series data from IMU, cut it according to the time segment, and calculate it according to the human acceleration data G 人 , robot acceleration data G 机 , human gyroscope data A 人 , robot gyroscope data A 机 Sequential splicing; S2-3, input the graph structure in S2-1 into the convolution layer and output the feature vector; S2-4, input the feature vector into the Transformer model; S2-5, input the S2-2 concatenated data into LSTM; S2-6, concatenate the outputs of the Transformer model and the LSTM model, and then input them into the fully connected layer to obtain the concatenated feature vector; S2-7, regress the concatenated feature vector to obtain the similarity between the robot and human gait data; S3, designing a reward function including walking stability, gait periodicity, imitation similarity and energy efficiency, wherein the reward function is a linear reward function; S4, uses a neural network as a policy model, inputs the robot state, and outputs gait control actions; S5, initialize the reward function; S6, use the PPO algorithm to train the policy model, combine the clipped probability ratio with the policy loss, value function loss and entropy regularization term to construct the optimization target, calculate the optimization target gradient and update the policy model parameters; S7, using maximum entropy inverse reinforcement learning to update the reward function; S8, simulation verifies the effectiveness of the gait control strategy.

2. The humanoid robot gait control method based on imitation and inverse reinforcement learning according to claim 1, characterized in that: In the collection of walking node and inertial data of humans and simulated humanoid robots, reflective marking points are installed on the main joints of the tester, and a force measurement platform is arranged on the ground to record gait contact force and impact force, and the inertial measurement element measures the target acceleration and speed; the main joints include ankles, knees, hips, waist, shoulders, and elbows.

3. The humanoid robot gait control method based on imitation and inverse reinforcement learning according to claim 1, characterized in that: The walking stability reward function is designed based on the trunk inclination angle: in, and are the pitch and roll angles of the trunk; The periodic reward function is designed based on the gait cycle consistency as follows: in, is the current gait cycle, is the target gait cycle; The energy efficiency reward function is designed based on the sum of squared joint torques: in, is the torque of the i-th joint; The trajectory similarity reward function uses a graph convolutional network to process joint nodes and is designed as follows: in, represents the coordinates of the robot's i-th joint node, Represents the coordinates of the i-th joint node of a person, represents the Euclidean distance; The overall reward function is expressed as: in, It is the robot state, It's action. is the weight.

4. The humanoid robot gait control method based on imitation and inverse reinforcement learning according to claim 1, characterized in that: The neural network includes an input layer, a hidden layer and an output layer, wherein the hidden layer uses a multi-layer perceptron combined with a long short-term memory network to process data.

5. The humanoid robot gait control method based on imitation and inverse reinforcement learning according to claim 1, characterized in that: The objective function of the clipping in step S6 is: in, is the advantage function, is the cropping range, is the clipping function, Calculates the strategy probability ratio based on the PPO algorithm.

6. The humanoid robot gait control method based on imitation and inverse reinforcement learning according to claim 5, characterized in that: The specific process of S6 includes: S6-1, the robot's current gait state and its corresponding control instructions Input the experience pool and calculate the reward value according to the reward function designed by S2 , record state-action pairs and reward value In the experience pool; S6-2, the state-action pairs in the experience pool Input the policy model designed in S4 , calculate the action probability distribution of the current strategy output, and calculate the strategy ratio based on the PPO algorithm: in, Is the current strategy in state Select Action probability; Is the strategy in state before the last update Select Action probability; S6-3, the strategy ratio calculated in the previous step Input cropping strategy update module; S6-4, input the policy ratio after the previous step into the target loss calculation module, combined with the policy loss and entropy regularization term and value function loss , construct the complete PPO optimization objective as: To improve the exploration and convergence of gait strategy, , is a hyperparameter; S6-5, input the optimization target calculated in the previous step into the gradient calculation module, use the back propagation algorithm to calculate the gradient of the strategy model, and update the parameters of the strategy model to make the gait control strategy more stable and adaptable to different environments.

7. A system for implementing the humanoid robot gait control method based on imitation and inverse reinforcement learning as described in any one of claims 1 to 6, characterized in that: include: Data acquisition module, which collects walking node and inertial data of humans and simulated humanoid robots, and extracts gait features; The gait similarity calculation module represents the robot's gait data and expert demonstration data as a graph structure and calculates the gait imitation similarity through a graph convolutional network; Simultaneously calculate robot stability, gait periodicity, and energy efficiency; Calculating gait imitation similarity includes: S2-1, representing the robot's joint data and the human's joint data as graph structures; S2-2, collect the sensor time series data from IMU, cut it according to the time segment, and calculate it according to the human acceleration data G 人 , robot acceleration data G 机 , human gyroscope data A 人 , robot gyroscope data A 机 Sequential splicing; S2-3, input the graph structure in S2-1 into the convolution layer and output the feature vector; S2-4, input the feature vector into the Transformer model; S2-5, input the S2-2 concatenated data into LSTM; S2-6, concatenate the outputs of the Transformer model and the LSTM model, and then input them into the fully connected layer to obtain the concatenated feature vector; S2-7, regress the concatenated feature vector to obtain the similarity between the robot and human gait data; The data processing module designs a reward function that considers walking stability, gait periodicity, imitation similarity, and energy efficiency. A neural network is used as the policy model, which inputs the robot state and outputs gait control actions. The reward function is initialized. The policy model is trained using the PPO algorithm. The specific process includes: Store interaction data and calculate the probability ratio of new and old strategies , by clipping the objective function to the probability ratio Perform trimming and set the trimmed probability ratio Combine the policy loss, value function loss and entropy regularization term to construct the optimization target, calculate the optimization target gradient and update the policy model parameters; Use maximum entropy inverse reinforcement learning to update the reward function; Simulation verification module, used to simulate and verify the effect of gait control strategy.

8. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs; When one or more programs are executed by the one or more processors, the one or more processors implement the humanoid robot gait control method based on imitation and inverse reinforcement learning as described in any one of claims 1 to 6.

9. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the humanoid robot gait control method based on imitation and inverse reinforcement learning as described in any one of claims 1 to 6 is implemented.

Images