Quadruped robot motion control method and system based on reinforcement learning

Abstract

The invention discloses a quadruped robot motion control method and system based on reinforcement learning, and the method can simulate and analyze the motion behavior of a quadruped robot in a virtual environment through building a simulation model of the quadruped robot, remarkably reduces the research and development cost, remarkably shortens the research and development period, converts a balance control problem into a Markov decision process, defines a state, an action and a reward function, and improves the reliability of the quadruped robot. A more intelligent and adaptive balance strategy can be realized, a deterministic strategy gradient algorithm and a simulation model are adopted for interactive training, an efficient balance control strategy can be obtained, robot behaviors are adjusted in real time to keep balance, a walking task is decomposed into two sub-tasks of foothold selection and gait generation by utilizing a hierarchical control idea, and the walking efficiency is improved. Through reinforcement learning, walking flexibility and stability of the robot on irregular terrains are improved, the irregular terrains are built on a simulation platform for experimental verification, robot performance can be safely and repeatedly evaluated, and a strategy is further optimized.

Description

Technical Field

[0001] The present invention relates to the technical field of robot motion control, and in particular to a quadruped robot motion control method and system based on reinforcement learning. Background Art

[0002] With the rapid development of robotics technology, quadruped robots have shown great application potential in exploration, rescue, transportation and other fields due to their excellent terrain adaptability and motion flexibility. However, the motion control of quadruped robots, especially walking on irregular terrain, is still a challenging topic. Traditional control methods often rely on precise mathematical models and complex control algorithms, which not only increases the complexity of the system and leads to increased costs, but also limits the robot's adaptability in unknown or complex environments. Summary of the invention

[0003] In view of this, the present invention proposes a quadruped robot motion control method and system based on reinforcement learning, which can effectively solve the defects of the prior art, such as complex system and weak adaptability in unknown or complex environments.

[0004] The technical solution of the present invention is achieved in this way:

[0005] A quadruped robot motion control method based on reinforcement learning, comprising:

[0006] Build a quadruped robot simulation model: Build a quadruped robot simulation model on the robot simulation platform to simulate the movement behavior of the quadruped robot;

[0007] Design a Markov decision process for balance control: Analyze the balance control problem of a quadruped robot, transform the balance control problem into a Markov decision process, and define the state space, action space, and reward function;

[0008] Training balance control strategy: In the Markov decision process of designing balance control, a deterministic policy gradient algorithm is used to interactively train with the constructed quadruped robot simulation model to obtain the balance control strategy;

[0009] Foothold selection and gait generation: Using the hierarchical control concept, the walking task of the quadruped robot on irregular terrain is decomposed into two levels, and the foothold selection and gait generation are realized through reinforcement learning.

[0010] Simulation verification: Build irregular terrain on the simulation platform and deploy the trained balance control strategy to the quadruped robot simulation model for experimental verification.

[0011] As a further optional solution of the quadruped robot motion control method based on reinforcement learning, the Markov decision process for designing balance control specifically includes:

[0012] Define the state space and select the features that can fully reflect the equilibrium state of the quadruped robot as state variables;

[0013] Determine the action space and define the set of actions that the quadruped robot can perform;

[0014] Design a reward function based on the quadruped robot's balance state, motion stability, and task completion;

[0015] The robot starts from the initial state and selects an action to execute based on the current state and strategy;

[0016] After executing the action, the robot transfers to the next state according to the state transition probability, and the robot obtains the corresponding reward according to the reward function;

[0017] Repeat the action execution process until the termination condition is reached.

[0018] As a further optional solution of the quadruped robot motion control method based on reinforcement learning, in the Markov decision process of designing balance control, a deterministic policy gradient algorithm is used to perform interactive training with the constructed quadruped robot simulation model to obtain a balance control strategy, which specifically includes:

[0019] Initialize the deterministic policy gradient algorithm, set the algorithm parameters, build the Actor network and Cr it ic network, and initialize their weights and biases;

[0020] In the simulation environment, perform actions based on the current state and the actions output by the Actor network, record the state, action, reward, and next state of each time step, and store these data in the experience replay buffer;

[0021] Extract data from the experience replay buffer as training samples, and use the training samples to update the weights of the Actor network and the Cr it ic network;

[0022] Use the updated Actor network to output a new action strategy;

[0023] Execute new actions in the simulation environment and continue to collect data and train the network;

[0024] The above steps are repeated until the predetermined number of training rounds is reached or the balance control strategy of the quadruped robot reaches a stable state.

[0025] As a further optional solution of the quadruped robot motion control method based on reinforcement learning, the hierarchical control idea is used to decompose the walking task of the quadruped robot on irregular terrain into two levels, and the foothold selection and gait generation are respectively realized through reinforcement learning, which specifically includes:

[0026] The walking task of the quadruped robot on irregular terrain is decomposed into two levels. The first level is foothold selection, which is responsible for planning suitable footholds in complex terrain. The second level is gait generation, which is responsible for generating stable gaits based on the selected footholds, so that the quadruped robot can move to these footholds in an orderly manner and maintain balance.

[0027] Reinforcement learning frameworks are constructed for the foothold selection problem and gait generation problem respectively, and the value function approximation method is used to build an advanced controller to complete the foothold selection task. The proximal strategy optimization algorithm is used to move the feet to the planned footholds in an orderly manner while maintaining the balance of the quadruped robot.

[0028] As a further optional solution of the quadruped robot motion control method based on reinforcement learning, the method of using value function approximation to construct an advanced controller to complete the foothold selection task specifically includes:

[0029] Define the current state space of the robot, including position, posture and velocity;

[0030] Define a set of optional footholds, each of which is represented by its position in three-dimensional space;

[0031] Design a reward function to encourage the robot to choose stable and safe landing points;

[0032] Use a deep neural network as a value function approximator, input the state space and optional landing points, and output the value estimate of each landing point;

[0033] According to the output of the value function approximator, the foothold with the highest value is selected as the robot's next action;

[0034] The optimization strategy is continuously iterated until the robot can stably select a suitable foothold on irregular terrain.

[0035] As a further optional solution of the reinforcement learning-based quadruped robot motion control method, the use of the proximal strategy optimization algorithm to orderly move the feet to the planned landing points while maintaining the balance of the quadruped robot specifically includes:

[0036] Define the current state space of the robot, including position, posture, speed, and angle and angular velocity of each joint;

[0037] Define the action space as the motion instructions for each joint;

[0038] Design a reward function to encourage the robot to generate stable gaits;

[0039] Use the proximal policy optimization algorithm to optimize the gait generation strategy;

[0040] According to the optimized gait generation strategy, the motion instructions of each joint are output. By adjusting the movement of the leg joints, the robot can move to the planned landing point in an orderly manner and maintain balance.

[0041] As a further optional solution of the reinforcement learning-based quadruped robot motion control method, the method also includes optimizing the center of gravity movement distance and speed of the quadruped robot in a static gait to achieve optimization of the static gait speed.

[0042] A quadruped robot motion control system based on reinforcement learning, comprising:

[0043] The quadruped robot simulation module is used to build a simulation model of the quadruped robot on the robot simulation platform and simulate the movement behavior of the quadruped robot;

[0044] The Markov decision process design module is used to analyze the balance control problem of the quadruped robot, transform the balance control problem into a Markov decision process, and define the state space, action space and reward function;

[0045] A balance control strategy training module is connected with the quadruped robot simulation module and the Markov decision process design module, and is used to interactively train the quadruped robot simulation model using a deterministic policy gradient algorithm to obtain a balance control strategy;

[0046] The foothold selection and gait generation module uses the hierarchical control concept to decompose the walking task of the quadruped robot on irregular terrain into two levels, and implements foothold selection and gait generation respectively through reinforcement learning;

[0047] The simulation verification module is used to build irregular terrain on the simulation platform and deploy the trained balance control strategy to the quadruped robot simulation model for experimental verification.

[0048] A computing device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements any one of the steps of the above-mentioned reinforcement learning-based quadruped robot motion control method when executing the computer program.

[0049] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any one of the above-mentioned reinforcement learning-based quadruped robot motion control methods.

[0050] The beneficial effects of the present invention are as follows: by constructing a simulation model of a quadruped robot, the movement behavior of the quadruped robot can be simulated and analyzed without actually manufacturing and testing the physical robot, which greatly reduces the R&D cost and shortens the R&D cycle, and allows developers to test the robot under various extreme conditions in a virtual environment to improve its performance and stability. The balance control problem of the quadruped robot is converted into a Markov decision process, making the problem more structured and easier to solve. By defining the state space, action space and reward function, the robot's behavior choices in different states and the consequences of the behavior (rewards or penalties) can be clearly described. This method helps to achieve a more intelligent and adaptive balance control strategy. By using a deterministic policy gradient algorithm and a simulation model for interactive training, an efficient balance control strategy can be obtained. The strategy can adjust the robot's behavior in real time according to its current state to maintain balance. Through a large number of training iterations, the strategy will be gradually optimized to improve the robot's balance ability and motion stability. Using the hierarchical control idea, the complex walking task is decomposed into two-level hierarchical problems, namely foothold selection and gait generation. This method helps to simplify the problem so that each sub-problem can be solved independently. By implementing these two sub-tasks separately through reinforcement learning, the robot can walk more flexibly and stably on irregular terrain. At the same time, this method also improves the robot's environmental adaptability and autonomous decision-making ability. By building irregular terrain on the simulation platform and experimentally verifying the trained balance control strategy, the robot's performance in various complex environments can be evaluated. This verification method is not only safe and reliable, but can also be repeated to further optimize and adjust the strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0052] Figure 1 A schematic diagram of a flow chart of a quadruped robot motion control method based on reinforcement learning according to the present invention;

[0053] Figure 2 A schematic diagram of the composition of a quadruped robot motion control system based on reinforcement learning according to the present invention;

[0054] Figure 3 The figure is a schematic diagram of the composition of a computing device according to the present invention. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] refer to Figures 1 to 3 , a quadruped robot motion control method based on reinforcement learning, comprising:

[0057] Build a quadruped robot simulation model: Build a quadruped robot simulation model on the robot simulation platform to simulate the movement behavior of the quadruped robot;

[0058] Design a Markov decision process for balance control: Analyze the balance control problem of a quadruped robot, transform the balance control problem into a Markov decision process, and define the state space, action space, and reward function;

[0059] Training balance control strategy: In the Markov decision process of designing balance control, a deterministic policy gradient algorithm is used to interactively train with the constructed quadruped robot simulation model to obtain the balance control strategy;

[0060] Foothold selection and gait generation: Using the hierarchical control concept, the walking task of the quadruped robot on irregular terrain is decomposed into two levels, and the foothold selection and gait generation are realized through reinforcement learning.

[0061] Simulation verification: Build irregular terrain on the simulation platform and deploy the trained balance control strategy to the quadruped robot simulation model for experimental verification.

[0062] In this embodiment, by constructing a simulation model of a quadruped robot, the movement behavior of the quadruped robot can be simulated and analyzed without actually manufacturing and testing a physical robot, which greatly reduces R&D costs and shortens the R&D cycle, and allows developers to test the robot under various extreme conditions in a virtual environment to improve its performance and stability. The balance control problem of the quadruped robot is converted into a Markov decision process, making the problem more structured and easier to solve. By defining the state space, action space, and reward function, the robot's behavior choices in different states and the consequences of the behavior (rewards or penalties) can be clearly described. This method helps to achieve a more intelligent and adaptive balance control strategy. By using a deterministic policy gradient algorithm and a simulation model for interactive training, an efficient balance control strategy can be obtained. The strategy can adjust the robot's behavior in real time according to its current state to maintain balance. Through a large number of training iterations, the strategy will be gradually optimized to improve the robot's balance ability and motion stability. Using the hierarchical control idea, the complex walking task is decomposed into two-level hierarchical problems, namely foothold selection and gait generation. This method helps to simplify the problem so that each sub-problem can be solved independently. By implementing these two sub-tasks separately through reinforcement learning, the robot can walk more flexibly and stably on irregular terrain. At the same time, this method also improves the robot's environmental adaptability and autonomous decision-making ability. By building irregular terrain on the simulation platform and experimentally verifying the trained balance control strategy, the robot's performance in various complex environments can be evaluated. This verification method is not only safe and reliable, but can also be repeated to further optimize and adjust the strategy.

[0063] Preferably, the Markov decision process for designing the balance control specifically includes:

[0064] Define the state space and select the features that can fully reflect the equilibrium state of the quadruped robot as state variables;

[0065] Determine the action space and define the set of actions that the quadruped robot can perform;

[0066] Design a reward function based on the quadruped robot's balance state, motion stability, and task completion;

[0067] The robot starts from the initial state and selects an action to execute based on the current state and strategy;

[0068] After executing the action, the robot transfers to the next state according to the state transition probability, and the robot obtains the corresponding reward according to the reward function;

[0069] Repeat the action execution process until the termination condition is reached.

[0070] In this embodiment, selecting features that can fully reflect the balance state of the quadruped robot as state variables is a key step in the design of the Markov decision process (MDP). By accurately describing the robot's posture, speed, position and other key information, the state space can accurately reflect the robot's current status, which helps the strategy to fully consider all relevant information when making decisions, thereby making more reasonable choices; defining the set of actions that the quadruped robot can perform provides a feasible behavior option for the strategy. The reasonable design of the action space can ensure that the robot has corresponding actions to perform in various states, thereby maintaining balance and stability. At the same time, the restriction of the action space also helps to simplify the problem and reduce the complexity of strategy learning; the reward function is a key element in guiding strategy learning in MDP. By designing the reward function according to the quadruped robot's balance state, motion stability and task completion, the strategy can be clearly told. The strategy knows which behaviors are beneficial and which are harmful. This clear feedback mechanism helps the strategy quickly learn the correct behavior pattern, thereby improving the robot's balance ability and motion stability. The robot starts from the initial state, selects an action to execute based on the current state and strategy, and transfers to the next state based on the state transition probability. This process simulates the movement process of the robot in the actual environment, allowing the strategy to continuously learn and optimize in the interaction with the environment. Through continuous trial and error and adjustment, the strategy can gradually adapt to various complex environments and improve the robot's autonomous decision-making ability. Repeating the action execution process until the termination condition is reached is the core link of MDP learning. Through a large amount of iterative training, the strategy can continuously accumulate experience and knowledge and gradually optimize its behavior pattern. This iterative learning mechanism enables the strategy to cope with various unknown and complex situations and improve the robot's adaptability and robustness.

[0071] Preferably, in the Markov decision process of designing the balance control, a deterministic policy gradient algorithm is used to perform interactive training with the constructed quadruped robot simulation model to obtain a balance control strategy, which specifically includes:

[0072] Initialize the deterministic policy gradient algorithm, set the algorithm parameters, build the Actor network and Cr it ic network, and initialize their weights and biases;

[0073] In the simulation environment, perform actions based on the current state and the actions output by the Actor network, record the state, action, reward, and next state of each time step, and store these data in the experience replay buffer;

[0074] Extract data from the experience replay buffer as training samples, and use the training samples to update the weights of the Actor network and the Cr it ic network;

[0075] Use the updated Actor network to output a new action strategy;

[0076] Execute new actions in the simulation environment and continue to collect data and train the network;

[0077] The above steps are repeated until the predetermined number of training rounds is reached or the balance control strategy of the quadruped robot reaches a stable state.

[0078] In this embodiment, by initializing the deterministic policy gradient algorithm and setting the algorithm parameters, such as learning rate, discount factor, experience replay buffer size, etc., the Actor network and the Cr it ic network are constructed and their weights and biases are initialized, which provides a basis for subsequent interactive training. This initialization process ensures that the starting state of the algorithm and the network is consistent, which helps to stabilize and control the subsequent training process; in the simulation environment, actions are performed according to the current state and the actions output by the Actor network, and the state, action, reward and next state of each time step are recorded. These data are stored in the experience replay buffer, which provides rich samples for subsequent network training. In this way, the algorithm can learn the actions that should be taken in different states and the consequences of these actions (i.e., rewards or penalties); data are extracted from the experience replay buffer as training samples, and these samples are used to update the Actor network and the Cr it ic network. The weights of the itic network and the experience replay mechanism help to break the time correlation between data, making network training more stable and efficient. At the same time, by continuously updating the network weights, the algorithm can gradually optimize its strategy and improve the balance control performance of the quadruped robot. Use the updated Actor network to output new action strategies and execute these actions in the simulation environment. Continue to collect data and conduct network training. This process is an iterative process. Through continuous trial and error and adjustment, the algorithm can gradually find the optimal strategy. This strategy iteration and optimization mechanism enables the quadruped robot to maintain balance and stability in various complex environments. Repeat the above steps until the predetermined number of training rounds is reached or the balance control strategy of the quadruped robot reaches a stable state. This training termination condition ensures that the algorithm has enough time to learn and optimize the strategy, while also avoiding the overfitting problem caused by overtraining.

[0079] It should be noted that for the Critic network, the extracted data is used to calculate its loss function, and its weights are updated by the gradient descent method. For the Actor network, its loss function is calculated based on the output of the Critic network and the current strategy, and its weights are updated by the gradient ascent method (or the corresponding optimization algorithm).

[0080] Preferably, the hierarchical control concept is used to decompose the walking task of the quadruped robot on irregular terrain into two levels, and the foothold selection and gait generation are respectively realized through reinforcement learning, which specifically includes:

[0081] The walking task of the quadruped robot on irregular terrain is decomposed into two levels. The first level is foothold selection, which is responsible for planning suitable footholds in complex terrain. The second level is gait generation, which is responsible for generating stable gaits based on the selected footholds, so that the quadruped robot can move to these footholds in an orderly manner and maintain balance.

[0082] Reinforcement learning frameworks are constructed for the foothold selection problem and gait generation problem respectively, and the value function approximation method is used to build an advanced controller to complete the foothold selection task. The proximal strategy optimization algorithm is used to move the feet to the planned footholds in an orderly manner while maintaining the balance of the quadruped robot.

[0083] In this embodiment, by constructing a reinforcement learning framework and applying the value function approximation method, the high-level controller can quickly plan a suitable foothold, which enables the quadruped robot to quickly find a safe walking path in complex terrain, avoiding unnecessary stops and detours, thereby improving walking efficiency. This technical solution enables the quadruped robot to flexibly adjust the foothold selection strategy according to different terrain characteristics. Whether it is on rugged mountains, soft sand or flat roads, the quadruped robot can find the walking route that suits it best; through the proximal strategy optimization algorithm, the quadruped robot can generate a stable gait according to the selected footholds. This gait not only enables the quadruped robot to move to these footholds in an orderly manner, but also maintains balance during walking, avoiding uneven terrain or sudden changes. To prevent falls or imbalances caused by the gait generation controller, the quadruped robot can automatically adjust the walking posture of the quadruped robot according to different walking speeds and terrain characteristics, which enables the quadruped robot to maintain the best posture during walking, thereby improving the walking comfort and stability; the combination of hierarchical control ideas and reinforcement learning methods makes the entire system more robust when facing complex environments and unknown challenges. Even in some extreme cases, such as sudden terrain changes or sensor failures, the quadruped robot can maintain walking stability and efficiency through learning and adjustment. Through simulation and reinforcement learning methods, the walking strategy of the quadruped robot can be verified and optimized without actually manufacturing and testing the physical robot, which greatly reduces the R&D cost, shortens the R&D cycle, and improves the reliability and performance of the system.

[0084] Preferably, the method of using the value function approximation method to construct an advanced controller to complete the foothold selection task specifically includes:

[0085] Define the current state space of the robot, including position, posture and velocity;

[0086] Define a set of optional footholds, each of which is represented by its position in three-dimensional space;

[0087] Design a reward function to encourage the robot to choose stable and safe landing points;

[0088] Use a deep neural network as a value function approximator, input the state space and optional landing points, and output the value estimate of each landing point;

[0089] According to the output of the value function approximator, the foothold with the highest value is selected as the robot's next action;

[0090] The optimization strategy is continuously iterated until the robot can stably select a suitable foothold on irregular terrain.

[0091] In this embodiment, by defining the robot's current state space (including position, posture and speed) and a set of optional footholds, this technical solution enables the robot to flexibly adjust its foothold selection strategy according to different terrain features. The value function approximator can learn the value distribution of footholds under different terrains, thereby guiding the robot to select stable and safe footholds, avoiding falls or imbalances caused by uneven terrain or sudden changes. A reward function is designed to encourage the robot to choose stable and safe footholds, which helps the robot avoid unnecessary stops and detours during walking, thereby optimizing the walking path. Through the output of the value function approximator, the robot The robot can choose the landing point with the highest value as the next action, which ensures that each step is optimal, thereby improving the overall walking efficiency; using deep neural networks as value function approximators, this technical solution can handle high-dimensional state spaces and complex landing point selection problems. The powerful representation ability of deep neural networks enables the robot to quickly and accurately evaluate the value of each landing point, thereby making the best decision; the process of continuously iterating the optimization strategy enables the robot to gradually adapt to various complex terrains and unknown challenges. Even in some extreme cases, such as sudden changes in terrain or sensor failures, the robot can maintain walking stability and efficiency through learning and adjustment.

[0092] It should be noted that the reinforcement learning implementation steps for the foothold selection problem are:

[0093] State space definition: define the current state space of the robot, including key information such as position, posture, speed, etc., and extract terrain features such as height, slope, ground material, etc. as a supplement to the state space;

[0094] Action space definition: define a set of optional footholds, each of which is represented by its position in three-dimensional space, and the robot's action is to move from the current state to a selected foothold;

[0095] Reward function design: Design a reward function to encourage the robot to choose a stable and safe landing point. The reward function may include rewards for reaching the target landing point, rewards for maintaining balance, rewards for avoiding collisions, etc.

[0096] Value function approximation: Use a deep neural network as a value function approximator, input the state space and optional landing points, output the value estimate of each landing point, and update the weight of the value function through a reinforcement learning algorithm (such as Qlearning, DQN, etc.) so that it can accurately estimate the value of each landing point;

[0097] Strategy optimization: Based on the output of the value function approximator, the foothold with the highest value is selected as the robot's next action, and the strategy is continuously iterated until the robot can stably select a suitable foothold on irregular terrain.

[0098] Preferably, the method of using the proximal strategy optimization algorithm to orderly move the feet to the planned footholds while maintaining the balance of the quadruped robot specifically includes:

[0099] Define the current state space of the robot, including position, posture, speed, and angle and angular velocity of each joint;

[0100] Define the action space as the motion instructions for each joint;

[0101] Design a reward function to encourage the robot to generate stable gaits;

[0102] Use the proximal policy optimization algorithm to optimize the gait generation strategy;

[0103] According to the optimized gait generation strategy, the motion instructions of each joint are output. By adjusting the movement of the leg joints, the robot can move to the planned landing point in an orderly manner and maintain balance.

[0104] In this embodiment, through the proximal policy optimization (PPO) algorithm, the robot can learn an efficient gait generation strategy, which can flexibly adjust the motion instructions of each joint according to different terrains and obstacles, so as to generate a stable gait. The PPO algorithm avoids the problems of training instability and low sample efficiency that may occur in the traditional policy gradient algorithm by limiting the amplitude of the strategy update, making the optimization of the gait generation strategy more stable and efficient; according to the optimized gait generation strategy, the robot can move to the planned foothold in an orderly manner. During this process, the robot can accurately judge the position and distance of each foothold, and adjust the movement of the leg joints accordingly to ensure that each step falls on the predetermined foothold. Through the optimization of the PPO algorithm, the robot can maintain a stable gait and rhythm during movement. The PPO algorithm optimizes the gait generation strategy, so that the robot can quickly respond to terrain changes and external interference during movement, and maintain body stability and balance by adjusting the torque and angle of the leg joints. The gait generation strategy learned by the PPO algorithm can not only maintain balance on flat terrain, but also show good balance in complex terrain (such as slopes, rugged roads, etc.); through the optimization of the PPO algorithm, the robot can quickly learn efficient gait generation strategies in a simulation environment without a lot of actual testing and adjustments, which greatly reduces R&D costs and time and improves R&D efficiency.

[0105] It should be noted that the reinforcement learning implementation steps for the gait generation problem are:

[0106] State space and action space definition: define the current state space of the robot, including position, posture, speed, angle and angular velocity of each joint, etc., and define the action space as the motion instructions of each joint, such as torque or angle adjustment;

[0107] Reward function design: Design a reward function to encourage the robot to generate a stable and efficient gait. The reward function may include rewards for reaching the target foothold, rewards for maintaining balance, rewards for reducing energy consumption, etc.

[0108] Proximal strategy optimization algorithm: Use the PPO algorithm to optimize the gait generation strategy, define the strategy network and value network, output the action probability distribution and value estimation respectively, and continuously iterate and optimize the gait generation strategy through steps such as collecting data, calculating the advantage function, updating the strategy network and value network;

[0109] Gait generation and balance control: According to the optimized gait generation strategy, the motion instructions of each joint are output. By adjusting the movement of the leg joints, the robot can move to the planned landing point in an orderly manner and maintain balance.

[0110] Preferably, the method further comprises optimizing the distance and speed of movement of the center of gravity of the quadruped robot in a static gait to achieve optimization of the static gait speed.

[0111] In this embodiment, through the optimization algorithm, the robot can calculate the shortest center of gravity movement trajectory in a static gait, thereby reducing the distance the center of gravity moves during walking. This not only reduces the robot's energy consumption, but also improves its walking efficiency. Reducing the center of gravity movement distance also helps the robot maintain a more stable posture during walking, reducing the risk of falling or imbalance caused by center of gravity fluctuations. By solving the maximum body and foot movement speeds under the joint angular velocity constraints, the robot's walking speed is improved, which enables the robot to move to the target position faster while maintaining static gait stability. Increasing the walking speed not only enhances the robot's mobility, but also This enables it to respond to external instructions and task requirements more quickly, improving overall work efficiency; in the process of optimizing the center of gravity moving distance and speed, the technical solution also takes into account the stability constraints of the robot, by ensuring that the center of gravity of the robot is always in the stable area during walking, thereby ensuring its stability in static gait. The consideration of stability constraints enables the robot to maintain a more stable posture when facing complex terrain and obstacles, avoiding falls or failures caused by loss of balance; by optimizing the center of gravity moving distance and speed, the technical solution also reduces the robot's energy consumption during walking. Reducing unnecessary energy consumption not only extends the robot's working time, but also reduces its operating costs.

[0112] A quadruped robot motion control system based on reinforcement learning, comprising:

[0113] The quadruped robot simulation module is used to build a simulation model of the quadruped robot on the robot simulation platform and simulate the movement behavior of the quadruped robot;

[0114] The Markov decision process design module is used to analyze the balance control problem of the quadruped robot, transform the balance control problem into a Markov decision process, and define the state space, action space and reward function;

[0115] A balance control strategy training module is connected with the quadruped robot simulation module and the Markov decision process design module, and is used to interactively train the quadruped robot simulation model using a deterministic policy gradient algorithm to obtain a balance control strategy;

[0116] The foothold selection and gait generation module uses the hierarchical control concept to decompose the walking task of the quadruped robot on irregular terrain into two levels, and implements foothold selection and gait generation respectively through reinforcement learning;

[0117] The simulation verification module is used to build irregular terrain on the simulation platform and deploy the trained balance control strategy to the quadruped robot simulation model for experimental verification.

[0118] A computing device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements any one of the steps of the above-mentioned reinforcement learning-based quadruped robot motion control method when executing the computer program.

[0119] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any one of the above-mentioned reinforcement learning-based quadruped robot motion control methods.

[0120] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A quadruped robot motion control method based on reinforcement learning, characterized in that: include: Build a quadruped robot simulation model: Build a quadruped robot simulation model on the robot simulation platform to simulate the movement behavior of the quadruped robot; Design a Markov decision process for balance control: Analyze the balance control problem of a quadruped robot, transform the balance control problem into a Markov decision process, and define the state space, action space, and reward function; Training balance control strategy: In the Markov decision process of designing balance control, a deterministic policy gradient algorithm is used to interactively train with the constructed quadruped robot simulation model to obtain the balance control strategy; Foothold selection and gait generation: Using the hierarchical control concept, the walking task of the quadruped robot on irregular terrain is decomposed into two levels, and foothold selection and gait generation are achieved through reinforcement learning. Simulation verification: Build irregular terrain on the simulation platform and deploy the trained balance control strategy to the quadruped robot simulation model for experimental verification.

2. A quadruped robot motion control method based on reinforcement learning according to claim 1, characterized in that: The Markov decision process for designing equilibrium control specifically includes: Define the state space and select the features that can fully reflect the equilibrium state of the quadruped robot as state variables; Determine the action space and define the set of actions that the quadruped robot can perform; Design a reward function based on the quadruped robot's balance state, motion stability, and task completion; The robot starts from the initial state and selects an action to execute based on the current state and strategy; After executing the action, the robot transfers to the next state according to the state transition probability, and the robot obtains the corresponding reward according to the reward function; Repeat the action execution process until the termination condition is reached.

3. A quadruped robot motion control method based on reinforcement learning according to claim 2, characterized in that: In the Markov decision process of designing balance control, a deterministic policy gradient algorithm is used to perform interactive training with the constructed quadruped robot simulation model to obtain a balance control strategy, which specifically includes: Initialize the deterministic policy gradient algorithm, set the algorithm parameters, build the Actor network and Critic network, and initialize their weights and biases; In the simulation environment, perform actions based on the current state and the actions output by the Actor network, record the state, action, reward, and next state of each time step, and store these data in the experience replay buffer; Extract data from the experience replay buffer as training samples, and use the training samples to update the weights of the Actor network and the Critic network; Use the updated Actor network to output a new action strategy; Execute new actions in the simulation environment and continue to collect data and train the network; The above steps are repeated until the predetermined number of training rounds is reached or the balance control strategy of the quadruped robot reaches a stable state.

4. A quadruped robot motion control method based on reinforcement learning according to claim 3, characterized in that: The hierarchical control concept is used to decompose the walking task of the quadruped robot on irregular terrain into two levels, and the foothold selection and gait generation are realized respectively through reinforcement learning, which specifically includes: The walking task of the quadruped robot on irregular terrain is decomposed into two levels. The first level is foothold selection, which is responsible for planning suitable footholds in complex terrain. The second level is gait generation, which is responsible for generating stable gaits based on the selected footholds, so that the quadruped robot can move to these footholds in an orderly manner and maintain balance. Reinforcement learning frameworks are constructed for the foothold selection problem and gait generation problem respectively, and the value function approximation method is used to build an advanced controller to complete the foothold selection task. The proximal strategy optimization algorithm is used to move the feet to the planned footholds in an orderly manner while maintaining the balance of the quadruped robot.

5. A quadruped robot motion control method based on reinforcement learning according to claim 4, characterized in that: The method of using value function approximation to construct an advanced controller to complete the foothold selection task specifically includes: Define the current state space of the robot, including position, posture and velocity; Define a set of optional footholds, each of which is represented by its position in three-dimensional space; Design a reward function to encourage the robot to choose stable and safe landing points; Use a deep neural network as a value function approximator, input the state space and optional landing points, and output the value estimate of each landing point; According to the output of the value function approximator, the foothold with the highest value is selected as the robot's next action; The optimization strategy is continuously iterated until the robot can stably select a suitable foothold on irregular terrain.

6. A quadruped robot motion control method based on reinforcement learning according to claim 5, characterized in that: The use of the proximal strategy optimization algorithm to orderly move the feet to the planned footholds while maintaining the balance of the quadruped robot specifically includes: Define the current state space of the robot, including position, posture, speed, and angle and angular velocity of each joint; Define the action space as the motion instructions for each joint; Design a reward function to encourage the robot to generate stable gaits; Use the proximal policy optimization algorithm to optimize the gait generation strategy; According to the optimized gait generation strategy, the motion instructions of each joint are output. By adjusting the movement of the leg joints, the robot can move to the planned landing point in an orderly manner and maintain balance.

7. A quadruped robot motion control method based on reinforcement learning according to claim 6, characterized in that: The method also includes optimizing the center of gravity moving distance and speed of the quadruped robot in a static gait to achieve optimization of the static gait speed.

8. A quadruped robot motion control system based on reinforcement learning, characterized in that: include: The quadruped robot simulation module is used to build a simulation model of the quadruped robot on the robot simulation platform and simulate the movement behavior of the quadruped robot; The Markov decision process design module is used to analyze the balance control problem of the quadruped robot, transform the balance control problem into a Markov decision process, and define the state space, action space and reward function; A balance control strategy training module is connected with the quadruped robot simulation module and the Markov decision process design module, and is used to interactively train the quadruped robot simulation model using a deterministic policy gradient algorithm to obtain a balance control strategy; The foothold selection and gait generation module uses the hierarchical control concept to decompose the walking task of the quadruped robot on irregular terrain into two levels, and implements foothold selection and gait generation respectively through reinforcement learning; The simulation verification module is used to build irregular terrain on the simulation platform and deploy the trained balance control strategy to the quadruped robot simulation model for experimental verification.

9. A computing device, characterized in that The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the reinforcement learning-based quadruped robot motion control method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, implements the steps of the reinforcement learning-based quadruped robot motion control method described in any one of claims 1 to 7.

Images