Reinforcement learning motion planning method and system for quadruped robots based on depth vision

Abstract

The present invention discloses a method and system for motion planning of a quadruped robot based on reinforcement learning of deep vision, and relates to the technical field of motion planning of quadruped robots. The method comprises the steps of: building a simulation environment, modeling staircase terrains with different slopes and robot dynamic parameters, wherein the staircase terrain model includes staircase terrains with different difficulty differences; loading the robot model to move in the staircase terrain model, tracking the robot's motion status, and obtaining a speed tracking effect, wherein the speed tracking effect is obtained by strategically evaluating the joint torque value and the corresponding linear motion velocity output by the teacher strategy network during the robot's movement; evaluating the speed tracking effect, and adaptively adjusting the environmental parameters in the staircase terrain model according to the evaluation results. The present invention can effectively learn irregular staircase terrains through a reinforcement learning framework based on deep visual information, so that the quadruped robot can adaptively operate stably in complex environments.

Description

Technical Field

[0001] The present invention relates to the technical field of quadruped robot motion planning, and in particular to a quadruped robot reinforcement learning motion planning method and system based on depth vision. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Compared to wheeled robots, quadruped robots offer significant advantages in navigating complex and rugged terrain. This is because quadruped robots can dynamically adjust their footholds based on the terrain, allowing them to better navigate rugged, irregular, or dynamically changing terrain. Consequently, quadruped robots hold broad application prospects in diverse and complex scenarios, including mountainous areas, urban ruins, rescue missions, and outdoor adventures. However, motion planning for quadruped robots based on external perception remains a challenging task. The motion planner must process large amounts of sensory data and perform complex calculations and decision-making to ensure smooth movement in dynamically changing environments.

[0004] In recent years, with the development of deep learning and reinforcement learning technologies, research on motion planning for quadruped robots has made significant progress. Learning-based methods for motion planning and control enable robots to autonomously explore and learn without human intervention, thus achieving more flexible locomotion. These methods not only help robots complete tasks in known environments but also improve their ability to adapt to changes in unknown environments, especially in complex terrain and unforeseen environmental conditions. However, despite the powerful autonomous learning capabilities provided by reinforcement learning, existing motion planning methods for quadruped robots still face a number of challenges in practical applications.

[0005] Existing methods for reinforcement learning motion planning for quadruped robots operating on complex terrain, particularly irregular staircases, often suffer from poor terrain training adaptability, low learning efficiency, and an over-reliance on manually specified movement directions. These methods are unable to effectively address the dynamic challenges of staircases, resulting in insufficient autonomous decision-making and adaptive capabilities. Therefore, effectively learning irregular staircases to enable quadruped robots to adaptively and stably operate in complex environments has become a pressing challenge for existing technologies. Summary of the Invention

[0006] In response to the shortcomings of the existing technology, the purpose of the present invention is to provide a quadruped robot reinforcement learning motion planning method and system based on deep vision, which adds a visual perception module to the motion planning task in an irregular stair environment. The robot can obtain terrain information from visual information, autonomously predict heading and plan actions.

[0007] In order to achieve the above object, the present invention is implemented through the following technical solutions:

[0008] A first aspect of the present invention provides a quadruped robot reinforcement learning motion planning method based on depth vision, comprising the following steps:

[0009] Building a simulation environment, modeling stairs with different slopes to obtain a stair terrain model, and modeling the robot's dynamic parameters to obtain a robot model, wherein the stair terrain model includes stairs with different levels of difficulty;

[0010] Load the robot model and let it move in the stair terrain model. Track the robot's motion to obtain the speed tracking effect. The speed tracking effect is obtained by evaluating the joint torque value and the corresponding linear velocity output by the teacher strategy network during the robot's motion.

[0011] The speed tracking effect is evaluated, and the environmental parameters in the stair terrain model are adaptively adjusted according to the evaluation results.

[0012] Furthermore, the specific steps of loading the robot model and tracking the robot's motion status in the stair terrain model are as follows:

[0013] Obtain robot depth vision information and robot body perception information;

[0014] Reconstructing a local height map based on the robot's deep vision information;

[0015] The teacher strategy network is used to process the local height map, robot proprioception information and privilege information to obtain the torque value of each joint of the robot. The torque value of each joint is combined with the linear velocity of motion to perform strategy evaluation and obtain the speed tracking effect.

[0016] Furthermore, the specific steps for reconstructing the local height map based on the robot's depth vision information are as follows:

[0017] Initial 3D point cloud data is obtained, filtered and coordinate transformed, and then a local 2D height map is generated as a local height map through projection and bilinear interpolation algorithms.

[0018] Furthermore, the specific steps of using the teacher strategy network to process the local height map, robot body perception information and privilege information are as follows:

[0019] Building a teacher strategy network;

[0020] The teacher strategy network is trained based on reinforcement learning to obtain a trained teacher strategy network;

[0021] The teacher strategy network is used to process the local height map, robot proprioception information and privileged information to obtain the torque value of each joint of the robot.

[0022] Furthermore, the specific steps for training the teacher strategy network based on reinforcement learning are as follows:

[0023] Using multi-layer perceptrons to train teacher strategies;

[0024] Build a supervised reinforcement learning training framework to distill the student policy network from the trained teacher policy network.

[0025] Furthermore, the specific steps for evaluating the strategy by combining the torque value of each joint with the linear velocity of motion are as follows:

[0026] The torque values ​​of each joint of the robot's legs and the corresponding linear motion speed output by the teacher strategy network are input into the strategy evaluation network. The reward signal of the strategy evaluation network is used to generate the robot's motion decision in complex terrain as a speed tracking effect.

[0027] Furthermore, the speed tracking effect is evaluated, and the specific steps for adaptively adjusting the environmental parameters in the stair terrain model according to the evaluation results are as follows:

[0028] At the end of each round of training, the root mean square error (RMSE) between the actual speed of the robot and the target speed is calculated. If the RMSE is lower than the preset threshold, the terrain difficulty signal is increased; if it is higher than the threshold, the terrain difficulty signal is decreased; otherwise, the current terrain difficulty signal is maintained.

[0029] A second aspect of the present invention provides a quadruped robot reinforcement learning motion planning system based on depth vision, comprising:

[0030] The simulation building module is configured to build a simulation environment, model stair terrains with different slopes to obtain a stair terrain model, and model robot dynamic parameters to obtain a robot model, wherein the stair terrain model includes stair terrains with different difficulty differences;

[0031] A motion tracking module is configured to load a robot model and move it in a staircase terrain model, track the robot's motion status, and obtain a speed tracking effect, wherein the speed tracking effect is obtained by strategically evaluating the joint torque values ​​and corresponding motion linear velocities output by the teacher strategy network during the robot's motion;

[0032] The effect evaluation module is configured to evaluate the speed tracking effect and adaptively adjust the environmental parameters in the stair terrain model according to the evaluation results.

[0033] A third aspect of the present invention provides a medium having a program stored thereon, which, when executed by a processor, implements the steps in the depth vision-based quadruped robot reinforcement learning motion planning method as described in the first aspect of the present invention.

[0034] The fourth aspect of the present invention provides a device comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the method for reinforcement learning motion planning of a quadruped robot based on depth vision as described in the first aspect of the present invention are implemented.

[0035] One or more of the above technical solutions have the following beneficial effects:

[0036] The present invention discloses a method and system for reinforcement learning motion planning of a quadruped robot based on depth vision. First, based on the three-dimensional point cloud data obtained by the depth camera carried by the quadruped robot, a local height map centered on the robot is constructed, and the proprioceptive information (such as joint position, speed, angular velocity, etc.) and other privileged information (such as terrain friction coefficient, target path points, etc.) are combined as input to train the teacher strategy; the teacher strategy adopts a model-free reinforcement learning method, and outputs appropriate leg joint torque with the support of local height map and proprioceptive information; during the training process, a double distillation training framework is adopted, and the knowledge transfer of the teacher-student model is used to effectively improve the training efficiency and reduce the system deployment cost; the student strategy gradually optimizes its motion decision-making process by learning the behavior of the teacher strategy, thereby realizing efficient motion control of the robot in complex terrain. ; In order to improve the efficiency and adaptability of the training process, a terrain course improvement module is added to dynamically adjust the difficulty of the training terrain to ensure that the robot learns in more difficult terrain and avoid premature training saturation, and adjust the terrain difficulty according to the robot's performance to ensure the gradual progression of training; the local height map and proprioception information are updated in real time during the training process to ensure that the robot obtains the latest environmental information in order to accurately predict and execute the next movement; in addition, the present invention reduces the dependence on external perception sensors and utilizes the image data of the depth camera to reduce the amount of calculation and improve the robot's autonomous navigation capability, so that the robot can better cope with complex and dynamically changing environments; through the comprehensive application of the above technologies, the present invention provides an efficient and reliable motion planning method for quadruped robots, which improves the robot's movement ability and decision-making efficiency in complex terrain.

[0037] Compared with traditional quadruped robot motion planning methods, the present invention adopts a reinforcement learning framework based on deep visual information, which significantly improves the robot's motion ability in complex environments. First, in the teacher strategy training stage, the point cloud data collected by the depth camera is used to reconstruct the local height map, and combined with proprioception information and other privileged information as input to train the robot strategy; at the same time, the target path point is preset in the terrain environment as a guide for the robot's forward direction, which improves the accuracy and adaptability of terrain perception and avoids the problem of over-reliance on manually specified travel directions; the three-dimensional point cloud data of the depth camera can reconstruct a more accurate height map, which makes up for the problem of missing details or errors that may exist when directly calling the API. Secondly, in the student strategy training stage, the CNN-GRU network architecture is used instead of the preset path point input, and the depth image is used as input. It is combined with the DAgger algorithm for training to autonomously predict the optimal heading, thereby improving the flexibility and autonomy of path planning; through the distillation method, the student strategy learns the behavior of the teacher strategy to ensure that when relying only on depth images and proprioception information, it generates motion decisions consistent with the teacher strategy. Finally, to address the problems of static terrain design and non-targeted training in traditional reinforcement learning, the present invention designs a terrain course improvement module, which dynamically adjusts the terrain difficulty according to the robot's performance, evaluates the strategy performance in real time, optimizes the terrain parameters, enhances the robot's ability to cope with more complex environments, and improves the adaptability and efficiency of training.

[0038] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0040] Figure 1 This is an overall flow chart of a method for motion planning for a quadruped robot using reinforcement learning based on depth vision according to the first embodiment of the present invention;

[0041] Figure 2 This is a flow chart of the policy update control in accordance with the first embodiment of the present invention;

[0042] Figure 3 This is a teacher-student strategy framework diagram of Example 1 of the present invention;

[0043] Figure 4 This is a schematic diagram of the movement of a quadruped robot on a staircase terrain according to Example 1 of the present invention. DETAILED DESCRIPTION

[0044] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0045] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations;

[0046] Example 1:

[0047] With the widespread adoption of quadruped robots in various fields, including industrial manufacturing, medical services, logistics and distribution, and military applications, their intelligence and environmental adaptability are continuously improving. Compared to wheeled robots, quadruped robots possess greater dynamic stability and environmental adaptability, enabling them to navigate complex terrain and perform more challenging tasks. When faced with dynamic and irregular terrain, especially in complex environments such as rugged stairways, quadruped robots must possess a high degree of autonomy and flexibility to complete their tasks.

[0048] Currently, for the problem of quadruped robot motion planning under irregular stair terrain, a reinforcement learning motion planning method for quadruped robots based on deep visual information is being studied. To achieve the reinforcement learning motion planning problem for quadruped robots, multiple issues need to be considered:

[0049] One is the accuracy of terrain data collection. Directly collecting terrain data through the terrain height reading function in Isaac Gym and using it as privileged information for motion planning has limitations when dealing with complex and dynamic terrain. This is especially true when the environment changes rapidly or details change. The terrain data directly generated in the simulation struggles to reflect subtle changes in the local terrain or new obstacles, resulting in the robot being unable to perceive and make appropriate motion decisions in a timely manner. Furthermore, strategies trained using terrain data read by the function are typically only applicable in simulation environments and cannot be effectively applied in the real world. Furthermore, differences between simulation and actual environments can lead to insufficient adaptability and autonomous decision-making capabilities in actual robot deployment.

[0050] Second, the autonomy and flexibility of path planning. Robots typically rely on externally input direction and speed commands. This reliance on manual instructions limits the robot's autonomous navigation capabilities, especially in complex environments, where it cannot independently determine its direction of travel or autonomously avoid obstacles. Robots can only follow predetermined paths and lack the ability to respond to environmental changes in real time. This results in insufficient flexibility and adaptability in dynamic or unknown environments. In particular, when faced with dynamic obstacles or sudden environmental changes, the robot's path cannot be adjusted in a timely manner, affecting its overall performance.

[0051] The third issue is the dynamic adaptability of movement strategies. During reinforcement learning training, terrain and environmental design are typically static, lacking dynamic adjustment and response mechanisms. Robots are trained under fixed environmental conditions, which makes them unable to effectively adapt to changing terrain or unknown challenges in the environment. This is especially true in dynamic environments where terrain and obstacles change, and robots may encounter unseen terrain. This results in insufficient generalization capabilities for trained models, making it difficult to cope with the complex changes in real-world applications.

[0052] In order to solve the problems of poor terrain training adaptability, low efficiency and over-reliance on manually specified travel directions, the first embodiment of the present invention provides a quadruped robot reinforcement learning motion planning method based on depth vision, such as Figure 1 As shown. First, create irregular staircase terrains of different specifications, import the robot model and randomize the parameters. Afterwards, obtain the robot's body perception information and deep visual information, build a terrain reconstructor to construct a local height map. In addition, the privileged information is combined with the multi-layer perceptron (MLP) to train the teacher strategy and construct a terrain improvement module. Finally, a supervised training framework is built to distill the student strategy. The present invention adopts a double distillation training framework, which improves the training efficiency and reduces the system deployment cost with the help of knowledge transfer between the teacher and student models; through the deep visual network architecture, it can autonomously infer the optimal heading from the low-frequency depth image, while reducing the external perception calculation amount, and improving the robot's autonomous navigation ability, so that it can cope with unknown changes in dynamic and complex environments; at the same time, a terrain course improvement module is introduced, which encourages the robot to learn in more difficult terrains by dynamically adjusting the terrain difficulty during training, thereby improving the targeted training and avoiding premature training saturation.

[0053] The method specifically comprises the following steps:

[0054] Step 1: Build a simulation environment, model the terrain of stairs with different slopes to obtain a stair terrain model, model the robot dynamic parameters to obtain a robot model.

[0055] Step 1.1: Model the staircase terrain with different slopes to obtain a staircase terrain model. The staircase terrain model includes staircase terrains with different difficulty levels.

[0056] In one specific implementation, the Isaac Gym physics engine is used to build a simulation environment, including modeling staircases with varying slopes and robot dynamics. The staircases constructed within the simulation environment are composed of rectangular blocks of varying sizes, friction coefficients, and quantities, defined in code. This embodiment designs three types of staircases with distinctly different levels of difficulty as the building blocks of complex terrain: low stairs, standard stairs, and high stairs. Each type of terrain is further categorized by stair inclination: mildly inclined or severely inclined, for a total of six terrain types. A target path point is set at the center of each step.

[0057] Step 1.2: Identify the robot's dynamic parameters and build a robot model.

[0058] The robot model is loaded by importing the URDF file, and the domain randomization method is used to dynamically adjust the robot's dynamic parameters, including the mass of each joint of the robot, the mass of the body, the body inertia and the terrain friction coefficient, etc., and additional torque and force are added as disturbance terms to improve the robot's anti-interference ability.

[0059] According to the simulation environment in which the quadruped robot is located, the depth camera information equipped on the robot head and the robot's proprioception information are obtained to train the autonomous movement strategy for irregular stair terrain.

[0060] Step 2: Load the robot model and move it through the staircase terrain model. Track the robot's motion to obtain the speed tracking effect. The speed tracking effect is obtained by evaluating the joint torque values ​​and corresponding linear velocity output by the teacher strategy network during the robot's motion.

[0061] Step 2.1: Obtain robot depth vision information and robot proprioception information.

[0062] The robot model is loaded at the starting position of each type of terrain, and the robot's current proprioception information is obtained, and the three-dimensional point cloud information in the depth camera worn by the robot is used as depth vision information.

[0063] In a specific embodiment, proprioception information is obtained by sensors inside the robot. The IMU, joint encoders, and torque sensors inside the robot body are used to provide proprioception information such as the position and speed of each joint of the robot's legs, the angular velocity of the body, the posture angle, and the sole contact judgment, providing the strategy network with accurate numerical values ​​of the robot's current state.

[0064] The robot's proprioceptive information can be represented as a 53-dimensional feature vector, denoted as:

[0065] .

[0066] in, is a set of feature vectors, including 3D angular velocity , 2D pitch and roll angles read by imu , yaw angle , yaw angle scaling , the yaw angle at the next moment , 3D control commands , 2D environment type judgment , 12-dimensional joint position , 12-dimensional joint velocity , 12-dimensional history commands and 4-dimensional plantar contact judgment .

[0067] The depth visual information is presented as three-dimensional point cloud data in the teacher strategy stage. After filtering and coordinate transformation of the initial point cloud data, a local two-dimensional height map is generated through projection and bilinear interpolation algorithm to reflect the change of terrain height. In the student strategy stage, eight consecutive images of size The depth camera on the robot's head can provide terrain environment information to help the robot plan its actions.

[0068] Step 2.2: Reconstruct the local height map based on the robot's depth vision information.

[0069] In one specific embodiment, the robot reads the 3D point cloud information from the depth camera and reconstructs a local height map centered on the robot. Specifically, initial 3D point cloud data is obtained from the depth camera on the robot's head. After filtering and coordinate conversion, a projection and bilinear interpolation algorithm are used to generate a local 2D height map, which serves as the local map to reflect terrain height changes.

[0070] The generated local height map is combined with proprioception information and other privileged information and input into the policy network composed of MLP to train the teacher motion policy. During each training or operation, the module continuously updates the local map to ensure that the robot's motion decisions are based on the latest terrain data.

[0071] Step 2.3: Use the teacher strategy network to process the local height map, robot body perception information and privilege information to obtain the torque value of each joint of the robot and combine it with the linear velocity of the robot body to evaluate the strategy and obtain the speed tracking effect.

[0072] like Figure 3 As shown, it includes the teacher strategy network of the first stage Training process and student strategy network in the second stage Training process. In the first stage, the teacher policy is trained using an MLP, taking as input the local height map, preset path points, robot proprioception information (53 dimensions), and other privileged information such as body inertia encoded by the privileged information encoder. The teacher policy outputs actions. The depth camera obtains the local height map via a terrain reconstructor, and the robot proprioception information is obtained via the IMU and joint encoder. In the second stage, a CNN-GRU network is used to extract the direction and speed from the depth image obtained by the depth camera. This information is combined with the robot proprioception information as input, and the student policy network is initialized with parameters set to a deep copy from the first stage. The output of the first stage is used for supervised learning to obtain the output action.

[0073] Among them, the privileged information includes terrain friction coefficient, target path point, body inertia, body mass, motor strength, motor torque output ratio and communication delay time.

[0074] In one specific embodiment, during the teacher policy network training process, the environment's privileged information, a local height map centered on the robot, and the robot's current proprioception information are input into the teacher policy. The torque of each joint of the robot's legs is then output and transmitted together with the corresponding linear velocity of motion to the policy evaluation network. The policy evaluation network determines the robot's next state and action based on reward feedback. Furthermore, the network weight parameters in the teacher policy are initialized into the student policy to accelerate training and perform supervised training. The student policy requires encapsulated proprioception information and a depth image captured by a depth camera as input. The CNN-GRU visual network architecture is used to predict the target heading from the depth image. The final output of the policy is the joint torque value.

[0075] Step 2.3.1: Construct the teacher-student network. The teacher-student network includes the teacher-strategy network and Student Policy Network .

[0076] Step 2.3.2: Train the teacher strategy network based on reinforcement learning to obtain the trained teacher-student network. The training process of the teacher strategy network is the first stage, and the training process of the student strategy network is the second stage.

[0077] Step 2.3.2.1: Train the teacher strategy using MLP.

[0078] The teacher policy network consists of a gated recurrent unit (GRU) and an MLP. The GRU processes time series data and extracts motion features, while the MLP serves as the policy network, calculating the robot joint torque. The parameters of the policy network are optimized using the proximal policy optimization (PPO) network architecture.

[0079] Because robot motion decisions involve time series data (historical states influence current decisions), using an MLP alone cannot capture temporal dependencies. Therefore, a GRU is used to process the time series, and weights are optimized through backpropagation through time (BPTT). This involves backpropagating from the final time step to the past 24 time steps, calculating the gradient of the loss function with respect to the GRU and MLP weights. This is used to update the time series modeling parameters and the output parameters of the policy network, ultimately optimizing the robot's motion policy.

[0080] In the first stage, the local height map is used, combined with the robot's proprioceptive information and other privileged information, and input into the teacher policy network. The teacher policy network is trained using model-free reinforcement learning. The GRU and MLP process the policy input, extract the terrain features and motion state, and output the robot's leg joint torque. Specifically, the teacher policy network is trained using model-free reinforcement learning. , the teacher policy network has access to the local height map , proprioceptive information and environmental parameters Privileged information such as GRU and MLP is used to extract feature vectors and , the final output action , that is, the torque values ​​of the twelve joints of the robot's legs at each moment.

[0081] First, the local height map The point cloud in is compressed into , and then passed to the GRU for predicting joint angles along with the rest of the observations, using GRU to process the environment point cloud , respectively and , as input to the basic feed-forward strategy.

[0082] The basic feedforward network is the strategy structure used to generate joint torque in the teacher strategy. It combines GRU and MLP for processing, and is used to extract features from three-dimensional point cloud data and robot proprioception information, and ultimately outputs the torque of the 12 joints of the robot's legs.

[0083] The specific formula is:

[0084] ,

[0085] ,

[0086] .

[0087] This embodiment uses PPO for training. During the strategy training process, BPTT is used to calculate the loss function and backpropagate from the final time step to the past 24 time steps. After that, the gradient of the loss function with respect to the GRU and MLP weights is calculated to update the time series modeling parameters of the GRU and the output parameters of the MLP to optimize the robot motion strategy.

[0088] The output joint torque and linear velocity will serve as input to the strategy evaluation network, which will evaluate and adjust the robot behavior based on reward value feedback.

[0089] During the training process of the teacher policy network, a double distillation approach is used to transfer the teacher policy network's knowledge to the student policy network. The student policy network is initialized using the teacher policy network's weights and trained through supervised learning, optimizing the student policy network's parameters to improve learning efficiency. During this process, the teacher policy network's action outputs are used as reward signals by the policy evaluation network to generate the robot's motion decisions in complex terrain.

[0090] Step 2.3.2.2: Build a supervised reinforcement learning training framework and distill the student policy network from the trained teacher policy network.

[0091] In the second phase, in the student strategy network During training, the student policy network is trained using DAgger, using a time-truncated backpropagation algorithm to minimize the mean squared error between the speed predicted by the student policy network and the speed output by the teacher policy network. The student policy network is trained through supervised learning, enabling the quadruped robot to accurately track speed commands. The outputs of the student policy network include the required linear velocity and yaw angle, ensuring precise motion on various staircase terrains. The required linear velocity includes the basic speed for forward and lateral movement. During the teacher policy training phase, to obtain more accurate environmental information, deep vision information is used with 3D point cloud data to train the optimal policy. Simultaneously, a state estimation network based on a CNN-GRU network architecture is constructed for feature extraction during the student policy training phase. During the student policy training phase, to reduce the computational cost of external perception, deep vision information is used with 2D depth images. The state estimation network generated in the first phase is embedded as a trained module within the student policy network to extract terrain and temporal features from continuous depth images and predict target direction and speed.

[0092] The student policy network receives encapsulated proprioception information and depth images captured by the depth camera as input. It utilizes a CNN-GRU visual network architecture to extract spatial features from the depth images and predict the target heading. The output joint torque values ​​are optimized using a regression loss function to ensure that the student policy network's actions are consistent with those of the teacher policy network. The student policy network's weights are initialized using the teacher policy network, and supervised learning is performed during training, gradually adjusting the strategy to adapt to different terrain conditions. During training, the student policy network uses depth images from the depth camera, velocity commands output by the state estimation network, and robot proprioception information as input. DAgger is used to optimize the student policy network's output to approximate the output of the teacher policy network. Through continuous iterative updates, the student policy network learns more precise motion control, enabling efficient autonomous decision-making in diverse terrain environments.

[0093] The input of the student policy network includes external perception information (depth image ) and proprioceptive information And use the state estimation network generated in the first stage as the convolution layer Conv depth image information in the second stage, GRU extracts time series features, and finally predicts the output direction of travel and speed instructions.

[0094] The main goal of the second stage is to use supervised learning to distill the teacher policy network generated in the first stage into an architecture that only relies on onboard sensor information, without retraining the entire controller, only training the feature vector and and uses the base policy trained in the first stage.

[0095] In the second stage, DAgger is used for training to minimize the mean square error between the student strategy prediction action and the teacher strategy network output action through BPTT. ,in, Output actions for the teacher strategy network, i.e., real actions, To predict actions. Specifically, the depth image is first preprocessed and visual features are extracted through a convolutional network. Afterwards, the GRU network is used to process the historical proprioceptive information and historical deep visual information to estimate the latent features of the terrain geometry. Since the camera is located in front of the robot, the combination of proprioceptive information and depth enables the GRU to implicitly track and estimate the terrain below the robot, obtaining the estimated latent features. Afterwards, the external features are estimated using the historical proprioceptive information and the direction and speed instructions predicted by the state estimation network. Finally, the MLP network combines the estimated latent features and external features to generate the predicted action .

[0096] In order to minimize the behavioral deviation between the teacher strategy and the student strategy, this embodiment sets the initialization parameters of the student strategy network to the copy from the first stage. The teacher-student network architecture is as follows: Figure 3 Doing so ensures that the student strategy starts out behaving as similarly as possible to the teacher strategy, thus accelerating the learning process.

[0097] The trained student policy extracts environmental features from the depth images collected by the robot and combines them with proprioception information to generate an estimate of the current motion state. The student policy then outputs torque values ​​for each joint to control the robot's motion based on the current proprioception information and external environmental information. Through reinforcement learning optimization, the student policy continuously adjusts to maximize long-term cumulative rewards. In the second phase, this embodiment combines real-time environmental information with motion planning goals, ensuring that the robot can make autonomous decisions and move efficiently in dynamic, irregular staircase terrain.

[0098] Since the CNN-GRU network is not trained at the initial training stage, directly using the target path point predicted by the CNN-GRU network as the robot's heading will cause the robot to walk aimlessly. Therefore, a hybrid method of teacher and student strategies is selected to predict the movement direction. Specifically, when the angle between the movement direction predicted by the student strategy network and the movement direction predicted by the teacher strategy network is less than a preset threshold, the movement direction predicted by the student strategy network is used; otherwise, the movement direction predicted by the teacher strategy network is used.

[0099] Step 2.3.3: Output the torque value of each joint of the robot according to the training process.

[0100] Step 2.3.4: Input the torque values ​​of each joint of the robot's legs and the corresponding linear motion speed output by the teacher strategy network into the strategy evaluation network, and generate the robot's motion decision in complex terrain as a speed tracking effect through the reward signal of the strategy evaluation network.

[0101] In one specific implementation, the torque values ​​and linear velocity of each robot leg joint output by the teacher-student strategy network are input into a strategy evaluation network, which consists of four modules: an input layer, a feature extraction module, a reward calculation module, and an output layer. First, the input layer receives the torque values ​​and linear velocity of the robot's leg joints output by the teacher strategy network. Then, an MLP is used to extract motion features and calculate the reward value for the robot's current behavior. The output layer then provides this reward value to the PPO network, which optimizes the parameters of the teacher strategy network through iterative training with the goal of maximizing the action reward.

[0102] The basic principle of designing the reward function in this embodiment is to guide the robot's stable movement in a complex environment through the robot's ability to track heading, control body balance, and perform precise movements, and to encourage the robot to make necessary dynamic adjustments during the task, ensuring that it can autonomously optimize its movement strategy and demonstrate high autonomy and intelligence.

[0103] To meet design requirements, the reward function consists of multiple different reward terms. The total reward term for each control time step is defined as the weighted sum of these individual reward terms. In this embodiment, the total reward terms are primarily divided into two categories: tracking rewards and regularization rewards. Tracking rewards include target velocity tracking rewards and target yaw rewards, while regularization rewards include vertical linear velocity penalties, posture error penalties, collision penalties, hip position penalties, and foot edge penalties. The sum of each reward term multiplied by its corresponding weight coefficient is the total reward function value.

[0104] In this embodiment, the termination step length for each round is designed to be 50,000, and the training process of this round is considered to be completed when the robot reaches the end position of the current terrain, and the quadruped robot is reset to the starting coordinates of the training. If the quadruped robot fails to successfully move to the desired position, it will also be reset to the starting coordinate position of the movement after the termination step length is reached in this round. The overall control process of the strategy update is as follows Figure 2 As shown in the figure, the quadruped robot uses the depth vision information and proprioception information collected by sensors as the input state s for reinforcement learning. The PPO network calculates and outputs the robot's control signal a in the action space based on the input state s. The control signal a is then mapped to the robot's 12 joint angles, and position control is used to enable the quadruped robot to complete the target action. At the same time, the system decides whether to reset the robot to its starting position based on whether it reaches the end point or falls. All state, action, and reward information is stored in an experience replay pool. When the data accumulates to a set threshold, a PPO policy update is triggered to optimize the robot's movement in complex terrain until the reward value stabilizes and training is complete.

[0105] Step 3: Evaluate the speed tracking effect and adaptively adjust the environmental parameters in the stair terrain model based on the evaluation results.

[0106] In a specific embodiment, Figure 4As shown, by evaluating the robot's performance in the current terrain partition, the terrain difficulty is dynamically adjusted. At the end of each round of training, the root mean square error (RMSE) between the actual speed of the robot's movement and the target speed is calculated. If the RMSE is lower than the preset threshold, the terrain difficulty signal is increased and the terrain complexity is increased; if it is higher than the threshold, the terrain difficulty signal is lowered and the terrain complexity is reduced; otherwise, the current terrain difficulty signal is maintained. According to the robot's performance in different terrain partitions, the terrain difficulty is gradually increased or decreased to ensure that the training process is challenging and adaptable. In this embodiment, in each new stair terrain partition, the corresponding terrain features, such as slope, number of steps, ground friction coefficient, etc., are adjusted to achieve the adjustment of terrain complexity. The specific terrain corresponding to the difficulty signal can be customized according to actual conditions, which can help the robot adapt to more complex environments and update the target difficulty of training in real time according to actual performance.

[0107] Specifically, the speed tracking effect is evaluated. For each terrain, if the root mean square error of speed tracking is less than 0.2, the difficulty of the terrain is increased. Set to 1. If it is higher than 0.5, the terrain difficulty is increased. Set to -1. Otherwise, the terrain difficulty will be increased. Set to 0.

[0108] Example 2:

[0109] A second embodiment of the present invention provides a quadruped robot reinforcement learning motion planning system based on depth vision, comprising:

[0110] The simulation building module is configured to build a simulation environment, model stair terrains with different slopes to obtain a stair terrain model, and model robot dynamic parameters to obtain a robot model, wherein the stair terrain model includes stair terrains with different difficulty differences;

[0111] A motion tracking module is configured to load a robot model and move it in a staircase terrain model, track the robot's motion status, and obtain a speed tracking effect, wherein the speed tracking effect is obtained by strategically evaluating the joint torque values ​​and corresponding motion linear velocities output by the teacher strategy network during the robot's motion;

[0112] The effect evaluation module is configured to evaluate the speed tracking effect and adaptively adjust the environmental parameters in the stair terrain model according to the evaluation results.

[0113] Example 3:

[0114] Embodiment 3 of the present invention provides a medium on which a program is stored. When the program is executed by a processor, the steps in the quadruped robot reinforcement learning motion planning method based on depth vision as described in Embodiment 1 of the present invention are implemented.

[0115] Example 4:

[0116] Embodiment 4 of the present invention provides a device, including a memory, a processor, and a program stored in the memory and runnable on the processor. When the processor executes the program, the steps in the depth vision-based quadruped robot reinforcement learning motion planning method as described in Embodiment 1 of the present invention are implemented.

[0117] The steps involved in the above embodiments 2, 3 and 4 correspond to those in the method embodiment 1. For the specific implementation methods, please refer to the relevant description part of the embodiment 1.

[0118] Those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computer device. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0119] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without any creative work are still within the scope of protection of the present invention.

Claims

1. A deep vision-based reinforcement learning motion planning method for a quadruped robot, characterized in that: The following steps are involved: Building a simulation environment, modeling stairs with different slopes to obtain a stair terrain model, and modeling the robot's dynamic parameters to obtain a robot model, wherein the stair terrain model includes stairs with different levels of difficulty; The robot model is loaded and moved in the stair terrain model, and the robot's motion status is tracked to obtain the speed tracking effect. The speed tracking effect is obtained by strategically evaluating the joint torque values ​​and corresponding motion linear velocities output by the teacher strategy network during the robot's motion. Specifically, the process includes the first stage of teacher strategy network training and the second stage of student strategy network training. In the first stage, the local height map is combined with the robot's proprioceptive information and other privileged information and input into the teacher policy network. The policy input is processed by GRU and MLP to extract terrain features and motion state, and output the robot's leg joint torque. The privileged information includes terrain friction coefficient, target path point, body inertia, body mass, motor strength, motor torque output ratio, and communication delay time. In the second stage, the CNN-GRU network is used to extract the direction and speed from the depth image obtained by the depth camera. Combined with the robot's proprioception information as input, the initialization parameters of the student policy network are set to the deep copy from the first stage. The output of the first stage is used for supervised learning to obtain the output action. Obtaining initial 3D point cloud data, filtering and coordinate conversion the initial 3D point cloud data, and then generating a local 2D height map as a local height map through projection and bilinear interpolation algorithms; The speed tracking effect is evaluated, and the environmental parameters in the stair terrain model are adaptively adjusted according to the evaluation results.

2. The method for motion planning of a quadruped robot based on deep vision reinforcement learning according to claim 1, wherein: Load the robot model and let it move in the staircase terrain model. The specific steps to track the robot's movement are as follows: Obtain robot depth vision information and robot body perception information; Reconstructing a local height map based on the robot's deep vision information; The teacher strategy network is used to process the local height map, robot proprioception information and privilege information to obtain the torque value of each joint of the robot. The torque value of each joint is combined with the linear velocity of motion to perform strategy evaluation and obtain the speed tracking effect.

3. The method for motion planning of a quadruped robot based on deep vision reinforcement learning according to claim 2, wherein: The specific steps of using the teacher strategy network to process the local height map, robot body perception information and privilege information are as follows: Building a teacher strategy network; The teacher strategy network is trained based on reinforcement learning to obtain a trained teacher strategy network; The teacher strategy network is used to process the local height map, robot proprioception information and privileged information to obtain the torque value of each joint of the robot.

4. The method for motion planning of a quadruped robot based on reinforcement learning of depth vision according to claim 3, wherein: The specific steps for training the teacher strategy network based on reinforcement learning are: Using multi-layer perceptrons to train teacher strategies; Build a supervised reinforcement learning training framework to distill the student policy network from the trained teacher policy network.

5. The method for motion planning of a quadruped robot based on reinforcement learning of depth vision according to claim 3, wherein: The specific steps for evaluating the strategy by combining the torque value of each joint with the linear velocity of motion are as follows: The torque values ​​of each joint of the robot's legs and the corresponding linear motion speed output by the teacher strategy network are input into the strategy evaluation network. The reward signal of the strategy evaluation network is used to generate the robot's motion decision in complex terrain as a speed tracking effect.

6. The method for motion planning of a quadruped robot based on reinforcement learning of depth vision according to claim 1, wherein: The specific steps for evaluating the speed tracking effect and adaptively adjusting the environmental parameters in the stair terrain model based on the evaluation results are as follows: At the end of each round of training, the root mean square error (RMSE) between the actual speed of the robot and the target speed is calculated. If the RMSE is lower than the preset threshold, the terrain difficulty signal is increased; if it is higher than the threshold, the terrain difficulty signal is decreased; otherwise, the current terrain difficulty signal is maintained.

7. A quadruped robot reinforcement learning motion planning system based on depth vision, using the quadruped robot reinforcement learning motion planning method based on depth vision according to any one of claims 1 to 6, characterized in that: include: The simulation building module is configured to build a simulation environment, model stair terrains with different slopes to obtain a stair terrain model, and model robot dynamic parameters to obtain a robot model, wherein the stair terrain model includes stair terrains with different difficulty differences; A motion tracking module is configured to load a robot model and move it in a staircase terrain model, track the robot's motion status, and obtain a speed tracking effect, wherein the speed tracking effect is obtained by strategically evaluating the joint torque values ​​and corresponding motion linear velocities output by the teacher strategy network during the robot's motion; The effect evaluation module is configured to evaluate the speed tracking effect and adaptively adjust the environmental parameters in the stair terrain model according to the evaluation results.

8. A computer-readable storage medium, characterized in that Multiple instructions are stored therein, which are suitable for being loaded by a processor of a terminal device and executing the quadruped robot reinforcement learning motion planning method based on depth vision as described in any one of claims 1-6.

9. A terminal device, characterized in that: The invention comprises a processor and a computer-readable storage medium, the processor being used to implement each instruction; the computer-readable storage medium being used to store multiple instructions, the instructions being suitable for being loaded by the processor and executing the quadruped robot reinforcement learning motion planning method based on depth vision according to any one of claims 1 to 6.

Images