Motion control method for quadruped robot based on topographic map reinforcement learning

Abstract

The invention discloses a quadruped robot motion control method based on topographic map reinforcement learning. The quadruped robot motion control method comprises the following steps: S1, building a quadruped robot model; s2, constructing a topographic map confidence estimation model; s3, constructing a state estimation model; s4, constructing an actor network and a commentator network to train the quadruped robot model; and S5, acquiring data of the laser radar and the inertial measurement unit by using a sensor fusion technology, generating a height map around the quadruped robot in real time, and updating environmental terrain information. By introducing the topographic map confidence estimation model and combining topographic map generation and topographic uncertainty estimation, the motion strategy of the quadruped robot in a complex terrain can be adjusted in real time, the adaptability of the robot to irregular terrains, obstacles and topographic changes is remarkably improved, and the reliability of the quadruped robot is improved. And control errors and falling risks caused by topographic changes or sensor noise are reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of quadruped robots, and in particular relates to a quadruped robot motion control method based on terrain map reinforcement learning. Background Art

[0002] Existing deep reinforcement learning-based quadruped robot motion control methods commonly use elevation map-based terrain perception methods, which primarily rely on fixed elevation map inputs. While these methods can provide effective terrain information for the robot, the use of elevation maps can lead to reduced accuracy in complex terrain or when sensor data is incomplete. This is especially true in the event of sensor failure or occlusion, as the robot cannot fully utilize environmental information. They also perform poorly in handling terrain uncertainty and drastic elevation changes, impacting the robot's motion performance. In particular, in complex and irregular terrain, the robot struggles to effectively plan based on the real-time terrain, resulting in reduced accuracy and efficiency in motion control.

[0003] To address this issue, recent research has proposed multimodal data fusion and uncertainty estimation solutions. However, existing methods often suffer from two drawbacks: inaccurate modeling of terrain information when dealing with complex terrain, and poor robustness to sensor noise or occlusion. Consequently, existing quadruped robot motion control methods based on perceptual reinforcement learning cannot effectively achieve stable motion in environments with unstable generated terrain maps, especially when the terrain changes dramatically or sensor errors are present. Summary of the Invention

[0004] In order to overcome the shortcomings of the existing technology, the present invention provides a quadruped robot motion control method based on terrain map reinforcement learning, which improves the robustness and obstacle surmounting ability of the quadruped robot in different terrains by combining terrain map with proprioception information.

[0005] The technical solution adopted by the present invention to solve the technical problem is: a quadruped robot motion control method based on terrain map reinforcement learning, comprising the following steps:

[0006] S1, building a quadruped robot model on the simulation platform;

[0007] S2, build a topographic map confidence estimation model, set the noise model in the simulation to obtain a noisy height map Simulating multiple real-world environments, using a quadruped robot with noise to perceive historical information and heightmap As input, it outputs terrain confidence weights and reconstructed terrain features;

[0008] S3, build a state estimation model based on the historical state observations perceived by the quadruped robot body and the latent vector encoded by the map confidence estimation module in step S2 As input, output is the estimated speed of the quadruped robot at the current moment , the next state of the quadruped robot , which serves as the basis for motion control decision-making of quadruped robots in complex terrain;

[0009] S4, builds an actor network and a critic network to train the quadruped robot model;

[0010] S5, uses sensor fusion technology to obtain lidar and inertial measurement unit data, generate a height map around the quadruped robot in real time, and update the environmental terrain information.

[0011] Furthermore, the step S2 includes the following sub-steps:

[0012] S21, add random noise to the height map and build a noise model ,in is the noise parameter;

[0013] S22, setting mapping conditions and configuring different noise parameters z for different conditions to simulate different terrain qualities and error sources;

[0014] S23, the height map with noise The historical state observations of the quadruped robot's proprioception Input noise model.

[0015] Furthermore, in step S22, the noise model includes at least one of the following noise types:

[0016] Normal noise, assuming good map quality and suitable for routine operations, simulates map measurement errors through Gaussian noise, and the standard deviation of Gaussian noise under the normal noise condition is σ∈[0.03m, 0.08m];

[0017] Large offset noise: simulates map offsets caused by pose estimation drift or terrain deformation through higher noise, and simulates pose estimation drift through spatial offset noise; the mean spatial offset of the large offset noise condition is μ∈[0.1m, 0.2m];

[0018] Large noise amplitude applies larger noise to each scan point to simulate the complete loss of terrain information due to occlusion or sensor failure, and simulates sensor failure scenarios through random occlusion noise. The large noise amplitude condition applies random zero-value noise to at least 20% of the scan points in the height map.

[0019] Furthermore, the step S23 includes the following sub-steps:

[0020] S231, heightmap with noise and historical state observations Through a network , Perform feature compression to obtain vector ;

[0021] S232, will Splicing is performed and then fed into the fully connected network , get the height map confidence , Determines the proprioceptive information of the quadruped robot and heightmap The calculation process of how much weight is generated for the final height map is expressed as:

[0022]

[0023]

[0024]

[0025] Where ⊙ is element-wise multiplication, resulting in a vector , during the training process, As input, it is fed into the decoder network to reconstruct the height map. The calculation formula is as follows:

[0026]

[0027] in, is to get the estimated height map, A true height map.

[0028] Furthermore, the height map and the real height map The error is calculated using the mse loss function, and the loss function is .

[0029] Furthermore, the step S3 includes the following sub-steps:

[0030] S31, the state estimation model uses a deep network architecture, and the input includes the historical state observations of the quadruped robot proprioception and the latent vector from the topographic map confidence estimation module , the input together encodes the current state of the quadruped robot, the specific observation value is a 45-dimensional vector defined as follows;

[0031]

[0032] represents the robot's body angular velocity, represents the gravity vector of the body, Indicates the robot's linear velocity and angular velocity instructions, represents the joint angular velocity, represents the joint angular acceleration, Indicates the offset of the joint position output by the strategy at the previous moment relative to the joint position at the initial moment;

[0033] S32, the state estimation model uses the structure of the twin network and the variational autoencoder framework to extract deep features and generate the current quadruped robot speed and latent vector ,in The potential representation extracted by the variational autoencoder structure can effectively capture the complex characteristics of the environment and robot state; the state estimation module outputs the estimated speed at the current moment , represents the motion state of the robot; the latent vector , contains implicit information about the current environment and the quadruped robot;

[0034] S33, using quadruped robot speed and latent vector Jointly predict the next moment's quadruped robot state , the decoder network will and Combine and output the state prediction for the next moment .

[0035] Furthermore, in step 3, the loss function in the variational autoencoder is used to train the network by minimizing the reconstruction error and the KL divergence of the latent variables. The loss function of the state estimation is: , used to evaluate the accuracy of the model in predicting the state at the next moment, the KL divergence of the latent variable is: ; The total loss is composed of the weighted reconstruction loss and KL divergence .

[0036] Furthermore, in step 4, a reinforcement learning algorithm is used for training, including constructing an actor network and a critic network based on the PPO reinforcement learning algorithm to train the quadruped robot model, and the strategy input includes the quadruped robot proprioception information. , the estimated speed of the output of the state estimation module and features extracted by variational autoencoder , and the output of the topographic map confidence estimation module , represents the confidence of the current map, which is used to weight the quadruped robot's proprioceptive information and the noisy height map information.

[0037] Furthermore, in step 4, the actor network uses the policy input information to generate the action of the quadruped robot at the current moment to maximize the cumulative reward; the critic network accepts the same policy input information, evaluates the value of the current state by calculating the state value function, and provides the expected return of the action.

[0038] Furthermore, in step 5, the lidar and inertial measurement unit data are fused through an extended Kalman filter, and the odometer output frequency reaches 200 Hz.

[0039] By generating and updating terrain map information in real time and estimating the uncertainty of the terrain map, and combining terrain map generation with terrain uncertainty estimation, it can help quadrupeds make more accurate and stable motion decisions under different environmental conditions, thereby improving the adaptability and execution efficiency of quadruped robots in complex terrain.

[0040] The present invention has the following beneficial effects: by introducing a terrain map confidence estimation model and combining terrain map generation with terrain uncertainty estimation, the motion strategy of a quadruped robot in complex terrain can be adjusted in real time, significantly improving the robot's adaptability to irregular terrain, obstacles, and terrain changes, and reducing the risk of control errors and falls caused by terrain changes or sensor noise. Noise is added and processed by the noise model to the height map data, effectively estimating and compensating for sensor noise and terrain uncertainty, thereby improving the robustness of the quadruped robot to terrain information. In particular, in the case of sensor failure or occlusion, the noise compensation and confidence assessment mechanism can ensure that the robot can still perform stable motion. By modeling terrain data at different noise levels and combining historical data with a confidence estimation module for weighted fusion, more accurate terrain perception and estimation is achieved. By combining proprioceptive information (such as joint angular velocity, gravity vector, etc.) with environmental information (such as terrain confidence, estimated speed, etc.), a reinforcement learning strategy is used to optimize the robot's movements. The multimodal information fusion approach significantly improves the robot's understanding of the current environment and decision-making capabilities, especially in changing environments, showing greater stability and autonomy. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 Schematic diagram of the topographic map confidence estimation model in the present invention.

[0042] Figure 2 Schematic diagram of the state estimation model in the present invention.

[0043] Figure 3 This is a schematic diagram of the actual machine deployment framework in the present invention.

[0044] Figure 4 Schematic diagram of the training framework under simulation in the present invention. DETAILED DESCRIPTION

[0045] In order to enable those skilled in the art to better understand the solutions of the present invention, the following will provide a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0046] A quadruped robot motion control method based on terrain map reinforcement learning includes the following steps:

[0047] S1, build a quadruped robot model on the simulation platform, including the overall structure of the robot and the motion model of each leg;

[0048] The quadruped robot model includes the kinematic and dynamic models of four independent robot legs and their joints. The quadruped robot utilizes a high-degree-of-freedom mechanical structure, with each leg driven by multiple joints (such as the hip, knee, and ankle) and motors. The model's dimensions, mass, and center of gravity match those of an actual robot, ensuring realism and accuracy in the simulation environment.

[0049] Specifically, the simulation platform can be Isaac Gym; in the Isaac Gym platform, a physical engine is used to simulate and control the robot, and different terrains of the environment are set, such as flat ground, stairs, slopes, irregular ground, etc., to ensure that the robot can be trained in various complex environments.

[0050] S2, constructed as Figure 1 The terrain map confidence estimation model shown in the figure sets the noise model in the simulation to obtain the height map with noise Simulate various real-world environments, including scenarios such as sensor failure and terrain changes. Noise types include lateral scan point offset and height perturbation. Lateral scan point offset applies lateral displacement noise to each scan point, while height perturbation applies perturbation noise to the height value of each scan point. The noise amplitude is adjusted using the noise parameter z. The noise offset in each simulation environment is initially set to zero and updated at each training step. The noise amplitude is a random value generated based on a Gaussian distribution.

[0051] A historical information graph of quadruped robot proprioception with noise and heightmap As input, through the encoding and decoding networks, the terrain confidence weights and reconstructed terrain features are output. Specifically, the estimated height map information can be output to reduce the impact of terrain uncertainty on the strategy.

[0052] Specifically, step S2 includes the following sub-steps:

[0053] S21, add random noise to the height map during training to build a noise model ,in is the noise parameter, which determines the variance of different noises;

[0054] S22 sets the mapping conditions and configures different noise parameters z for different conditions to simulate different terrain qualities and error sources. The noise parameter z is learned as part of the training process and increases linearly over time, which helps the robot gradually adapt to more complex and uncertain terrain environments.

[0055] Specifically, the noise model sets three mapping conditions, including normal noise, large offset noise, and large noise amplitude. Normal noise assumes good map quality and is suitable for routine operations. Map measurement errors are simulated using Gaussian noise, with a Gaussian noise standard deviation of σ∈[0.03m, 0.08m].

[0056] Large offset noise: simulates map offsets caused by pose estimation drift or terrain deformation through higher noise, and simulates pose estimation drift through spatial offset noise; the mean spatial offset of the large offset noise condition is μ∈[0.1m, 0.2m];

[0057] Large noise amplitude applies larger noise to each scan point to simulate the complete loss of terrain information due to occlusion or sensor failure, and simulates sensor failure scenarios through random occlusion noise. The large noise amplitude condition applies random zero-value noise to at least 20% of the scan points in the height map.

[0058] The selection of noise type is randomly controlled by a probability value. According to the following rules, different noise standard deviations are added to select appropriate values within the above range: Normal noise (60% probability), which simulates normal sensor errors, and the noise standard deviation is set to 0.03 meters. Large offset noise (30% probability), which simulates large offsets caused by sensor drift or terrain changes (such as ground deformation, slippage, etc.). The noise standard deviation is increased to 0.12 meters. Specifically, the noise amplitude will gradually increase with training time, and its calculation method is, the noise standard deviation , the more training steps, the greater the amplitude of the noise. Large noise amplitude (10% probability): This noise simulates serious sensor failure or occlusion problems, resulting in distortion or loss of height data. The noise standard deviation is set to 0.6 meters and gradually increases as training progresses. Each training environment is divided into multiple small cells (for example, an area of 0.5 meters x 0.5 meters), each of which represents a terrain block. Within these small cells, the system adds a random offset to each cell to simulate the irregularity and local changes of the terrain. The size of the noise is achieved by a randomly generated offset, and its range is set to [-0.12,0.12]m. The height map taken in this example The size is 1.6mX1.0m, the resolution is 0.1m, and the dimension of the height map is 187.

[0059] S23, setting mapping conditions and configuring different noise parameters z for different conditions to simulate different terrain qualities and error sources.

[0060] Specifically, step S23 includes the following sub-steps:

[0061] S231, the above height map with noise and the historical state observations of the quadruped robot proprioception enter Figure 1 The topographic map confidence estimation module shown, where Using 5 frames of historical information, the dimension is 225, first with the height map of noise and historical state observations Through a fully connected network , Perform feature compression to obtain vector ,in The hidden layer dimensions are set to {128, 64}, {256, 128}, The dimensions are all 64;

[0062] S232, the vector Splicing is performed and then fed into the fully connected network , get the height map confidence vector , The dimension of is 64, where The hidden layer dimensions are set to {128, 64}, Determines the proprioceptive information of the quadruped robot and heightmap The calculation process of how much weight is generated for the final height map is expressed as:

[0063]

[0064]

[0065]

[0066] Where ⊙ is element-wise multiplication, resulting in a vector , during the training process, As input, it is fed into the decoder network to reconstruct the height map. The calculation formula is as follows:

[0067]

[0068] in, is to get the estimated height map, is the real height map. In order to train this module, it is necessary to calculate the estimated loss of the height map. and the real height map The error is calculated using the mse loss function for supervised learning. The loss function can be defined as follows: .

[0069] S3, build a state estimation model, such as Figure 2 As shown, the historical state observation value of the quadruped robot proprioception is and the latent vector encoded by the map confidence estimation module in step S2 As input, output is the estimated speed of the quadruped robot at the current moment , the next state of the quadruped robot , and use this as the basis for motion control decision-making of quadruped robots in complex terrain;

[0070] Specifically, step S3 includes the following sub-steps:

[0071] S31, the state estimation model uses a deep network architecture, and the input includes the historical state observations of the quadruped robot proprioception and the latent vector from the topographic map confidence estimation module , the input together encodes the current state of the quadruped robot, the specific observation value is a 45-dimensional vector defined as follows;

[0072]

[0073] represents the robot's body angular velocity, represents the gravity vector of the body, Indicates the robot's linear velocity and angular velocity instructions, represents the joint angular velocity, represents the joint angular acceleration, Indicates the offset of the joint position output by the strategy at the previous moment relative to the joint position at the initial moment. In this example, is the body angular velocity of the three axes of roll, pitch, and yaw, The dimension is 3, which is the gravity vector of the quadruped robot, The linear velocity command in the x and y directions, in m / s, and the yaw angle velocity command; are the angles and angular accelerations of the 12 joints, The value output by the policy network at the previous moment.

[0074] S32, the state estimation model uses the structure of the twin network and the variational autoencoder (VAE) framework to extract deep features, input 5 frames of historical data, and generate the current quadruped robot speed and latent vector ,in The potential representation extracted by the variational autoencoder (VAE) structure can effectively capture the complex characteristics of the environment and robot state; the state estimation module outputs the estimated speed at the current moment , represents the motion state of the robot; the latent vector , contains implicit information about the current environment and the quadruped robot;

[0075] S33, using quadruped robot speed and latent vector Jointly predict the next moment's quadruped robot state , this process is completed by the decoder part, the decoder network will and Combine and output the state prediction for the next moment In this example, The dimension is 3, is 19, is 45.

[0076] In order to optimize the state estimation module, the loss function in the variational autoencoder (VAE) is used to train the network by minimizing the reconstruction error and the KL divergence of the latent variables. Specifically, the loss function of the state estimation is: , used to evaluate the accuracy of the model in predicting the state at the next moment, the KL divergence of the latent variable is: The total loss is composed of the weighted reconstruction loss and KL divergence .

[0077] S4, build an actor network and a critic network to train the quadruped robot model;

[0078] Specifically, a reinforcement learning algorithm is used for training. Based on the PPO reinforcement learning algorithm, an actor network and a critic network are constructed to train the quadruped robot model. The strategy input includes 45-dimensional proprioception information. , the estimated speed of the output of the state estimation module and features extracted by variational autoencoder , and the output of the topographic map confidence estimation module , represents the confidence of the current map, which is used to weight the quadruped robot's proprioceptive information and the noisy height map information.

[0079] The actor network uses policy input to generate the quadruped's current action, specifically the difference in joint angles relative to the initial default joint angles, to maximize cumulative reward. The critic network accepts the same policy inputs, such as proprioception information, estimated velocity, latent vectors, and confidence vectors, but its goal is to evaluate the value of the current policy and provide an expected reward for an action. The critic network evaluates the value of the current state by calculating a state-value function and provides the expected reward for the action. During training, the policy and value networks are updated using the PPO algorithm, using the advantage function to adjust network parameters.

[0080] In this example, the input to the actor network is 45-dimensional proprioceptive information , the output in step 3 and , output in step 2 The actor network has a hidden layer dimension of {256, 256, 256} and outputs 12-dimensional joint motions. The critic network inputs proprioceptive information. , 187-dimensional height map true value , PPO is used for training optimization.

[0081] S5 utilizes sensor fusion technologies, such as Fast-LIO2, to acquire LiDAR odometry and IMU data, generating a real-time height map of the quadruped robot's surroundings and updating the surrounding terrain. Fast-LIO2 outputs point cloud and IMU data at a frequency of 10Hz. In practical applications, using a 10Hz LiDAR odometry can result in an error of approximately 10cm. This timing error can affect the accuracy of task execution, especially when the robot is performing jumping tasks. To further improve accuracy, the LiDAR odometry and IMU data are fused using an extended Kalman filter (EKF), achieving a high-frequency odometry output of 200Hz.

[0082] The above specific embodiments are used to illustrate the present invention rather than to limit the present invention. Any modifications and changes made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A quadruped robot motion control method based on terrain map reinforcement learning, characterized in that: The following steps are involved: S1, building a quadruped robot model on the simulation platform; S2, build a topographic map confidence estimation model, set the noise model in the simulation to obtain a noisy height map Simulating multiple real-world environments, using a quadruped robot with noise to perceive historical information and heightmap As input, it outputs terrain confidence weights and reconstructed terrain features; S3, build a state estimation model based on the historical state observations perceived by the quadruped robot body and the latent vector encoded by the map confidence estimation module in step S2 As input, output is the estimated speed of the quadruped robot at the current moment , the next state of the quadruped robot , which serves as the basis for motion control decision-making of quadruped robots in complex terrain; S4, builds an actor network and a critic network to train the quadruped robot model; S5, uses sensor fusion technology to obtain lidar and inertial measurement unit data, generate a height map around the quadruped robot in real time, and update the environmental terrain information.

2. The quadruped robot motion control method according to claim 1, characterized in that: The step S2 includes the following sub-steps: S21, add random noise to the height map and build a noise model ,in is the noise parameter; S22, setting mapping conditions and configuring different noise parameters z for different conditions to simulate different terrain qualities and error sources; S23, the height map with noise The historical state observations of the quadruped robot's proprioception Input noise model.

3. The quadruped robot motion control method according to claim 2, characterized in that: In step S22, the noise model includes at least one of the following noise types: Normal noise, assuming good map quality and suitable for routine operations, simulates map measurement errors through Gaussian noise, and the standard deviation of Gaussian noise under the normal noise condition is σ∈[0.03m, 0.08m]; Large offset noise: simulates map offset caused by pose estimation drift or terrain deformation through higher noise, and simulates pose estimation drift through spatial offset noise; the mean spatial offset of the large offset noise condition is μ∈[0.1m,0.2m]; Large noise amplitude applies larger noise to each scan point to simulate the complete loss of terrain information due to occlusion or sensor failure, and simulates sensor failure scenarios through random occlusion noise. The large noise amplitude condition applies random zero-value noise to at least 20% of the scan points in the height map.

4. The quadruped robot motion control method according to claim 2, wherein: The step S23 includes the following sub-steps: S231, heightmap with noise and historical state observations Through a network , Perform feature compression to obtain vector ; S232, will Splicing is performed and then fed into the fully connected network , get the height map confidence , Determines the proprioceptive information of the quadruped robot and heightmap The calculation process of how much weight is generated for the final height map is expressed as: in To multiply by elements, we get vector , during the training process, As input, it is fed into the decoder network to reconstruct the height map. The calculation formula is as follows: in, is the estimated height map, A true height map.

5. The quadruped robot motion control method according to claim 4, characterized in that: The height map and the real height map The error is calculated using the mse loss function, and the loss function is .

6. The quadruped robot motion control method according to claim 1, wherein: The step S3 includes the following sub-steps: S31, the state estimation model uses a deep network architecture, and the input includes the historical state observations of the quadruped robot proprioception and the latent vector from the topographic map confidence estimation module , the input together encodes the current state of the quadruped robot, the specific observation value is a 45-dimensional vector defined as follows; represents the robot's body angular velocity, represents the gravity vector of the body, Indicates the robot linear velocity and angular velocity instructions, represents the joint angular velocity, represents the joint angular acceleration, Indicates the offset of the joint position output by the strategy at the previous moment relative to the joint position at the initial moment; S32, the state estimation model uses the structure of the twin network and the variational autoencoder framework to extract deep features and generate the current quadruped robot speed and latent vector ,in The potential representation extracted by the variational autoencoder structure can effectively capture the complex characteristics of the environment and robot state; the state estimation module outputs the estimated speed at the current moment , represents the motion state of the robot; the latent vector , contains implicit information about the current environment and the quadruped robot; S33, using quadruped robot speed and latent vector Jointly predict the next moment's quadruped robot state , the decoder network will and Combine and output the state prediction for the next moment .

7. The quadruped robot motion control method according to claim 6, characterized in that: In step 3, the loss function in the variational autoencoder is used to train the network by minimizing the reconstruction error and the KL divergence of the latent variables. The loss function of the state estimation is: , used to evaluate the accuracy of the model in predicting the state at the next moment, the KL divergence of the latent variable is: ; The total loss is composed of the weighted reconstruction loss and KL divergence .

8. The quadruped robot motion control method according to claim 1, wherein: In step 4, a reinforcement learning algorithm is used for training, including constructing an actor network and a critic network based on the PPO reinforcement learning algorithm to train the quadruped robot model. The strategy input includes the quadruped robot proprioception information. , the estimated speed of the output of the state estimation module and features extracted by variational autoencoder , and the output of the topographic map confidence estimation module , represents the confidence of the current map, which is used to weight the quadruped robot's proprioceptive information and the noisy height map information.

9. The quadruped robot motion control method according to claim 8, characterized in that: In step 4, the actor network uses the policy input information to generate the action of the quadruped robot at the current moment to maximize the cumulative reward; the critic network accepts the same policy input information, evaluates the value of the current state by calculating the state value function, and provides the expected return of the action.

10. The quadruped robot motion control method according to claim 1, characterized in that: In step 5, the lidar and inertial measurement unit data are fused through an extended Kalman filter, and the odometer output frequency reaches 200 Hz.

Images