Motion generation method for quadruped robot in sparse terrain based on terrain reconstruction

Abstract

The invention discloses a quadruped robot motion generation method under sparse terrain based on terrain reconstruction, and relates to the technical field of quadruped robot motion generation. Training a motion planning controller with a terrain perception memory mechanism based on multi-modal perception input; the elevation map of the terrain around the current quadruped robot is displayed and estimated from the body sensing information and the visual observation information of the airborne depth camera through a visual terrain reconstruction module; then, feature fusion is carried out on elevation map reconstruction results of the ontology perception information and the visual observation information, and comprehensive perception representation is obtained; and taking the comprehensive perception representation as input, constructing a corresponding reward function, and giving the expected moment of force of each joint of the quadruped robot in a complex discrete terrain with a sparse passable area in an end-to-end manner. The self gait can be adjusted in real time or in advance, the flexibility, adaptability and reliability of self movement in complex discrete terrains are improved, and the application of the quadruped robot is expanded.

Description

Technical Field

[0001] The present invention relates to the technical field of quadruped robot motion generation, and in particular to a quadruped robot motion generation method under sparse terrain based on terrain reconstruction. Background Art

[0002] Due to the excellent mobility and terrain adaptability of quadruped robots, bionic quadruped robots have been a research hotspot in the field of robotics in the past decade. Excellent quadruped robot research teams at home and abroad include Tokyo Institute of Technology in Japan, Boston Dynamics in the United States, ETH Zurich in Switzerland, Zhejiang University, Hangzhou Yunshenchu ​​Technology, Hangzhou Yushu Technology and Shandong Youbote Intelligent Robot Company in China. Research results include: ANYmal, a quadruped robot based on series elastic actuators, launched by the ANYbotics team of ETH Zurich in 2017.

[0003] Due to the movement advantage of legged robots in complex discrete terrain, quadruped robots inevitably have to perform tasks in extremely dangerous and complex terrains with sparse traversable areas such as plum blossom piles, single-plank bridges, and gullies. This requires them to have a high understanding of the environment while having agile movement and flexible adaptability. Most existing works use model-based controllers to plan footholds online, but they cannot be used in actual scenarios due to the high computational cost, the mismatch between simplified models and reality, and the reliance on manual data adjustment. At the same time, although model-free reinforcement learning methods have been widely used in the generation of quadruped motion control skills, due to the inevitable requirement of extremely dangerous discrete terrain for robots to accurately select footholds and quickly adjust gaits, they need to accurately perceive the environment, accurately select footholds, and execute with less observation input information. At present, no one has used end-to-end reinforcement learning methods to handle motion generation tasks in such terrains. Therefore, designing and developing a set of model-free end-to-end reinforcement learning methods that can adapt to complex discrete terrains with sparse traversable areas will help improve the movement ability of quadruped robots, expand their scope of application, and promote their application and development in a wider range of practical scenarios. Summary of the invention

[0004] In order to solve at least one of the technical problems mentioned in the background technology, the purpose of the present invention is to provide a method for generating quadrupedal robot motion in sparse terrain based on terrain reconstruction, which can adjust its own gait in real time or in advance to further improve its flexibility, adaptability and reliability of movement in complex discrete terrain, and expand the application of quadrupedal robots in industrial production and daily life.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A method for generating quadruped robot motion in sparse terrain based on terrain reconstruction, training a motion planning controller with terrain perception memory mechanism based on multimodal perception input;

[0007] Through the visual terrain reconstruction module with memory mechanism, the elevation map of the terrain around the current quadruped robot is displayed and estimated from the proprioceptive information and the visual observation information of the onboard depth camera; then, a multimodal feature encoder is used to fuse the features of the elevation map reconstruction results of the proprioceptive information and the visual observation information to obtain a comprehensive perception representation;

[0008] Taking the comprehensive perception representation as input and constructing a corresponding reward function, the motion planning controller gives the desired torque of each joint of the quadruped robot in a complex discrete terrain with sparse navigable areas in an end-to-end manner.

[0009] In some embodiments of the present invention, the method for displaying and estimating the terrain elevation map around the current quadruped robot is as follows:

[0010] The proprioceptive information O t , a gated recurrent unit is used for encoding to obtain the latent vector of the proprioceptive feature

[0011] For the visual observation information D t , the visual terrain reconstruction module is used for encoding, and the latent vector z of the proprioceptive feature is fused prop The motion prior information contained in it reconstructs an elevation map of the robot's surroundings, including the bottom and back of the body

[0012] In some embodiments of the present invention, a convolutional neural network is used to reconstruct the elevation map. Perform feature extraction to obtain a two-dimensional terrain feature vector Where C represents the feature dimension of each channel, W and H represent the feature vector width and height respectively; the two-dimensional terrain feature vector z hmap Split into W×H different C-dimensional feature vectors <z hmap,1 ,z hmap,2 ,...,z hmap,W×H >, Where each C-dimensional feature vector z hmap,i Represents a local terrain feature area; through the proprioceptive information O t and visual observation information D t After encoding, a total of W×H+1 C-dimensional feature vectors are obtained.

[0013] In certain embodiments of the present invention, a method for performing feature fusion on the elevation map reconstruction results of proprioceptive information and visual observation information to obtain a comprehensive perception representation is as follows:

[0014] A set of Transformer encoder layers are used to fuse the ontology perception feature latent vector x prop and the two-dimensional terrain feature vector z hmap ; Each Transformer encoder layer consists of a self-attention layer and a forward propagation layer with a residual connection mechanism; z prop With z hmap The feature dimensions are connected as the input sequence and encoded through multiple Transformer encoder layers to obtain a comprehensive perceptual representation with the same dimension as the input sequence. as output.

[0015] In some embodiments of the present invention, the method in which the motion planning controller gives the desired torque of each joint of the quadruped robot in a complex discrete terrain with sparse navigable areas end to end is as follows:

[0016] The comprehensive perception representation z fuse Input an Actor network composed of MLP to get the action output a t , output the action a t The result is transmitted to the joint PD controller for execution, thus achieving the desired torque of each joint of the quadruped robot.

[0017] In some embodiments of the present invention, the proprioceptive information O t , visual observation information D t to the comprehensive perceptual representation z fuse The perceptual information encoding process is optimized using a hybrid loss function, which is defined as follows:

[0018] L=L hmap +L explict +L implicit +L obs

[0019] Where: L, L hmap , L explict , L implicit , L obs They represent mixed loss, elevation map reconstruction loss, explicit state estimation loss, implicit state estimation loss and observation reconstruction loss respectively.

[0020] In some embodiments of the present invention, the working method of the visual terrain reconstruction module is as follows:

[0021] A two-dimensional CNN feature encoder is used to encode the visual observation information Dt =<

[0022] d t ,d t-1 ,...,d t-M+1 > Encode and obtain the visual feature latent vector z depth ; Then, the visual feature latent vector z depth and the latent vector z of the proprioceptive feature prop The output of the gated recurrent unit is then input into a decoder with an MLP structure to roughly reconstruct the elevation map and reshape it into two dimensions, obtaining a rough reconstruction result of the elevation map. Then, the elevation map is roughly reconstructed. Input into the decoder network for fine reconstruction to obtain the elevation map fine reconstruction result

[0023] In some embodiments of the present invention, during the terrain reconstruction process, when the average reward obtained in a trajectory is low, the elevation map truth value H is used. truth Replace the elevation map reconstruction result Perform feature fusion; when the average reward obtained in a trajectory is high, use the elevation map to accurately reconstruct the result Perform feature fusion.

[0024] In certain embodiments of the present invention, the reward function includes a velocity tracking reward and an auxiliary reward, and the auxiliary reward includes a body angular velocity tracking reward, a body z-direction linear velocity penalty, a body roll / pitch axis angular velocity penalty, a body orientation penalty, a power penalty, a joint velocity penalty, a joint acceleration penalty, a collision penalty, a motion change rate penalty, a motion smoothness penalty, a foot ground clearance penalty, and a landing point terrain edge penalty.

[0025] In certain embodiments of the present invention, the complex discrete terrain with sparse traversable areas includes flat land, gullies, single-plank bridge terrain, continuous gully terrain, plum blossom pile terrain of the same height, and plum blossom pile terrain with high undulation; in the training stage, pre-training is first performed on flat land, narrow gully terrain, and plum blossom pile terrain of the same height, and then secondary training is performed on single-plank bridge terrain, continuous gully terrain, wide gully terrain, and plum blossom pile terrain with high undulation.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] The present invention introduces a visual terrain reconstruction mechanism including a memory mechanism and an explicit and implicit estimation mechanism of environmental states through proprioceptive information and visual observation information of an onboard depth camera, so as to help the quadruped robot better understand the environmental characteristics around the body and make up for the performance loss of the robot due to perception lag or partial observation limitations, so as to adjust its own gait in real time or in advance, so as to further improve its flexibility, adaptability and reliability of movement in complex discrete terrains, and expand the application of quadruped robots in industrial production and daily life. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a structural diagram of the motion planning controller model of the present invention.

[0029] Figure 2 This is a structural diagram of the visual terrain reconstruction module of the present invention.

[0030] Figure 3 Schematic diagram of elevation map information around a robot body and feet according to an embodiment of the present invention.

[0031] Figure 4 Schematic diagram of the movement gait of a quadruped robot on flat ground.

[0032] Figure 5 Schematic diagram of the movement gait of a quadruped robot on a plum blossom pile terrain.

[0033] Figure 6 Schematic diagram of the movement gait of a quadruped robot on a single-plank bridge terrain.

[0034] Figure 7 Schematic diagram of the movement gait of a quadruped robot on continuous gully terrain. DETAILED DESCRIPTION

[0035] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] This embodiment provides a method for generating motion of a quadruped robot in sparse terrain based on terrain reconstruction, which mainly includes the following contents: training a motion planning controller with a terrain perception memory mechanism based on multimodal perception input;

[0037] Through the visual terrain reconstruction module with memory mechanism, the elevation map of the terrain around the current quadruped robot is displayed and estimated from the proprioceptive information and the visual observation information of the onboard depth camera; then, a multimodal feature encoder is used to fuse the features of the elevation map reconstruction results of the proprioceptive information and the visual observation information to obtain a comprehensive perception representation;

[0038] Taking the comprehensive perception representation as input and constructing a corresponding reward function, the motion planning controller gives the desired torque of each joint of the quadruped robot in a complex discrete terrain with sparse navigable areas in an end-to-end manner.

[0039] Please refer to Figure 1 The following is a detailed introduction to this solution:

[0040] The method for displaying and estimating the terrain elevation map around the current quadruped robot is as follows:

[0041] In the model training phase, the proprioceptive information O t , a gated recurrent unit (GRU) is used for encoding to obtain the latent vector of the proprioceptive feature The GRU module is used to process proprioceptive information in the hope that its gated structure with a memory mechanism can retain historical observation information, thereby providing a better understanding of the characteristics of the environment and motion information.

[0042] For the visual observation information D t , the visual terrain reconstruction module (Heightmap Estimator) is used for encoding, and the proprioceptive feature latent vector z is fused prop The included motion prior information explicitly reconstructs an elevation map of the robot's surroundings, including the area below and behind the body It is worth mentioning that the specific content of the visual terrain reconstruction module (Heightmap Estimator) will be elaborated in detail below.

[0043] Then, a Convolutional Neural Network (CNN) is used to reconstruct the elevation map. Perform feature extraction to obtain a two-dimensional terrain feature vector Where C represents the feature dimension of each channel and is related to the latent vector z of the proprioceptive feature prop The dimensions of are consistent, W and H represent the width and height of the feature vector respectively.

[0044] The two-dimensional terrain feature vector z hmap Split into W×H different C-dimensional feature vectors <z hmap,1 ,zhmap,2 ,...,z hmap,W×H >, Where each C-dimensional feature vector z hmap,i Represents a local terrain feature area; through the proprioceptive information O t and visual observation information D t After encoding, a total of W×H+1 C-dimensional feature vectors are obtained. The calculation formula of the above process can be expressed as follows:

[0045] z prop =GRU(O t )

[0046]

[0047]

[0048] Then, the feature fusion of the elevation map reconstruction results of the proprioceptive information and the visual observation information is performed to obtain the comprehensive perception representation. The specific method is as follows:

[0049] A set of Transformer encoder layers are used to fuse the latent vector z of the ontology perception feature prop and the two-dimensional terrain feature vector z hmap ; Each Transformer encoder layer consists of a self-attention layer and a forward propagation layer with a residual connection mechanism; z prop With z hmap The feature dimensions are connected as the input sequence and encoded through multiple Transformer encoder layers to obtain a comprehensive perceptual representation with the same dimension as the input sequence. as output.

[0050] The self-attention mechanism of the Transformer encoder enables it to fuse and encode multimodal information by assigning attention weights, fusing the feature information of the two modalities at multiple abstract levels. hmap,i Each represents a local area corresponding to the terrain elevation map around the body in reality. The self-attention mechanism of Transformer also enables the model to perform spatial reasoning on terrain information, allowing the model to explicitly pay attention to terrain areas that need attention, such as plum blossom piles, gullies, or uneven ground, so as to better understand the environmental perception. In addition, since the input terrain feature vector z hmapThe number of two-dimensional codes from the estimated information of the elevation map far exceeds the number of proprioceptive feature vectors, so directly using the Transformer encoder output as the final perceptual feature may dilute the proprioceptive information. In order to balance the perceptual information of the two modalities, the average value of all output feature vectors corresponding to the terrain feature vector is calculated to obtain the encoding result of a single terrain feature vector. The two feature vectors representing different modal features are then concatenated and projected using a single-layer MLP to obtain the final output vector z representing the comprehensive perceptual feature. fuse The calculation formula of the above process can be expressed as follows:

[0051] z fuse = Transformer(z prop ,z hmap )

[0052] Human-generated integrated perception representation fuse Divide into z feet 、z hmap , Z obs , respectively represent the body speed, the terrain feature latent vectors around the four feet, the terrain perception latent vector around the body, and the observation perception latent vector, that is, Will This estimate with obvious real physical meaning is called explicit estimate, corresponding to Figure 1 In z feet 、z hmap The implicit expression of these high-dimensional estimators with real physical meaning in low-dimensional feature space is called implicit estimation, corresponding to Figure 1 In For the display estimate, the model estimates the body velocity by This important privileged information that cannot be directly obtained from the input observation data enables it to better understand the current body motion state information and significantly improve the robustness of the motion strategy. The accurate estimation of can eliminate the accumulated estimation drift of the robot's inertial navigation module (IMU) on the body motion state, so that the robot can accurately know its current motion speed and better plan the motion at the current moment. For the implicit estimation term, this paper assumes that the implicit vector z of the terrain features around the robot's four feet is feet and the terrain-aware latent vector z around the body hmap Contains information that can fully express the terrain features around the robot's four legs and body, so two MLP decoders are used to respectively use z feet and z hmap To reconstruct the elevation map around the four legs and body and And calculate the true value H of the elevation map in the simulatorfeet and H hmap The reconstruction loss is used to train the model, encouraging the model to make the perceptual feature vector z fuse The terrain features are fully included and expressed. In addition, the estimation of the elevation map around the four feet can also enable the model to better understand the terrain features around each foot during the learning process. By coordinating with the penalty term of the landing point terrain edge, it can consciously choose a landing point far away from the edge in complex and discrete terrain, and will not fall due to the foot landing on the edge of the terrain during the migration to reality.

[0053] Finally, the method of the motion planning controller to give the desired torque of each joint of the quadruped robot in a complex discrete terrain with sparse traversable areas end-to-end is as follows:

[0054] The comprehensive perception representation z fuse Input an Actor network (Actor Policy) composed of MLP to get the action output a t , output the action a t The result is transmitted to the joint PD controller for execution, achieving the desired torque of each joint of the quadruped robot. The calculation formula can be expressed as follows:

[0055] a t =Actor(z fuse )

[0056] The above-mentioned information O is perceived from the proprioceptive t , visual observation information D t to the comprehensive perceptual representation z fuse The perceptual information encoding process is optimized using a hybrid loss function, which is defined as follows:

[0057] L=L hmap +L explict +L implicit +L obs

[0058] Where: L, L hmap , L explict , L implicit , L obs They represent mixed loss, elevation map reconstruction loss, explicit state estimation loss, implicit state estimation loss and observation reconstruction loss respectively.

[0059] Specifically, the elevation map reconstruction loss L hmap This will be explained in detail in the visual terrain reconstruction module. The remaining three items are calculated using mean squared error (MSE). Display state estimation loss L explict Represents the above comprehensive perceptual representation z fuseDisplay state estimation item body speed and the truth value v base The loss is defined as follows:

[0060]

[0061] Implicit state estimation loss L implicit Indicates the use of comprehensive perception to represent z fuse The implicit state estimation term z feet With z hmap Reconstructed elevation map of the robot's four legs and surrounding body and and the truth value H feet and H hmap The loss is defined as follows:

[0062]

[0063] Observation reconstruction loss L obs Indicates the use of comprehensive perception to represent z fuse Estimated next moment proprioception and the next moment proprioceptive observation truth value o t+1 By accurately predicting the observations at future moments, the model can rely on the current moment observations to have a more accurate understanding of the environmental state transition, including richer environmental privileged information, thereby better utilizing the comprehensive perception feature vector z fuse Guide the action output. Observation reconstruction loss L obs is defined as follows:

[0064]

[0065] Using the comprehensive perception feature vector z fuse Information about physical status base , H feet and H hmap Make explicit or implicit estimates and observe o at the next moment t+1 The essence of making predictions is to encourage the model to encode the current observation information in such a way that it can contain as much privileged environmental information as possible, making up for the problem of the model's weak understanding of the environment caused by not using privileged learning methods to provide priors during training, thereby better guiding the generation of motion.

[0066] After the model training is completed, the model can be deployed on the robot. Since all decoder modules are only used for state estimation and reconstruction loss calculation in the training phase, in the deployment phase, only the Actor strategy network and the forward transfer module in all perception information encoding need to be deployed on the robot.

[0067] The visual terrain reconstruction module aims to use the limited perception information of the quadruped robot's onboard sensors to reconstruct the privileged information of the elevation map around the robot body, so as to provide accurate terrain perception for subsequent motion strategy generation. This scheme is inspired by HeleiDuan et al.

[13] Inspired by the method of reconstructing elevation maps using deep visual information in our work, we receive visual observation information D t and the latent vector z of the proprioceptive feature prop As input, the motion prior information contained in the recent visual observation stream is fused with the proprioceptive features to perform feature encoding including a memory mechanism, and then a height map estimation of the robot's surroundings including the bottom and back of the body is explicitly reconstructed through two stages of coarse reconstruction and fine reconstruction. It is passed to the subsequent strategy generation module to guide the state encoder state estimation and motion generation.

[0068] For details, please refer to Figure 2 , the working method of the visual terrain reconstruction module is as follows:

[0069] A simplified two-dimensional CNN feature encoder is used to encode the visual observation information D t = <d t ,d t-1 ,...,d t-M+1 > Encode and obtain the visual feature latent vector z depth ; Then, the visual feature latent vector z depth and the latent vector z of the proprioceptive feature prop The GRU module is connected and input into a GRU module, which retains the historical proprioceptive information and visual observation information through its gated structure with a memory mechanism; then, the output of the GRU is input into a decoder (Rough Decoder) with an MLP structure to roughly reconstruct the elevation map and reshape it into two dimensions to obtain the rough reconstruction result of the elevation map. Then, considering the symmetrical structure of the U-Net network and its advantages in the field of image pixel-level transformation, the rough reconstruction result of the elevation map is The input is then sent to a reference U-Net network with the same input and output dimensions and a simplified decoder network (Refined Decoder) for fine reconstruction to obtain the refined reconstruction result of the elevation map.

[0070] The process of visual terrain reconstruction module to reconstruct the elevation map can be expressed as follows:

[0071] z depth =CNN(D t )

[0072] z mem =GRU(z depth ,zprop )

[0073]

[0074] The above visual terrain reconstruction module adopts the elevation map reconstruction loss L hmap Optimize. Elevation map reconstruction loss L hmap Contains the coarse reconstruction loss L calculated using mean squared error (MSE) rough And the fine reconstruction loss L calculated using the norm regularization error (L1 regularization, L1) refined , respectively represent the rough reconstruction elevation map and precisely reconstruct elevation maps and the true value H of the elevation map truth The estimated loss is defined as follows:

[0075]

[0076] During the training process, two processes, the rough reconstruction and the fine reconstruction of the elevation map, are designed because the experiment found that only using the decoder of the MLP structure to reconstruct the elevation map in only one stage will have limited feature expression capabilities, resulting in the estimated elevation map having smooth non-flat surfaces at the corners and edges, which is difficult to correct through the loss function. Therefore, the fine estimation module using the U-Net architecture can effectively clean up the rough estimated elevation map. At the same time, the linear nature of the L1 loss also enables it to refine the corners, edges and plane information that are difficult to accurately express during the rough estimation stage, and output a more accurate elevation map reconstruction result, so as to reduce the difficulty of subsequent modules in extracting features from the elevation map.

[0077] In addition, in the early stage of training, since the visual terrain reconstruction module is difficult to converge quickly, the reconstruction effect of the elevation map is poor and contains a lot of noise, and contains less effective terrain information. At this time, if the poor elevation map reconstruction results are directly sent to the subsequent strategy generation module, it may be difficult to generate good actions. Poor actions will cause the visual terrain reconstruction module to obtain poor observation results and become more difficult to converge, thus falling into a vicious circle. To avoid this problem, this paper proposes an adaptive bootstrap method that uses the true value H of the elevation map observation obtained from the simulation environment truth The core idea is that when the mean reward obtained in a trajectory is low, it means that the model training effect is poor. At this time, the elevation map truth value H should be used more. truth Replace the elevation map reconstruction result The input is then fed into the subsequent motion generation network, so that the motion generation model can better learn to effectively use terrain information to guide the generation of motion; when the average reward obtained by the model is high, it means that the model training effect is good, and the elevation map reconstruction results are more often used. As input, the motion generation model has a higher robustness to the estimation results of the visual terrain reconstruction module. Specifically, it is achieved by sampling the Bernoulli distribution according to formula (3.22). In the final deployment, the true value H of the elevation map is not used. truth , using the elevation map output by the visual terrain reconstruction module to accurately reconstruct the results As the input of the subsequent strategy generation part. In training, the model uses the elevation map truth value H truth Replace the elevation map reconstruction result The probability as input is defined as:

[0078] p boot =1-sigmoid(R)

[0079] Among them, R represents the average reward value obtained by the robot in each parallel environment in each trajectory in an episode.

[0080] For the above methods, choosing the right input and output is crucial. The input, i.e. the observation part of the model, needs to contain sufficient information that can represent the body state and terrain characteristics of the quadruped robot, so that the model can better learn the environmental characteristics through observation input and use it to improve gait flexibility and adaptability. In order for the motion strategy to perceive the environment ahead and adjust the gait in advance for possible obstacles or uneven ground, the model uses proprioceptive information o t and visual observation information t as input.

[0081] For proprioceptive information t , for each time step t, we have The meaning and dimensions of each parameter are shown in Table 1:

[0082] Table 1 Proprioceptive input

[0083]

[0084] The gravity projection in the body coordinate system represents the posture of the robot body. The proprioceptive input has a total of 45 dimensions. The proprioceptive observation sequence of N time steps is used as the actual input of the policy network. In this paper, N=10 is selected, so the actual input dimension of proprioceptive perception at each time step is 45×10=450.

[0085] For visual observation information, the depth image is regarded as a two-dimensional vector, and each depth image has a dimension of W×H. In this paper, the width and height of the depth image are both 64 pixels, so the dimension of each depth image is 64×64. Similar to the proprioceptive information, the visual external perception observation sequence of M time steps is used as the actual input of the strategy network. In this paper, M=6 is selected, so the actual input dimension of the visual external perception at each time step is 6×64×64.

[0086] According to the motion strategy architecture, the motion planning controller model outputs the expected position of each joint of the current robot and sends it to the motion PD controller for execution. Therefore, the output of each time step model is 12-dimensional, representing the expected position of each joint. In summary, the proprioceptive input of each time step model in this paper is 450-dimensional, the visual external perception input dimension is 6×64×64, and the output is 12-dimensional.

[0087] In addition, the terrain elevation map around the robot body used in the training phase of this paper uses a 5cm resolution elevation map within a range of 1.55m×0.75m around the center of the robot body, with a dimension of 32×16, such as Figure 3 The yellow dot in the middle shows the topographic elevation map around the four feet. The elevation map with a resolution of 5 cm in the range of 0.2 m × 0.2 m around each foot is used. The dimension is 4 × 5 × 5 = 100. Figure 3 Shown by the purple dot.

[0088] The reward function mainly includes two parts: speed tracking reward and auxiliary reward. Among them, the speed tracking reward is the most important reward, which encourages the model to track the input motion instructions and complete the corresponding motion tasks. The auxiliary rewards include body angular velocity tracking reward, body z-direction linear velocity penalty, body roll / pitch axis angular velocity penalty, body orientation penalty, power penalty, joint velocity penalty, joint acceleration penalty, collision penalty, action change rate penalty, action smoothness penalty, foot ground clearance penalty and landing point terrain edge penalty. The purpose is to enable the model to maintain body balance and stability while tracking motion instructions, and have a certain robustness to various interferences. Specifically, this article uses the following reward function for model training.

[0089] 1) Body Line Speed ​​Tracking Rewards: where v x 、v y is the body's x and y speed, Represents the speed command of the body in the x and y directions. The tracking loss is calculated after taking the minimum value of the body's x-direction speed and the speed command in order not to penalize the body's forward speed that is higher than the speed command, so that the robot can learn to move forward at the highest possible speed.

[0090] 2) Body angular velocity tracking reward: where ω yaw is the angular velocity of the body's yaw axis, Represents the body yaw axis angular velocity command;

[0091] 3) Penalty of linear velocity in the z direction of the body: where v z is the body's z-direction velocity;

[0092] 4) Body roll and pitch axis angular velocity penalty: where ω roll ,ω pitch is the body roll and pitch axis angular velocity;

[0093] 5) Body orientation penalty: where g x , g y The first two dimensions represent the projection of the gravity vector in the body coordinate system, which is expected to keep the body posture stable;

[0094] 6) Power penalty: Where N represents the total number of body joints, τ i represents the joint torque, represents the joint velocity;

[0095] 7) Joint speed penalty:

[0096] 8) Joint acceleration penalty: in represents the joint acceleration;

[0097] 9) Collision Penalty: Where n represents the total number of collision bodies (such as body, knees, etc.) that need to be punished by collision, and F i Indicates the external force on the collision body;

[0098] 10) Action change rate penalty: where a t is the action at the current time step, a t-1 is the action of the previous time step;

[0099] 11) Action smoothness penalty: where a t-2 is the action at the previous time step. This term can be approximated as the rate of change of the action change rate, that is, the second-order derivative of the action, which indicates the smoothness of the action change;

[0100] 12) Ground clearance penalty: where v i represents the linear velocity of the foot in the x, y plane, z i Indicates the height of the foot from the ground. represents the expected height of the foot from the ground. This expectation enables the robot to lift its feet to a higher height during movement to improve the robustness to uneven ground.

[0101] 13) Landing point terrain edge penalty: where c i Indicates the foot contact with the ground, e i Indicates whether the foot is within a range of less than a threshold distance from the edge of the terrain. This item expects the robot to actively choose a foothold farther from the edge of the terrain when moving, so that it is less likely to fall due to stepping on the edge of the terrain when migrating to the real object.

[0102] In actual training, the specific coefficients of each reward item used in this paper are shown in Table 2.

[0103] Table 2 Coefficients of various reward items

[0104]

[0105] The complex discrete terrain with sparse traversable areas described in this scheme includes flat land, gullies, terrain with plum blossom piles of the same height, single-plank bridge terrain, continuous gully terrain and highly undulating plum blossom pile terrain. When faced with complex discrete terrains with higher difficulty such as single-plank bridge terrain, continuous gully terrain and highly undulating plum blossom pile terrain, it is difficult for the robot to learn strategies that can better adapt to these terrains from scratch due to the sparsity of rewards and the conflict of optimal gait patterns between some terrains. Therefore, this scheme divides the training process into two stages, namely: first pre-training on flat land, narrow gully terrain, and plum blossom pile terrain of the same height, and then secondary training on single-plank bridge terrain, continuous gully terrain, wide gully terrain and highly undulating plum blossom pile terrain.

[0106] It is worth mentioning that the width of the wider gully terrain is greater than the set threshold TH, while the width of the narrower gully terrain is less than or equal to the set threshold TH.

[0107] Experimental results analysis:

[0108] Figures 4 to 7 The figures show the contact between the robot's four feet and the ground when it moves on a flat surface, a plum blossom pile, a single-plank bridge, and a continuous gully terrain (FL represents the left front foot, HL represents the left rear foot, FR represents the right front foot, and HR represents the right rear foot).

[0109] It can be found that although there is no constraint on the gait in any form during training, and the model is only encouraged to learn the best movement strategy through rewards such as command tracking and body stability, the model can learn to move in a Trot gait under different terrains, that is, a group of feet on the diagonal line (such as HL and FR) first enter the swing phase and move forward, and after landing and entering the contact phase, another group of feet on the diagonal line (such as FL and HR) enters the swing phase and moves forward, and so on. The Trot gait is one of the gaits with the highest energy efficiency among the gaits with a large range of movement speeds, and is the most common movement gait of quadrupeds in nature. Therefore, it can be considered that the model gradually learns a movement strategy that can meet the task requirements as much as possible during the training process and obtains a gait pattern close to the optimal one.

[0110] At the same time, although the model moves in a trot-like gait under different terrains, it can adaptively adjust the gait frequency according to the terrain characteristics. It can be seen from the figure that the gait frequency of the model is low in the plum blossom pile terrain. This is because the plum blossom pile terrain is relatively fragmented, and there are very few optional areas for the robot to maintain a stable foothold. When one foot of the robot selects a foothold and enters the contact phase, there are often no other plum blossom piles nearby that can be used as alternative footholds for the foot. The foot will only enter the swing phase and step out when the robot body moves forward and another optional foothold area appears in the motion space. Therefore, the farther distance between two adjacent stable footholds than other terrains leads to a lower gait frequency of the model in the plum blossom pile terrain. The model has a high gait frequency in the single-plank bridge terrain because the passable area in the single-plank bridge terrain is narrow, and the robot's left and right feet can only land near the center axis of the body, so it is in an unstable "inverted triangle" state when moving; and the model can continuously adjust its posture during movement through a higher frequency of gait, so that the body can effectively respond to random external interference in dynamic stability and maintain its own stability. The above experimental results show that the model can respond to different terrain features by adjusting its own gait frequency, and has learned effective movement strategies that can meet the expected tasks such as stability as much as possible.

[0111] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, the embodiments should be considered exemplary and non-restrictive in all respects, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims be included in the present invention.

Claims

1. A method for generating quadruped robot motion in sparse terrain based on terrain reconstruction, characterized in that: Train a motion planning controller with terrain-aware memory based on multimodal sensory input; Through the visual terrain reconstruction module with memory mechanism, the elevation map of the terrain around the current quadruped robot is displayed and estimated from the proprioceptive information and the visual observation information of the onboard depth camera; then, a multimodal feature encoder is used to fuse the features of the elevation map reconstruction results of the proprioceptive information and the visual observation information to obtain a comprehensive perception representation; Taking the comprehensive perception representation as input and constructing a corresponding reward function, the motion planning controller gives the desired torque of each joint of the quadruped robot in a complex discrete terrain with sparse navigable areas in an end-to-end manner.

2. The method for generating quadruped robot motion in sparse terrain based on terrain reconstruction according to claim 1, characterized in that: The method for displaying and estimating the terrain elevation map around the current quadruped robot is as follows: The proprioceptive information O t , a gated recurrent unit is used for encoding to obtain the latent vector of the proprioceptive feature For the visual observation information D t , the visual terrain reconstruction module is used for encoding, and the latent vector z of the proprioceptive feature is fused prop The motion prior information contained in it reconstructs an elevation map of the robot's surroundings, including the bottom and back of the body 3. The method for generating quadruped robot motion in sparse terrain based on terrain reconstruction according to claim 2, characterized in that: Using a convolutional neural network to reconstruct the elevation map Perform feature extraction to obtain a two-dimensional terrain feature vector Where C represents the feature dimension of each channel, W and H represent the feature vector width and height respectively; the two-dimensional terrain feature vector z hmap Split into W×H different C-dimensional feature vectors where each C-dimensional feature vector z hmap,i Represents a local terrain feature area; through the proprioceptive information O t and visual observation information D t After encoding, a total of W×H+1 C-dimensional feature vectors are obtained.

4. The method for generating quadruped robot motion in sparse terrain based on terrain reconstruction according to claim 3 is characterized in that: The method of fusing the features of the elevation map reconstruction results of proprioceptive information and visual observation information to obtain a comprehensive perception representation is as follows: A set of Transformer encoder layers are used to fuse the latent vector z of the ontology perception feature prop and the two-dimensional terrain feature vector z hmap ; Each Transformer encoder layer consists of a self-attention layer and a forward propagation layer with a residual connection mechanism; z prop With z hmap The feature dimensions are connected as the input sequence and encoded through multiple Transformer encoder layers to obtain a comprehensive perceptual representation with the same dimension as the input sequence. as output.

5. The method for generating quadruped robot motion in sparse terrain based on terrain reconstruction according to claim 4, characterized in that: The method of the motion planning controller to give the desired torque of each joint of the quadruped robot in a complex discrete terrain with sparse traversable areas end to end is as follows: The comprehensive perception representation z fuse Input an Actor network composed of MLP to get the action output a t , output the action a t The result is transmitted to the joint PD controller for execution, thus achieving the desired torque of each joint of the quadruped robot.

6. The method for generating quadruped robot motion in sparse terrain based on terrain reconstruction according to claim 5, characterized in that: The proprioceptive information O t , visual observation information D t to the comprehensive perceptual representation z fuse The perceptual information encoding process is optimized using a hybrid loss function, which is defined as follows: L=L hmap +L explict +L implicit +L obs Where: L, L hmap , L exolict , L implicit , L obs They represent mixed loss, elevation map reconstruction loss, explicit state estimation loss, implicit state estimation loss and observation reconstruction loss respectively.

7. The method for generating quadruped robot motion in sparse terrain based on terrain reconstruction according to claim 6, characterized in that: The working method of the visual terrain reconstruction module is as follows: A two-dimensional CNN feature encoder is used to encode the visual observation information D t =< d t ,d t-1 ,...,d t-M+1 > Encode and obtain the visual feature latent vector z depth ; Then, the visual feature latent vector z depth and the latent vector z of the proprioceptive feature prop The output of the gated recurrent unit is then input into a decoder with an MLP structure to roughly reconstruct the elevation map and reshape it into two dimensions, obtaining a rough reconstruction result of the elevation map. Then, the elevation map is roughly reconstructed. Input into the decoder network for fine reconstruction to obtain the elevation map fine reconstruction result 8. The method for generating quadruped robot motion in sparse terrain based on terrain reconstruction according to claim 7, characterized in that: In the process of terrain reconstruction, when the mean reward obtained in a trajectory is low, the elevation map truth value H is used truth Replace the elevation map reconstruction result Perform feature fusion; when the average reward obtained in a trajectory is high, use the elevation map to accurately reconstruct the result Perform feature fusion.

9. The method for generating quadruped robot motion in sparse terrain based on terrain reconstruction according to claim 1 or 8, characterized in that: The reward function includes a speed tracking reward and an auxiliary reward, and the auxiliary reward includes a body angular velocity tracking reward, a body z-direction linear velocity penalty, a body roll / pitch axis angular velocity penalty, a body orientation penalty, a power penalty, a joint velocity penalty, a joint acceleration penalty, a collision penalty, a motion change rate penalty, a motion smoothness penalty, a foot ground clearance penalty, and a landing point terrain edge penalty.

10. The method for generating quadruped robot motion in sparse terrain based on terrain reconstruction according to claim 7, characterized in that: The complex discrete terrain with sparse traversable areas includes flat land, gullies, plum blossom pile terrain of the same height, single-plank bridge terrain, continuous gully terrain and highly undulating plum blossom pile terrain; in the training stage, pre-training is first performed on flat land, gullies, and plum blossom pile terrain of the same height, and then secondary training is performed on single-plank bridge terrain, continuous gully terrain and highly undulating plum blossom pile terrain.

Images