Four-foot robot three-dimensional environment omni-directional motion control method based on mixed representation learning

Abstract

The invention provides a hybrid representation learning-based omni-directional motion control method for a quadruped robot in a three-dimensional environment, aiming at solving the problem that an existing motion control method for the quadruped robot is limited by a narrow forward view angle and external perception noise of the robot, so that the omni-directional motion capability in the three-dimensional environment is difficult to obtain. Comprising the steps of designing various input and output contents of a first-stage end-to-end neural network learning framework, designing a training process including mixed representation learning based on a reinforcement learning framework, constructing a specific neural network model of each module, and constructing a quadruped robot simulation motion control training environment. The method has the advantages that network input and output content beneficial to efficiently learning the relation between actual observation of the robot and the environment truth value is provided, and a mixed representation learning method including supervised learning and comparative learning is introduced, so that the robot can effectively deduce environment information beyond the visual field range of the robot in the training process.

Description

Technical Field

[0001] The present invention relates to the field of quadruped robot motion control based on reinforcement learning, and in particular to a method for omnidirectional motion control of a quadruped robot in a three-dimensional environment based on hybrid representation learning. Background Art

[0002] In recent years, with the continuous development of quadruped robot motion control algorithms, quadruped robots have gained more and more attention due to their superior motion capabilities. Quadruped robots have a body structure similar to that of animals, and can flexibly adjust their footholds and postures when facing various terrains to adapt to the needs of passing through complex environments. In addition, compared with humanoid robots and multi-legged robots such as hexapods, quadruped robots can achieve more robust and fast motion skills due to their stable and efficient body structure design, and are more suitable for deployment in very challenging environments such as mountains, forests, tunnels and caves to complete exploration, search and rescue tasks. However, unlike structured urban environments, quadruped robots need to use visual information to assist in passing various uneven terrains and three-dimensional obstacles on the ground or underground when performing operations in harsh environments such as the wild and narrow underground spaces, and can also demonstrate their motion flexibility when their field of vision is blocked or the direction of movement exceeds the field of vision.

[0003] Traditional motion control methods for quadruped robots are represented by optimization-based model predictive control, which has decent stability in walking tasks on flat terrain. However, when faced with highly uneven terrain and complex obstacles, traditional control methods rely on offline planning using pre-generated elevation maps. Therefore, when faced with the inaccurate mapping problem often encountered in unstructured outdoor environments, the robustness of traditional control methods will be severely reduced, and it is difficult to adaptively select dynamic and agile behaviors to pass through unstructured terrain.

[0004] Data-driven methods based on reinforcement learning are currently popular methods in the field of quadruped robot motion control. Inspired by the way animals learn robust motion skills, reinforcement learning methods skip the complex modeling process and train a neural network in a simulation environment built for a specific task. The goal is to maximize the set reward function return, learn motion strategies that meet the task requirements with high anti-disturbance performance and high dynamic performance, and finally deploy them on physical robots. Nahrendra et al. proposed a motion control learning framework that relies only on the proprioception of quadruped robots. By building an action network to learn and implicitly infer terrain attributes such as terrain height and friction, the robot can adaptively adjust its gait on rugged ground to obtain stable terrain passing ability. Fan Yong et al. (publication number: CN118192254A) proposed a teacher-student network trained with robot state observations and privileged information. By using a state estimation network module that predicts the linear velocity of motion with proprioceptive observations and an incremental terrain partitioning training method, the physical quadruped robot can move in different complex terrains. Zhao Xiaoqing et al. (publication number: CN118012077A) proposed a two-stage reinforcement learning method that introduced motion imitation learning, and improved the efficiency of quadruped robot motion strategy learning through joint tracking reward incentives. However, most of these control methods only focus on the body state of the quadruped robot without using visual information feedback such as depth cameras, making it difficult for the quadruped robot to pass obstacles of comparable or even higher size.

[0005] In addition, Agarwal et al. proposed a teacher-student network learning method that uses robot-centric depth camera visual information. The network is guided by privileged information such as elevation maps to learn the terrain perception required for the movement of quadruped robots from depth maps with unclear features, and achieves movement skills in complex terrains such as going up and down stairs and through cracks. Zhang Wei et al. (publication number: CN113867333A) proposed a method for quadruped robots to climb stairs based on visual perception, which obtains the geometric properties of stairs through three-dimensional point clouds and plans the robot's reference speed, landing position, target posture and climbing trajectory accordingly. However, these methods only achieve forward motion with the assistance of visual information, ignoring the characteristics of the quadruped robot's omnidirectional and flexible movement, and cannot cope with harsh field environment conditions where visual perception is disturbed or even the field of vision is completely blocked.

[0006] In summary, current quadruped robot motion control methods have achieved good traversability and stability on flat and rugged terrains. However, these methods cannot build an end-to-end efficient training framework, and thus cannot achieve omnidirectional motion skills of quadruped robots in three-dimensional environments using only limited visual perception. Summary of the invention

[0007] In view of the fact that the existing quadruped robot motion control methods are limited by the robot's narrow forward field of view and external sensory noise, making it difficult to obtain omnidirectional motion capabilities in a three-dimensional environment, the present invention proposes a three-dimensional environment omnidirectional motion control method for a quadruped robot based on hybrid representation learning, especially for the generation of motion skills of the quadruped robot in an environment containing three-dimensional obstacles when the robot's field of view is blocked or the motion direction is inconsistent with the field of view.

[0008] The present invention provides a method for controlling omnidirectional motion of a quadruped robot in a three-dimensional environment based on hybrid representation learning, which specifically includes the following steps:

[0009] S1. Design an overall framework of reinforcement learning with an asymmetric action network-strategy network structure, and design the input and output content of each part of the neural network in the overall framework of reinforcement learning; the neural network includes a standard input feature extraction encoder, a privileged information input feature extraction encoder, an action network, and a value network;

[0010] S2. Based on the overall framework of reinforcement learning in step S1, a hybrid representation learning training process is designed; the hybrid representation learning includes supervised learning, contrastive learning and reinforcement learning methods;

[0011] S3, based on the training process in step S2, building a neural network model for the standard input feature extraction encoder, the privileged information input feature extraction encoder, the action network and the value network to achieve the reasoning and gradient update functions required for training;

[0012] S4. Build a simulation training environment and design training terrain and reward functions suitable for quadruped robots to learn omnidirectional movement skills in a three-dimensional environment under different conditions;

[0013] S5. Based on the overall reinforcement learning framework of step S1, the training process of step S2, the neural network model of step S3 and the simulation training environment of step S4, reinforcement learning training is performed to obtain an omnidirectional motion control strategy for a three-dimensional environment suitable for limited visual perception, and the strategy is migrated and deployed on a quadruped robot.

[0014] Furthermore, the step S1 includes the following sub-steps:

[0015] S11. Design the overall framework of reinforcement learning, in which the standard input feature extraction encoder is used to process the standard input directly obtained from the physical quadruped robot and output a hidden layer vector as the extracted feature for the action network to generate actions in the next step; the privileged information input feature extraction encoder uses the input obtained in the simulation to provide the encoding of the surrounding environment; the value network receives the privileged information input, evaluates the results of the action network output, and guides and optimizes the action network;

[0016] S12. Based on the internal and external sensors of the quadruped robot, design the standard input content in the overall reinforcement learning framework serving the motion control task, including the quadruped robot's proprioception and depth vision images;

[0017] S13. Based on the standard input content of step S12, design privileged information input content that can be obtained in the simulation environment, which is used to characterize the local information of the three-dimensional environment and the state of the robot body, including body privileged observation and privileged external perception;

[0018] S14. Design the output content of the overall reinforcement learning framework, including the action network output for quadruped robot motion control and the value network output for guiding policy gradient updates.

[0019] Furthermore, the step S2 includes the following sub-steps:

[0020] S21. Design state estimation learning based on supervised learning method: construct an encoder-decoder structure before the action network, where the encoder accepts standard input and extracts features, reducing the input dimension to a hidden layer vector; then the decoder decodes the hidden layer vector into various state estimation values ​​that meet the requirements of the motion control task, and calculates the mean square error between the state true value obtained in the simulation, thereby realizing gradient propagation and guiding the update of the standard input encoder network;

[0021] S22. Design terrain understanding learning based on contrastive learning method: construct an encoder in front of the action network to accept privileged information input and extract features, and reduce the input dimension to a hidden layer vector; then, based on the contrastive learning method, calculate the contrast error between the hidden layer vector and the contrast vector in the output of the standard input feature extraction encoder, and then realize gradient propagation to guide the update of the standard input feature extraction encoder network and the privileged information input feature extraction encoder network;

[0022] S23. Design motion control learning based on reinforcement learning method: optimize the reinforcement learning algorithm through proximal strategy, calculate the value function loss, alternative strategy loss and entropy loss, and their sum constitutes PPO loss, and then realize gradient propagation to guide the update of action network and value network.

[0023] Furthermore, the step S3 includes the following sub-steps:

[0024] S31. Build a standard input feature extraction encoder neural network model: The standard input feature extraction encoder is based on the proprioception of the quadruped robot and the deep visual image, and uses the self-attention mechanism to extract the fusion features of these two modal inputs, and inputs their output into the gated recurrent unit to generate a hidden layer vector;

[0025] S32. Build a neural network model of a privileged information input feature extraction encoder: The privileged information input feature extraction encoder uses a cross-attention mechanism to extract features from the privileged ontology observation and privileged external perception obtained in the simulation, and obtains privileged external perception features extracted with the assistance of privileged ontology perception;

[0026] S33. Build an action network model and a value network model with asymmetric input: the action network extracts the hidden layer vector output by the encoder based on the standard input and the standard input feature, and outputs the expected position of each joint of the quadruped robot; the value network outputs the value of the current state based on the privileged information input.

[0027] Furthermore, in step S4, the training terrain includes: learning to effectively use vision to pass through high platforms, gullies and stair environments with challenging obstacles in front, learning rugged terrain where vision is disturbed and blocked, learning to crawl through floating obstacle terrain placed above the robot, and learning to pass through comprehensive three-dimensional environment terrain in the form of combined obstacles in all directions.

[0028] Furthermore, in step S4, the reward function mainly includes a task reward function item and a stability reward function item; wherein, the task reward function item is used to satisfy the motion control task instructions; the stability reward function item is used to keep the body posture of the quadruped robot stable during the movement and constrain the amplitude of the action output by the strategy.

[0029] Furthermore, in step S5, the motion control strategy obtained through training is exported into a format executable by the quadruped robot, and deployed on the quadruped robot for actual machine testing.

[0030] Compared with the prior art, the method of the present invention has the following beneficial effects: (1) The one-stage end-to-end training framework including hybrid representation learning proposed by the present invention enables the quadruped robot to fully understand the local environment information by using limited visual perception due to the efficient and comprehensive input encoding method, thereby being able to achieve omnidirectional mobility on a variety of challenging terrains; (2) The multimodal feature fusion extraction method introducing an asymmetric attention mechanism proposed by the present invention has good environmental adaptability. When the movement direction is consistent with the field of view, the quadruped robot can accept visual guidance and show excellent obstacle passing ability. When the movement direction is inconsistent with the field of view or the field of view is blocked, the quadruped robot can overcome the interference caused by vision and complete the movement task; (3) The present invention is theoretically applicable to various quadruped robot platforms, such as Jueying Lite3, Jueying X30, ANYmal C, Spot, Cheetah3, Go1, etc., and is adaptable to various common motion control tasks, such as jumping onto a high platform, jumping over a gully, climbing stairs, passing through a short hole, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a flow chart of the present invention for describing a method for omnidirectional control of a quadruped robot in a three-dimensional environment based on hybrid representation learning;

[0032] Figure 2 is a diagram of the overall framework structure of reinforcement learning in step S1 used in an embodiment of the present invention;

[0033] Figure 3 is a schematic diagram for describing visual perception in standard input of a quadruped robot according to an embodiment of the present invention;

[0034] Figure 4 is a schematic diagram for describing privileged external perception in privileged information input of a quadruped robot according to an embodiment of the present invention;

[0035] Figure 5 is a schematic diagram of a neural network framework of each module in step S3 used in an embodiment of the present invention;

[0036] Figure 6 is a schematic diagram of an embodiment of the present invention used to describe a quadruped robot passing through various obstacles in a good field of vision;

[0037] Figure 7 This is a schematic diagram of an embodiment of the present invention used to describe a quadruped robot passing through an obstacle when its field of vision is blocked and its movement direction is inconsistent with its field of vision. DETAILED DESCRIPTION

[0038] The present invention will be described in detail below according to the accompanying drawings and preferred embodiments, and the purpose and effect of the present invention will become more clear. The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0039] The present invention proposes a method for controlling the omnidirectional motion of a quadruped robot in a three-dimensional environment based on hybrid representation learning. Figure 1 As shown, the specific steps include:

[0040] S1. Design the overall framework of asymmetric action network-strategy network structure reinforcement learning, and design the input and output content of each part of the neural network in the framework.

[0041] S11. Model the reinforcement learning training environment as an infinite-horizon partially observable Markov decision process (POMDP) ​​and design the overall reinforcement learning framework. Divide the neural network framework into two parts with asymmetric inputs during training: the action network pathway that mainly accepts standard inputs and the policy network pathway that accepts privileged information inputs, in order to stimulate the implicit imagination process of local environment information when learning movement strategies.

[0042] In the embodiment of the present invention, Figure 2 As shown in the figure, the overall framework of reinforcement learning consists of four main components: standard input feature extraction encoder, privileged information input feature extraction encoder, action network and value network. The standard input feature extraction encoder is used to process the standard input directly obtained from the physical quadruped robot and output a hidden layer vector as the extracted feature for the action network to generate actions in the next step. The privileged information input feature extraction encoder uses input that can only be obtained in simulation to provide a more comprehensive encoding of the surrounding environment. Through the interaction between this encoder and the standard input feature extraction encoder, it is ensured that the hidden layer vector output by the standard input feature extraction encoder contains sufficient understanding of the local environment around the quadruped robot to guide the action decision process. The value network also receives privileged information input, and uses more comprehensive and realistic input to evaluate the results of the action network output, guiding the action network to optimize in a better direction.

[0043] S12. Based on the internal and external sensors carried by the quadruped robot, design standard input content in the overall reinforcement learning framework that serves the motion control task, including the quadruped robot's proprioception and depth vision images.

[0044] In the embodiment of the present invention, the standard input consists of two parts: the quadruped robot body observation o t and the deep visual image d t . The ontological observation is defined as:

[0045]

[0046] This includes the robot body angular velocity ω t , the direction of gravity g in the robot coordinate system t , Linear speed command on the xy plane in the robot coordinate system Z-axis angular velocity command in the world coordinate system The position of each joint of the robot t , the robot joint speed and the joint position command a issued in the previous frame t-1In order to retain transient memory, the proprioceptive perception of the quadruped robot in the past H frames is stacked. As the internal perception part in the standard input. Deep visual image d t The single-frame image captured by the front-mounted depth camera of the quadruped robot is acquired in the simulation environment by a simulated depth camera placed in the same position as the real object, such as Figure 3 shown.

[0047] S13. Based on the standard input content described in step S12, design privileged information input content that can be obtained in the simulation environment to further characterize the local information of the three-dimensional environment and the robot body state, including body privileged observation and privileged external perception.

[0048] In the embodiment of the present invention, the privileged information also consists of two parts: the privileged observation s of the quadruped robot body t and privileged external perception t . where privileged ontology observations are defined as:

[0049] s t =[o t ,v t ] T #(2)

[0050] This includes standard ontological observations o t and the body velocity v in the robot coordinate system t , the latter is privileged information that can only be obtained in simulation. In omnidirectional motion tasks in a three-dimensional environment, it is difficult to fully describe the terrain characteristics by relying solely on elevation maps. Since the 2.5D elevation map information is incomplete and the 3D voxel map has a large computational and storage overhead, the embodiment of the present invention proposes a new privileged external perception m t , which provides a lightweight and efficient three-dimensional terrain representation and is defined as:

[0051]

[0052] Among them, c t Represents a cubic depth map, which consists of five depth images centered on the quadruped robot facing different directions, facing forward Above Below Left and right This cubic depth map sampling method captures depth information in space more evenly than the equirectangular projection obtained by lidar-like sampling methods, which tend to be oversampled at the poles and undersampled at the equator. Contains the sparse depth information around the robot’s four legs. Similarly, mt It can only be obtained in a simulation environment, such as Figure 4 The above-mentioned privileged information input setting is designed to guide the standard input feature extraction encoder to better extract environmental information through rich and efficient 3D terrain representation.

[0053] S14. Design the output content of the overall reinforcement learning framework, including the action network output for quadruped robot motion control and the value network output for guiding policy gradient updates.

[0054] In the embodiment of the present invention, the output of the overall reinforcement learning framework can be mainly regarded as two parts: the action network output and the value network output. The action network outputs a vector of size 12×1, denoted as a t , represents the expected position of each joint of the quadruped robot. To improve learning efficiency, the embodiment of the present invention will a t Assume that the joint positions θ of the quadruped robot are in a stationary standing posture stand The relative value of , that is, the final expected position of each joint of the quadruped robot is defined as:

[0055] θ desired =θ stand +a t #(5)

[0056] The desired position of each joint of the quadruped robot is tracked by a proportional-derivative (PD) controller, which is converted into the desired torque of each joint of the quadruped robot. The value network outputs an estimate of the value of the current state V(s t ).

[0057] S2. Design a hybrid representation learning training process to obtain an overall training method that includes supervised learning, contrastive learning, and reinforcement learning methods.

[0058] S21. Design state estimation learning based on supervised learning methods. Construct an encoder-decoder structure before the action network, where the encoder accepts standard input and extracts features, reducing the input dimension to a hidden layer vector; the decoder then decodes the hidden layer vector into various state estimates that meet the requirements of the motion control task, and calculates the mean square error (MSE) between the true state values ​​obtained in the simulation, and then gradient propagation can be performed to guide the update of the standard input encoder network.

[0059] In the embodiment of the present invention, the state estimation learning based on the supervised learning method focuses on the features that are relatively easy to extract. In the standard input feature extraction encoder, the standard input is encoded into a hidden layer vector, which is composed of the estimated value of the quadruped robot body velocity. Hidden state estimation vector of quadruped robot Latent vector estimation of forward-looking depth map for quadruped robot And contrastive learning latent vector The features in the hidden layer vector can contain the true value of the estimated value e t Specifically, the estimated body velocity in the hidden layer vector Directly return to the true value v t , hidden state estimation vector The variational autoencoder structure is used, and the decoder is reconstructed into And return to the true value o t+1 In order to enhance the robot's intuitive understanding of depth vision, another forward-looking depth map estimates the latent vector Decoding into forward depth map from cube depth map Then with the truth value Calculate the regression error. The above regression loss L supervise It is calculated by Mean-Square Error (MSE), and the formula is:

[0060]

[0061] Among them, D KL represents the Kullback-Leibler (KL) divergence used in the variational autoencoder to calculate the similarity between two distributions, q represents the posterior distribution of the hidden state estimation vector, and is determined by the input With d t It is calculated that p represents the prior distribution of the hidden state vector, and the embodiment of the present invention adopts the standard normal distribution as the prior distribution of the hidden state vector.

[0062] S22. Design terrain understanding learning based on contrastive learning method. Independent of the standard input feature extraction encoder, build another encoder before the action network to accept privileged information input and extract features, and also reduce the input to a hidden layer vector; then based on the contrastive learning method, calculate the contrast error between the hidden layer vector and the contrast vector in the output of the standard input feature extraction encoder, and then perform gradient propagation to guide the update of the standard input feature extraction encoder network and the privileged information input feature extraction encoder network.

[0063] Although the supervised learning method can make the encoding result of the standard input contain actual and clear meaning, the supervised learning method alone will lose the function of feature extraction when there is no direct connection between the input and the estimated value. In the embodiment of the present invention, the terrain understanding learning based on the contrastive learning method focuses on establishing an abstract connection between the standard input of the quadruped robot and the privileged information input, aiming to establish a certain understanding of the state of the quadruped robot and the local environment through limited and noisy proprioception and visual observation. Specifically, the contrastive learning hidden vector in the hidden layer vector of the standard input encoding result Encoding result with privileged information input The shared characteristics between standard input and privileged information input should be captured. To achieve this, and Through a predictor, the two output vectors are recorded as and The embodiment of the present invention achieves the contrastive learning optimization goal by minimizing their negative cosine similarity, and the formula is:

[0064]

[0065] Among them, L contrast It represents the loss function used in the contrastive learning method, and stopgrad represents the gradient truncation operator, which aims to prevent the collapse phenomenon in the contrastive learning process.

[0066] S23. Design motion control learning based on reinforcement learning method. Implement the proximal policy optimization (PPO) reinforcement learning algorithm, calculate the value function loss, alternative policy loss and entropy loss, and the sum of them constitutes the PPO loss L ppo , and then gradient propagation can be performed to guide the update of the action network and the value network.

[0067] In summary, the comprehensive loss function L based on hybrid representation learning used in the embodiment of the present invention is total Defined as:

[0068] L total =L supervise +L contrast +L ppo #(8)

[0069] S3. Build a neural network model for each module to realize the reasoning and gradient update functions required for training.

[0070] S31. Build a standard input feature extraction encoder neural network model. The standard input feature extraction encoder is based on the proprioception and deep visual images of the quadruped robot. It uses the self-attention mechanism to extract fusion features from the above two modal inputs, and uses a recurrent neural network (RNN) to maintain long-term memory for limited observations to retain the understanding of local environmental information.

[0071] In an embodiment of the present invention, the standard input feature extraction encoder uses a multi-layer perceptron (MLP) to process the observation history To obtain proprioceptive features, a convolutional neural network (CNN) is used to extract the deep visual image d t These proprioceptive and external perception features are then concatenated and processed through a Transformer encoder with a self-attention mechanism to improve the effect of extracting fused features from cross-modal inputs. In order to preserve the quadruped robot’s long-term memory of the input, the output of the Transformer encoder is input into a gated recurrent unit (GRU) to generate a hidden layer vector.

[0072] S32. Build a neural network model for the privileged information input feature extraction encoder. The privileged information input feature extraction encoder uses a cross-attention mechanism to extract features from the privileged ontology observation and privileged external perception obtained in the simulation, and obtains privileged external perception features extracted with the assistance of privileged ontology perception.

[0073] In the embodiment of the present invention, the privileged information input feature extraction encoder is designed to be similar to the network structure of the standard input feature extraction encoder to introduce an inductive bias of outputting similar feature encodings in contrastive learning. However, a key difference between the two is that a cross-attention mechanism is introduced in the privileged information input feature extraction encoder to focus on the environmental information features when processing multimodal inputs. Specifically, the query input q is obtained by converting the privileged ontology perception observation s t is projected into a multilayer perceptron, where the key input k and value input v are computed by transforming the privileged visual observation m t The visual embedding (extracted by convolutional neural network CNN) is projected. In the subsequent skip connection and normalization (Add&Norm) module, only the visual embedding participates in the skip connection. This mechanism is designed to ensure that the privileged information input feature extraction encoder focuses on its visual input, while the privileged proprioceptive observation is supported as auxiliary information through the attention mechanism.

[0074] S33. Build an action network model and a value network model with asymmetric input. The action network extracts the hidden layer vector output by the encoder based on the standard input and the standard input feature, and outputs the expected position of each joint of the quadruped robot; the value network outputs the value of the current state based on the privileged information input.

[0075] In the embodiment of the present invention, the action network is based on the ontology perception o t The hidden layer vector output by the standard input feature extraction encoder is used as input, and a 12-dimensional action vector a is output through a multi-layer perceptron. tThe value network uses multi-layer perceptron and convolutional neural network to respectively t and privileged external perception t After projection processing, the state value V(s) is calculated using a multilayer perceptron. t ).

[0076] The neural network framework and training method used in the embodiments of the present invention are as follows: Figure 5 shown.

[0077] S4. Build a simulation training environment and design training terrain and reward functions suitable for quadruped robots to learn omnidirectional movement skills in a three-dimensional environment under different conditions.

[0078] S41. In an embodiment of the present invention, the Isaac Gym simulator is used to train the entire neural network framework. In order to develop a comprehensive strategy, an embodiment of the present invention designs a series of simulation environments, covering the learning of skills such as long jump, crawling, climbing stairs, and omnidirectional walking on discrete terrain. Based on the omnidirectional motion control task of a quadruped robot in a three-dimensional environment, four simulation training terrains are constructed, including: learning to effectively use vision to pass through high platforms, gullies and staircase environments with challenging obstacles in the front; learning rugged terrain where vision is disturbed and blocked; learning to crawl through floating obstacle terrain placed above the robot; learning omnidirectional comprehensive three-dimensional environment terrain in the form of combined obstacles. In addition, the embodiment of the present invention introduces regular noise into the depth image during the simulation process to simulate the noise that may be generated in the visual output due to possible field of view occlusion of the quadruped robot in actual scenes. In each environment, the embodiment of the present invention introduces terrain courses with increasing difficulty to achieve a smooth motor skill learning process.

[0079] S42. Based on the simulation training environment described in step S41, a reasonable reward function term is designed to motivate the quadruped robot to learn to track the speed instruction through the combination of three-dimensional obstacles.

[0080] In the embodiment of the present invention, a simple but efficient reward function is designed, which mainly includes a task reward function term and a stability reward function term. The former is designed to meet the motion control task instructions, specifically, to track the linear velocity in the xy plane and the z-axis angular velocity in the coordinate system of the quadruped robot issued; the latter is designed to keep the quadruped robot's body posture stable during the movement process and constrain the amplitude of the action output by the strategy to achieve a better physical robot deployment effect.

[0081] S5. Complete reinforcement learning training to obtain an omnidirectional motion control strategy for traversing a three-dimensional environment that can adapt to limited visual perception situations such as when the field of view is blocked or the direction of movement is inconsistent with the field of view, and migrate and deploy it on a quadruped robot.

[0082] S51. In an embodiment of the present invention, based on the training framework and simulation environment described above, reinforcement learning training is carried out to obtain a motion control strategy that meets the task requirements.

[0083] S52. In the embodiment of the present invention, the trained strategy is exported into a format that can be run by a quadruped robot, and deployed on the quadruped robot for real-machine testing to verify the method's ability to migrate from zero-sample simulation to reality. The robot used in the embodiment of the present invention is the quadruped robot "Jueying Lite3", which has a body height of 0.35m when standing, weighs 11kg, and has 12 degrees of freedom. Figure 6 As shown, in the forward motion with good visual conditions, the strategy implemented by the embodiment of the present invention can enable the quadruped robot to jump onto a 0.6m high platform, cross a 0.9m long gully, climb a 0.25m high continuous staircase, and drill through a 0.2m high short hole, demonstrating the extremely superior obstacle-passing motion control capability using visual information. Figure 7 As shown, in omnidirectional movement, the strategy implemented by the embodiment of the present invention can enable the quadruped robot to climb 0.15m high continuous stairs and pass through 0.25m high short holes, and can quickly respond to visual interference through proprioceptive feedback such as collisions to overcome the problem of difficulty in effectively obtaining visual information.

[0084] Those skilled in the art can understand that the above are only preferred examples of the invention and are not intended to limit the invention. Although the invention is described in detail with reference to the above examples, those skilled in the art can still modify the technical solutions recorded in the above examples or replace some of the technical features therein with equivalents. Any modification, equivalent replacement, etc. made within the spirit and principle of the invention shall be included in the protection scope of the invention.

Claims

1. A method for omnidirectional motion control of a quadruped robot in a three-dimensional environment based on hybrid representation learning, characterized in that: The following steps are involved: S1. Design an overall framework of reinforcement learning with an asymmetric action network-strategy network structure, and design the input and output content of each part of the neural network in the overall framework of reinforcement learning; the neural network includes a standard input feature extraction encoder, a privileged information input feature extraction encoder, an action network, and a value network; S2. Based on the overall framework of reinforcement learning in step S1, a hybrid representation learning training process is designed; the hybrid representation learning includes supervised learning, contrastive learning and reinforcement learning methods; S3, based on the training process in step S2, building a neural network model for the standard input feature extraction encoder, the privileged information input feature extraction encoder, the action network and the value network to achieve the reasoning and gradient update functions required for training; S4. Build a simulation training environment and design training terrain and reward functions suitable for quadruped robots to learn omnidirectional movement skills in a three-dimensional environment under different conditions; S5. Based on the overall reinforcement learning framework of step S1, the training process of step S2, the neural network model of step S3 and the simulation training environment of step S4, reinforcement learning training is performed to obtain an omnidirectional motion control strategy for a three-dimensional environment suitable for limited visual perception, and the strategy is migrated and deployed on a quadruped robot.

2. The method for controlling omnidirectional motion of a quadruped robot in a three-dimensional environment based on hybrid representation learning according to claim 1, characterized in that: The step S1 comprises the following sub-steps: S11. Design the overall framework of reinforcement learning, in which the standard input feature extraction encoder is used to process the standard input directly obtained from the physical quadruped robot and output a hidden layer vector as the extracted feature for the action network to generate actions in the next step; the privileged information input feature extraction encoder uses the input obtained in the simulation to provide the encoding of the surrounding environment; the value network receives the privileged information input, evaluates the results of the action network output, and guides and optimizes the action network; S12. Based on the internal and external sensors of the quadruped robot, design the standard input content in the overall reinforcement learning framework serving the motion control task, including the quadruped robot's proprioception and depth vision images; S13. Based on the standard input content of step S12, design privileged information input content that can be obtained in the simulation environment, which is used to characterize the local information of the three-dimensional environment and the state of the robot body, including body privileged observation and privileged external perception; S14. Design the output content of the overall reinforcement learning framework, including the action network output for quadruped robot motion control and the value network output for guiding policy gradient updates.

3. The method for controlling omnidirectional motion of a quadruped robot in a three-dimensional environment based on hybrid representation learning according to claim 2, characterized in that: The step S2 comprises the following sub-steps: S21. Design state estimation learning based on supervised learning method: construct an encoder-decoder structure before the action network, where the encoder accepts standard input and extracts features, reducing the input dimension to a hidden layer vector; then the decoder decodes the hidden layer vector into various state estimation values ​​that meet the requirements of the motion control task, and calculates the mean square error between the state true value obtained in the simulation, thereby realizing gradient propagation and guiding the update of the standard input encoder network; S22. Design terrain understanding learning based on contrastive learning method: construct an encoder in front of the action network to accept privileged information input and extract features, and reduce the input dimension to a hidden layer vector; then, based on the contrastive learning method, calculate the contrast error between the hidden layer vector and the contrast vector in the output of the standard input feature extraction encoder, and then realize gradient propagation to guide the update of the standard input feature extraction encoder network and the privileged information input feature extraction encoder network; S23. Design motion control learning based on reinforcement learning method: optimize the reinforcement learning algorithm through proximal strategy, calculate the value function loss, alternative strategy loss and entropy loss, and their sum constitutes PPO loss, and then realize gradient propagation to guide the update of action network and value network.

4. The method for controlling omnidirectional motion of a quadruped robot in a three-dimensional environment based on hybrid representation learning according to claim 3 is characterized in that: The step S3 comprises the following sub-steps: S31. Build a standard input feature extraction encoder neural network model: The standard input feature extraction encoder is based on the proprioception of the quadruped robot and the deep visual image, and uses the self-attention mechanism to extract the fusion features of these two modal inputs, and inputs their output into the gated recurrent unit to generate a hidden layer vector; S32. Build a neural network model of a privileged information input feature extraction encoder: The privileged information input feature extraction encoder uses a cross-attention mechanism to extract features from the privileged ontology observation and privileged external perception obtained in the simulation, and obtains privileged external perception features extracted with the assistance of privileged ontology perception; S33. Build an action network model and a value network model with asymmetric input: the action network extracts the hidden layer vector output by the encoder based on the standard input and the standard input feature, and outputs the expected position of each joint of the quadruped robot; the value network outputs the value of the current state based on the privileged information input.

5. The method for controlling omnidirectional motion of a quadruped robot in a three-dimensional environment based on hybrid representation learning according to claim 1, characterized in that: In step S4, the training terrain includes: learning to effectively use vision to pass through high platforms, gullies and stair environments with challenging obstacles in the front, learning rugged terrain where vision is disturbed and blocked, learning to crawl through floating obstacle terrain placed above the robot, and learning to pass through comprehensive three-dimensional environment terrain in the form of combined obstacles in all directions.

6. The method for controlling omnidirectional motion of a quadruped robot in a three-dimensional environment based on hybrid representation learning according to claim 1, characterized in that: In step S4, the reward function mainly includes a task reward function item and a stability reward function item; wherein the task reward function item is used to satisfy the motion control task instructions; the stability reward function item is used to keep the quadruped robot's body posture stable during the movement and constrain the amplitude of the action output by the strategy.

7. The method for controlling omnidirectional motion of a quadruped robot in a three-dimensional environment based on hybrid representation learning according to claim 1, It is characterized in that In step S5, the motion control strategy obtained through training is exported into a format that can be run by the quadruped robot. And it is deployed on a quadruped robot for actual machine testing.

Images