Quadruped robot foot end terrain prediction method and system based on long short-term memory network and reinforcement learning

Abstract

The invention provides a quadruped robot foot end terrain prediction method and system based on a long short-term memory network and reinforcement learning, and relates to the technical field of machine learning. Firstly, a teacher-student strategy model for predicting the foot end terrain of the quadruped robot is constructed, and the teacher-student strategy model comprises a teacher strategy model and a student strategy model; body sensing and privilege information and radar point cloud information of the quadruped robot are obtained, and a teacher strategy model is trained through the body sensing and privilege information, the radar point cloud information and reinforcement learning; the teacher strategy model is utilized to guide and train the student strategy model, and in the training process, the student strategy model uses a long short-term memory network to fit the privilege information coding value and the radar point cloud coding value in the teacher model; and the trained student strategy model is deployed on the quadruped robot for terrain prediction, stable walking in the unstructured terrain is realized, and the prediction robustness is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine learning, and more specifically relates to a method and system for predicting terrain at the foot end of a quadruped robot based on long short-term memory networks and reinforcement learning. Background Art

[0002] With the progress of society and the development of science and technology, the application field of mobile robots has gradually expanded. Traditional wheeled robots and tracked robots are widely used due to their advantages such as simple control and mature technology. However, they are greatly affected by terrain factors and cannot work in many environments. Quadruped robots came into being. Quadruped robots have the ability to walk on unstructured terrain and can perform tasks that wheeled or tracked robots cannot complete.

[0003] Reinforcement learning is an important branch of machine learning, which achieves intelligent decision-making through interactive learning with the environment. With the continuous development of artificial intelligence technology, reinforcement learning, as a method of autonomous learning, has gradually become a research hotspot and has been widely used in various fields.

[0004] At present, two training stages and multiple networks are required to achieve the basic walking of a quadruped robot and the ability to predict the surrounding foot-end terrain. The prior art discloses a quadruped robot motion planning method based on privileged knowledge distillation. The scheme is divided into two stages to train a teacher network with privileged information and a student network that only uses proprioception; for the teacher network, the state update network, action execution network, and reward function of the quadruped robot are designed based on the PPO algorithm; for the student network, a TCN encoder that can accommodate a long history is used. By imitating the teacher's behavior supervision training, the student network can infer terrain information from historical proprioception to make decisions. This scheme enables the quadruped robot to quickly and stably cross a variety of unstructured complex terrains in a real environment using only proprioception.

[0005] However, in the prior art, the student network uses a TCN encoder, which processes time series data through causal convolution and dilated convolution, which can effectively capture the robot's proprioceptive information and fit the encoding value. However, TCN needs to specify the length of the historical sequence, so the training occupies a large amount of video memory, resulting in longer training time and low efficiency in predicting the foot-end terrain. Summary of the invention

[0006] In order to solve the problem of low terrain prediction efficiency of current quadruped robots, the present invention provides a quadruped robot foot-end terrain prediction method and system based on long short-term memory network and reinforcement learning, which better improves the quadruped robot's terrain prediction ability and improves prediction efficiency.

[0007] In order to achieve the above technical effects, the technical solution of the present invention is as follows:

[0008] A method for predicting terrain at the foot end of a quadruped robot based on long short-term memory network and reinforcement learning, the detection method comprising the following steps:

[0009] S1: constructing a teacher-student strategy model for terrain prediction at the foot end of a quadruped robot, wherein the teacher-student strategy model includes a teacher strategy model and a student strategy model;

[0010] S2: Obtain proprioception, privileged information and radar point cloud information of the quadruped robot, and use proprioception, privileged information, radar point cloud information and reinforcement learning to train the teacher strategy model;

[0011] S3: Use the teacher strategy model to guide the training of the student strategy model. During the training process, the student strategy model uses the long short-term memory network to fit the privileged information encoding value and the radar point cloud encoding value in the teacher model;

[0012] S4: Deploy the trained student strategy model on a quadruped robot for terrain prediction.

[0013] Furthermore, the teacher strategy model consists of three multi-layer perceptron components: terrain encoder Privileged Encoder and a teacher-executor network; the terrain encoder Privileged Encoder Connect in parallel and then connect the teacher actuator network.

[0014] Furthermore, the teacher strategy model also includes a discriminator network, and the evaluator network is used to judge the similarity between the current state of the quadruped robot and the reference motion.

[0015] Furthermore, the student strategy model consists of a long short-term memory network, an encoder And the student actuator network is connected in sequence.

[0016] Furthermore, the privileged information includes the linear velocity of the substrate, the ground friction coefficient, the ground elastic coefficient, the contact force of the four-legged ends, the external force exerted on the fuselage, and the collision state.

[0017] Furthermore, step S2 also includes: designing a state space, an action space and a reward function for reinforcement learning;

[0018] Among them, the state of the state space Proprioceptive observations Privileged Information and terrain information Composition; proprioceptive observations Including: the direction of the gravity vector in the quadruped robot base coordinate system, the base angular velocity, joint position, velocity, and the last action a selected by the current strategy t-1 and the desired base velocity command vector; Privileged Information Including: substrate linear velocity, ground friction coefficient, ground elastic coefficient, four-leg contact force, external force and collision state of the fuselage; terrain information represents the measurements of 187 terrain points sampled from a grid around the base of the quadruped robot;

[0019] Action space uses policy action a t Denotes , as the target joint position offset, which is added to the time-invariant nominal joint positions to specify the target motor position for each joint of the quadruped robot;

[0020] Reward function:

[0021]

[0022] In the formula, Representative task rewards, Rewards for representative style, represents the regularization reward; the task reward consists of linear velocity and angular velocity tracking; the style reward is used to evaluate the similarity between the behavior of the quadruped robot and the behavior in the reference dataset; the style reward is composed of parameters The discriminator of the neural network representation Output.

[0023] Furthermore, the process of training the teacher strategy model using reinforcement learning is as follows: Input to privileged encoder Terrain information Input to Terrain Coder Privileged Information and terrain information In the privileged coder Terrain Coder are compressed into low-dimensional latent representations and Then and Concatenate into a complete latent representation Will and proprioceptive observations Input to the teacher executor network, output Gaussian distribution in, represents the mean, Represents the variance of the action.

[0024] The process of training the teacher strategy model using reinforcement learning is as follows: collect the teacher's current state and previous state in the training environment, input them into the discriminator to obtain the similarity between the two-step state of the quadruped robot and the reference action, and then add them to the reward function. After collecting the current state, action, expected state, reward result and termination judgment condition of the specified round, use the general reinforcement learning PPO algorithm to update the teacher strategy model;

[0025] After the teacher strategy model is updated, the training environment is reset and the training process is repeated until the upper limit of the training round is reached and the teacher strategy model is saved.

[0026] Furthermore, in step S3, the past hidden state of the long short-term memory network is h t -1, cell state is c t-1 , proprioceptive observations The previous hidden state h of the LSTM network t -1. Cell state c t-1 Encoded by the LSTM network as the current hidden state h t Pass to encoder Output encoding value Then and Concatenate input student executor network output

[0027]

[0028] The present invention also provides an electronic device, comprising a memory, a processor, and a quadruped robot foot-end terrain prediction program based on a long short-term memory network and reinforcement learning, which is stored in the memory and runs on the processor. When the quadruped robot foot-end terrain prediction program based on a long short-term memory network and reinforcement learning is executed by the processor, the steps of a quadruped robot foot-end terrain prediction method based on a long short-term memory network and reinforcement learning are implemented.

[0029] The present invention also provides a quadruped robot foot-end terrain prediction system based on long short-term memory network and reinforcement learning, comprising:

[0030] A terrain prediction model construction module is used to construct a teacher-student strategy model for terrain prediction at the foot of a quadruped robot, wherein the teacher-student strategy model includes a teacher strategy model and a student strategy model;

[0031] The first training module is used to obtain proprioception, privileged information and radar point cloud information of the quadruped robot, and train the teacher strategy model using proprioception, privileged information, radar point cloud information and reinforcement learning;

[0032] The second training module uses the teacher strategy model to guide the training of the student strategy model. During the training process, the student strategy model uses the long short-term memory network to fit the privileged information encoding value and the radar point cloud encoding value in the teacher model.

[0033] The terrain prediction module is used to deploy the trained student strategy model on the quadruped robot for terrain prediction.

[0034] Compared with the prior art, the beneficial effects of this method are:

[0035] The present invention proposes a method and system for predicting terrain at the foot of a quadruped robot based on a long short-term memory network and reinforcement learning. First, a teacher-student strategy model for predicting terrain at the foot of a quadruped robot is constructed, wherein the student strategy model extracts the historical sequence of the quadruped robot's body information through a long short-term memory network, so that the quadruped robot can extract surrounding terrain information based on the historical sequence information and achieve stable walking in unstructured terrain. The method proposed in the present invention uses a long short-term memory network to encode historical information, which saves more onboard resources than a multi-layer perceptron and a temporal convolutional network, and only uses body perception measurements without the need for additional sensors, thereby improving prediction robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a flow chart of a method for predicting terrain at the foot end of a quadruped robot based on a long short-term memory network and reinforcement learning according to an embodiment of the present invention;

[0037] Figure 2 It is a framework diagram of the teacher-student strategy model proposed in the embodiment of the present invention;

[0038] Figure 3 It is a basic structure diagram of the long short-term memory network proposed in an embodiment of the present invention;

[0039] Figure 4 This is a terrain information map proposed in an embodiment of the present invention.

[0040] Figure 5 It is a structural diagram of a quadruped robot foot-end terrain prediction system based on long short-term memory network and reinforcement learning proposed in an embodiment of the present invention.

[0041] Figure 6 It is a schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0043] In order to better illustrate the present embodiment, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the actual size;

[0044] It is understandable to those skilled in the art that descriptions of certain well-known contents in the drawings may be omitted.

[0045] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0046] The positional relationships described in the drawings are only for illustrative purposes and should not be construed as limiting the present patent;

[0047] Example 1

[0048] This embodiment proposes a method for predicting terrain at the foot end of a quadruped robot based on long short-term memory network and reinforcement learning. Figure 1 , including the following steps:

[0049] S1: constructing a teacher-student strategy model for terrain prediction at the foot end of a quadruped robot, wherein the teacher-student strategy model includes a teacher strategy model and a student strategy model;

[0050] S2: Obtain proprioception, privileged information and radar point cloud information of the quadruped robot, and use proprioception, privileged information, radar point cloud information and reinforcement learning to train the teacher strategy model;

[0051] S3: Use the teacher strategy model to guide the training of the student strategy model. During the training process, the student strategy model uses the long short-term memory network to fit the privileged information encoding value and the radar point cloud encoding value in the teacher model;

[0052] S4: Deploy the trained student strategy model on a quadruped robot for terrain prediction.

[0053] In this embodiment, a method for training a teacher strategy model and a student strategy model is proposed. Based on the above steps, this embodiment provides Figure 2 Framework diagram of the teacher-student strategy model shown.

[0054] The teacher policy model consists of three multi-layer perceptron components: terrain encoder Privileged Encoder and a teacher-executor network; the terrain encoder Privileged Encoder Connect in parallel and then connect the teacher actuator network.

[0055] The student strategy model consists of a long short-term memory network, an encoder And the student actuator network is connected in sequence.

[0056] In this embodiment, for step S2, the state space, action space and reward function of reinforcement learning are first designed.

[0057] The learning goal of reinforcement learning is to find an optimal state sequence. The minimum unit of a state sequence can be defined as (S t-1 ,A t ,S t ) That is, the robot dog takes an action according to the state of the previous step and enters the current state. For each state, it needs to be input into the teacher evaluator network to obtain the goodness (value) of this state. For each action, the goodness (reward value) of the action can be directly calculated according to the reward function.

[0058] Among them, the state of the state space Proprioceptive observations Privileged Information and terrain information Composition; proprioceptive observations Including: the direction of the gravity vector in the quadruped robot base coordinate system, the base angular velocity, joint position, velocity, and the last action a selected by the current strategy t-1 and the desired base velocity command vector; such as Figure 2 As shown, in this embodiment, the proprioceptive observation value is 48 dimensions, the privileged information is 30 dimensions, and the radar point cloud is 187 dimensions. Including: substrate linear velocity, ground friction coefficient, ground elastic coefficient, four-leg contact force, external force and collision state of the fuselage; terrain information represents the measurements of 187 terrain points sampled from a grid around the base of the quadruped robot, as Figure 3 shown.

[0059] Action space uses policy action a t Denotes the target joint position offset, which is added to the time-invariant nominal joint position to specify the target motor position for each joint; the joint proportional-derivative controller uses these target motor positions with fixed gains (Kp=20, Kd=0.5) to calculate the torque command.

[0060] Reward function:

[0061]

[0062] In the formula, Representative task rewards, Rewards for representative style, represents the regularization reward; the task reward consists of linear velocity and angular velocity tracking; the style reward is used to evaluate the similarity between the behavior of the quadruped robot and the behavior in the reference dataset; the style reward is composed of parameters The discriminator of the neural network representation Output.

[0063] In reinforcement learning, the use of a discriminator can simplify the reward function. It can calculate the degree of similarity between the actions of the quadruped robot during training and the reference actions. The more similar, the higher the reward value.

[0064] The teacher obtains the new state and the previous state each time it moves in the environment. This state sequence is input to the discriminator network To obtain Used to calculate gait style similarity reward This award is added to other awards received by teachers.

[0065] Discriminator training objectives:

[0066]

[0067] The first two are least squares generative adversarial network formulas that encourage the discriminator to distinguish whether a given input state transition comes from the quadruped robot or the reference action. Both the reference action and the robot dog action are expressed as It means that when the reference action is input into the discriminator, the discriminator is encouraged to output 1, and when the robot dog action is input into the discriminator, the discriminator is encouraged to output 0. After a period of training, the discriminator can distinguish the similarity between the robot dog's current action and the reference action. The more similar, the more the output value tends to be 1. The last term is the gradient penalty, which reduces the tendency of the discriminator to assign non-zero gradients on the real data sample manifold. gp is a manually specified coefficient, using α gp = 10. The style reward is then defined as:

[0068]

[0069] Using reinforcement learning to train a teacher strategy model to privilege information Input to privileged encoder Terrain information Input to Terrain Coder Privileged Information and terrain information In the encoder, they are compressed into low-dimensional latent representations and Then and Concatenate into a complete latent representation Will and proprioceptive observations Input to the teacher executor network, output Gaussian distribution in, represents the mean, Represents the variance of the action.

[0070] The teacher’s current state and previous state in the environment are collected and input into the discriminator to obtain the similarity between the two-step state of the quadruped robot and the reference action, which is then added to the reward function. After the current state, action, expected state, reward result and termination judgment condition of the specified round are collected, the teacher’s strategy model is updated.

[0071] After the teacher policy model is updated, reset the environment and repeat the training process until the training round limit is reached and save the teacher policy model.

[0072] The teacher strategy model also includes an evaluator network, which takes as input the ontology perception, privileged information, and the concatenated vector of the radar point cloud. The output is a value to obtain the degree of goodness of this state. The larger the value, the better the state.

[0073] The input of the teacher-executor network is the concatenated vector of the proprioception, privileged information encoding value, and radar point cloud encoding value, and the output is 12-dimensional actions. The reward value corresponding to the actions is calculated by the reward function.

[0074] The teacher evaluator network and the teacher actuator network are trained together using the PPO algorithm. During training, for normal walking gaits (the quality of walking is evaluated by both value and reward), the teacher evaluator network will output a higher value, and for gaits that are prone to falling, it will output a lower value. Training encourages the robot dog to find state sequences with high value and reward (S t-1 ,A t ,S t ).

[0075] Reinforcement learning objective formula:

[0076]

[0077] where γ_t is the discount factor.

[0078] During student training, since the real machine cannot obtain privileged information and radar point cloud, a long short-term memory network is used to extract proprioceptive perception. The history sequence is fitted to the privileged information encoding value and the radar point cloud encoding value.

[0079] The past hidden state of the LSTM network is h t -1, cell state is c t-1 , proprioceptive observations The previous hidden state h t -1. Cell state c t-1 Encoded by the LSTM network as the current hidden state h t Pass to encoder Output encoding value Then and Concatenate input student executor network output

[0080] Figure 4 This is the basic structure diagram of the long short-term memory network. The input of the long short-term memory network is 40 steps. (t-40 to t), the output is 256-dimensional ht, which is then encoded to obtain 16-dimensional encoding values ​​and privileged information encoding values, and the radar point cloud encoding values ​​are fitted.

[0081] In a LSTM network, the first step is to decide what information to discard from the cell state. This decision is made by a sigmoid layer called the "forget gate layer". It looks at the hidden state h at the previous time step. t-1 and the current input x t , and for each number in the cell state, outputs a number between 0 and 1. 1 means "totally keep this", while 0 means "totally discard this".

[0082] f t =σ(W f ·[h t-1 ,x t ]+b f )

[0083] The second step is to decide what new information to store in the cell state. This has two parts. First, a sigmoid layer, called the “input gate layer,” decides which values ​​we will update. Next, a tanh layer creates a new candidate value vector These values ​​can be added to the state. In the next step, the two will be combined to update the state.

[0084] i t =σ(W i ·[h t-1 ,x t ]+b i )

[0085]

[0086] The third step is to multiply the old state by f t , forget what we decided to forget before. Then add Here are the new candidate values, which are scaled by how much to update each state value.

[0087] Finally, we need to decide what to output. This output will be based on our cell state, but will be a filtered version of it. First, we run a sigmoid layer, which decides which parts of the cell state we will output. We then pass the cell state through a tanh function (which pushes the values ​​to between -1 and 1) and multiply it with the output of the sigmoid gate so that we only output the parts we decided to output.

[0088] The student actuator network input is proprioceptive perception As well as the encoding value of the network, the output is actions. When the student is trained, the encoding value of the network is compared with the privileged information encoding value of the teacher, the radar point cloud encoding value generates the reconstruction loss, and the student actions and the teacher's actions generate the imitation loss.

[0089] The trained student policy model is deployed and tested. The quadruped robot can stably cross different terrains, demonstrating the superior performance of the model.

[0090] Example 2

[0091] Figure 5 For this embodiment, a quadruped robot foot-end terrain prediction system based on long short-term memory network and reinforcement learning is proposed, including:

[0092] A terrain prediction model construction module is used to construct a teacher-student strategy model for terrain prediction at the foot of a quadruped robot, wherein the teacher-student strategy model includes a teacher strategy model and a student strategy model;

[0093] The first training module is used to obtain proprioception, privileged information and radar point cloud information of the quadruped robot, and train the teacher strategy model using proprioception, privileged information, radar point cloud information and reinforcement learning;

[0094] The second training module uses the teacher strategy model to guide the training of the student strategy model. During the training process, the student strategy model uses the long short-term memory network to fit the privileged information encoding value and the radar point cloud encoding value in the teacher model.

[0095] The terrain prediction module is used to deploy the trained student strategy model on the quadruped robot for terrain prediction.

[0096] Example 3

[0097] like Figure 6 As shown, an embodiment of the present invention further proposes an electronic device, comprising a memory 101, a processor 102, and a computer program stored in the memory 101 and running on the processor 102, wherein when the processor 102 executes the computer program, the steps of the quadruped robot foot-end terrain prediction method based on long short-term memory network and reinforcement learning proposed in this embodiment are implemented.

[0098] Specifically, in the present embodiment, the processor 102 may include a central processing unit (CPU) or a specific integrated circuit, or be configured to implement one or more integrated circuits of the present embodiment, and the memory 101 may include a large-capacity memory for data or instructions. It may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 101 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 101 may be inside or outside the integrated gateway disaster recovery device.

[0099] The memory 101 may include a read-only memory (ROM), a random access memory (RAM), a disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Therefore, generally, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the steps of implementing the method for predicting the terrain at the foot end of a quadruped robot based on a long short-term memory network and reinforcement learning proposed in this embodiment.

[0100] The embodiments are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A quadruped robot foot-end terrain prediction method based on long short-term memory network and reinforcement learning, characterized in that: The detection method comprises the following steps: S1: constructing a teacher-student strategy model for terrain prediction at the foot end of a quadruped robot, wherein the teacher-student strategy model includes a teacher strategy model and a student strategy model; S2: Obtain proprioception, privileged information and radar point cloud information of the quadruped robot, and use proprioception, privileged information, radar point cloud information and reinforcement learning to train the teacher strategy model; S3: Use the teacher strategy model to guide the training of the student strategy model. During the training process, the student strategy model uses the long short-term memory network to fit the privileged information encoding value and the radar point cloud encoding value in the teacher model; S4: Deploy the trained student strategy model on a quadruped robot for terrain prediction.

2. A method for predicting terrain at the foot end of a quadruped robot based on long short-term memory network and reinforcement learning according to claim 1, characterized in that: The teacher policy model consists of three multi-layer perceptron components: terrain encoder Privileged Encoder and a teacher-executor network; the terrain encoder Privileged Encoder Connect in parallel and then connect the teacher actuator network.

3. A method for predicting terrain at the foot end of a quadruped robot based on long short-term memory network and reinforcement learning according to claim 2, characterized in that: The teacher strategy model also includes a discriminator network, and the evaluator network is used to judge the similarity between the current state of the quadruped robot and the reference action.

4. The method for predicting terrain at the foot end of a quadruped robot based on long short-term memory network and reinforcement learning according to claim 1, characterized in that: The student strategy model consists of a long short-term memory network, an encoder And the student actuator network is connected in sequence.

5. The method for predicting terrain at the foot end of a quadruped robot based on long short-term memory network and reinforcement learning according to claim 1, characterized in that: The privileged information includes: substrate linear velocity, ground friction coefficient, ground elastic coefficient, four-legged end contact force, and external force and collision state of the fuselage.

6. The method for predicting terrain at the foot end of a quadruped robot based on long short-term memory network and reinforcement learning according to claim 1, characterized in that: Step S2 also includes: designing a state space, an action space, and a reward function for reinforcement learning; Among them, the state of the state space Proprioceptive observations Privileged Information and terrain information Composition; proprioceptive observations Including: the direction of the gravity vector in the quadruped robot base coordinate system, the base angular velocity, joint position, velocity, and the last action a selected by the current strategy t-1 and the desired base velocity command vector; Privileged Information Including: substrate linear velocity, ground friction coefficient, ground elastic coefficient, four-leg contact force, external force and collision state of the fuselage; terrain information represents the measurements of 187 terrain points sampled from a grid around the base of the quadruped robot; Action space uses policy action a t Denotes , as the target joint position offset, which is added to the time-invariant nominal joint positions to specify the target motor position for each joint of the quadruped robot; Reward function: In the formula, Representative task rewards, Rewards for representative style, represents the regularization reward; the task reward consists of linear velocity and angular velocity tracking; the style reward is used to evaluate the similarity between the behavior of the quadruped robot and the behavior in the reference dataset; the style reward is composed of parameters The discriminator of the neural network representation Output.

7. A method for predicting terrain at the foot end of a quadruped robot based on long short-term memory network and reinforcement learning according to claim 6, characterized in that: Privileged Information Input to privileged encoder Terrain information Input to Terrain Coder Privileged Information and terrain information In the privileged coder Terrain Coder are compressed into low-dimensional latent representations and Then and Concatenate into a complete latent representation Will and proprioceptive observations Input to the teacher executor network, output Gaussian distribution in, represents the mean, represents the variance of the action; The process of training the teacher strategy model using reinforcement learning is as follows: collect the teacher's current state and previous state in the training environment, input them into the discriminator to obtain the similarity between the two-step state of the quadruped robot and the reference action, and then add them to the reward function. After collecting the current state, action, expected state, reward result and termination judgment condition of the specified round, use the general reinforcement learning PPO algorithm to update the teacher strategy model; After the teacher strategy model is updated, the training environment is reset and the training process is repeated until the upper limit of the training round is reached and the teacher strategy model is saved.

8. The method for predicting terrain at the foot end of a quadruped robot based on long short-term memory network and reinforcement learning according to claim 7, characterized in that: In the student strategy model, the past hidden state of the LSTM network is h t -1, cell state is c t-1 , proprioceptive observations The previous hidden state h of the LSTM network t -1. Cell state c t-1 Encoded by the LSTM network as the current hidden state h t Pass to encoder Output encoding value Then and Concatenate input student executor network output 9. An electronic device, characterized in that: The method comprises a memory, a processor, and a quadruped robot foot-end terrain prediction program based on a long short-term memory network and reinforcement learning, which is stored in the memory and runs on the processor. When the quadruped robot foot-end terrain prediction program based on a long short-term memory network and reinforcement learning is executed by the processor, the steps of the method as described in any one of claims 1 to 8 are implemented.

10. A quadruped robot foot-end terrain prediction system based on long short-term memory network and reinforcement learning, characterized in that: include: A terrain prediction model construction module is used to construct a teacher-student strategy model for terrain prediction at the foot of a quadruped robot, wherein the teacher-student strategy model includes a teacher strategy model and a student strategy model; The first training module is used to obtain proprioception, privileged information and radar point cloud information of the quadruped robot, and train the teacher strategy model using proprioception, privileged information, radar point cloud information and reinforcement learning; The second training module uses the teacher strategy model to guide the training of the student strategy model. During the training process, the student strategy model uses the long short-term memory network to fit the privileged information encoding value and the radar point cloud encoding value in the teacher model. The terrain prediction module is used to deploy the trained student strategy model on the quadruped robot for terrain prediction.

Images