Quadruped robot control method and device and medium

Abstract

The invention relates to a quadruped robot control method, which realizes efficient and stable control of a quadruped robot by combining a basic behavior controller (BBC), a task specific controller (TSC) and an evolutionary strategy simulator optimization method. The BBC learns diverse and natural behavior patterns through a semi-supervised information maximized generation adversarial imitation learning method; the TSC generates a control instruction through privileged learning and a self-supervised robust optimization method in combination with the depth image and proprietary perception information so as to realize efficient control of the quadruped robot; according to the evolutionary strategy simulator optimization method, by combining the ideas of the evolutionary strategy and the generative adversarial network, efficient optimization of simulator parameters is achieved, and the gap between the simulator and the real world is reduced. According to the method, the adaptability and task execution capability of the quadruped robot in a complex environment are remarkably improved.

Description

Technical Field

[0001] This application relates to the field of robot control technology, and particularly to a quadruped robot control method, device and medium. Background Art

[0002] With the continuous development of robot technology, quadruped robots are increasingly widely used in complex environments. However, existing quadruped robots still have deficiencies in aspects such as behavioral diversity, task adaptability, and the gap between the simulator and the real world. Traditional control methods often struggle to achieve efficient adaptation to complex environments and have significant performance differences during the transfer between the simulator and the real world. Summary of the Invention

[0003] This application provides a quadruped robot control method, device and medium, and its advantage is a controller that can efficiently and stably control a quadruped robot to perform specific tasks in a complex environment.

[0004] The technical solution of this application is as follows:

[0005] On the one hand, this application provides a quadruped robot control method, including the following steps:

[0006] (1) Use the basic behavior controller BBC to learn diverse and natural behavior patterns;

[0007] (2) Use the task-specific controller TSC to generate control instructions to achieve efficient control of the quadruped robot;

[0008] (3) Use the evolutionary strategy simulator optimization method to optimize simulator parameters and narrow the gap between the simulator and the real world.

[0009] Further, in the step (1), the BBC uses the semi-supervised information maximization generative adversarial imitation learning method to learn diverse and natural behavior patterns and achieve precise control of the quadruped robot's behavior,

[0010] including the following steps:

[0011] (1.1) Motion capture data preparation: Collect the behavior data of real animals and map it to the target robot skeleton;

[0012] (1.2) Semi-supervised learning of discrete behavior patterns: Maximize the mutual information through the variational lower bound to learn discrete behavior patterns, including walking, pacing, trotting, jogging, and jumping;

[0013] (1.3) Unsupervised learning of continuous behavior styles: Maximize the mutual information between continuous variables and behavior data to capture continuous style changes in behavior;

[0014] (1.4) Unbalanced data optimization: Using discriminant clustering technology and combining with the method of regularized information maximization to extract the intrinsic information from unbalanced data;

[0015] (1.5) Adversarial training and stability enhancement: Introducing a gradient penalty term to optimize the stability of the generative adversarial network.

[0016] Furthermore, in the step (1.1), the preparation of motion capture data is carried out, including the following steps:

[0017] (1.1.1) Collecting the motion capture data of real animal dogs, including five behavior patterns, namely walking, pacing, trotting, jogging and jumping; The motion capture data includes labeled data and unlabeled data. The labeled motion capture data clearly indicates the behavior pattern to which the motion capture data belongs; The unlabeled motion capture data has no annotation of the behavior pattern.

[0018] (1.1.2) Using motion redirection technology to map the original skeletal animation in the motion capture data to the quadruped robot; Defining the centroid and five key points of the quadruped on the original skeleton and mapping them to the robot skeleton, and using inverse kinematics to calculate the joint rotation to obtain the angular positions of each joint of the quadruped robot.

[0019] In the step (1.2), the semi-supervised mutual information is maximized in the following way:

[0020] (1.2.1) Formulating the control problem of the quadruped robot as a discrete-time dynamics problem satisfying the partially observable Markov process: At each time step t, using the policy π to control the robot to execute an action, which makes the robot transfer to the next state; On this basis, using the latent skill variable c to represent the five common animal dog behavior patterns: walking, pacing, trotting, jogging and jumping;

[0021] Maximizing the mutual information between c and the state of the quadruped robot, denoted as I(c; o I ), where o I includes the pitch angle, roll angle, yaw angular velocity, joint angles, joint velocities and foot-end contact states of the quadruped robot; Maximizing the mutual information between the true label and the state of the quadruped robot, denoted as I(y; o I ), where y is the true label of the motion capture data.

[0022] Subsequently, by calculating the variational lower bound and to approximately process the mutual information, the specific form is:

[0023]

[0024] where, Q(·|o I) is an estimate of the true posterior probability P(·|o I ), and the variational lower bound is tight when Q = P; d EL is the state transition distribution of labeled motion capture data; d π is the state transition distribution of the quadruped robot obtained by the policy π; p(y) and p(c) represent the prior probabilities of y and c respectively, which are independent of the policy π; and represent the entropies of y and c respectively;

[0025] Let Q1 = Q2, denoted as Q c , and the final semi-supervised regularization term L SS is obtained:

[0026]

[0027] (1.2.2) In Equation (3), is the supervised term, and Q c is implemented using a neural network. Using d EL to predict y, and optimizing Q c according to the cross-entropy loss function; is the unsupervised term, and Q c is fixed. Learning the latent semantic variables of y according to d π , and using reinforcement learning to optimize the policy π. The output of Q c will be used as the semi-supervised imitation reward r ss , and its form is:

[0028] r SS = logQ c (c|o I )(25)

[0029] In step (1.3) described above, the unsupervised mutual information is maximized in the following way:

[0030] (1.3.1) The motion capture data contains data of five behavioral patterns: walking, pacing, trotting, jogging, and jumping. There are also style variations within the same behavioral pattern; introducing a continuous variable ∈ to capture this internal continuous style variation, and realizing it by maximizing the mutual information I(∈; o I ) between ∈ and the quadruped robot state; using the variational lower bound L US (Q ∈ ) to approximate the mutual information, and the calculation method is:

[0031]

[0032] where, ∈ is sampled from the uniform distribution U(-1, 1), and Q ∈ (∈|o I) is an estimate of the true posterior probability;

[0033] (5) In equation (1.3.2), L US is the unsupervised term, which is used as the objective function to optimize Q ∈ and, at the same time, calculate the unsupervised imitation reward r ∈ according to the output of Q US , and use reinforcement learning to optimize the policy π. The specific form is:

[0034] r US = logQ ∈ (∈|o I ). (27)

[0035] In step (1.4), the imbalanced data is optimized in the following way:

[0036] (1.4.1) Adjust the sampling distribution of the latent variable c to align it with the state transition distribution of the imbalanced data; The specific method is to use Q c to predict the empirical label distribution based on the unlabeled data where N is the total number of unlabeled data, is the predicted label, and ψ is the parameter of Q c ; Then use the empirical label distribution as the sampling distribution of the latent skill variable c;

[0037] (1.4.2) Use the regularized information maximization method to calculate the penalty term L RIM to automatically identify the boundaries between different behavioral patterns in the unlabeled motion capture data. The specific form is:

[0038]

[0039] where the first term is the clustering hypothesis, represents the entropy of; The second term is to avoid degenerate solutions by minimizing the KL divergence between and the distribution p(c) of the latent variable c; The third term R(ψ) is the parameter regularization performed on ψ to avoid complex solutions;

[0040] In step (1.5), adversarial training and stability enhancement are carried out in the following way:

[0041] (1.5.1) The overall framework of the algorithm training is optimized based on the generative adversarial imitation learning method; Minimize a discriminator objective L GAIL similar to that of the least squares generative adversarial network: where D is the discriminator;

[0042] (1.5.2) Introduce a gradient penalty term LGP To improve the training stability, in the specific form of:

[0043]

[0044] where D is the discriminator and φ is the discriminator parameter;

[0045] (1.5.3) Combine all the optimization items mentioned in the above steps to obtain the integrated objective function of the algorithm, which serves as the optimization objective for the discriminator D and the estimator Q c and Q ∈ and is updated by the gradient descent method:

[0046]

[0047] where the first term is a discriminator objective similar to the least squares generative adversarial network, that is

[0048] (1.5.4) After updating the discriminator D and the estimator Q c and Q ∈ , update the policy π by maximizing the total reward through the proximal policy optimization algorithm in reinforcement learning; where the total reward includes the imitation reward r T and the task reward r T :

[0049] r = w I r I + w T r T (31)

[0050] The imitation reward includes the discriminator reward, the semi-supervised imitation reward, and the unsupervised imitation reward:

[0051] r I = r D + r SS + r US (32)

[0052] where the first term r D is the discriminator reward. For the samples generated by the policy of the quadruped robot, the discriminator objective is to predict its score as -1; while for the demonstration samples in the motion capture data, the objective is to predict its score as 1. The specific reward formula is r D = max[0, 1 - 0.25(D(o I )) - 1) 2 , and the semi-supervised imitation reward r SS and the unsupervised imitation reward r US are given by equations (4) and (6) respectively;

[0053] The task reward includes the linear velocity tracking reward Angular velocity tracking reward Jump height reward And stable height reward The specific form is as follows:

[0054]

[0055]

[0056] Among them, And v xy Represent the command and the actual linear velocity; And ω z Represent the command and the actual angular velocity; h cmd And h represent the command and the actual center height, Is the indicator function;

[0057] (1.5.5) Continuously repeat the adversarial training process, and cyclically update the discriminator, estimator, and policy until convergence; finally, obtain the policy π that generates five behavioral pattern actions, realizing the imitation of the natural behavioral patterns of real animals, dogs, by the quadruped robot.

[0058] Furthermore, in the step (2), a task-specific controller TSC is used to generate control commands to achieve efficient control of the quadruped robot. The TSC generates control commands to achieve efficient control of the quadruped robot by means of privileged learning and self-supervised robust optimization methods, combined with depth images and proprietary perception information.

[0059] It includes the following steps:

[0060] (2.1) Sample the terrain height around the quadruped robot;

[0061] (2.2) The teacher policy is trained based on privileged information;

[0062] (2.3) Enhance the robustness of the depth image through self-supervised robust optimization methods;

[0063] (2.4) The student policy is trained to imitate the teacher policy based on historical state privileged information.

[0064] Furthermore, the training of the teacher policy based on privileged information in the step (2.2) includes the following steps:

[0066] (2.2.1) Use the privileged learning architecture to divide the algorithm into two stages: teacher policy training and student policy training;

[0067] During the teacher policy training process, collect data including privileged information as the input of the teacher policy network π TSC Specifically including the following content:

[0068] (a) External perception information: The surrounding terrain elevation map obtained by sampling;

[0069] (b) Privilege information: The type of obstacle the quadruped robot is currently facing, the yaw angle error Δ yaw with the current target point, and the yaw angle error Δ′ yaw ;

[0070] (c) Proprioceptive information: including the pitch angle, roll angle, yaw angular velocity, joint angles, joint velocities, and foot-end contact states of the quadruped robot;

[0071] (2.2.2) Use the hybrid proximal policy optimization algorithm in reinforcement learning to implement the training of the hybrid action space policy; The teacher policy hybrid action space a T represents that the action output by the teacher policy is a mixture of discrete and continuous variable commands: a T = {a D , a C}, where the teacher policy discrete command a D represents the behavior pattern command; The teacher policy continuous command a C represents the speed command, root height command, and action style command;

[0072] The hybrid proximal policy optimization algorithm needs to maximize the reward r in the following form TSC :

[0073] r TSC = w I r I + w TSC r TSC (37)

[0074] In Equation (1), w I and w TSC are the coefficients of the imitation reward and the task reward respectively; r I is the imitation reward calculated by the discriminator network pre-trained, and the calculation formula is r I = max[0, 1 - 0.25(D(o I )) - 1) 2 , where D is the discriminator network, o I is the proprioceptive information of the quadruped robot, and o I is used as the input of D. If the action of the quadruped robot is closer to the action of the natural animal dog, then the output D(o I ) of D will be close to 1, otherwise close to -1; The task reward r TSC includes the linear velocity tracking reward the yaw angle tracking reward the waypoint arrival reward and termination reward The formula is as follows:

[0075]

[0076] where v is the linear velocity, ν target is the target linear velocity, d wpt is the direction of the target path point, and θ z are the yaw angles of the target path point and the quadruped robot respectively, p wpt and p represent the positions of the target path point and the quadruped robot respectively, is the indicator function;

[0077] The trained teacher policy is combined with the underlying controller with the assistance of privileged information;

[0078] In step (2.3), the robustness of the depth image is enhanced through the following steps by a self-supervised robust optimization method:

[0079] (2.3.1) Perform probability enhancement processing on the depth image collected by the binocular depth camera on the quadruped robot, including white noise, background noise, random cropping, edge noise, and Gaussian blur;

[0080] (2.3.2) Use the self-supervised learning method BYOL to maximize the similarity between two enhanced views of the same depth image to learn task-related features, and integrate this self-supervised loss into the depth image encoder network;

[0081] In step (2.4), the student policy is trained to imitate the teacher policy based on historical state privileged information, including the following steps:

[0082] (2.4.1) When training the student policy network , use the robustified depth image to replace the surrounding terrain elevation map, and at the same time collect the historical sequence of proprioceptive information instead of the difficult-to-obtain privileged information as the input;

[0083] (2.4.2) Use the gated recurrent unit to process the input sequence and output the predicted privileged information and the latent vector of environmental information, and output the final mixed student action through the gated recurrent unit and the multi-layer perceptron where is the student policy mixed action space, is the student policy discrete command, is the student policy continuous command; Training the student policy uses supervised learning to imitate the teacher policy, and the minimized objective function is in the following form:

[0084]

[0085] where K is the number of discrete actions; the first term of the objective function corresponds to the cross-entropy loss between the teacher policy and the discrete commands of the student policy, while the second term is used to capture the mean squared error between the teacher policy and the continuous commands of the student policy;

[0086] The trained student policy is combined with the underlying controller.

[0087] Furthermore, in step (3), the evolutionary strategy simulator optimization method is used to optimize the simulator parameters, reducing the gap between the simulator and the real world, including the following steps:

[0088] (3.1) Parameter initialization: Initialize the parameter distribution of the simulator;

[0089] (3.1) Parameter initialization: Initialize the parameter distribution of the emulator;

[0090] (3.2) Real data collection: Collect state-action trajectory data in the real environment;

[0091] (3.3) Simulation data collection: Collect state-action trajectory data in the simulation environment;

[0092] (3.4) Adversarial training: Use a discriminator to distinguish between the real world and the state-action transitions of the simulator, and at the same time optimize the parameter distribution through evolutionary strategies to maximize the discriminator's score for real samples;

[0093] (3.5) Parameter update: Update the simulator according to the optimized parameter distribution;

[0094] (3.6) Iterative optimization: Repeat simulation data collection, adversarial training, and parameter update until the parameter distribution converges.

[0095] (3.7) Policy optimization: Retrain or fine-tune the control policy using the optimized simulator.

[0096] Furthermore, the parameter initialization in step (3.1) includes the following:

[0097] (3.1.1) Define the initial range of the simulator parameters;

[0098] (3.1.2) Randomly sample the initial parameters;

[0099] The data collection in step (3.2) includes the following steps:

[0100] (3.2.1) In the real environment, use a predefined controller to collect state-action trajectories;

[0101] (3.2.2) Use the collected data for adversarial training;

[0102] The data acquisition in the step (3.3) includes the following steps:

[0103] (3.3.1) In the simulation environment, use the same controller as in step (1.1) to collect state-action trajectories;

[0104] (3.3.2) Use the collected data for adversarial training;

[0105] The adversarial training in the step (3.4) includes the following:

[0106] (3.4.1) The optimization objective of the discriminator is to distinguish the state-action transitions between the real world and the emulator, in the specific form of:

[0107]

[0108] where d B (s, a, s′) is the simulation data, and d M (s, a, s′) is the real data;

[0109] (3.4.2) The optimization objective of the evolutionary strategy is to optimize the physical parameters of the emulator, so as to maximize the score of the discriminator for the emulator samples;

[0110] The parameter update in the step (3.5) includes the following steps:

[0111] (3..5.1) Update the physical parameters of the emulator according to the optimized parameter distribution.

[0112] The iterative optimization in the step (3.6) includes the following:

[0113] (3.6.1) Repeat data acquisition, adversarial training, and parameter update until the discriminator can no longer distinguish between real and simulation data;

[0114] (3.6.2) Verify whether the optimized emulator meets the preset convergence conditions, such as the change amplitude of the parameter distribution is less than the threshold;

[0115] The iterative optimization in the step (3.7) includes the following:

[0116] (3.7.1) Use the optimized physical parameters of the emulator as the parameter range for domain randomization, retrain the control strategy, or fine-tune for the same controller as in (3.2.1), so that the strategy adapts to a simulation closer to the real environment.

[0117] On the other hand, the present application provides a quadruped robot control device, including a memory and a processor. When the computer program stored in the memory is called and executed by the processor, the above-mentioned method is implemented.

[0118] On the other hand, the present application provides a computer-readable medium storing a computer program, which when executed by a computer, implements the method described above.

[0119] In summary, the beneficial effects of the present application are as follows: (1) Through the learning of BBC, the quadruped robot can exhibit diverse and natural behavior patterns, including discrete behavior categories and continuous behavior style changes. (2) Through the control of TSC, the quadruped robot can efficiently and stably execute specific tasks and adapt to different task requirements in complex environments. (3) Through the optimization of the simulation optimization method, the simulator parameters are closer to the real world, improving the performance and robustness of the robot in the real environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0120] Figure 1 Schematic diagram for training the integrated control system of the quadruped robot;

[0121] Figure 2 Flowchart of the method of BBC;

[0122] Figure 3 Schematic diagram for processing motion capture data using the motion remapping technique;

[0123] Figure 4 Schematic diagram for the quadruped robot to imitate five behavior patterns of a real animal dog;

[0124] Figure 5 Flowchart of the method of TSC;

[0125] Figure 6 Schematic diagram of the terrain elevation map, privilege information, and depth image;

[0126] Figure 7 Comparison schematic diagram for robust optimization of the depth image;

[0127] Figure 8 Flowchart framework of the simulator optimization process. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0128] The following describes the detailed embodiments of the present application with reference to the accompanying drawings.

[0129] As Figure 1 shown, an integrated quadruped robot leg controller includes the following steps:

[0130] (1) Using a basic behavior controller (BBC) to learn diverse and natural behavior patterns.

[0131] (2) Using a task-specific controller (TSC) to generate control instructions to achieve efficient control of the quadruped robot.

[0132] (3) Optimize the simulator parameters using the evolutionary strategy simulator optimization method to narrow the gap between the simulator and the real world.

[0133] In the above step (1), as Figure 2 shown, a basic behavior controller (BBC) for a quadruped robot includes the following steps:

[0134] Step 1: Motion capture data preparation: Collect the behavior data of real animals and map it onto the target robot skeleton;

[0135] (1.1) Collect the motion capture data of real animal dogs, including five behavior patterns: walking, pacing, trotting, jogging, and jumping. Among them, only a small part of the motion capture data is labeled, that is, the behavior pattern to which this piece of motion capture data belongs is clearly marked; while the vast majority of the motion capture data is unlabeled and there is no annotation of the behavior pattern.

[0136] (1.2) Use motion redirection technology to map the original skeletal animation in the motion capture data onto the quadruped robot. As Figure 3 shown, define the center of mass and five key points of the four legs on the original skeleton, map them onto the robot skeleton, and use inverse kinematics to calculate the joint rotation to obtain the angular positions of the joints of the quadruped robot.

[0137] Step 2: Semi-supervised learning of discrete behavior patterns: Maximize the mutual information through the variational lower bound to learn the five discrete behavior patterns as Figure 4 shown, including walking, pacing, trotting, jogging, and jumping; specifically including the following steps:

[0138] (2.1) Since the annotation cost of motion capture data is relatively high, it is not appropriate to use the method of supervised learning to obtain a large amount of labeled data. Also, because there are multiple behavior patterns in the motion capture data, it is very difficult to directly use the unsupervised method to extract behavior features from the original data. Therefore, choose to use the semi-supervised learning method, and on the basis of using a small amount of labeled data to guide the disentanglement of multi-mode behaviors, make full use of a large amount of unlabeled data for learning.

[0139] Specifically, formulate the control problem of the quadruped robot as a discrete-time dynamics problem that satisfies the partially observable Markov process: at each time step t, use the policy π to control the robot to execute an action, and this action makes the robot transfer to the next state. On this basis, use the latent skill variable c to represent the five common animal dog behavior patterns: walking, pacing, trotting, jogging, and jumping, as Figure 5 shown. By changing the latent skill variable c of the input policy π, the quadruped robot can show corresponding different behavior patterns.

[0140] To ensure that c can guide the policy π to generate different actions, it is necessary to maximize the mutual information between c and the state of the quadruped robot, denoted as I(c; o I ), where o I includes the pitch angle, roll angle, yaw angular velocity, joint angles, joint velocities, and foot contact states of the quadruped robot. On the other hand, when using a small amount of labeled motion capture data for supervised learning, it is necessary to maximize the mutual information between the true label and the state of the quadruped robot, denoted as I(y; o I ), where y is the true label of the motion capture data.

[0141] Subsequently, the variational lower bounds and are calculated to approximate the mutual information, and the specific form is:

[0142]

[0143] where Q(·|o I ) is an estimate of the true posterior probability P(·|o I ), and the variational lower bound is tight when Q = P; d EL is the state transition distribution of the labeled motion capture data; d π is the state transition distribution of the quadruped robot obtained by the policy π; p(y) and p(c) represent the prior probabilities of y and c respectively, which are independent of the policy π; and represent the entropies of y and c respectively.

[0144] In practice, let Q1 = Q2, denoted as Q c , and the final semi-supervised regularization term L SS is obtained:

[0145]

[0146] (2.2) In equation (3), is the supervised term, and Q c is implemented using a neural network. Using d EL to predict y, and optimizing Q c according to the cross-entropy loss function; is the unsupervised term, and Q c is fixed. Learning the latent semantic variables of y according to d π , and using reinforcement learning to optimize the policy π. The output of Q c will be used as the semi-supervised imitation reward r ss , and its form is:

[0147] r SS = logQ c (c|o I)(46)

[0148] Step 3, Unsupervised learning of continuous behavior style: Capture the continuous style changes in behavior by maximizing the mutual information between continuous variables and behavior data. Specifically, it includes the following steps:

[0149] (3.1) The motion capture data contains data of five behavior patterns: walking, pacing, trotting, jogging, and jumping. However, there are also style changes within the same behavior pattern. Therefore, it is necessary to introduce a continuous variable ∈ to capture this internal continuous style change by maximizing the mutual information I(∈; o I ) between ∈ and the state of the quadruped robot. Use the variational lower bound L US to approximate the mutual information, and the calculation method is:

[0150]

[0151] where ∈ is sampled from the uniform distribution U(-1, 1), and Q ∈ (∈|o I ) is an estimate of the true posterior probability, implemented using a neural network.

[0152] (3.2) Since there are no style labels within the same behavior pattern in the motion capture data, the L in equation (5) US is an unsupervised term and is used as the objective function to optimize Q ∈ . At the same time, calculate the unsupervised imitation reward r ∈ according to the output of Q US , and use reinforcement learning to optimize the policy π. The specific form is:

[0153] r US = logQ ∈ (∈|o I )(48)

[0154] Step 4, Unbalanced data optimization: Adopt discriminant clustering technology and combine the method of maximizing regularization information to extract intrinsic information from unbalanced data. Specifically, it includes the following steps:

[0155] (4.1) Since in the motion capture data, the proportions of data of different behavior patterns are different. When performing semi-supervised learning of behavior patterns, it is necessary to adjust the sampling distribution of the latent variable c to align it with the state transition distribution of the unbalanced data. The specific method is to use Q c to predict the empirical label distribution based on unlabeled data where N is the total number of unlabeled data, is the predicted label, and ψ is the parameter of Q c . Then use the empirical label distribution as the sampling distribution of the latent skill variable c.

[0156] (4.2) To better utilize the unlabeled motion capture data, the regularized information maximization method is selected to calculate the penalty term L RIM , automatically identify the boundaries between various behavior patterns in the unlabeled motion capture data, and the specific form is:

[0157]

[0158] Among them, the first term is the clustering hypothesis, representing the entropy of. The second term is to avoid the degenerate solution by minimizing the KL divergence between and the distribution p(c) of the latent variable c. The third term R(ψ) is the parameter regularization performed on ψ to avoid complex solutions.

[0159] Step 5, Adversarial Training and Stability Enhancement: Introduce a gradient penalty term to optimize the stability of the generative adversarial network and make the training process more robust; specifically, it includes the following steps:

[0160] (5.1) The overall framework of the algorithm training is optimized based on the generative adversarial imitation learning method. To obtain more stable training and higher-quality results, it is necessary to minimize a discriminator objective L similar to the least-squares generative adversarial network GAIL : where D is the discriminator.

[0161] (5.2) Due to the function approximation error in the discriminator, the adversarial imitation learning method often has instability. The discriminator will assign non-zero gradients to actual data samples, which may lead to overshooting and oscillation of the generator. Therefore, a gradient penalty term L GP is introduced to improve the training stability, and the specific form is:

[0162]

[0163] where D is the discriminator and φ is the discriminator parameter;

[0164] (5.3) Combine all the optimization terms mentioned in the above steps to obtain the integrated objective function of the algorithm, which is used as the optimization objective of the discriminator D and the estimator Q c and Q ∈ and is updated by the gradient descent method:

[0165]

[0166] Among them, the first term is a discriminator objective similar to the least-squares generative adversarial network, that is

[0167] (5.4) When updating the discriminator D and the estimator Q c and Q∈ After that, the proximal policy optimization algorithm in reinforcement learning is used to maximize the total reward to update the policy π. The total reward includes the imitation reward r T and the task reward r T :

[0168] r = w I r I + w T r T (52)

[0169] The imitation reward includes the discriminator reward, semi-supervised imitation reward, and unsupervised imitation reward:

[0170] r I = r D + r SS + r US (53)

[0171] Among them, the first term r D is the discriminator reward. For the samples generated by the policy of the quadruped robot, the discriminator's goal is to predict its score as -1; while for the demonstration samples in the motion capture data, the goal is to predict its score as 1. The specific reward formula is r D = max[0, 1 - 0.25(D(o I ) - 1) 2 . The semi-supervised imitation reward r SS and the unsupervised imitation reward r US are given by equations (4) and (6) respectively.

[0172] The task reward includes the linear velocity tracking reward angular velocity tracking reward jump height reward and stable height reward The specific forms are as follows:

[0173]

[0174] Among them, and v xy represent the command and the actual linear velocity; and ω z represent the command and the actual angular velocity; h cmd and h represent the command and the actual central height, is the indicator function.

[0175] (5.5) Continuously repeat the process of adversarial training, and cycle through updating the discriminator, estimator, and policy until convergence. Finally, the policy π that generates five behavioral pattern actions is obtained, realizing the imitation of the natural behavioral patterns of the real animal dog by the quadruped robot.

[0176] In step (2), as Figure 5 shown, a task-specific controller (TSC) for a quadruped robot includes the following steps:

[0177] Step 1: Sample the height of the terrain around the quadruped robot. As Figure 6 shown, each orange point around the quadruped robot represents the location of that point as a sampling point for the height of the surrounding terrain, which will be updated in real time as the robot moves, helping the robot continuously obtain the elevation map of the surrounding terrain.

[0178] Step 2: The teacher policy is trained based on privileged information, including:

[0179] (2.1) A complete basic behavior controller has been deployed on the quadruped robot as the underlying controller, which can control the robot to perform different actions after receiving the corresponding command inputs. The required commands include behavior mode commands, speed commands, root height commands, and action style commands. The behavior mode commands are discrete and can enable the quadruped robot to select one of the five behavior modes of walking, pacing, trotting, jogging, and jumping; the speed commands are used to control the forward linear speed and yaw angular speed of the quadruped robot; the root height commands are used to control the body height of the robot; the action style commands are used to control the stride amplitude and frequency of the robot.

[0180] On this basis, a task-specific controller is required as the top-level policy to output the commands required by the underlying controller to control the quadruped robot to perform corresponding actions to complete specific tasks. To solve the problem of sparse reward signals in complex scenarios of specific tasks, the present invention uses a privileged learning architecture, and divides the algorithm into two stages: teacher policy training and student policy training.

[0181] During the teacher policy training process, it is necessary to collect data including privileged information in the simulator as the input of the teacher policy network π TSC as follows:

[0182] (a) External perception information: according to the sampled elevation map of the surrounding terrain;

[0183] (b) Privileged information: the type of obstacle currently faced by the quadruped robot, the yaw angle error Δ yaw with the current target point, and the yaw angle error Δ′ yaw with the next target point; as Figure 6 shown, the blue point represents the current target point that the quadruped robot needs to reach, and the red point after the blue point represents the next target point that needs to be reached. At the same time, the type of obstacle to which the robot currently belongs will also be provided in real time;

[0184] (c) Proprioceptive information: including the pitch angle, roll angle, yaw angular velocity, joint angles, joint velocities, and foot contact states of the quadruped robot.

[0185] It is difficult to directly train in the simulation environment according to the real - world scenario conditions due to problems such as sparse reward signals. Although the surrounding terrain elevation map and privileged information are difficult to obtain in reality, they can be obtained in real - time in the simulation environment. By using this information additionally in the teacher - policy training, the speed and effect of policy training will be improved.

[0186] (2.2) Use the hybrid proximal policy optimization algorithm in reinforcement learning to implement the hybrid action - space policy training. The teacher - policy hybrid action - space a T represents that the output action of the teacher policy is a mixture of discrete and continuous variable commands: a T ={a D , a C}, where the teacher - policy discrete command a D represents the behavior - pattern command; the teacher - policy continuous command a C represents the speed command, root - height command, and action - style command.

[0187] The hybrid proximal policy optimization algorithm needs to maximize the reward r in the following form TSC :

[0188] r TSC =w I r I +w TSC r TSC (58)

[0189] In Equation (1), w I and w TSC are the coefficients of the imitation reward and the task reward respectively. r I is the imitation reward calculated by the discriminator network pre - trained, and the calculation formula is r I =max[0,1 - 0.25(D(o I )) - 1) 2 , where D is the discriminator network, o I is the proprioceptive information of the quadruped robot, and o I is used as the input of D. If the action of the quadruped robot is closer to the action of the natural animal dog, then the output D(o I ) of D will be close to 1, otherwise close to - 1. The task reward r TSC includes the linear - velocity tracking reward yaw - angle tracking reward way - point arrival reward and termination reward The formulas are as follows:

[0190]

[0191]

[0192] where v is the linear velocity, and v target is the target linear velocity, and d wpt is the direction of the target waypoint, and θ z are the yaw angles of the target waypoint and the quadruped robot respectively, and p wpt and p represent the positions of the target waypoint and the quadruped robot respectively, is the indicator function.

[0193] The trained teacher policy, combined with the underlying controller with the assistance of privileged information, can manipulate the quadruped robot in the simulation environment to complete different task requirements.

[0194] In step (3), the robustness of the depth image is enhanced through the following steps by the self-supervised robust optimization method:

[0195] (3.1) The depth image is the external perception information commonly used when the quadruped robot is deployed in real-world scenario tasks. The depth images in the real environment usually contain noises from various sources, making them different from the images in the simulation environment. To improve the robustness of the algorithm to the depth image, probability enhancement processing is performed on the depth images collected by the binocular depth camera on the quadruped robot, such as Figure 3 shown, including white noise, background noise, random cropping, edge noise, and Gaussian blur.

[0196] (3.2) Using the self-supervised learning method Bootstrap Your Own Latent (BYOL), maximize the similarity between two augmented views of the same depth image to learn task-related features, such as Figure 7 shown, integrate this self-supervised loss into the depth image encoder network to enhance the robustness of the top-level policy to complex real-world environments.

[0197] In step (4), the student policy is trained to imitate the teacher policy based on historical state privileged information, including the following steps:

[0198] (4.1) To obtain the top-level policy of the quadruped robot that can be deployed in the real-world scenario, when training the student policy network , use the robustified depth image to replace the surrounding terrain elevation map, and at the same time collect the historical sequence of proprioceptive information instead of the difficult-to-obtain privileged information as the input.

[0199] (4.2) Process the input sequence using a gated recurrent unit and output the predicted privileged information and the latent vector of environmental information, and output the final hybrid student action through the gated recurrent unit and the multi-layer perceptron wherein is the student policy hybrid action space, is the student policy discrete command, is the student policy continuous command. Training the student policy uses supervised learning to imitate the teacher policy, and the form of the objective function to be minimized is as follows:

[0200]

[0201] where K is the number of discrete actions. The first term of the objective function corresponds to the cross-entropy loss between the teacher policy and the student policy discrete command, while the second term is used to capture the mean square error between the teacher policy and the student policy continuous command.

[0202] The trained student policy combined with the underlying controller enables the quadruped robot to successfully execute a variety of complex tasks in the real environment.

[0203] In the said step (three), as Figure 8 shown, a simulator optimization method based on evolutionary strategy and generative adversarial imitation learning includes the following steps:

[0204] Step 1, parameter initialization: Initialize the parameter distribution of the simulator;

[0205] (1.1) Define the initial range of the simulator parameters;

[0206] (1.2) Randomly sample the initial parameter distribution.

[0207] Step 2, real data collection: Collect state-action trajectory data in the simulation environment and the real environment respectively; specifically including the following steps:

[0208] (2.1) In the real environment, use a predefined controller to generate state-action trajectories;

[0209] (2.2) Use the collected data for adversarial training.

[0210] Step 3, simulation data collection: Collect state-action trajectory data in the simulation environment and the real environment respectively; specifically including the following steps:

[0211] (3.1) In the simulation environment, use the same controller as in step (2.1) to generate state-action trajectories;

[0212] (3.2) Use the collected data for adversarial training.

[0213] Step 4. Adversarial training: The discriminator is used to distinguish the state-action transitions between the real world and the emulator. Meanwhile, the parameter distribution is optimized by Evolutionary Strategies (ES) to maximize the discriminator's score for real samples. The specific steps are as follows:

[0214] (4.1) The optimization objective of the discriminator D is to distinguish the state-action transitions between the real world and the emulator, and the specific form is:

[0215]

[0216] where d B (s,a,s′) is simulation data, and d M (s,a,s′) is real data;

[0217] (4.2) The optimization objective of Evolutionary Strategies (ES) is to maximize the discriminator's score for real samples

[0218] Step 5. Parameter update: Update the emulator according to the optimized parameter distribution. The specific steps are as follows: The specific steps are as follows:

[0219] (5.1) Update the physical parameters of the emulator according to the optimized parameter distribution;

[0220] Step 6. Iterative optimization: Repeat data collection, adversarial training, and parameter update until the parameter distribution converges. The specific steps are as follows:

[0221] (6.1) Repeat data collection and adversarial training until the discriminator can no longer distinguish real and simulation data;

[0222] (6.2) Verify whether the optimized emulator meets the preset convergence conditions, such as the change range of the parameter distribution is less than the threshold.

[0223] Step 7. Policy optimization: Retrain or fine-tune the control policy using the optimized emulator. The specific steps are as follows:

[0224] (7.1) Use the physical parameters of the optimized emulator as the parameter range for domain randomization to retrain the control policy or fine-tune the same controller as in (2.1) to make the policy adapt to a simulation closer to the real environment.

[0225] The above are only the preferred embodiments of the present application. It should be noted that for those of ordinary skill in the art, without departing from the creative concept of the present application, several modifications and improvements can be made, and these all belong to the protection scope of the present application.

Claims

1. A control method for a quadruped robot, characterized in that It includes the following steps: (1) Use the Basic Behavior Controller (BBC) to learn diverse and natural behavior patterns; (2) Use the Task-Specific Controller (TSC) to generate control instructions for efficient control of the quadruped robot; (3) Use the evolutionary strategy simulator optimization method to optimize the simulator parameters and narrow the gap between the simulator and the real world.

2. The quadruped robot control method according to claim 1, wherein, In step (1), the BBC uses the semi-supervised information maximization generative adversarial imitation learning method to learn diverse and natural behavior patterns and achieve precise control of the quadruped robot's behavior, which includes the following steps: (1.1) Motion capture data preparation: Collect the behavior data of real animals and map it to the target robot's skeleton; (1.2) Semi-supervised learning of discrete behavior patterns: Maximize the mutual information through the variational lower bound to learn discrete behavior patterns, including walking, pacing, trotting, jogging, and jumping; (1.3) Unsupervised learning of continuous behavior styles: Maximize the mutual information between the continuous variable and the behavior data to capture the continuous style changes in the behavior; (1.4) Imbalanced data optimization: Adopt discriminative clustering techniques and combine the regularized information maximization method to extract the intrinsic information from imbalanced data; (1.5) Adversarial training and stability enhancement: Introduce a gradient penalty term to optimize the stability of the generative adversarial network.

3. The quadruped robot control method according to claim 1, characterized in that, In step (1.1) for motion capture data preparation, it includes the following steps: (1.1.1) Collect the motion capture data of real animal dogs, including five behavior patterns, namely walking, pacing, trotting, jogging, and jumping; The motion capture data includes labeled data and unlabeled data. The labeled motion capture data clearly indicates the behavior pattern to which the motion capture data belongs; the unlabeled motion capture data has no annotation of the behavior pattern to which it belongs; (1.1.2) Use the motion redirection technology to map the original skeletal animation in the motion capture data to the quadruped robot; define the centroid and five key points of the quadruped on the original skeleton and map them to the robot's skeleton, and use inverse kinematics to calculate the joint rotation to obtain the angular positions of each joint of the quadruped robot; In step (1.2), the semi-supervised mutual information is maximized in the following way: (1.2.1) Formulate the control problem of the quadruped robot as a discrete-time dynamics problem that satisfies the partially observable Markov process: At each time step t, use the policy π to control the robot to execute an action, which makes the robot transfer to the next state; on this basis, use the latent skill variable c to represent five common animal dog behavior patterns: walking, pacing, trotting, jogging, and jumping; Maximize the mutual information between c and the state of the quadruped robot, denoted as I(c; o I ), where o I includes the pitch angle, roll angle, yaw angular velocity, joint angles, joint velocities, and foot contact states of the quadruped robot; Maximize the mutual information between the true label and the state of the quadruped robot, denoted as I(y; o I ), where y is the true label of the motion capture data; Subsequently, the variational lower bound is calculated and to approximate the mutual information, and the specific form is as follows: Among them, Q(·|o I ) is an estimate of the true posterior probability P(·|o I ), and the variational lower bound is tight when Q = P; d EL is the state transition distribution of the labeled motion capture data; d π is the state transition distribution of the quadruped robot obtained by the policy π; p(y) and p(c) represent the prior probabilities of y and c respectively, which are independent of the policy π; and represent the entropies of y and c respectively; Let Q1 = Q2, denoted as Q c , and the final semi-supervised regularization term L SS is obtained: (1.2.2) In formula (3), is the supervised item, Q c is implemented using a neural network. Using d EL to predict y, and optimizing Q according to the cross-entropy loss function c ; is the unsupervised item, Q c is fixed. According to d π to learn the latent semantic variables of y, using reinforcement learning to optimize the policy π, and the output of Q c will be used as the semi-supervised imitation reward r ss , and its form is: r SS = logQ c (c|o I )(4) In step (1.3), the unsupervised mutual information is maximized in the following way: (1.3.1) The motion capture data contains data of five behavioral patterns: walking, pacing, trotting, jogging, and jumping. There are also style variations within the same behavioral pattern. A continuous variable ∈ is introduced to capture this internal continuous style variation, and it is achieved by maximizing the mutual information I(∈; o I ) between ∈ and the state of the quadruped robot; the variational lower bound L US (Q ∈ ) is used to approximate the mutual information, and the calculation method is as follows: where ∈ is sampled from a uniform distribution U(-1, 1), and Q ∈ (∈|o I ) is an estimate of the true posterior probability; In formula (5) of (1.3.2), L US is the unsupervised term, which is used as the objective function to optimize Q ∈ and simultaneously calculates the unsupervised imitation reward r according to the output of Q ∈ . The reinforcement learning is used to optimize the policy π, and its specific form is as follows: US ​ r US = logQ ∈ (∈|o I )。(6) In step (1.4), the imbalanced data is optimized in the following way: (1.4.1) Adjust the sampling distribution of the latent variable c to align it with the state transition distribution of the imbalanced data; specifically, use Q c Predict the empirical label distribution based on the unlabeled data where N is the total number of unlabeled data, is the predicted label, and ψ is the parameter of Q c ; then use the empirical label distribution as the sampling distribution of the latent skill variable c; (1.4.2) Calculate the penalty term L using the regular information maximization method RIM , and automatically identify the boundaries between various behavioral patterns in the unlabeled motion capture data, in the specific form of: Among them, the first item is the clustering hypothesis, representing the entropy of; the second item is to avoid degenerate solutions by minimizing the KL divergence between and the distribution p(c) of the latent variable c; the third item R(ψ) is the parameter regularization performed on ψ to avoid complex solutions; In step (1.5), the adversarial training and stability enhancement are carried out in the following way: (1.5.1) The overall framework of algorithm training is optimized based on the generative adversarial imitation learning method; minimize a discriminator objective L similar to the least squares generative adversarial network GAIL : where D is the discriminator; (1.5.2) introduced the gradient penalty term L GP to improve training stability, and its specific form is: Among them, D is the discriminator, and φ is the discriminator parameter; (1.5.3) Combine all the optimization items mentioned in the above steps to obtain the integrated objective function of the algorithm, which serves as the optimization objectives for the discriminator D and the estimator Q c and Q ∈ and update them through the gradient descent method: Among them, the first item is a discriminator objective similar to the least squares generative adversarial network, that is (1.5.4) When updating the discriminator D and the estimator Q c and Q ∈ After that, the policy π is updated by maximizing the total reward through the proximal policy optimization algorithm in reinforcement learning; where the total reward includes the imitation reward r T and the task reward r T : r = w I r I + w T r T (10) The imitation rewards include discriminator rewards, semi-supervised imitation rewards, and unsupervised imitation rewards: r I ＝r D +r SS +r US (11) Among them, the first item r D is the discriminator reward. For the samples generated by the quadruped robot through the policy, the discriminator's goal is to predict its score as -1; while for the demonstration samples in the motion capture data, the goal is to predict its score as 1. The specific reward formula is r D = max[0, 1 - 0.25(D(o I ) - 1) 2 . The semi-supervised imitation reward r SS and the unsupervised imitation reward r US are given by equations (4) and (6) respectively; The task rewards include a linear speed tracking reward an angular velocity tracking reward a jump height reward and a stable height reward The specific forms are as follows: Among them, and v xy represent the commanded and actual linear velocity; and ω z represent the commanded and actual angular velocity; h cmd and h represent the commanded and actual center height, is the indicator function; (1.5.5) Continuously repeat the process of adversarial training, and cyclically update the discriminator, estimator, and policy until convergence; finally, obtain the policy π for generating five behavioral pattern actions, realizing the imitation of the natural behavioral patterns of real animals, dogs, by the quadruped robot.

4. The quadruped robot control method according to claim 1, characterized in that In step (2), a task-specific controller TSC is used to generate control commands to achieve efficient control of the quadruped robot. The TSC generates control commands to achieve efficient control of the quadruped robot through privileged learning and self-supervised robust optimization methods, combined with depth images and proprietary perception information, including the following steps: (2.1) Sample the terrain height around the quadruped robot. (2.2) The teacher policy is trained based on privileged information. (2.3) Enhance the robustness of the depth image through self-supervised robust optimization methods. (2.4) The student policy is trained by imitating the teacher policy based on historical state privileged information.

5. The quadruped robot control method according to claim 4, characterized in that The training of the teacher policy based on privileged information in step (2.2) includes the following steps: (2.2.1) Use a privileged learning architecture to divide the algorithm into two stages: teacher policy training and student policy training. During the teacher strategy training process, data including privileged information, etc. is collected as the input of the teacher strategy network π TSC and specifically includes the following content: (a) External perception information: According to the sampled surrounding terrain elevation map. (b) Privileged information: the type of obstacle the quadruped robot is currently facing, the yaw angle error Δ yaw with the current target point, and the yaw angle error Δ yaw with the next target point; (c) Proprioceptive information: including the pitch angle, roll angle, yaw angular velocity, joint angles, joint velocities, and foot-end contact states of the quadruped robot. (2.2.2) Implement the training of the hybrid action space policy using the Hybrid Proximal Policy Optimization algorithm in reinforcement learning; the teacher policy hybrid action space a T represents that the output action of the teacher policy is a mixture of discrete and continuous variable commands: a T = {a D , a C}, where the discrete command a D of the teacher policy represents the behavior pattern command; the continuous command a C of the teacher policy represents the speed command, the root height command, and the action style command; The Hybrid Proximal Policy Optimization algorithm needs to maximize the reward r in the following form TSC : In Equation (1), w I and w TSC are the coefficients of the imitation reward and the task reward respectively; r I is the imitation reward calculated by the discriminator network pre-trained, and the calculation formula is r I = max[0, 1 - 0.25(D(o I )) - 1) 2 , where D is the discriminator network, o I is the proprioceptive information of the quadruped robot, and o I is used as the input of D. If the actions of the quadruped robot are closer to those of the natural animal dog, then the output D(o I ) of D will be close to 1, otherwise close to -1; the task reward r TSC includes the linear velocity tracking reward the yaw angle tracking reward the waypoint arrival reward and the termination reward The formulas are as follows: where v is the linear velocity, v target is the target linear velocity, d wpt is the direction of the target path point, and θ z are the yaw angles of the target path point and the quadruped robot respectively, p wpt and p represent the positions of the target path point and the quadruped robot respectively, is the indicator function; The trained teacher policy is combined with the underlying controller with the assistance of privileged information. In step (2.3), the following steps are used to enhance the robustness of the depth image through self-supervised robust optimization methods: (2.3.1) Perform probability enhancement processing on the depth images collected by the binocular depth camera on the quadruped robot, including white noise, background noise, random cropping, edge noise, and Gaussian blur. (2.3.2) Use the self-supervised learning method BYOL to maximize the similarity between two enhanced views of the same depth image to learn task-related features, and integrate this self-supervised loss into the depth image encoder network. The training of the student policy by imitating the teacher policy based on historical state privileged information in step (2.4) includes the following steps: (2.4.1) When training the student policy network use the robustified depth image to replace the surrounding terrain elevation map, and collect the historical sequence of proprioceptive information instead of the difficult-to-obtain privileged information as the input; (2.4.2) Process the input sequence using a gated recurrent unit and output the predicted privileged information and the latent vector of environmental information. Output the final mixed student action through the gated recurrent unit and the multi-layer perceptron. where is the student policy mixed action space, is the student policy discrete command, is the student policy continuous command; Train the student policy to imitate the teacher policy using supervised learning, and use the minimized objective function in the following form: where K is the number of discrete actions; the first term of the objective function corresponds to the cross-entropy loss between the discrete commands of the teacher policy and the student policy, while the second term is used to capture the mean square error between the continuous commands of the teacher policy and the student policy. The trained student policy is combined with the underlying controller.

6. The quadruped robot control method according to claim 1, characterized in that, In step (3), the evolutionary strategy simulator optimization method is used to optimize the simulator parameters and narrow the gap between the simulator and the real world, including the following steps: (3.1) Parameter initialization: Initialize the parameter distribution of the simulator. (3.2) Real data collection: Collect state-action trajectory data in the real environment. (3.3) Simulation data collection: Collect state-action trajectory data in the simulation environment. (3.4) Adversarial training: Use the discriminator to distinguish the state-action transitions between the real world and the simulator, and at the same time optimize the parameter distribution through evolutionary strategies to maximize the discriminator's score for real samples. (3.5) Parameter update: Update the simulator according to the optimized parameter distribution. (3.6) Iterative optimization: Repeat simulation data collection, adversarial training, and parameter update until the parameter distribution converges. (3.7) Policy optimization: Retrain or fine-tune the control policy using the optimized simulator.

7. The quadruped robot control method according to claim 1, characterized in that The parameter initialization in step (3.1) includes the following: (3.1.1) Define the initial range of the simulator parameters; (3.1.2) Randomly sample the initial parameters; The data collection in step (3.2) includes the following steps: (3.2.1) In the real environment, use the predefined controller to collect the state-action trajectories; (3.2.2) Use the collected data for adversarial training; The data collection in step (3.3) includes the following steps: (3.3.1) In the simulation environment, use the same controller as in step (1.1) to collect the state-action trajectories; (3.3.2) Use the collected data for adversarial training; The adversarial training in step (3.4) includes the following: (3.4.1) The optimization objective of the discriminator is to distinguish the state-action transitions between the real world and the simulator, and the specific form is: where d B (s, a, s′) is simulation data, and d M (s, a, s′) is real data; (3.4.2) The optimization objective of the evolutionary strategy is to optimize the physical parameters of the simulator to maximize the discriminator's score for the simulator samples; The parameter update in step (3.5) includes the following steps: (3..5.1) Update the physical parameters of the simulator according to the optimized parameter distribution. The iterative optimization in step (3.6) includes the following: (3.6.1) Repeat data collection, adversarial training, and parameter update until the discriminator can no longer distinguish between real and simulation data; (3.6.2) Verify whether the optimized simulator meets the preset convergence conditions, such as the change range of the parameter distribution is less than the threshold; The iterative optimization in step (3.7) includes the following: (3.7.1) Use the optimized physical parameters of the simulator as the parameter range for domain randomization, retrain the control policy, or fine-tune the same controller as in (3.2.1) to make the policy adapt to a simulation closer to the real environment.

8. A four-legged robot control device, characterized in that, It includes a memory and a processor. When the computer program stored in the memory is called and executed by the processor, the method described in any one of claims 1-4 is implemented.

9. A computer-readable medium, characterized in that, The computer-readable medium stores a computer program. When the computer program is called and executed by the computer, the method described in any one of claims 1-4 is implemented.

Images