A method for constructing a jumping control model for a single-legged robot

Abstract

The present invention relates to a method for constructing a jumping control model for a single-leg robot, which belongs to the field of robot technology and solves the problem that the existing single-leg robot jumps unstably due to the difficulty in obtaining high-quality expert data. The method includes initializing the robot simulation environment after loading the single-leg robot model; obtaining multiple expert strategies by training a reinforcement learning model based on the robot simulation environment; sampling multiple expert trajectories by interacting the expert strategies with the robot simulation environment; constructing a generator and the same number of discriminators as the expert strategies; sampling imitator trajectories by interacting the generator with the robot simulation environment during the training process; using the imitator trajectory to train a discriminator with each expert trajectory respectively, calculating the reward value according to the discriminant value output by each discriminator, and training the generator using the PPO algorithm according to the imitator trajectory and the reward value; and obtaining a jumping control model according to the trained generator. The stable and continuous jumping of the single-leg robot is achieved.

Description

Technical Field

[0001] The present invention relates to the field of robot technology, and in particular to a method for constructing a jumping control model for a single-legged robot. Background Art

[0002] As robots are increasingly used in various fields, in industrial applications of robots, the robot's motion trajectory is usually predefined, or a certain task environment is pre-set, so that the robot can repeat the execution according to the plan. In this control mode, the robot cannot cope with changes in the task environment or sudden disturbances. More importantly, the robot's motion trajectory does not imply human operating habits. Robot imitation learning is an important method to solve these problems.

[0003] Imitation learning is an important technique in machine learning, which aims to learn a task or solve a problem by observing and imitating the behavior of experts. Compared with traditional rule-based or reward-based learning methods, imitation learning can directly learn high-quality behavioral strategies from expert demonstrations.

[0004] However, imitation learning methods all assume that the expert's strategy is the best and that the data is obtained from high-quality experts. However, in reality, there are problems such as incomplete expert data and uneven expert experience, which makes it difficult to obtain high-quality expert data. In the process of imitation learning, if the quality of expert data is diverse and the distribution of quality is unclear, and there is no additional information to evaluate and screen expert data of different qualities, imitation learning becomes very difficult. Therefore, how to make multi-quality expert imitation learning training tend to approach higher-quality expert trajectories and better control the robot's movements is an urgent problem that needs to be solved. Summary of the invention

[0005] In view of the above analysis, an embodiment of the present invention aims to provide a method for constructing a jumping control model for a single-legged robot, so as to solve the problem that the single-legged robot's jumping is unstable due to the difficulty in obtaining high-quality expert data.

[0006] An embodiment of the present invention provides a method for constructing a jumping control model for a single-legged robot, comprising the following steps:

[0007] After loading the single-leg robot model, initialize the robot simulation environment;

[0008] Based on the robot simulation environment, multiple expert strategies are obtained by training the reinforcement learning model; multiple expert trajectories are sampled by interacting the expert strategy with the robot simulation environment;

[0009] Construct a generator and the same number of discriminators as the expert strategy; during the training process, the generator interacts with the robot simulation environment to sample the imitator trajectory; use the imitator trajectory and each expert trajectory to train a discriminator, calculate the reward value based on the discriminant value output by each discriminator, and use the PPO algorithm to train the generator based on the imitator trajectory and the reward value;

[0010] The jump control model is obtained based on the trained generator.

[0011] Based on the further improvement of the above method, the robot simulation environment is obtained by building the MuJoCo physics engine and introducing the Gym library; the single-legged robot model is an XML format file, which is used to define the composition and connection relationship of the single-legged robot, wherein the single-legged robot consists of a body, thigh, calf and foot, the body and thigh are connected through a thigh joint, the thigh and calf are connected through a calf joint, and the calf and foot are connected through a foot joint.

[0012] Based on the further improvement of the above method, the robot simulation environment is initialized, including: based on the jumping scene of the single-legged robot, defining the state space, action space and reward function; and initializing the state space of the single-legged robot when the simulation environment is reset.

[0013] Based on further improvement of the above method, the state space includes: the height of the highest point of the robot body, the angle of the body, the angle of the thigh joint, the angle of the calf joint, the angle of the foot joint, the speed of the body's x coordinate, the speed of the body's z coordinate, the angular velocity of the body, the angular velocity of the thigh joint, the angular velocity of the calf joint, and the angular velocity of the foot joint; the action space includes: the torque of the thigh joint, the calf joint and the foot joint.

[0014] Based on the further improvement of the above method, through the interaction between expert strategies and robot simulation environment, multiple expert trajectories are sampled based on the same task, and the expert strategies are used to interact with the robot simulation environment respectively. The state space input and the action space output in each interaction are taken as a state-action pair. After the same number of interactions, the state-action pair sequence obtained is taken as the expert trajectory corresponding to the expert strategy; the imitator trajectory is the state-action pair sequence generated by the generator interacting with the robot simulation environment multiple times.

[0015] Based on the further improvement of the above method, a discriminator is trained using the imitator trajectory and each expert trajectory respectively, that is, the state-action pairs in the imitator trajectory are concatenated with the state-action pairs in each expert trajectory and passed into a discriminator, and each discriminator outputs the discrimination values ​​of the two state-action pairs; the gradient of the corresponding discriminator is calculated according to the discrimination values ​​of the two state-action pairs to update the discriminator parameters.

[0016] Based on the further improvement of the above method, the reward value is calculated according to the discriminant value output by each discriminator, including:

[0017] According to the discriminant value output by each discriminator, the probability that the discriminator is correct is calculated, and the expert accuracy and imitator accuracy of each discriminator are calculated respectively, and the sum of the two is taken as the accuracy of each discriminator;

[0018] The ratio of the accuracy of each discriminator to the sum of the accuracy of all discriminators is taken as the expert reliability coefficient of each discriminator, and the reward value is calculated according to the expert reliability coefficient of each discriminator and the discriminant value corresponding to the imitator's trajectory in each discriminator.

[0019] Based on the further improvement of the above method, the discriminant value output by each discriminator is judged to be correct by the discriminator, including:

[0020] For the state-action pairs in the expert trajectory, when the discriminant value output by the discriminator is greater than or equal to 0.5, it is considered that the discriminator has judged correctly; for the state-action pairs in the imitator trajectory, when the discriminant value output by the discriminator is less than 0.5, it is considered that the discriminator has judged correctly.

[0021] Based on the further improvement of the above method, the reward value is calculated by the following formula according to the expert reliability coefficient of each discriminator and the discriminant value corresponding to the imitator trajectory in each discriminator:

[0022]

[0023] Among them, r i represents the reward value during the i-th round of training; n represents the number of expert trajectories, represents the expert reliability coefficient of the discriminator corresponding to the jth expert trajectory during the i-th round of training; It represents the discriminant value of the state-action pair (s, a) in the imitator trajectory input to the discriminator output corresponding to the j-th expert trajectory during the i-th round of training; s represents the state space in the trajectory, and a represents the action space in the trajectory.

[0024] Based on the further improvement of the above method, the reward function includes: stability reward, jumping reward and action reward; among them, the stability reward is a positive reward given according to the number of interactions; the jumping reward is a positive reward or negative reward given by judging whether the robot jumps forward or backward; the action reward is a negative reward of a specified proportion to the sum of the squares of actions in each dimension in the action space.

[0025] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0026] 1. Based on the simulation platform, we obtain multi-quality expert trajectories, which are close to the actual application scenarios. We input the robot imitation trajectory and the expert trajectories of different qualities into different discriminators at the same time, neutralize the high-quality expert data and the low-quality expert data, and make the multi-quality expert imitation learning training tend to approach the higher-quality expert trajectory, so as to achieve better learning effect and enhance the robustness of the model.

[0027] 2. The reliability coefficients of different expert data are calculated based on the accuracy of different discriminators, and used as weights to remove the noise influence of different trajectories and improve the stability of single-legged robot jumping.

[0028] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0030] Figure 1 This is a flow chart of a method for constructing a jumping control model for a single-legged robot in an embodiment of the present invention;

[0031] Figure 2 This is an example diagram of the jump control model training process in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0033] A specific embodiment of the present invention discloses a method for constructing a jumping control model for a single-legged robot, such as Figure 1 As shown, the following steps are included:

[0034] S1. After loading the single-leg robot model, initialize the robot simulation environment.

[0035] It should be noted that in this embodiment, the robot simulation environment is constructed by building the MuJoCo physics engine and introducing the Gym library. According to the needs of the actual application scenario, the single-leg robot model is customized through an XML format file to describe the composition and connection relationship of the single-leg robot.

[0036] Specifically, the single-legged robot consists of a body, a thigh, a calf and a foot. The body and the thigh are connected through a thigh joint, the thigh and the calf are connected through a calf joint, and the calf and the foot are connected through a foot joint.

[0037] Furthermore, the robot simulation environment is initialized, including: based on the jumping scene of the single-legged robot, defining the state space, action space and reward function, and initializing the state space of the single-legged robot when the simulation environment is reset.

[0038] Specifically, the state space of a single-legged robot is an 11-dimensional continuous state space, including: the height of the highest point of the robot body, the angle of the body, the angle of the thigh joint, the angle of the calf joint, the angle of the foot joint, the speed of the body's x-coordinate, the speed of the body's z-coordinate, the angular velocity of the body, the angular velocity of the thigh joint, the angular velocity of the calf joint, and the angular velocity of the foot joint; all of the angles are angles with the positive direction of the z-axis. The action space is a 3-dimensional continuous action space, including: the torque of the thigh joint, calf joint, and foot joint.

[0039] This embodiment takes training a single-legged robot to jump forward continuously as the task, and the reward function includes stability reward, jumping reward and action reward, that is, the sum of these three reward values ​​is taken as the total reward.

[0040] It should be noted that the stability reward is a positive reward given according to the number of interactions. Each time the one-legged robot interacts with the simulation environment, if the termination condition is not met, it means that the robot has made a continuous jump, the stability reward value increases by 1, and the "flag of whether the execution is completed" is set to 0; if the termination condition is met, it means that the one-legged robot falls, and the "flag of whether the execution is completed" is set to 1, ending the interaction with the robot simulation environment.

[0041] Specifically, the termination condition is reached when the following conditions are met at the same time: there are null values ​​in the state space obtained from the simulation environment, the height of the highest point of the robot body in the state space exceeds the maximum height threshold, and the data of other dimensions are all less than their respective minimum thresholds.

[0042] It should be noted that the jumping reward is a positive or negative reward given by judging whether the robot jumps forward or backward. The jumping direction of the robot can be determined by calculating the direction of the current speed based on the difference between the current position and the previous position and the time interval. If the direction of the speed is in the positive direction of the x-axis, it is a forward jump and a specified positive reward is given. Otherwise, it is a backward jump and a specified negative reward is given.

[0043] It should be noted that the action reward is a negative reward of a specified proportion to the sum of the squares of the actions in each dimension in the action space. In other words, the greater the jumping action of the single-legged robot, the greater the penalty. For example, the sum of the square of the thigh joint torque, the square of the calf joint torque, and the square of the foot joint torque multiplied by -0.001 is the action reward.

[0044] S2. Based on the robot simulation environment, multiple expert strategies are obtained by training the reinforcement learning model; multiple expert trajectories are sampled by interacting the expert strategies with the robot simulation environment.

[0045] It should be noted that the expert strategy of this embodiment is to use the PPO algorithm to train the reinforcement learning model for different rounds to obtain expert strategies of different qualities. This embodiment does not limit the number of expert strategies. Preferably, three expert strategies are trained to represent experts of three levels. For example, the expert strategy obtained after less than 10 rounds of training has a low task completion rate, representing an expert of poor level; the expert strategy obtained after 2000 rounds of training has a good task completion rate, representing an expert of average level; the expert strategy obtained after 4000 rounds of training has the best task completion rate, representing an expert of the best level.

[0046] Specifically, the reinforcement learning model includes a policy network and a value network, wherein the policy network is used to output an action space according to the state space of the input single-legged robot in a two-dimensional plane area, and the value network is used to estimate the value function value of the state space of the input single-legged robot.

[0047] In this embodiment, the policy network and the value network are both two-layer fully connected networks, using the Relu activation function. After setting the hyperparameters for training the reinforcement learning model, the one-legged robot performs corresponding actions according to the action space and continuously interacts with the simulation environment to obtain the next state, and calculates the reward value according to the reward function in step S1, inputs it into the PPO algorithm model, and updates the policy network based on the PPO algorithm.

[0048] Exemplarily, the hyperparameters include: the discount factor gamma is set to 0.98, the GAE hyperparameter lmbda is set to 0.95, the batch size batch_size is set to 64, the policy network learning rate learning_rate_p is set to 3e-4, the value network learning rate learning_rate_v is set to 1e-3, the number of iterations ppo_epochs when updating the algorithm is set to 10, the step size per round is set to 50, and the number of hidden layer neurons hidden_size is set to 64.

[0049] After training, the model structures of multiple expert strategies are the same, but the network parameters are different. Next, through the interaction between the expert strategy and the robot simulation environment, multiple expert trajectories are sampled, including:

[0050] Based on the same task, expert strategies are used to interact with the robot simulation environment. During each interaction, the state space of the single-legged robot is input into the strategy network of the expert strategy, and the action space is output. The state space s and the output action space a of each interaction are taken as a state-action pair (s, a). E , after the same number of interactions m (i.e., the execution step length is m), the obtained state-action pair sequence is used as the expert trajectory corresponding to the expert strategy. The expert trajectories corresponding to different expert strategies represent expert data of different quality. For example, the jth expert trajectory is represented as

[0051] Compared with the prior art, this embodiment takes into account that in actual situations, the experience of experts varies greatly and it is difficult to obtain or evaluate expert data with the best quality. Therefore, multi-quality expert data is obtained for comprehensive consideration to improve the applicability of the technical solution.

[0052] S3. Construct a generator and the same number of discriminators as the expert strategy. During the training process, the generator interacts with the robot simulation environment to sample the imitator trajectory. Use the imitator trajectory to train a discriminator with each expert trajectory, calculate the reward value based on the discriminant value output by each discriminator, and use the PPO algorithm to train the generator based on the imitator trajectory and the reward value.

[0053] It should be noted that the generator in this embodiment also includes a strategy network and a value network, and the grid structure is the same as the network structure of the expert strategy. The number of discriminators is the same as the number of expert strategies, and is used to correspond to one expert trajectory respectively.

[0054] According to the simulation requirements, set the model parameters such as the policy network learning rate, value network learning rate, number of iterations when updating the algorithm, number of hidden layer neurons, etc. In an exemplary manner, the policy network learning rate learning_rate_p is set to 2e-4, the value network learning rate learning_rate_v is set to 1e-4, the number of iterations ppo_epochs when updating the algorithm is set to 10000, and the number of hidden layer neurons hidden_size is set to 64.

[0055] If three expert trajectories are derived using three different levels of expert strategies in step S2, the imitation learning process in step S3 is as follows: Figure 2 shown.

[0056] exist Figure 2In each round of training, the single-legged robot performs the same task. The generator generates actions according to the current state of the simulation environment, executes the actions in the simulation environment, obtains the next state, and transmits the current state, the executed action, the next state, and the flag data of whether the execution is completed to the experience pool; the state-action pair (s, a) sequence generated by the multiple interactions between the above generator and the robot simulation environment is used as the imitator trajectory, and the state-action pairs in the imitator trajectory are spliced ​​with the state-action pairs in an expert trajectory and transmitted to a different discriminator, which outputs the discrimination values ​​of the two state-action pairs; the gradient of the discriminator is calculated according to the discrimination values ​​of the two state-action pairs to update the discriminator parameters, and the accuracy of each discriminator is calculated according to the discrimination values ​​output by each discriminator, and then the reward value is calculated according to the accuracy and transmitted to the experience pool, and the loss function of the generator is calculated according to the data stored in the experience pool based on the PPO algorithm, and the generator parameters are updated using the gradient descent algorithm.

[0057] Specifically, the discriminator uses a three-layer fully connected network. The input is the state-action pairs in the imitator trajectory and the state-action pairs in the expert trajectory. The output is a two-dimensional real number, which corresponds to the discriminant value of the state-action pairs in the imitator trajectory and the discriminant value of the state-action pairs in the expert trajectory. The range of the discriminant value is [0,1], which is used to indicate whether the discriminator believes that the state-action pair comes from the imitator trajectory or the expert trajectory.

[0058] The gradient of each discriminator is calculated according to the following formula to update the parameters of each discriminator:

[0059]

[0060] in, represents the calculation expectation, ▽ represents the calculation gradient, w(j) i represents the discriminator parameters corresponding to the j-th expert trajectory (i.e., the j-th discriminator) during the i-th round of training, T i represents the imitator trajectory sampled during the i-th round of training, represents the j-th expert trajectory, They represent the discriminator D corresponding to the j-th expert trajectory during the i-th round of training. w(j)i For the imitator state-action pair (s, a), the expert state pair (s, a) E The output discriminant value.

[0061] Furthermore, the reward value is calculated according to the discriminant value output by each discriminator, including:

[0062] ① According to the discriminant value output by each discriminator, the probability that the discriminator is correct is calculated, and the expert accuracy and imitator accuracy of each discriminator are calculated respectively, and the sum of the two is taken as the accuracy of each discriminator.

[0063] It should be noted that the discriminant value output by each discriminator is considered correct by the discriminator, including: for the state-action pair in the expert trajectory, when the discriminant value output by the discriminator is greater than or equal to 0.5, it is considered that the discriminator has judged it to be correct; for the state-action pair in the imitator trajectory, when the discriminant value output by the discriminator is less than 0.5, it is considered that the discriminator has judged it to be correct.

[0064] Furthermore, the probability that the state-action pair in the expert trajectory is judged as correct by the discriminator is calculated to obtain the expert accuracy, that is, the number of state-action pairs judged as correct by the discriminator in the expert trajectory divided by the total number of state-action pairs in the expert trajectory, expressed by formula (2); the probability that the state-action pair in the imitator trajectory is judged as correct by the discriminator is calculated to obtain the imitator accuracy, that is, the number of state-action pairs judged as correct by the discriminator in the imitator trajectory divided by the total number of state-action pairs in the imitator trajectory, expressed by formula (3):

[0065]

[0066]

[0067] in, represents the expert accuracy of the j-th discriminator during the i-th round of training, represents the imitator accuracy of the jth discriminator during the i-th round of training; in formula (2), It indicates that the state-action pair in the expert trajectory is judged correct by the discriminator, It represents the probability that the state-action pair in the expert trajectory is correctly judged by the discriminator; in formula (3), It indicates that the state-action pair in the imitator's trajectory is judged to be correct by the discriminator. It represents the probability that the state-action pair in the imitator's trajectory is judged to be correct by the discriminator.

[0068] The accuracy of the discriminator is the sum of the expert accuracy and the imitator accuracy, expressed by formula (4):

[0069]

[0070] in, represents the accuracy of the j-th discriminator during the i-th round of training.

[0071] ② The ratio of the accuracy of each discriminator to the sum of the accuracy of all discriminators is taken as the expert reliability coefficient of each discriminator, and the reward value is calculated based on the expert reliability coefficient of each discriminator and the discriminant value corresponding to the imitator's trajectory in each discriminator.

[0072] It should be noted that, through formula (5), the ratio of the accuracy of each discriminator to the sum of the accuracy of all discriminators is taken as the expert reliability coefficient of each discriminator, and through formula (6), the reward value is calculated according to the expert reliability coefficient of each discriminator and the discriminant value corresponding to the imitator trajectory in each discriminator.

[0073]

[0074]

[0075] Among them, r i represents the reward value during the i-th round of training; n represents the number of expert trajectories, represents the expert reliability coefficient of the jth discriminator during the i-th round of training; It represents the discriminant value output by the discriminator corresponding to the j-th expert trajectory for the state-action pair (s, a) in the imitator trajectory during the i-th round of training.

[0076] According to formulas (5) and (6), this embodiment targets multi-quality expert trajectories. During training, high-quality and low-quality expert trajectories are integrated. The higher the accuracy of the discriminator, the higher the expert reliability coefficient. This allows the imitator trajectory output by the generator to learn the experience of the best expert in the current situation while learning the experience of different experts.

[0077] The calculated reward value is passed into the experience pool, and matched with the current state, executed action, next state, and flag data indicating whether the execution is completed in the experience pool. The loss function of the generator is calculated based on these data based on the PPO algorithm, and the generator parameters are updated using the gradient descent algorithm. When the number of training rounds reaches the maximum number, the trained generator is obtained.

[0078] S4. Obtain the jump control model based on the trained generator.

[0079] It should be noted that the policy network in the trained generator is used as the jump control model. The jump control model is deployed to the control system of the single-legged robot in the real environment. When the single-legged robot interacts with the real environment, the state space is constructed according to the collected robot posture information, and the state space is passed to the jump control model. The single-legged robot is controlled to jump according to the output action space, and the continuous jump control of the single-legged robot is achieved through continuous interaction.

[0080] Compared with the prior art, the present embodiment provides a method for constructing a jumping control model for a single-legged robot. Multi-quality expert trajectories are obtained based on a simulation platform, which is close to the actual application scenario. The robot imitation trajectory is simultaneously input into different discriminators together with expert trajectories of different qualities, and high-quality expert data and low-quality expert data are neutralized, so that multi-quality expert imitation learning training tends to approach higher-quality expert trajectories, thereby achieving better learning effects and enhancing the robustness of the model. The reliability coefficients of different expert data are calculated according to the accuracy of different discriminators, and are used as weights to remove the noise influence of different trajectories, thereby improving the stability of the single-legged robot's jumping.

[0081] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0082] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for constructing a jumping control model for a single-legged robot, characterized in that: The following steps are involved: After loading the single-leg robot model, the robot simulation environment is initialized, including: defining the state space, action space and reward function based on the jumping scene of the single-leg robot; initializing the state space of the single-leg robot when the simulation environment is reset; the state space of the single-leg robot is an 11-dimensional continuous state space, including: the height of the highest point of the robot body, the angle of the body, the angle of the thigh joint, the angle of the calf joint, the angle of the foot joint, the speed of the body x coordinate, the speed of the body z coordinate, the angular velocity of the body, the angular velocity of the thigh joint, the angular velocity of the calf joint, and the angular velocity of the foot joint; the action space is a 3-dimensional continuous action space, including: the torque of the thigh joint, the calf joint and the foot joint; taking the training of the single-leg robot to jump forward continuously as the task, the reward function includes stability reward, jumping reward and action reward; the stability reward is a positive reward given according to the number of interactions; the jumping reward is a positive reward or negative reward given by judging whether the robot jumps forward or backward; the action reward is a negative reward given in a specified proportion to the sum of the squares of actions in each dimension in the action space; Based on the robot simulation environment, multiple expert strategies are obtained by training the reinforcement learning model; multiple expert trajectories are sampled by interacting the expert strategy with the robot simulation environment; Construct a generator and the same number of discriminators as the expert strategy; the generator includes a strategy network and a value network, both of which are two-layer fully connected networks; during the training process, the generator interacts with the robot simulation environment to sample the imitator trajectory; use the imitator trajectory and each expert trajectory to train a discriminator, calculate the reward value according to the discriminant value output by each discriminator, and use the PPO algorithm to train the generator according to the imitator trajectory and the reward value; A jumping control model is obtained according to the trained generator, that is, the policy network in the trained generator is used as the jumping control model; the jumping control model is deployed to the control system of a single-legged robot in a real environment. When the single-legged robot interacts with the real environment, a state space is constructed according to the collected robot posture information, and the state space is passed into the jumping control model. The single-legged robot is controlled to jump according to the output action space, and continuous jumping control of the single-legged robot is achieved through continuous interaction.

2. The method for constructing a jumping control model for a single-legged robot according to claim 1, characterized in that: The robot simulation environment is obtained by building the MuJoCo physics engine and introducing the Gym library; the single-leg robot model is an XML format file, which is used to define the composition and connection relationship of the single-leg robot, wherein the single-leg robot consists of a body, a thigh, a calf and a foot, the body and the thigh are connected through a thigh joint, the thigh and the calf are connected through a calf joint, and the calf and the foot are connected through a foot joint.

3. The method for constructing a jumping control model for a single-legged robot according to claim 1, characterized in that: The method interacts with the robot simulation environment through expert strategies, samples multiple expert trajectories based on the same task, and uses expert strategies to interact with the robot simulation environment respectively. The state space input and the action space output in each interaction are taken as a state-action pair. After the same number of interactions, the state-action pair sequence obtained is taken as the expert trajectory corresponding to the expert strategy; the imitator trajectory is the state-action pair sequence generated by the generator interacting with the robot simulation environment multiple times.

4. The method for constructing a jumping control model for a single-legged robot according to claim 3, characterized in that: The method of using the imitator trajectory and each expert trajectory to train a discriminator is to concatenate the state-action pairs in the imitator trajectory with the state-action pairs in each expert trajectory and pass them into a discriminator, and each discriminator outputs the discriminant value of the two state-action pairs; The discriminant parameters of the corresponding discriminator are updated by calculating the gradient of the discriminator according to the discriminant values ​​of the two state-action pairs.

5. The method for constructing a jumping control model for a single-legged robot according to claim 4, characterized in that: The step of calculating the reward value according to the discriminant value output by each discriminator includes: According to the discriminant value output by each discriminator, the probability that the discriminator is correct is calculated, and the expert accuracy and imitator accuracy of each discriminator are calculated respectively, and the sum of the two is taken as the accuracy of each discriminator; The ratio of the accuracy of each discriminator to the sum of the accuracy of all discriminators is taken as the expert reliability coefficient of each discriminator, and the reward value is calculated according to the expert reliability coefficient of each discriminator and the discriminant value corresponding to the imitator's trajectory in each discriminator.

6. The method for constructing a jumping control model for a single-legged robot according to claim 5, characterized in that: The discriminant values ​​output by each discriminator are judged by the discriminator to be correct, including: For the state-action pair in the expert trajectory, when the discriminant value output by the discriminator is greater than or equal to 0.5, it is considered that the discriminator has judged it correctly; for the state-action pair in the imitator trajectory, when the discriminant value output by the discriminator is less than 0.5, it is considered that the discriminator has judged it correctly.

7. The method for constructing a jumping control model for a single-legged robot according to claim 5, characterized in that: The reward value is calculated according to the expert reliability coefficient of each discriminator and the discriminant value corresponding to the imitator trajectory in each discriminator by the following formula: Among them, r i represents the reward value during the i-th round of training; n represents the number of expert trajectories, represents the expert reliability coefficient of the discriminator corresponding to the jth expert trajectory during the i-th round of training; It represents the discriminant value of the state-action pair (s, a) in the imitator trajectory input to the discriminator output corresponding to the j-th expert trajectory during the i-th round of training; s represents the state space in the trajectory, and a represents the action space in the trajectory.

Images