A method for autonomous emergence of motion trajectories of legged robots based on prior experience

Abstract

The present invention provides a method for autonomous emergence of motion trajectories of a legged robot based on prior experience, establishes a deep reinforcement learning model of a legged robot intelligent body, and designs a semi-autonomous emergence model of motion trajectory and a fully autonomous emergence model of motion trajectory for robot motion control after reinforcement learning, together forming a method for autonomous emergence of motion trajectories of a legged robot based on prior experience. After the model is deployed, the autonomous motion and walking of a physical prototype can be successfully realized, and the autonomous generation of the robot's motion trajectory can be effectively realized without any human intervention and remote operation command control, which can reduce the cost and risk of on-orbit verification, provide a reference for the intelligent and autonomous motion and walking of space robots in the complex environment of the Moon and Mars, and point out the direction of cooperation between traditional control and learning control.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot motion control, and in particular relates to a method for autonomous emergence of motion trajectories of a legged robot based on prior experience. Background Art

[0002] One of the main goals of my country's deep space exploration mission is to establish permanent bases on the surfaces of the moon and Mars. However, the construction of extraterrestrial bases is an unprecedented major project, which requires sufficient preparation before implementation. The most critical part of this is the development of space robots that can adapt to the lunar and Martian environments, paving the way for the establishment of bases and even manned landings.

[0003] Considering that robot technology has gradually matured and has been widely used in recent years, for space robots serving space missions, due to their unique mission environment and usage requirements, the operation and control methods of traditional robots have encountered many challenges in space missions. On the one hand, compared with robots working on the ground and near-Earth space, deep space and extraterrestrial exploration robots can only be subject to limited human intervention, and the environment is unknown or limited, and there are strict communication constraints, resulting in many mature control methods that have been successfully applied on the ground no longer applicable; on the other hand, with the increasing complexity of space missions, space robots have evolved from simple systems that perform a single task to complex nonlinear systems that integrate multiple tasks. The formulation of traditional model-based control strategies generally requires accurate dynamic descriptions and boundary conditions, and the dynamic modeling process of nonlinear systems is often more complicated, resulting in differences in the same control method due to different usage environments, and a large amount of repeated modeling work. Therefore, how to solve the motion control problem of robots with the help of artificial intelligence methods is of great significance for space missions. Summary of the invention

[0004] To solve the above problems, the present invention provides a method for autonomous emergence of motion trajectories of a legged robot based on prior experience, which can realize autonomous generation of motion trajectories of a legged robot without any human intervention and remote operation command control.

[0005] A method for autonomous emergence of motion trajectories of a legged robot based on prior experience, which uses a fully autonomous emergence model or a semi-autonomous emergence model obtained through reinforcement learning training to obtain the motion trajectory of the legged robot;

[0006] The method for obtaining the motion trajectory of the fully autonomous emergence model is as follows: inputting the rotation angle of the motor on each leg of the legged robot at the current moment into the fully autonomous emergence model, obtaining the rotation angle of the motor on each leg of the legged robot at the next moment, so that the legged robot can autonomously complete the walking task;

[0007] The method for the semi-autonomous emergence model to obtain the motion trajectory is: input the reference value of the motor rotation angle on each leg of the legged robot and the motor rotation angle on each leg of the legged robot at the current moment into the semi-autonomous emergence model, and obtain the motor rotation angle on each leg of the legged robot at the next moment, so that the legged robot can complete the walking task semi-autonomously.

[0008] Furthermore, the training method of the fully autonomous emergence model is:

[0009] The roll angle, pitch angle, roll angular velocity, pitch angular velocity, position of the legged robot at the current moment, and the motor speed and motor torque on each leg of the legged robot are constructed as a state space;

[0010] The motor torque on each leg of the legged robot at the current moment is constructed as the action space;

[0011] Construct a reward function r1 based on the optimal patrol speed:

[0012]

[0013] Among them, x, y, z are the current positions of the legged robot, x0, y0, z0 are the positions of the legged robot at the previous moment, τ i is the motor torque on each leg of the legged robot, ω i is the motor speed of each leg of the legged robot, i = 1, 2, ..., n, n is the number of legs of the legged robot;

[0014] In the current terrain environment, different state spaces, action spaces and the motor rotation angles of each leg of the legged robot corresponding to different state spaces and action spaces at the next moment are obtained as training sample sets, and the fully autonomous emergence model is trained by reinforcement learning based on the training sample sets until the reward function r1 based on the optimal patrol speed converges to obtain the final fully autonomous emergence model.

[0015] Furthermore, the training method of the fully autonomous emergence model is:

[0016] The roll angle, pitch angle, roll angular velocity, pitch angular velocity, position of the legged robot at the current moment, and the motor speed and motor torque on each leg of the legged robot are constructed as a state space;

[0017] The motor torque on each leg of the legged robot at the current moment is constructed as the action space;

[0018] Construct a reward function r2 based on optimal joint energy consumption:

[0019]

[0020] Among them, x, y, z are the current positions of the legged robot, x0, y0, z0 are the positions of the legged robot at the previous moment, τ i is the motor torque on each leg of the legged robot, ω i is the motor speed of each leg of the legged robot, i = 1, 2, ..., n, n is the number of legs of the legged robot;

[0021] In the current terrain environment, different state spaces, action spaces and the motor rotation angles of each leg of the legged robot corresponding to different state spaces and action spaces at the next moment are obtained as training sample sets, and the fully autonomous emergence model is trained by reinforcement learning based on the training sample sets until the reward function r2 based on the optimal joint energy consumption converges to obtain the final fully autonomous emergence model.

[0022] Furthermore, the training method of the semi-autonomous emergence model is:

[0023] The roll angle, pitch angle, roll angular velocity, pitch angular velocity, position of the legged robot at the current moment, and the motor speed and motor torque on each leg of the legged robot are constructed as a state space;

[0024] The motor torque on each leg of the legged robot at the current moment is constructed as the action space;

[0025] Construct the reward function r1 based on the optimal patrol speed and the reward function r2 based on the optimal joint energy consumption:

[0026]

[0027]

[0028] Among them, x, y, z are the current positions of the legged robot, x0, y0, z0 are the positions of the legged robot at the previous moment, τ i is the motor torque on each leg of the legged robot, ω i is the motor speed of each leg of the legged robot, i = 1, 2, ..., n, n is the number of legs of the legged robot;

[0029] In the current terrain environment, the difference between the motor rotation angle of each leg of the legged robot at the next moment corresponding to different state spaces and action spaces and the set reference value is obtained, and the different state spaces, action spaces and the differences corresponding to different state spaces and action spaces are used as training sample sets. Based on the training sample sets, the semi-autonomous emergence model is trained by reinforcement learning until the reward function r1 based on the optimal patrol speed or the reward function r2 based on the optimal joint energy consumption converges to obtain the final semi-autonomous emergence model.

[0030] Furthermore, the training method of the semi-autonomous emergence model is:

[0031] The roll angle, pitch angle, roll angular velocity, pitch angular velocity, position of the legged robot at the current moment, and the motor speed and motor torque on each leg of the legged robot are constructed as a state space;

[0032] The motor torque on each leg of the legged robot at the current moment is constructed as the action space;

[0033] Construct the reward function r1 based on the optimal patrol speed and the reward function r2 based on the optimal joint energy consumption:

[0034]

[0035]

[0036] Among them, x, y, z are the current positions of the legged robot, x0, y0, z0 are the positions of the legged robot at the previous moment, τ i is the motor torque on each leg of the legged robot, ω i is the motor speed of each leg of the legged robot, i = 1, 2, ..., n, n is the number of legs of the legged robot;

[0037] In the current terrain environment, the difference between the motor rotation angle of each leg of the legged robot at the current moment corresponding to different state spaces and action spaces and the set reference value is obtained, and the different state spaces, action spaces and the differences corresponding to different state spaces and action spaces are used as training sample sets. Based on the training sample sets, the semi-autonomous emergence model is trained by reinforcement learning until the reward function r1 based on the optimal patrol speed or the reward function r2 based on the optimal joint energy consumption converges to obtain the final semi-autonomous emergence model.

[0038] Furthermore, the fully autonomous emergence model and the semi-autonomous emergence model belong to a neural network model, and the fully autonomous emergence model or the semi-autonomous emergence model is deployed on the legged robot according to the following steps:

[0039] By avoiding the hardware equipment required to deploy the neural network model, the driving data is directly injected into the controller of the legged robot to achieve pseudo transfer of reinforcement learning;

[0040] After completing the pseudo transfer of reinforcement learning, a static policy network is deployed on the controller of the legged robot that supports the neural network model to achieve offline transfer of reinforcement learning;

[0041] After completing the offline migration of reinforcement learning, the policy network and the value function network are deployed simultaneously on the controller of the legged robot that supports the neural network model to achieve online migration.

[0042] Furthermore, the reinforcement learning algorithm architecture used in training the fully autonomous emergence model or the semi-autonomous emergence model is the Actor-Critic architecture, and the Actor network and the Critic network in the Actor-Critic architecture are updated using the PPO algorithm.

[0043] Furthermore, the open source PyBullet physics engine is used to build a reinforcement learning model simulation environment for the legged robot, and OpenGL is used to display the graphics of the reinforcement learning model simulation environment. At the same time, TensorFlow is used to build a fully autonomous emergence model or semi-autonomous emergence model based on a neural network.

[0044] Beneficial effects:

[0045] 1. The present invention provides a method for autonomous emergence of motion trajectories of a legged robot based on prior experience, establishes a deep reinforcement learning model of a legged robot intelligent body, and designs a semi-autonomous emergence model of motion trajectory and a fully autonomous emergence model of motion trajectory for robot motion control after reinforcement learning, together forming a method for autonomous emergence of motion trajectories of a legged robot based on prior experience. After the model is deployed, the autonomous motion and walking of the physical prototype can be successfully realized, and the autonomous generation of the robot's motion trajectory can be effectively realized without any human intervention and remote operation command control, which can reduce the cost and risk of on-orbit verification, provide a reference for the intelligent and autonomous motion and walking of space robots in the complex environment of the Moon and Mars, and point out the direction of cooperation between traditional control and learning control.

[0046] 2. The present invention provides a method for autonomous emergence of motion trajectories of a legged robot based on prior experience, which can effectively realize the autonomous generation of motion trajectories of the legged robot at the optimal patrol speed without any human intervention and remote operation command control; specifically, the present invention can be implemented by semi-autonomous emergence of motion trajectories or fully autonomous emergence of motion trajectories. At this time, the reward function will focus on the weight of the patrol speed, and the legged robot will be trained to realize patrol speed control, thereby realizing autonomous walking of the legged robot at the optimal patrol speed; it can be seen that the present invention establishes an intelligent agent control model based on prior experience, which can reduce the training time of the intelligent agent model and improve the convergence performance of the model.

[0047] 3. The present invention provides a method for autonomous emergence of motion trajectories of a legged robot based on prior experience, which can effectively realize the autonomous generation of motion trajectories of the legged robot with optimal joint energy consumption without any human intervention and remote operation command control; specifically, the present invention can be implemented by semi-autonomous emergence of motion trajectories or fully autonomous emergence of motion trajectories. At this time, the reward function will focus on the weight of joint energy consumption, and the legged robot will be trained to achieve joint energy consumption control, thereby realizing autonomous walking of the legged robot with optimal joint energy consumption; it can be seen that the present invention establishes an intelligent body control model based on prior experience, which can reduce the training time of the intelligent body model and improve the convergence performance of the model.

[0048] 4. The present invention provides a method for autonomous emergence of motion trajectories of a legged robot based on prior experience. The migration process of deploying a fully autonomous emergence model or a semi-autonomous emergence model on a legged robot is divided into three stages and implemented step by step. That is, the model is deployed on a physical prototype in the order of data stream pseudo-migration, offline network migration, and online network migration. This can reduce the damage to the hardware of the physical prototype caused during the learning process. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A schematic diagram of the reinforcement learning model.

[0050] Figure 2 This is a diagram of the reinforcement learning algorithm architecture;

[0051] Figure 3 Design a schematic diagram for the Actor network;

[0052] Figure 4 Design a schematic diagram for the Critic network;

[0053] Figure 5 It is the non-autonomous emergence diagram of the motion trajectory;

[0054] Figure 6 It is a fully autonomous emergence map of the motion trajectory;

[0055] Figure 7 It is a semi-autonomous emergence diagram of the movement trajectory;

[0056] Figure 8 This is another semi-autonomous emergence diagram of the movement trajectory. DETAILED DESCRIPTION

[0057] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0058] The present invention proposes a solution to the motion and walking problems that robots may encounter in complex environments during future space exploration missions. Namely, a method for autonomous emergence of motion trajectories of legged robots based on prior experience is designed. The non-autonomous emergence of motion trajectories, semi-autonomous emergence of motion trajectories, and fully autonomous emergence of motion trajectories are verified based on digital simulation design, and relevant reinforcement learning training is carried out to effectively verify the autonomous generation capability of the robot reinforcement learning model without any human intervention and remote operation command control. Meanwhile, the intelligent agent control model based on prior experience is verified based on the physical prototype design. After the model is deployed, the prototype can successfully walk autonomously, effectively simulating the autonomous walking process of the robot in a complex environment, verifying the feasibility of the mission, and reducing the cost and risk of on-orbit verification.

[0059] The object of the present invention is to provide an intelligent autonomous motion method for a legged robot that can realize autonomous generation of motion trajectories based on prior experience or known control models without any human intervention or remote control instructions.

[0060] In order to achieve the above-mentioned objectives, the method adopted by the present invention is: a legged robot dynamics and control model and an intelligent agent reinforcement learning model are successively established and designed, and the robot's autonomous motion control in a simulation environment and an actual prototype is realized, the problem of intelligent autonomous motion of the space robot is solved, and the feasibility of applying the reinforcement learning method to future space patrol and detection missions is verified.

[0061] That is, the present invention uses a fully autonomous emergence model or a semi-autonomous emergence model obtained through reinforcement learning training to obtain the motion trajectory of the legged robot;

[0062] The method for obtaining the motion trajectory of the fully autonomous emergence model is as follows: inputting the rotation angle of the motor on each leg of the legged robot at the current moment into the fully autonomous emergence model, obtaining the rotation angle of the motor on each leg of the legged robot at the next moment, so that the legged robot can autonomously complete the walking task;

[0063] The method for the semi-autonomous emergence model to obtain the motion trajectory is: input the reference value of the motor rotation angle on each leg of the legged robot and the motor rotation angle on each leg of the legged robot at the current moment into the semi-autonomous emergence model, and obtain the motor rotation angle on each leg of the legged robot at the next moment, so that the legged robot can complete the walking task semi-autonomously.

[0064] Specifically, the process of acquiring the fully autonomous emergence model or semi-autonomous emergence model of the present invention includes building a virtual digital simulation environment, establishing a legged robot dynamics model, establishing an intelligent agent reinforcement learning model, iterative training of the reinforcement learning model, debugging the robot physical prototype, and deployment and migration of the reinforcement learning model, which consists of six main steps.

[0065] (1) Building a virtual digital simulation environment

[0066] Build a virtual digital simulation environment: Build a legged robot dynamics simulation environment and a terrain simulation environment based on the open source library PyBullet. Use the open source library OpenAI Gym to build an intelligent agent reinforcement learning simulation environment. OpenAI Gym and PyBullet are developed based on the Python language, which ensures the open source and white box nature of the digital simulation environment.

[0067] (2) Establishing the dynamic model of the legged robot

[0068] Establishing a dynamic model of a legged robot: In order to ensure that the robot can walk efficiently, stably and reliably in the task environment, it is necessary to establish kinematics, dynamics and control descriptions that meet certain accuracy requirements. Therefore, the present invention establishes high-precision robot dynamics and control models and joint motor models based on the dynamics simulation environment built by PyBullet, and gives mathematical descriptions of the kinematics, dynamics and controllers of the controlled robot, including the relationship between geometric parameters and time parameters related to motion, the forward and inverse kinematics equations of the legs and the base, the relationship between joint forces / torques and the generalized coordinates of each joint of the robot, the forward and inverse dynamics equations of the legs and the base, the joint controller model and the gait model, etc., which provide a basis for the design and establishment of subsequent reinforcement learning models. In addition, by reducing the deviation between the simulation model and the physical prototype through model correction, high-precision training samples are provided for simulation training, and a reliable model basis is also provided for subsequent prototype migration work.

[0069] 1) Kinematic model

[0070] The kinematic model studies the relationship between geometric parameters and time parameters related to motion. That is, based on the mechanical structure of the robot, the kinematics of its legs and the kinematics of the base are analyzed, and the forward and inverse kinematic equations are derived using geometric methods to obtain the mapping relationship between the leg and body postures and the angular positions of each joint of the robot, thereby providing a basis for dynamic calculations.

[0071] 2) Kinetic model

[0072] The dynamic model studies the relationship between joint forces / torques and the generalized coordinates of each joint of the robot. That is, the forward and inverse dynamic equations of the legs and the base are established based on the multi-body relationship and articulated body algorithm, and the node displacement and stress and strain distribution are obtained by solving the motion equations and algebraic constraint equations, thereby providing accurate input for the controller.

[0073] 3) Gait model

[0074] The gait model studies the design of gait parameters such as step length, step sequence, duty cycle, relative phase, motion cycle, body height, etc. according to task requirements, and then solves the multi-constraint equations established based on this to obtain the optimal solution of the terminal trajectory, and then solves the motion state of each joint based on the previously established inverse kinematics model to obtain the gait required for the robot to walk.

[0075] (3) Establishing an agent reinforcement learning model

[0076] Establish an intelligent agent reinforcement learning model: Before establishing the intelligent agent model, it is necessary to first design a reinforcement learning environment model including action space, state space, transfer function and reward function according to the motion model of the controlled object (legged robot) and the relevant parameters of the controller model. That is, based on the second step, by sorting out the relevant parameters of the dynamics and control models, based on the reinforcement learning simulation environment built by OpenAI Gym, appropriate parameters are selected to design the action space, state space, transfer function and reward mechanism required for reinforcement learning to establish a reinforcement learning model, which includes a non-autonomous emergence model of motion trajectory, a semi-autonomous emergence model of motion trajectory, and a fully autonomous emergence model of motion trajectory.

[0077] like Figure 1 As shown in Figure 1, Agent represents the research object of reinforcement learning, i.e., the intelligent agent; Environment represents the learning environment of reinforcement learning; t Indicates the action taken by the Agent at the current step; S t+1 Indicates the next state of the learning environment given according to the current step action; R t+1 Represents the reward for the next step given by the learning environment based on the current step action.

[0078] like Figure 2 As shown in FIG, the process of reinforcement learning is a process in which the intelligent agent interacts with the environment, continuously tries and makes mistakes, and improves its decision-making ability. In order to ensure that the learning process can converge, most mainstream reinforcement learning algorithms currently use the Actor-Critic architecture. Therefore, the reinforcement learning algorithm architecture adopted by the present invention is the Actor-Critic architecture (Actor-Critic Approaches); Figure 3 and Figure 4 As shown in the figure, the Actor network and Critic network in the Actor-Critic architecture are updated using the Proximal Policy Optimization (PPO) algorithm.

[0079] 2) Strengthen environmental design

[0080] In the present invention, PyBullet is used as the dynamics platform and OpenAI Gym is used as the reinforcement learning platform. Therefore, a reinforcement learning model containing a motor model is first established by combining OpenAI Gym and PyBullet, and relevant training is carried out to verify the feasibility and generalization ability of the reinforcement learning model's non-autonomous emergence of motion trajectories, semi-autonomous emergence of motion trajectories, and fully autonomous emergence of motion trajectories models.

[0081] The reinforcement learning model environment uses the open source PyBullet physics engine to build a robot simulation environment, uses OpenGL for graphics display, builds a general programming and graphics interface compatible with OpenAI Gym, and uses TensorFlow to build a neural network. The physical properties of the robot and the environment are added to the model, such as mass, friction coefficient, elastic coefficient, damping coefficient, etc. The motor model is provided by the open source solution of PyBullet. The robot's sensor information is added to the simulation environment in the form of a PyBullet plug-in and displayed in a visual way. All of the above links are implemented by open source tools and can be modified as needed.

[0082] Motion trajectory non-autonomous emergence design

[0083] like Figure 5 As shown in the figure, the motion trajectory of the non-autonomous emergence design adopts the traditional motion control based on the preset continuous periodic gait. Because the control scheme is not changed according to the changes in the environment in the design, it is impossible to complete the walking task on complex terrain. It is mainly used for comparison with reinforcement learning control. The motion trajectory of the robot in this design is given by the traditional motion control model.

[0084] Fully autonomous emergence design of motion trajectory

[0085] like Figure 6 As shown in the figure, the fully autonomous emergence design of the motion trajectory does not need to model the robot's gait model, but directly uses the reinforcement learning model to establish a mapping from the motor rotation angle at the current moment to the rotation angle at the next moment. The motor rotation angle at the current moment represents the current state of the quadruped robot, and the motor rotation angle at the next moment represents the current control strategy. The training process is to learn the strategy using the reinforcement learning method based on the sample data. In this design, the robot's motion trajectory is completely given autonomously by the reinforcement learning model.

[0086] Semi-autonomous emergence design of motion trajectory

[0087] like Figure 7 and Figure 8As shown, there are two semi-autonomous emergence modes of motion trajectories provided by the present invention. The semi-autonomous emergence design of motion trajectory is a reinforcement learning optimization control scheme that combines the advantages of the traditional motion control method for modeling the robot's legs and the reinforcement learning model without prior experience. Compared with the non-autonomous emergence design of motion trajectory and the fully autonomous emergence design of motion trajectory, this method has a faster convergence speed and better learning effect, and can enable the robot to walk better on complex terrain. In this design, the robot's motion trajectory is partially given autonomously by the reinforcement learning model.

[0088] 3) Strengthen model design

[0089] State space: body, i.e. the roll angle of the body of the legged robot Pitch angle φ

[0090] Rolling angular velocity of the aircraft Pitch angular velocity

[0091] The position of the body x,y,z

[0092] Assuming that the legged robot has four legs and each leg has two electrodes, the rotation angle θ of the motor is i (i=1…8)

[0093] Motor speedω i (i=1…8)

[0094] The torque of the motor τ i (i=1…8)

[0095] Action space: motor torque τ i (i=1…8)

[0096] State space reward function:

[0097]

[0098] Among them, r is the reward function; x, y, z are the current position of the body; x0, y0, z0 are the previous position of the body; τ is the motor torque; ω is the motor speed; is the roll angle of the aircraft; θ is the yaw angle of the aircraft; φ is the pitch angle of the aircraft; and the hyperparameters λ0, λ1,…, λ6 are the weight coefficients of each reward item.

[0099] (4) Iterative training of reinforcement learning model

[0100] Iterative training of reinforcement learning model: After building the robot dynamics and control model and reinforcement learning model, carry out training and verification of the simulation model. During the training process, the parameters such as state space, action space, reward function, network structure, etc. need to be continuously iterated and improved, that is, "training-iteration-retraining" to meet the convergence needs of the reinforcement learning model. After the training is completed and the converged model is obtained, verify its ability to autonomously generate motion trajectories and its adaptability after the dynamic changes of the system model.

[0101] Next, we will train and verify the autonomous movement and generalization ability of the reinforcement learning model based on prior experience under different terrain parameters. Specifically, we define a terrain environment as the training environment, conduct walking training on it, and after the training results converge, deploy the agent in other terrain environments. During the simulation process, the model is not modified or trained, and its autonomous generation and generalization capabilities of motion trajectories in unknown terrain environments are verified.

[0102] 1) Reinforcement learning model training based on optimal patrol speed

[0103] Through training, i.e. “training-iteration-retraining”, the weight coefficients of the reward function of the reinforcement learning model based on the optimal patrol speed in the under-state space (that is, all parameters are known during simulation, but many parameters of the prototype are unknown, so the prototype is under-state space) are given as [λ0,λ1,λ2,λ3]=[1,0.3,0.3,0.005], that is,

[0104] State space: roll angle of the body Pitch angle φ

[0105] Rolling angular velocity of the aircraft Pitch angular velocity

[0106] The position of the body x,y,z

[0107] The motor's rotation angle θ i (i=1…8)

[0108] Motor speedω i (i=1…8)

[0109] The torque of the motor τ i (i=1…8)

[0110] Action space: motor torque τ i (i=1…8)

[0111] Reward function:

[0112]

[0113] Among them, r1 is the reward function; x, y, z are the current position of the body; x0, y0, z0 are the previous position of the body; τ is the motor torque; ω is the motor speed.

[0114] The above is the reinforcement learning model with optimal patrol speed based on prior experience.

[0115] 2) Reinforcement learning model training based on optimal joint energy consumption

[0116] Through training, i.e. “training-iteration-retraining”, the weight coefficients of the reward function of the reinforcement learning model based on the optimal joint energy consumption in the under-state space are given as [λ0,λ1,λ2,λ3]=[1,0.3,0.3,0.1], that is

[0117] State space: Same as above

[0118] Action space: Same as above

[0119] Reward function:

[0120]

[0121] Among them, r2 is the reward function; x, y, z are the current position of the body; x0, y0, z0 are the previous position of the body; τ is the motor torque; ω is the motor speed.

[0122] The above is a reinforcement learning model based on prior experience and optimized for joint energy consumption.

[0123] It should be noted that the larger the reward value, the better. If the last term in the reward function determined by the motor torque and motor speed is If it is too large, the reward value will become smaller, and the reinforcement learning model will tend to reduce the motor speed or motor torque, making the legged robot energy consumption lower.

[0124] After determining the state space, action space, and reward function required for reinforcement learning, you can start training the fully autonomous emergence model and the semi-autonomous emergence model.

[0125] The training method of the fully autonomous emergence model is as follows: Figure 6 As shown, in the current terrain environment, different state spaces, action spaces and the motor rotation angles on each leg of the legged robot corresponding to different state spaces and action spaces at the next moment are obtained as training sample sets, and the fully autonomous emergence model is trained by reinforcement learning based on the training sample sets until the reward function r1 based on the optimal patrol speed or the reward function r2 based on the optimal joint energy consumption converges to obtain the final fully autonomous emergence model.

[0126] The training method of the semi-autonomous emergence model is as follows: Figure 7As shown, in the current terrain environment, the difference between the motor rotation angle of each leg of the legged robot at the next moment corresponding to different state spaces and action spaces and the set reference value is obtained, and the different state spaces, action spaces and the differences corresponding to different state spaces and action spaces are used as training sample sets, and the semi-autonomous emergence model is trained by reinforcement learning based on the training sample sets until the reward function r1 based on the optimal patrol speed or the reward function r2 based on the optimal joint energy consumption converges to obtain the final semi-autonomous emergence model.

[0127] Or Figure 8 As shown, in the current terrain environment, the difference between the motor rotation angle of each leg of the legged robot at the current moment corresponding to different state spaces and action spaces and the set reference value is obtained, and the different state spaces, action spaces and the differences corresponding to different state spaces and action spaces are used as training sample sets, and the semi-autonomous emergence model is trained by reinforcement learning based on the training sample sets until the reward function r1 based on the optimal patrol speed or the reward function r2 based on the optimal joint energy consumption converges to obtain the final semi-autonomous emergence model.

[0128] (5) Robot physical prototype debugging

[0129] Robot physical prototype debugging: This step is carried out simultaneously with the simulation process. Based on the actual physical prototype, motion control based on the preset continuous periodic gait is realized. Because the walking of the legged robot relies on the intermittent support work of multiple legs in a periodic rhythmic manner, the differences in support frequency and phase difference constitute different gait types. In addition to the basic multi-legged standing gait, typical gaits include the walking gait based on three legs (Walk), the diagonal gait based on diagonal two legs (Trot), the jumping gait based on the intermittent contact of the front and rear legs (Bound), the jumping gait based on the simultaneous jump of the front and rear legs (Pronk), and the differential turning gait based on the different step lengths of the left and right legs. In addition, in the debugging process, the robot simulation system model needs to be continuously corrected to shorten the gap between the simulation model and the physical prototype.

[0130] (6) Reinforcement Learning Model Deployment and Migration

[0131] Reinforcement learning model deployment migration: Considering that there is a certain gap between the simulation model and the actual prototype, and the reinforcement model deployed in this invention comes from the simulation, it is necessary to repeatedly adjust the parameters to shorten the gap between the simulation model and the actual prototype. The migration work will be carried out step by step from easy to difficult, that is, data flow migration, offline network migration and online network migration will be carried out successively to complete the on-board deployment and realize the autonomous movement of the robot.

[0132] Specifically, the migration process can be divided into three stages and implemented step by step, namely: 1) data flow pseudo-migration exploration stage, that is, avoiding the hardware equipment required for deploying neural networks and directly injecting driving data into the controller to realize reinforcement learning pseudo-migration; 2) offline network migration exploration stage, that is, deploying a static policy network on a controller that supports neural networks to realize reinforcement learning offline migration; 3) online network migration exploration stage, that is, simultaneously deploying a policy network and a value function network on a controller that supports neural networks to realize online migration.

[0133] 1) Pseudo-migration of data flow

[0134] Exploration of pseudo-migration of data flow, that is, avoiding the hardware equipment required for deploying neural networks and directly injecting drive data into the controller to achieve pseudo-migration of reinforcement learning. That is, without deploying neural networks, the traditional drive data writing method is used to drive the physical prototype, and the drive data comes from the output of the reinforcement learning model that has converged in the simulation. The advantage of this migration method is that it can quickly achieve migration, and the disadvantage is that the prototype will not have adaptability to the environment during walking, and can only adapt to the simulation condition where the drive data comes from.

[0135] 2) Offline network migration exploration

[0136] Offline network migration exploration, that is, deploying a static policy network on a controller that supports neural networks to achieve offline migration of reinforcement learning. That is, deploying the reinforcement learning model that has converged in the simulation on a physical prototype to achieve offline learning. The advantage of this migration method is that compared with the previous migration method, it truly realizes the migration of reinforcement learning. During the walking process, the prototype "looks up the table" according to the policy network to obtain driving data that is adaptive to the environment. The disadvantage is that it is limited by the static policy network and cannot realize the on-board online learning process. Therefore, when the prototype has a long task cycle or the environment suddenly changes, the prototype is no longer adaptable to the new environment.

[0137] 3) Online network migration exploration

[0138] Online network migration exploration, that is, deploying the policy network and the value function network on the controller that supports neural networks to achieve online migration. That is, the reinforcement learning model that has converged in the simulation is deployed on the physical prototype, and the policy network and the value function network are deployed at the same time to achieve online learning. The advantage of this migration method is that it realizes on-board online learning and can adapt to any environmental changes after deployment. It is suitable for tasks that are difficult to accurately model in simulation and the environment changes significantly over time. The disadvantage is that the online learning process may be affected by the performance of the on-board equipment, resulting in a slow learning process and a certain degree of damage to the hardware during the learning process.

[0139] 4) Migration effect verification

[0140] The training and correction of the reinforcement learning model through simulation and the implementation of the preset continuous periodic gait by hardware are limited to ensure the status of the simulation model and the robot prototype, and then the reinforcement learning model is written into the control board on the robot prototype, so that the robot prototype can realize autonomous motion control (autonomous walking and autonomous obstacle crossing) based on the onboard static strategy network without remote operation instructions and human intervention. After the early prototype has undergone early no-load tests and load tests, it can initially realize intelligent autonomous walking and obstacle crossing in the middle and late stages. This realization can play a certain exploratory and enlightening role in the engineering application of intelligent methods in space missions.

[0141] It can be seen that the method for autonomous emergence of motion trajectories of legged robots based on prior experience in the present invention is based on artificial intelligence methods. A method for autonomous emergence of motion trajectories of legged robots is designed in a digital simulation environment to realize autonomous generation of motion trajectories of robots without any human intervention and remote operation command control, verifying the feasibility of artificial intelligence methods in solving the motion control problems of legged robots, and providing a reference for intelligent and autonomous movement of space robots in the complex environment of the Moon and Mars. At the same time, the present invention establishes an intelligent agent control model based on the deep reinforcement learning method in machine learning theory and prior experience, which can reduce the training time of the intelligent agent model and improve the convergence performance of the model.

[0142] Of course, the present invention may have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may certainly make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for autonomous emergence of motion trajectories of a legged robot based on prior experience, characterized in that: The motion trajectory of the legged robot is obtained by using a fully autonomous emergence model or a semi-autonomous emergence model obtained through reinforcement learning training; The method for obtaining the motion trajectory of the fully autonomous emergence model is as follows: inputting the rotation angle of the motor on each leg of the legged robot at the current moment into the fully autonomous emergence model, obtaining the rotation angle of the motor on each leg of the legged robot at the next moment, so that the legged robot can autonomously complete the walking task; The method for obtaining the motion trajectory of the semi-autonomous emergence model is as follows: inputting the reference value of the motor rotation angle on each leg of the legged robot and the motor rotation angle on each leg of the legged robot at the current moment into the semi-autonomous emergence model, and obtaining the motor rotation angle on each leg of the legged robot at the next moment, so that the legged robot can complete the walking task semi-autonomously; The training method of the fully autonomous emergence model is: The roll angle, pitch angle, roll angular velocity, pitch angular velocity, position of the legged robot at the current moment, and the motor speed and motor torque on each leg of the legged robot are constructed as a state space; The motor torque on each leg of the legged robot at the current moment is constructed as the action space; Construct a reward function based on optimal patrol speed Or a reward function based on optimal joint energy consumption : in, is the current position of the legged robot, is the position of the legged robot at the last moment, τi is the motor torque on each leg of the legged robot, ωi is the motor speed of each leg of the legged robot, i =1,2,…, n , n is the number of legs of the legged robot; In the current terrain environment, different state spaces, action spaces and the motor rotation angles of each leg of the legged robot corresponding to different state spaces and action spaces at the next moment are obtained as training sample sets, and the fully autonomous emergence model is trained by reinforcement learning based on the training sample sets until the reward function based on the optimal patrol speed is obtained. Or a reward function based on optimal joint energy consumption Converge and obtain the final fully autonomous emergence model; The training method of the semi-autonomous emergence model is: The roll angle, pitch angle, roll angular velocity, pitch angular velocity, position of the legged robot at the current moment, and the motor speed and motor torque on each leg of the legged robot are constructed as a state space; The motor torque on each leg of the legged robot at the current moment is constructed as the action space; Construct a reward function based on optimal patrol speed and a reward function based on optimal joint energy consumption : in, is the current position of the legged robot, is the position of the legged robot at the last moment, τi is the motor torque on each leg of the legged robot, ωi is the motor speed of each leg of the legged robot, i =1,2,…, n , n is the number of legs of the legged robot; Under the current terrain environment, the difference between the motor rotation angle of each leg of the legged robot at the current moment or the next moment corresponding to different state spaces and action spaces is obtained, and the difference corresponding to different state spaces, action spaces and different state spaces and action spaces is used as a training sample set. Based on the training sample set, the semi-autonomous emergence model is trained by reinforcement learning until the reward function based on the optimal patrol speed is obtained. Or a reward function based on optimal joint energy consumption Converge and obtain the final semi-autonomous emergence model.

2. The method for autonomous emergence of motion trajectories of a legged robot based on prior experience as claimed in claim 1, characterized in that: The fully autonomous emergence model and the semi-autonomous emergence model are neural network models, and the fully autonomous emergence model or the semi-autonomous emergence model is deployed on the legged robot according to the following steps: By avoiding the hardware equipment required to deploy the neural network model, the driving data is directly injected into the controller of the legged robot to achieve pseudo transfer of reinforcement learning; After completing the pseudo transfer of reinforcement learning, a static policy network is deployed on the controller of the legged robot that supports the neural network model to achieve offline transfer of reinforcement learning; After completing the offline migration of reinforcement learning, the policy network and the value function network are deployed simultaneously on the controller of the legged robot that supports the neural network model to achieve online migration.

3. A method for autonomous emergence of motion trajectories of a legged robot based on prior experience as claimed in claim 1 or 2, characterized in that: The reinforcement learning algorithm architecture used in training the fully autonomous emergence model or the semi-autonomous emergence model is the Actor-Critic architecture, and the Actor network and the Critic network in the Actor-Critic architecture are updated using the PPO algorithm.

4. A method for autonomous emergence of motion trajectories of a legged robot based on prior experience as claimed in claim 1 or 2, characterized in that: The open source PyBullet physics engine is used to build a reinforcement learning model simulation environment for the legged robot, and OpenGL is used to display the graphics of the reinforcement learning model simulation environment. At the same time, TensorFlow is used to build a fully autonomous emergence model or semi-autonomous emergence model based on a neural network.

Images