Motion control method and system of foot type robot, electronic equipment and storage medium
By training the policy network model on the simulation platform, the problem of insufficient adaptability and movement diversity of foot robots in complex environments is solved, and its stability and reliability in the real world is improved.
Patent Information
- Application Number
- CN202410692161.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-30
- Publication Date
- 2024-11-08
AI Technical Summary
The existing intelligent control methods of foot-type robots have shortcomings in environmental adaptability and action diversity, resulting in low long-term stability and reliability in real-world environments.
By configuring the simulation environment on the preset simulation platform, performing model training to generate a policy network model, and using the model to perform motion control of foot robots, including training in the simulation environment and reinforcement learning training environment, generating an asymmetric actor - Critic policy network, used to determine the desired position and control amount of joints.
It realizes the efficient adaptability of foot-type robots in complex environments, improves environmental recognition and action diversity, reduces the demand for training data and computing resources, and saves human and financial resources.
Smart Images

Figure CN118915799A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of legged robots, and in particular to a motion control method and system, electronic equipment and storage medium for a legged robot. Background Art
[0002] A legged robot is a mobile robot with multiple legs that can move and perform tasks on various terrains. The design of a legged robot allows it to move freely on complex terrains, just like humans or animals.
[0003] Legged robots need to have good terrain adaptability, stability, freedom, balance control and other characteristics. Legged robots also need to have intelligent control capabilities to perform effective movements and tasks in unstructured environments.
[0004] However, the inventors of this application have found that the current intelligent control methods of legged robots have some problems in terms of environmental adaptability and action diversity. For example, current legged robots have technical problems such as limited environmental recognition capabilities, insufficient adaptive dynamic adjustment, and low action space diversity, which will make the long-term stability and reliability of legged robots relatively insufficient in real-world environments. Summary of the invention
[0005] According to one aspect of the present application, the present application provides a motion control method for a legged robot, comprising: configuring a preset simulation environment on a preset simulation platform, the preset simulation environment including a simulation environment and a simulation reinforcement learning training environment, the simulation environment including information of a simulated legged robot and simulation environment information, the simulation reinforcement learning training environment including preset task information, training parameter information and network structure information; performing model training in the preset simulation environment to obtain a policy network model; communicating the policy network model with the legged robot to obtain state observation information of the legged robot or sending control instructions to the legged robot; determining the expected joint position information corresponding to the state observation information based on the policy network model; determining the joint control amount based on the joint expected position information; and sending control instructions to the legged robot based on the joint control amount, so that the legged robot performs corresponding movements according to the joint control amount.
[0006] According to some embodiments of the present application, performing model training in a preset simulation environment to obtain a policy network model includes: configuring a policy network in a preset simulation environment, the policy network including a first network and a second network, the first network obtaining state observation information of a simulated legged robot based on environmental interaction, and the second network obtaining state observation information and privilege information of the simulated legged robot based on environmental interaction; generating motion actions based on the policy gradient of the policy network; obtaining updated state observation information and reward information of the simulated legged robot after performing motion actions based on environmental interaction, wherein the reward information is calculated based on the state observation information and a preset reward function; cyclically updating the state observation information and reward information of the simulated legged robot after performing motion actions until termination flag information is detected; calculating an updated reward and advantage of the policy network based on the state observation information, reward information and privilege information; calculating the policy network loss based on the updated reward and advantage, and updating the policy network to generate a policy network model.
[0007] According to some embodiments of the present application, the policy network is an asymmetric Actor-Critic policy network; the first network is an Actor network, and the second network is a Critic network.
[0008] According to some embodiments of the present application, the preset reward function includes a joint speed penalty, and the joint speed penalty is:
[0009]
[0010] In joint speed penalty:
[0011] dof_vel is the joint velocity, dof_vel_limits is the joint velocity limit, the value range of dof_vel_limits is [min_dof_vel_limit,1], hard_dof_vel_limits is the hard constraint of the joint velocity, and terrain_levels is the terrain level.
[0012] According to some embodiments of the present application, the preset reward function includes a posture penalty, and the posture penalty is:
[0013] 10(1+projected_gravity_z+projected_gravity_p.norm()*(projected_gravity_z<0))*(terrain_levels<0)
[0014] In the stance penalty:
[0015] projected_gravity_z is the projection of the Z axis of the current legged robot base coordinate system on the gravity direction (0,0,-1); projected_gravity_p.norm() is the norm of the projection vector of the X axis and Y axis of the current legged robot base coordinate system on the gravity direction (0,0,-1); terrain_levels is the terrain level.
[0016] According to some embodiments of the present application, the preset reward function includes an energy penalty, and the energy penalty is:
[0017] Σ(mechanical_energy+thermal_energy) / episode_length_buf.unsqueeze(1)
[0018] In Energy Penalty:
[0019] mechanical_energy is the mechanical energy consumed by the legged robot, which is the product of the joint speed and torque in the time period; thermal_energy is the thermal energy consumed by the legged robot, which is the absolute value of the product of the joint speed and torque in the time period; episode_length_buf is the length of the time period.
[0020] According to some embodiments of the present application, the preset reward function includes a foot clearance penalty, and the foot clearance penalty is:
[0021] {[(foot_position_z-measured_feet_heights.mean(dim=-1)
[0022] -desired_height).clip(min=-desired_height_p,max=0) / sigma] 2
[0023] *(foot_velocity.norm(dim=-1) / 0.5)}.mean(dim=-1)
[0024] *(terrain_levels>=-2)
[0025] In the foot clearance penalty:
[0026] foot_position_z is the component of the foot position of the legged robot along the Z-axis direction; measured_feet_heights.mean() is the average measured height of the terrain around the foot of the legged robot; desired_height is the desired height of the foot of the legged robot from the ground; sigma is the scaling factor; foot_velocity is the horizontal velocity of the foot of the legged robot; terrain_levels is the terrain level.
[0027] According to some embodiments of the present application, the preset reward function includes a collision penalty, and the collision penalty is:
[0028] ∑1.*(contact_forces_penalized>0.1).norm()*(projected_gravity_z<0)
[0029] In the collision penalty:
[0030] contact_forces_penalized is the magnitude of the collision force on all rigid bodies of the preset legged robot; projected_gravity_z is the projection of the Z axis of the base coordinate system of the legged robot in the direction of gravity (0,0,-1).
[0031] According to some embodiments of the present application, the preset reward function includes a foot contact penalty, and the foot contact penalty is:
[0032] |feet_velocity.mean()*(contact_forces_foot).mean()|
[0033] In the Foot Touchdown Penalty:
[0034] feet_velocity.mean() is the average value of the foot velocity of the legged robot; contact_forces_foot is the contact force of the foot of the legged robot.
[0035] According to some embodiments of the present application, the preset reward function includes a base height penalty, and the base height penalty is:
[0036] {1-[(base_height-target_base_height) / target_base_height) / sigma] 2 *(command_lin_vel_p.norm()<1}*(terrain_levels<0)
[0037] In the base height penalty:
[0038] base_height is the calculated base height; target_base_height is the expected base height; sigma is the scaling factor; command_lin_vel_p.morn() is the norm of the base horizontal velocity command signal; terrain_levels is the terrain level.
[0039] According to some embodiments of the present application, the preset reward function includes a standing position penalty, and the standing position penalty is:
[0040] (1-projection_error)*(terrain_levels==-2)+(1-projection_error)*(terrain_levels>=-1)*sigma
[0041] In the standing position penalty:
[0042] projection_error is the horizontal projection error of the vector from the shoulder to the foot of the legged robot; terrain_levels is the terrain level; sigma is the scaling factor.
[0043] According to some embodiments of the present application, the preset reward function includes a high degree of stability, and the high degree of stability is:
[0044] (base_lin_vel_z / tracking_sigma) 2
[0045] In high stability:
[0046] base_lin_vel_z is the component of the base linear velocity along the Z axis; tracking_sigma is the scaling factor;
[0047] The preset reward function also includes a horizontal line speed tracking penalty, which is:
[0048] leaky ReLU(1-lin vel error / tracking sigma, 0.1)
[0049] Among them, leaky_ReLU() is the activation function; lin_vel_error is the linear velocity tracking error;
[0050] The preset reward function also includes a vertical angular velocity tracking penalty, which is:
[0051] leaky ReLU(1-ang vel error / tracking sigma, 0.1)
[0052] Among them, ang_vel_error is the angular velocity tracking error.
[0053] According to another aspect of the present application, the present application provides a motion control system for a legged robot, the motion control system comprising a simulation environment configuration module, a model training module and an entity deployment module. The simulation environment configuration module is used to configure a preset simulation environment on a preset simulation platform, the preset simulation environment comprising a simulation simulation environment and a simulation reinforcement learning training environment, the simulation simulation environment comprising information of a simulated legged robot and simulation environment information of the legged robot, and the simulation reinforcement learning training environment comprising preset task information, training parameter information and network structure information. The model training module is used to perform model training in a preset simulation environment to obtain a policy network model. The entity deployment module is used to connect the policy network model to the legged robot in communication to obtain state observation information of the legged robot or send control instructions to the legged robot; the entity deployment module determines the joint expected position information corresponding to the state observation information based on the policy network model, determines the joint control amount based on the joint expected position information, and sends a control instruction to the legged robot based on the joint control amount, so that the legged robot performs corresponding movements according to the joint control amount.
[0054] According to another aspect of the present application, an electronic device is provided, which includes: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors can implement the motion control method described above.
[0055] According to another aspect of the present application, a non-volatile computer-readable storage medium is provided, wherein a computer program is stored on the storage medium, and the computer program can implement the motion control method described above.
[0056] The motion control method for a legged robot provided in the present application does not require real data. The strategy network can be trained through a simulation environment. By utilizing the simulation space to obtain all data for the intelligent control of the legged robot, a large amount of training data and computing resources can be avoided, thereby saving a large amount of manpower and financial resources. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0058] Figure 1 A flow chart showing a motion control method according to an embodiment of the present application;
[0059] Figure 2 Another flow chart showing the motion control method according to an embodiment of the present application;
[0060] Figure 3 Another flow chart showing the motion control method according to an embodiment of the present application;
[0061] Figure 4 A schematic structural diagram of a motion control system according to an embodiment of the present application is shown.
[0062] Description of reference numerals:
[0063] Motion control system 1; simulation environment configuration module 10; model training module 20; entity deployment module 30. DETAILED DESCRIPTION
[0064] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this application will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.
[0065] The described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of these specific details, or other modes, components, materials, devices, etc. may be adopted. In these cases, known structures, methods, devices, implementations, materials or operations will not be shown or described in detail.
[0066] In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.
[0067] The terms "first", "second" and the like in the specification and claims of this application and the above drawings are used to distinguish different objects rather than to describe a specific order.
[0068] The following is a clear and complete description of the technical solution of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.
[0069] Legged robots can include bipedal robots (such as humanoid robots) and quadrupedal robots (such as robot dogs, wheeled robots, etc.). Traditional legged robots are mainly controlled based on preset motion patterns and rigid control algorithms. For example, the legged robot control system drives the movement of the legged robot based on a walking strategy with preset rules, predefined gaits and motion sequences, and can determine the motion control data such as step length, step height and step frequency of the legged robot during walking based on the gait pattern and speed control algorithm.
[0070] With the development of science and technology, for some legged robot control systems, environmental information data can be measured by sensors (such as laser radar, induction camera, mechanical sensor, etc.) to adjust the walking strategy of the legged robot. For example, the environmental information data measured by the sensor is obtained, and the current environmental terrain or obstacles are determined according to the environmental information data to perform real-time terrain analysis and dynamic stability control, so that the legged robot can adapt to different ground conditions (such as uneven or smooth ground).
[0071] Furthermore, in some legged robot control systems, machine learning methods can be used to train legged robots to better cope with more complex environments. By pre-training legged robots in simulated environments, legged robots can cope with various terrains and obstacles before entering the real world environment.
[0072] However, the inventors found that the control method of the above-mentioned legged robot still has some problems:
[0073] 1. Limited environmental recognition ability
[0074] Traditional legged robot control methods based on preset rules or relying on sensors have limited adaptability in unknown or complex environments.
[0075] For example, when a legged robot faces unknown or complex environments such as sensor damage, irregular terrain, slippery surfaces, soft sand, etc., it is difficult for the legged robot to accurately identify and adapt, which can cause the legged robot to have difficulty walking in actual application environments, and may even cause it to get stuck or fall.
[0076] 2. Training legged robots based on machine learning methods requires a large amount of training data and computing resources, and the training method is relatively complicated.
[0077] 3. Insufficient adaptive dynamic adjustment
[0078] Current legged robot control methods have certain limitations in quickly adapting to environmental changes (including sudden changes in weather and ground conditions, etc.).
[0079] For example, current control methods for legged robots generally require a relatively stable and predictable environment to maintain optimal operating performance. In a complex, dynamically changing environment, the reaction speed and adjustment mechanism of the legged robot are difficult to quickly adapt to changes in the current environment, resulting in a decrease in the movement efficiency and stability of the legged robot.
[0080] 4. Low diversity in action space
[0081] The current control methods for legged robots generally set a limited set of actions. The limited action selection makes it impossible for legged robots to effectively adjust their walking strategies when dealing with non-standard terrain or complex obstacles.
[0082] For example, legged robots may not be able to adapt to traversing narrow spaces, climbing steep slopes, or passing through obstacles of varying sizes.
[0083] According to one aspect of the present application, the present application provides a motion control method for a legged robot. Figure 1 A flow chart showing the motion control method of the present application embodiment is shown as follows: Figure 1 As shown, the motion control method includes steps S100 to S600. Exemplarily, the motion control method can be executed by a motion control system of a legged robot.
[0084] In step S100, the motion control system configures a preset simulation environment on a preset simulation platform, where the preset simulation environment includes a simulation environment and a simulation reinforcement learning training environment.
[0085] The motion model training of the legged robot provided in this application is implemented based on a simulation platform.
[0086] According to an exemplary embodiment, the motion control system creates a preset simulation environment based on a reinforcement learning environment (IsaacGym) and a URDF (Unified Robot Description Format) file on a preset simulation platform. The preset simulation environment includes a simulation environment and a simulation reinforcement learning training environment.
[0087] On the one hand, the simulation environment includes information of the legged robot, the simulated legged robot and the simulation environment information.
[0088] For example, the motion control system responds to the user's instructions, loads the information of the legged robot simulation (such as the body configuration information of the legged robot) and loads the corresponding physical engine according to the URDF file of the legged robot; and according to the actual needs of the user, the motion control system sets the simulation environment information, such as terrain information, map scale information, collision type information and other environmental parameters.
[0089] On the other hand, the simulated reinforcement learning training environment includes preset task information, training parameter information and network structure information.
[0090] For example, the motion control system responds to the user's instructions and parses the preset task information. For example, the preset task information can be command line parameters set by the user, such as the name of the reinforcement learning task, the name of the current experiment, whether to continue training based on the existing model, and the specified device.
[0091] The training parameter information may include reward function weight, number of agents, random sampling instruction range, action course information, maximum number of training iterations, etc.
[0092] The network structure information may include input and output type and dimension, number of hidden layers and structure, activation function and other information.
[0093] By creating a preset simulation environment based on a preset simulation platform in the motion control system, a training environment is provided for the motion model training of the legged robot, so that the motion model training of the legged robot can be performed according to different task types, different environmental information, URDF files of different legged robots and different network structures, thereby making the motion control method of the legged robot provided in the present application scalable and easy to debug.
[0094] In step S200, the motion control system performs model training in a preset simulation environment to obtain a strategy network model.
[0095] For example, the motion control system performs simulation training according to the different training stages, task types and environmental changes of the legged robot (such as weather differences, terrain differences, etc.) to obtain a policy network model. According to the policy network configured in the policy network model, after model training, execution actions (such as walking, jumping and running) covering the motion space of the legged robot can be generated according to the policy gradient.
[0096] Exemplarily, the training phase of a legged robot may include a standing phase and a multi-task walking phase. The standing phase mainly includes the stable standing task of the legged robot in different initial states and different environments (weather, terrain); the multi-task walking phase mainly includes the walking, jumping and running tasks of the legged robot in different weather environments, different terrain environments and different speeds.
[0097] Figure 2 Another flow chart of the motion control method according to an embodiment of the present application is shown. Figure 2 As shown, step S200 may also include steps S210-S260.
[0098] In step S210, the motion control system configures a policy network in a preset simulation environment. The policy network includes a first network and a second network. The first network obtains state observation information of the simulated legged robot based on environmental interaction, and the second network obtains state observation information and privilege information of the simulated legged robot based on environmental interaction.
[0099] Optionally, the policy network is an asymmetric Actor-Critic policy network. The first network is an Actor network, and the second network is a Critic network.
[0100] For example, the policy network is generated by combining the Actor network and the Critic network. The Actor network is responsible for generating action strategies, which is the main part of the motion control model; the Critic network is responsible for evaluating the quality of the policy network in order to better update the Actor network.
[0101] Exemplarily, the Actor network can be constructed using a multi-layer perceptron (MLP) neural network. The Actor network includes four hidden layers, and the dimensions of the four hidden layers are 256, 128, 64, and 32, respectively. The input signal of the Actor network is the state observation information generated by the interaction between the legged robot and the environment. The state observation information includes, but is not limited to, joint state information, base body angular velocity information, command information, and action information. The output signal of the Actor network is the action control information of the legged robot, which can be converted into the expected position information of the legged robot joints, so that the joint control amount can be calculated according to the expected position information.
[0102] The critic network can also be built using a multi-layer perceptron (MLP) neural network. The critic network includes four hidden layers, and the dimensions of the four hidden layers are 256, 128, 64, and 32, respectively. The input signal of the critic network includes state observation information and privileged information of the legged robot. The privileged information of the legged robot includes, but is not limited to, body height information, foot end measurement height information, base linear velocity information, and external velocity information that propels the legged robot.
[0103] Since privileged information is state observation information that a legged robot cannot directly obtain, privileged information can help the Critic network estimate the value function and better guide the Actor network to update the strategy. The output signal of the Critic network is state value data, which can calculate the advantage, reward and loss function, thereby updating the entire strategy network.
[0104] According to an example embodiment, before step S210, the training data may be initialized, and the training data includes data such as learning iterations numbers and episode length information.
[0105] Exemplarily, the motion control system realizes data interaction between the simulated legged robot and the environment through environmental interaction, updates the motion state of the simulated legged robot according to the output action and generates corresponding state feedback information to generate state observation information.
[0106] On the one hand, the state observation information can be used to design the reward function, and on the other hand, it can be used by the policy network to generate motion control parameters for simulated legged robots.
[0107] Exemplarily, in step S210, the motion control system transmits the state observation information generated based on the environment interaction in the previous iteration process into the first network (such as the Actor network), and transmits the state observation information and privileged information into the second network (such as the Critic network) to form an asymmetric Actor-Critic strategy network.
[0108] In step S220, the motion control system generates motion actions based on the policy gradient of the policy network.
[0109] In step S230, the motion control system obtains updated state observation information and reward information after the simulated legged robot performs motion actions based on the environment interaction, wherein the reward information is calculated based on the state observation information and a preset reward function.
[0110] For example, the policy gradient based on the policy network generates motion actions, so that after the simulated legged robot performs the corresponding motion actions, the updated state observation information and reward information are collected again based on the environment interaction.
[0111] The preset reward function can be a user-defined reward function. The reward function can define the quality and goodness of the quadruped's behavior in a specific environment or task scenario. By designing the reward function, the optimization process of the policy network and the performance of the policy network on specific tasks can be affected.
[0112] In step S240, the motion control system cyclically updates the state observation information and reward information after the simulated legged robot performs the motion action until a termination mark is detected.
[0113] For example, within a complete interaction process (episode), the state observation information and the accumulated reward information are continuously updated until the interaction process ends. If the termination flag information is detected, the cyclic update is stopped.
[0114] The termination flag information can be customized by the user according to the needs. For example, the termination flag information can be that the preset number of iterations are completed, or the strategy network reaches the customized target accuracy and then stops the iteration, which is not limited in this application.
[0115] In step S250, the motion control system calculates the updated reward and advantage of the policy network based on the state observation information and the reward information.
[0116] In step S260 , the motion control system calculates the policy network loss according to the update reward and the advantage, and updates the policy network to generate a policy network model.
[0117] For example, the motion control system calculates updated rewards and advantages based on the input (such as state observation information and reward information) of the second network (such as the Critic network). And the motion control system calculates the policy network loss based on the rewards and advantages, optimizes and updates the policy network weights by maximizing the expected return, maximizing the policy entropy, and minimizing the value function loss, and finally generates a policy network model.
[0118] Figure 3 Another flow chart of the motion control method according to an embodiment of the present application is shown.
[0119] Alternatively, if Figure 3 As shown, in step S260, the motion control system updates the strategy network including steps S261-S262.
[0120] In step S261, the motion control system uses the current strategy network to forward propagate the state observation data to obtain action information, value function estimation information, action mean information, action standard deviation, entropy and other information.
[0121] In step S262, the motion control system adjusts the learning rate, specifies the target KL divergence (Kullback-Leibler divergence, relative entropy) and calculates the KL divergence between the current policy network and the old policy network, and adaptively adjusts the learning rate according to the calculation result.
[0122] In step S263, the motion control system calculates the policy network loss, where the network loss includes but is not limited to the agent loss, policy entropy loss and value function loss of the policy network.
[0123] In step S264, the motion control system uses the Adam optimizer to optimize and update the parameters of the policy network.
[0124] In step S300, the motion control system connects the policy network model to the legged robot for communication to obtain state observation information of the legged robot or send control instructions to the legged robot.
[0125] For example, the trained policy network model is configured in the onboard computer of the legged robot, and the legged robot is connected to the UDP (User Datagram Protocol) of the policy network model, so that the policy network model can receive the state observation information of the legged robot and send control instructions to the legged robot.
[0126] In step S400, the motion control system determines the expected joint position corresponding to the state observation information based on the strategy network model.
[0127] For example, the motion control system configures the corresponding strategy network model, determines the task mode information, control parameter information and control mode information. Exemplarily, the control mode information includes a position control mode and a torque control mode.
[0128] The state observation information of the legged robot is received through the policy network model, and the Actor network is applied to perform model reasoning. The policy probability is determined according to the input signal of the policy network, the action information is selected according to the policy probability, and the action information is converted into the expected position information of the joints of the legged robot.
[0129] In step S500, the motion control system determines the joint control amount according to the joint expected position information.
[0130] According to an exemplary embodiment, the motion control system calculates the joint control amount according to the joint expected position information transmitted by the strategy network. In different control modes, there are different control amount calculation methods.
[0131] For example, in the case of position control mode, proportional adjustment control is performed on the joint position; in the case of torque control mode, proportional-differential adjustment control is performed on the joint position.
[0132] In step S600, a control instruction is sent to the legged robot based on the joint control amount, so that the legged robot performs corresponding movements according to the joint control amount.
[0133] For example, the motion control system sends the calculated joint control amount to the joint motors of the legged robot so that the legged robot performs corresponding movements.
[0134] Through the above-mentioned example embodiments, the motion control method of the legged robot provided by the present application does not require real data. The strategy network can be trained through simulation environment training. By utilizing the simulation space to obtain all the data for the intelligent control of the legged robot, a large amount of training data and computing resources can be avoided, thereby saving a lot of manpower and financial resources.
[0135] During the model training process, the motion control method of the legged robot provided in the present application can provide a strategy network model that can cover the motion space of the legged robot according to different training stages, task types and terrain differences, and provide a legged robot control model with both environmental generalization capabilities and task generalization capabilities.
[0136] The motion control method of a legged robot provided in the present application can design corresponding reward functions according to different stages of training, and through an asymmetric Actor-critic strategy network, it can adapt to the ever-changing real environment according to a set of trained control strategy data, and can realize the motion control of the legged robot driven by strategy data, thereby improving the environmental adaptability of the legged robot.
[0137] The algorithm reasoning complexity of the motion control method of the legged robot provided in this application is low, the reasoning frequency can reach more than 100HZ, and the simple structure based on the Actor-critic policy network can achieve rapid deployment and real-time calculation of joint control quantity. When the external environment of the legged robot suddenly changes, there is no need to switch the policy network, and the policy network model can adapt to all terrains, weather and other environments. The legged robot can complete various actions such as standing, walking and running through the policy network model, and can perform robust control in various terrains, weather and other environments.
[0138] According to an exemplary embodiment, the preset reward function can be divided into a standing phase reward function and a multi-task phase reward function according to different task phases. The standing phase reward function is a subset of the multi-task phase reward function.
[0139] terrain_levels terrain level is a key value of the preset terrain course. Different values of terrain_levels correspond to the learning stage and task status of the legged robot in different terrain environments.
[0140] For example, when the data is initialized, terrain_levels = -2, which means that the legged robot is standing on flat ground. In the process of updating the terrain course, if the legged robot walks far enough from the origin, the terrain level is increased by one level, that is, terrain_levels = -1. If the distance walked by the legged robot from the origin is short, the terrain level is reduced by one level. terrain_levels ≥ 0 means that the legged robot has begun to learn specific terrain tasks, such as potholes or stepped terrain. terrain_levels = max_terrain_levels means that the legged robot has completed the highest level of tasks and will then be randomly placed in a new terrain level.
[0141] By setting the terrain level and terrain tasks according to the performance of the legged robot, the legged robot can learn the terrain courses better and faster.
[0142] Optionally, the preset reward function includes a joint speed penalty. The joint speed penalty can represent that the legged robot maintains a moderate joint speed in a standing state.
[0143] The joint velocity penalty is:
[0144]
[0145] In joint speed penalty:
[0146] dof_vel is the joint velocity, dof_vel_limits is the joint velocity limit, the value range of dof_vel_limits is [min_dof_vel_limit,1], hard_dof_vel_limits is the hard constraint of the joint velocity, and terrain_levels is the terrain level.
[0147] dof_vel_limits takes a random value in the range of [min_dof_vel_limit,1], which is a hard limit on the joint velocity parameter of the legged robot. If the hard limit is exceeded, the reward item will be truncated. The hard constraint of the joint velocity is a hard constraint on the joint velocity determined by the hardware or motor of the legged robot itself.
[0148] Exemplarily, (terrain_levels == -2) means that when the terrain level is -2 (when the legged robot is standing), the joint speed penalty takes effect.
[0149] Optionally, the preset reward function includes a posture penalty, which can be used to characterize whether the legged robot maintains the correct orientation of its body posture when standing or running on flat ground. The posture penalty can be:
[0150] 10(1+projected_gravity_z+projected_gravity_p.norm()*(projected_gravity_z<0))*(terrain_levels<0)
[0151] In the stance penalty:
[0152] projected_gravity_z is the projection of the Z axis of the base coordinate system of the legged robot on the direction of gravity (0,0,-1); projected_gravity_p.norm() is the norm of the projection vectors of the X axis and Y axis of the base coordinate system of the legged robot on the direction of gravity (0,0,-1). When the vertical component is less than zero, it indicates that the legged robot is flipped upside down. At this time, the horizontal component of gravity is penalized to make the legged robot complete the flip as soon as possible; terrain_levels is the terrain level.
[0153] Exemplarily, (terrain_levels<0) means that when the terrain level is less than zero (when the legged robot stands and walks on flat ground), the posture penalty takes effect.
[0154] Optionally, the preset reward function includes an energy penalty, which indicates that the average energy consumption of the legged robot in each step time period in each environment is small, and the energy penalty can be:
[0155] ∑(mechanical_energy+thermal_energy) / episode_length_buf.unsqueeze(1)
[0156] In Energy Penalty:
[0157] mechanical_energy is the mechanical energy consumed by the legged robot during the time period; thermal_energy is the thermal energy consumed by the legged robot during the time period; episode_length_buf is the length of the time period.
[0158] Exemplarily, the energy penalty calculates the average energy consumption of the legged robot during the entire movement period.
[0159] Mechanical energy mechanical_energy is related to the product of torque torques and joint velocity dof_vel; thermal energy thermal_energy is related to the absolute value of the product of torque torques and joint velocity dof_vel. For example:
[0160] mechanical_energy=(dof_vel*torques).clip(min=0)
[0161] thermal_energy=|dof_vel*torques|.clip(max=0)
[0162] Optionally, the preset reward function includes a foot clearance penalty, which represents the desire that the foot of the legged robot can maintain a stable foot clearance height when walking on different terrains when the terrain or ground height changes. The foot clearance penalty can be:
[0163] {[(foot_positionz-measured_feet_heights.mean(dim=-1)
[0164] -desired_height).clip(min=-desired_height_p,max=0) / sigma] 2
[0165] *(foot_velocity.norm(dim=-1) / 0.5)}.mean(dim=-1)
[0166] *(terrain_levels>=-2)
[0167] In the foot clearance penalty:
[0168] foot_position_z is the component of the foot position of the legged robot in the Z-axis direction; measured_feet_heights.mean() is the average measured height of the terrain around the foot of the legged robot; desired_height is the desired height of the foot of the legged robot from the ground; sigma is the scaling factor; foot_velocity is the horizontal velocity of the foot of the legged robot; terrain_levels is the terrain level.
[0169] For example, the ().clip operation means clipping the value in the brackets to the set range; the ().clip operation means that when the terrain level is greater than -2 (when the legged robot is walking), the foot clearance penalty takes effect. The foot clearance penalty can be used to penalize the horizontal speed of the foot end so that the legged robot can remain stable and reduce the friction between the foot end and the ground.
[0170] Optionally, the preset reward function includes a collision penalty, which indicates that when the legged robot is turned upside down on the base, collisions between rigid bodies can be reduced to reduce body wear and energy consumption. The collision penalty can be:
[0171] ∑1.*(contact_forces_penalized>0.1).norm()*(projected_gravity_z<0)
[0172] In the collision penalty:
[0173] contact_forces_penalized is the magnitude of the collision force applied to the preset legged robot rigid body; projected_gravity_z is the projection of the Z axis of the legged robot base coordinate system in the direction of gravity (0,0,-1).
[0174] Exemplarily, (projected_gravity_z<0) means that the collision penalty takes effect when the back of the legged robot base faces upward.
[0175] Optionally, the preset reward function includes a foot-end touchdown penalty, which represents the relationship between the foot-end velocity change and the foot-end contact force to prevent hardware loss caused by excessive foot-end stepping on the ground, so that the legged robot can land smoothly. The foot-end touchdown penalty can be:
[0176] |feet_velocity.mean()*(contact_forces_foot).mean()|
[0177] In the Foot Touchdown Penalty:
[0178] feet_velocity.mean() is the average value of the foot velocity of the legged robot; contact_forces_foot is the contact force of the foot of the legged robot.
[0179] Optionally, the preset reward function includes a base height penalty, which represents the desire for the legged robot to reduce the error between the base height and the expected height. The base height penalty can be:
[0180] {1-[(base_height-target_base_height) / target_base_height) / sigma] 2
[0181] *(command_lin_vel_p.norm()<1}*(terrain_levels<0)
[0182] In the base height penalty:
[0183] base_height is the calculated base height; target_base_height is the desired base height; sigma is the scaling factor; command_lin_vel_p is the base horizontal velocity command signal; terrain_levels is the terrain level.
[0184] Exemplarily, (terrain_levels<0) means that when the terrain level is less than zero, the base height penalty takes effect.
[0185] Optionally, the preset reward function includes a standing position penalty, which indicates that the legged robot can make the foot end and the shoulder on the same vertical projection line when standing, thereby quickly learning the relative position of the standing legs. The standing position penalty can be:
[0186] (1-projection_error)*(terrain_levels==-2)+(1-projection_error)
[0187] *(terrain_levels>=-1)*sigma
[0188] In the standing position penalty:
[0189] projection_error is the horizontal projection error of the direction vector from the shoulder to the foot of the legged robot; terrain_levels is the terrain level; sigma is the scaling factor.
[0190] Exemplarily, the standing position penalty is a full calculation error when the terrain level is -2 (when the legged robot is standing); the standing position penalty has an error scale sigma of 0.1 when the terrain level is greater than -2 (when the legged robot is walking).
[0191] Optionally, the preset reward function includes height stability, which indicates that the legged robot is expected to reduce the height change in the vertical direction of the base so as to walk more smoothly. The height stability can be:
[0192] (base_lin_vel_z / tracking_sigma) 2
[0193] In high stability:
[0194] base_lin_vel_z is the component of the base linear velocity along the Z axis; tracking_sigma is the scaling factor used to adjust the scale of the error.
[0195] Optionally, the preset reward function includes a horizontal linear velocity tracking penalty, and the horizontal linear velocity tracking penalty represents that the legged robot is expected to reduce the error between the horizontal linear velocity in the horizontal direction of the base and the expected linear velocity. The horizontal linear velocity tracking penalty is:
[0196] leaky_ReLU(1-lin_vel_error / tracking_sigma, 0.1)
[0197] Among them, leaky_ReLU() is the activation function used to correct the angular velocity error to ensure that the reward will not completely disappear even if the error is large; lin_vel_error is the linear velocity tracking error.
[0198] Optionally, the preset reward function also includes a vertical angular velocity tracking penalty, which represents the desire for the legged robot to reduce the error between the vertical angular velocity of the base and the desired angular velocity. The vertical angular velocity tracking penalty is:
[0199] leaky ReLU(1-ang vel error / tracking sigma, 0.1)
[0200] Among them, ang_vel_error is the angular velocity tracking error.
[0201] According to another aspect of the present application, the present application provides a motion control system for a legged robot. Figure 4 A schematic diagram showing the structure of the motion control system of an embodiment of the present application is shown in FIG. Figure 4 As shown, the motion control system 1 includes a simulation environment configuration module 10 , a model training module 20 and an entity deployment module 30 .
[0202] According to an exemplary embodiment, the simulation environment configuration module 10 is used to configure a preset simulation environment on a preset simulation platform, wherein the preset simulation environment includes a simulation environment and a simulation reinforcement learning training environment. The simulation environment includes information of a simulated legged robot and simulation environment information of a legged robot, and the simulation reinforcement learning training environment includes preset task information, training parameter information, and network structure information.
[0203] The model training module 20 is used to perform model training in a preset simulation environment to obtain a policy network model.
[0204] The entity deployment module 30 is used to connect the policy network model to the legged robot for communication so as to obtain the state observation information of the legged robot or send control instructions to the legged robot; the entity deployment module determines the expected joint position information corresponding to the state observation information based on the policy network model, determines the joint control amount according to the expected joint position information, and sends control instructions to the legged robot based on the joint control amount, so that the legged robot performs corresponding movements according to the joint control amount.
[0205] According to an exemplary embodiment, the motion control system provided by the present application is used to execute the motion control method as described above. Specifically, the motion control method has been described in detail above and will not be repeated here.
[0206] According to another aspect of the present invention, an electronic device is provided, which includes: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors can implement the motion control method described above.
[0207] According to another aspect of the present invention, a non-volatile computer-readable storage medium is provided, wherein a computer program is stored on the storage medium, and the computer program can implement the motion control method described above.
[0208] Finally, it should be noted that the above is only a preferred embodiment of the present application and is not intended to limit the present application. Although the present application is described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions of the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.
Claims
1. A motion control method for a legged robot, It is characterized in that include: A preset simulation environment is configured on a preset simulation platform, wherein the preset simulation environment includes a simulation environment and a simulation reinforcement learning training environment, wherein the simulation environment includes information of a simulated footed robot and simulation environment information of the legged robot, and the simulation reinforcement learning training environment includes preset task information, training parameter information, and network structure information; Performing model training in the preset simulation environment to obtain a policy network model; Connecting the strategy network model to the legged robot in communication to obtain state observation information of the legged robot or to send control instructions to the legged robot; Determine the joint expected position information corresponding to the state observation information based on the strategy network model; Determining a joint control amount according to the joint expected position information; A control instruction is sent to the legged robot based on the joint control amount, so that the legged robot performs corresponding movements according to the joint control amount.
2. The control method according to claim 1, It is characterized in that The performing model training in the preset simulation environment to obtain the policy network model includes: Configure a policy network in the preset simulation environment, the policy network comprising a first network and a second network, the first network acquiring state observation information of the simulated legged robot based on environment interaction, and the second network acquiring state observation information and privilege information of the simulated legged robot based on environment interaction; Generate motion actions based on the policy gradient of the policy network; Acquire state observation information and reward information updated after the simulated legged robot performs the motion action based on the environment interaction, wherein the reward information is calculated based on the state observation information and a preset reward function; cyclically updating the state observation information and reward information of the simulated legged robot after executing the motion action until the termination mark information is detected; Calculating an updated reward and advantage of the policy network based on the state observation information, the reward information, and the privilege information; The policy network loss is calculated according to the updated reward and the advantage, and the policy network is updated to generate the policy network model.
3. The control method according to claim 2, It is characterized in that The strategy network is an asymmetric Actor-Critic strategy network; The first network is an Actor network, and the second network is a Critic network.
4. The control method according to claim 2, It is characterized in that The preset reward function includes a joint speed penalty, which is: In the joint velocity penalty: dof_vel is the joint velocity, dof_vel_limits is the joint velocity limit, the value range of dof_vel_limits is [min_dof_vel_limit,1], hard_dof_vel_limits is the hard constraint of the joint velocity, and terrain_levels is the terrain level.
5. The control method according to claim 2, It is characterized in that The preset reward function includes a posture penalty, and the posture penalty is: 10(1+projected gravity z+projected gravity p.norm()*(projected gravity z<0))*(terrain levels<0) In the posture penalty: projected_gravity_z is the projection of the Z axis of the base coordinate system of the legged robot in the direction of gravity (0,0,-1); projected_gravity_p.norm() is the norm of the projection vectors of the X axis and Y axis of the base coordinate system of the legged robot in the direction of gravity (0,0,-1); terrain_levels is the terrain level.
6. The control method according to claim 2, It is characterized in that The preset reward function includes an energy penalty, and the energy penalty is: ∑(mechanical energy+thermal energy) / episode length buf.unsqueeze(1) In the energy penalty: mechanical_energy is the mechanical energy consumed by the legged robot, which is the product of the joint speed and torque in the time period; thermal_energy is the thermal energy consumed by the legged robot, which is the absolute value of the product of the joint speed and torque in the time period; episode_length_buf is the length of the time period.
7. The control method according to claim 2, It is characterized in that The preset reward function includes a foot clearance penalty, which is: {[(foot_position_z-measured_feet_heights.mean(dim=-1)-desired_height).clip(min=-desired_height_p,max=0) / sigma] 2 *(foot_velocity.norm(dim=-1) / 0.5)}.mean(dim=-1)*(terrain_levels>=-2) In the foot clearance penalty: foot_position_z is the component of the foot position of the legged robot along the Z-axis direction; measured_feet_heights.mean() is the measured height average of the terrain around the foot of the legged robot; desired_height is the desired height of the foot of the legged robot from the ground; sigma is the scaling factor; foot_velocity is the horizontal velocity of the foot of the legged robot; terrain_levels is the terrain level.
8. The control method according to claim 2, It is characterized in that The preset reward function includes a collision penalty, which is: ∑1.*(contact_forces_penalized>0.1).norm()*(projected_gravity_z<0) In the collision penalty: contact_forces_penalized is the preset collision force on all rigid bodies of the legged robot; projected_gravity_z is the projection of the Z axis of the base coordinate system of the legged robot in the direction of gravity (0,0,-1).
9. The control method according to claim 2, It is characterized in that The preset reward function includes a foot contact penalty, and the foot contact penalty is: |feet_velocity.mean()*(contact_forces_foot).mean()| In the foot contact penalty: feet_velocity.mean() is the average value of the foot velocity of the foot-type robot; contact_forces_foot is the contact force of the foot-type robot.
10. The control method according to claim 2, It is characterized in that The preset reward function includes a base height penalty, wherein the base height penalty is: {1-[(base_height-target_base_height) / target_base_height) / sigma] 2 *(command_lin_vel_p.norm()<1}*(terrain_levels<0) In the base height penalty: base_height is the calculated base height; target_base_height is the desired base height; sigma is the scaling factor; command_lin_vel_p is the base horizontal velocity command signal; terrain_levels is the terrain level.
11. The control method according to claim 2, It is characterized in that The preset reward function includes a standing position penalty, and the standing position penalty is: (1-projection_error)*(terrain_levels==-2)+(1-projection_error)*(terrain_levels>=-1)*sigma In the said standing position penalty: projection_error is the projection error of the vector from the shoulder to the foot of the legged robot in the horizontal direction; terrain_levels is the terrain level; sigma is the scaling factor.
12. The control method according to claim 2, It is characterized in that The preset reward function includes a high degree of stability, which is: (base_lin_vel_z / tracking_sigma) 2 In the described high stability: base_lin_vel_z is the component of the base linear velocity along the Z axis; tracking_sigma is the scaling factor; The preset reward function also includes a horizontal line speed tracking penalty, and the horizontal line speed tracking penalty is: leaky_ReLU(1-lin_vel_error / tracking_sigma, 0.1) Among them, leaky_ReLU() is the activation function; lin_vel_error is the linear velocity tracking error; The preset reward function also includes a vertical angular velocity tracking penalty, and the vertical angular velocity tracking penalty is: leaky_ReLU(1-ang_vel_error / tracking_sigma, 0.1) Among them, ang_vel_error is the angular velocity tracking error.
13. A motion control system for a legged robot. It is characterized in that The motion control system is used to execute the motion control method according to any one of claims 1 to 12, and the motion control system includes: A simulation environment configuration module, used to configure a preset simulation environment on a preset simulation platform, wherein the preset simulation environment includes a simulation environment and a simulation reinforcement learning training environment, wherein the simulation environment includes information of a simulated footed robot and simulation environment information of a legged robot, and the simulation reinforcement learning training environment includes preset task information, training parameter information, and network structure information; A model training module, used to perform model training in the preset simulation environment to obtain a policy network model; An entity deployment module is used to communicate the policy network model with the legged robot to obtain state observation information of the legged robot or send control instructions to the legged robot; the entity deployment module determines the joint expected position information corresponding to the state observation information based on the policy network model, determines the joint control amount according to the joint expected position information, and sends a control instruction to the legged robot based on the joint control amount, so that the legged robot performs corresponding movements according to the joint control amount.
14. An electronic device, It is characterized in that include: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the motion control method as described in any one of claims 1-12.
15. A non-volatile computer-readable storage medium having a computer program stored thereon, It is characterized in that The computer program implements the motion control method as described in any one of claims 1-12.
Citation Information
Cited By
Quadruped robot control method and system based on reinforcement learning and composite model
CN119200619A
Control method and system of quad-wheeled legged robot based on reinforcement learning and composite model
CN119200619B
Operation control model training method and robot operation control method and system
CN119748447A
Complex terrain self-adaptive motion control method and system for double-wheel-foot robot
CN120233686A
A complex terrain adaptive motion control method and system for a two-wheeled foot robot
CN120233686B
