Digital twin assisted crocodile robot gait control deep reinforcement learning method
By combining digital twin technology and edge devices, a crocodile-like robot twin system model was constructed and pre-trained and real-time trained. This solved the problems of low sample acquisition efficiency and slow training speed of deep reinforcement learning in gait control of crocodile-like robots, and realized efficient gait control strategy generation.
Patent Information
- Application Number
- CN202411745777.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-02
- Publication Date
- 2025-10-24
- Estimated Expiration
- 2044-12-02
AI Technical Summary
Existing deep reinforcement learning methods suffer from low sample acquisition efficiency and slow training speed in gait control of crocodile-like robots. Simulation strategies cannot be directly applied to real robots, and the robots themselves have limited computing and storage capabilities, resulting in response delays.
A crocodile-like robot twin system model was constructed using digital twin technology, and pre-training and real-time training were performed on edge devices. Gait control was achieved using an Actor-Critic network architecture, and data was collected in parallel through multiple processes. A reward function was designed to improve training efficiency.
This improves the resource-intensive utilization rate and training efficiency of deep reinforcement learning for gait control in the alligator-like robot system, reduces the number of real-time training sessions, shortens the data training cycle, and reduces the latency of gait control.
Smart Images
Figure CN119759070B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application belongs to the field of crocodile robot gait control, and particularly relates to a digital twin assisted crocodile robot gait control deep reinforcement learning method. BACKGROUND
[0002] The crocodile robot is inspired by crocodiles in nature. Compared with other mobile robots such as tracked robots and wheeled robots, the landing point of the crocodile robot is discrete, and the leg can realize the crossing of obstacles, and the flatness requirement of the ground environment is low. Therefore, the crocodile robot has a wide application prospect in complex terrains such as dangerous, space-limited and the like, such as space exploration, power inspection, search and rescue and the like.
[0003] The gait control of the crocodile robot is an important prerequisite for realizing the motion control. The traditional control methods such as virtual model control, model predictive control and central pattern generator depend on accurate kinematics and dynamics models, and are only suitable for specific scenes and need a lot of time for debugging and optimization, so the realization is difficult and the control precision is insufficient.
[0004] The machine learning (ML) paradigm provides a new idea for the gait control of the crocodile robot, especially the reinforcement learning (RL) for processing sequence decision problems and the deep learning (DL) for processing high-dimensional feature expression. The gait control of the crocodile robot is also a sequence decision problem, and the robot learns the optimal strategy of the gait control through continuous interaction with the environment state, while the deep learning can effectively solve the feature expression problem in the high-dimensional continuous state-action space in the complex environment state. However, the deep reinforcement learning (DRL) method still has some deficiencies in the gait control, such as low sample collection efficiency, slow training speed, simulation strategy cannot be directly applied to the real robot, simulation and reality gap exists, and the response delay exists due to the limited computing and storage capacity of the robot itself.
[0005] Therefore, the present application considers the digital twin (DT) technology, pre-training and edge device three factors, and proposes a digital twin assisted crocodile robot gait control deep reinforcement learning method to solve the above technical problems. SUMMARY
[0006] The application aims to provide a digital twin assisted crocodile robot gait control deep reinforcement learning method, which is used for improving the resource intensive utilization rate in the crocodile robot system and the training efficiency of DRL on gait control. In order to achieve the purpose, the application discloses a digital twin assisted crocodile robot gait control deep reinforcement learning method, which is divided into pre-training and real-time training two stages, and includes the following steps:
[0007] Step 1: a crocodile robot twin system model is constructed by using simulation and modeling technology, and is stored in an edge device;
[0008] The crocodile robot twin system model includes a crocodile robot twin model and a terrain environment model, the crocodile robot twin model is a digital replication of the physical entity of the crocodile robot, the terrain environment model is described by the terrain environment collected by the sensor loaded in the crocodile robot physical entity and built, and the edge device refers to a device for data processing and storage at the network edge, and information is transmitted between the twin physical entity through the communication module, which is composed of a data processing unit, a data storage unit and a communication module;
[0009] Step 2: constructing the state space and action space of crocodile robot gait control;
[0010] The state space is the state information of the crocodile robot twin system, including the crocodile robot state information and the terrain environment information, and the action space is the rotation angle of the motor of each joint of the crocodile robot;
[0011] Step 3: designing a reward function considering forward motion gait speed, stability and low energy consumption;
[0012] The reward function includes forward motion reward, energy consumption penalty and stability reward; the forward motion reward includes forward speed reward and forward direction reward; the energy consumption only considers the torque and speed factors of each joint motor; the stability reward includes the centroid height floating, falling and posture angle change of the crocodile robot; the final reward function is the linear superposition of the above reward functions;
[0013] Step 4: pre-training by using multi-process parallel data collection to obtain a twin decision model;
[0014] Each sub-process in the multi-process is responsible for exploring the environment and collecting historical data by using CPU resources, and the main process updates the strategy in the GPU; the DRL algorithm is deployed in the twin decision model, and the Actor-Critic network architecture is adopted;
[0015] Step 5: deploying the twin decision model in the edge device, using the communication module to transmit information with the twin physical entity, and training the crocodile robot in real time.
[0016] Further, the step 1, the specific method for constructing the crocodile-imitating robot twin system model by using simulation and modeling technology and storing it in the edge device is:
[0017] A Solidworks three-dimensional model of the quadruped crocodile-imitating robot is designed, which has 8 degrees of freedom, each leg has two degrees of freedom of hip joint and knee joint, the coordinate system and rotation axis of each joint are set, and the URDF file format is exported to obtain the crocodile-imitating robot twin model; the crocodile-imitating robot physical entity is obtained by 3D printing and installing sensors, and the point cloud data of the surrounding terrain is obtained by the laser radar loaded therein, the three-dimensional terrain environment model is obtained by processing the point cloud data; finally, the crocodile-imitating robot twin system model is stored in the edge device.
[0018] Further, the step 2, the specific method for constructing the state space and action space of the crocodile-imitating robot gait control is:
[0019] The state space is composed of crocodile-imitating robot state information and terrain environment information, which has 19 dimensions; the first part is the crocodile-imitating robot state information, which has 17 dimensions, including the rotation angles of the hip joints and knee joints of the four legs (s h1 ,s h2 ,s h3 ,s h4 ,s k1 ,s k2 ,s k3 ,s k4 ), the roll, pitch and yaw angles of the body (s r ,s p ,s y ), the roll, pitch and yaw angular velocities of the body The displacements of the body mass center in x, y and z axis directions (s x ,s y ,s z ); the second part is the terrain environment state information, which has 2 dimensions, which are the obstacle height and obstacle distance (s h ,s s ); the state space is expressed as follows:
[0020]
[0021] In formula (1), s h1 ,s h2 ,s h3 ,s h4 ,s k1 ,s k2 ,s k3 ,s k4 represent the hip joint angles and knee joint angles of the left front leg, the right front leg, the left rear leg and the right rear leg respectively;
[0022] In the state space, the value range of each state information is different, which is not conducive to the learning of the policy network. All state information is normalized. State normalization can accelerate the training process of the network. The normalization formula is as follows:
[0023]
[0024] In formula (2), x is the state value before normalization, and x∈(x min ,x max ), x′ is the normalized state value, and x′∈[-1,1];
[0025] The action space is the rotation angle of the hip joint and knee joint of the four legs of the crocodile robot. The continuous action space and position control mode are adopted. For the action space, formula (2) is also used for normalization processing, so that the output is a uniform type of value. After normalization, x ′ ∈[-1,1]; The action space is expressed as follows:
[0026] A=[s h1 ,s h2 ,s h3 ,s h4 ,s k1 ,s k2 ,s k3 ,s k4 ] (3)。
[0027] Further, the step 3, the design considers the reward function specific method of forward motion gait speed, stability and low energy consumption is:
[0028] (1) Forward motion reward: used to guide the crocodile robot to move towards the target direction, containing forward speed reward, forward direction reward, the target direction is the x axis of the world coordinate system;
[0029] ① Forward speed reward: encourage the robot to move towards the target direction, the distance of the robot centroid coordinates in the world coordinate system towards the target direction at the front and back time is as large as possible;
[0030] R x1 =k x1 (p x -p′ x ) / Δt (4)
[0031] In formula (4), p x is the position of the robot centroid in the world coordinate system on the x axis at the current time, p′ x is the position of the robot body on the x axis at the previous time, k x1 is the weight coefficient, and Δt represents the time step;
[0032] ② Forward direction reward: encourages the robot to move towards the target direction and punishes deviation;
[0033]
[0034] In formula (5), are the expressions of the current direction vector and the expected direction vector in the world coordinate system respectively. The angle between the two vectors indicates the degree of deviation of the current direction from the expected direction. ori is the weight parameter;
[0035] The robot's forward motion reward is R x =R x1 +R ori ;
[0036] (2) Energy consumption penalty: The robot is encouraged to achieve gait motion with the lowest possible energy consumption; the energy consumed by all motors in one time step is:
[0037]
[0038] In formula (6), i is the motor number, k is e is the weight parameter, τ i is the motor torque, ω i is the motor speed;
[0039] (3) Stability reward: encourages the robot to walk in a stable gait, including center of mass height fluctuation penalty, fall penalty and posture angle change penalty;
[0040] ① Penalty for center of mass height fluctuation: Encourage the robot to achieve stable movement, so that the center of mass height is close to the height of a stationary position;
[0041] R z =-k z |p z -p z_ref | (7)
[0042] In formula (7), p z_ref is the center of mass height of the robot when it is standing still, p z is the height of the robot's center of mass at the current moment, k z is the weight parameter;
[0043] ②Fall penalty: punish the robot for falling during training;
[0044] R z =R fall =-k fall ·r fall (8)
[0045] In formula (8), kfall r is the weight of the fall penalty term fall C is a constant, representing a fixed fall penalty value
[0046] ③Attitude angle change penalty: penalize the robot for too large attitude angle change when walking, increase the stability of walking
[0047] R rpy = -k rpy (k roll |α roll -α′ roll |+k pitch |β pitch -β′ pitch |+k yaw |γ yaw -γ′ yaw |) (9)
[0048] In equation (9), k rpy is the weight of the attitude angle change penalty term, k roll is the weight coefficient of roll angle change, k pitch is the weight coefficient of pitch angle change, k yaw is the weight coefficient of yaw angle change, α roll , β pitch , γ yaw are the roll angle, pitch angle and yaw angle of the robot at the current time, respectively, α′ roll , β′ pitch , γ′ yaw are the roll angle, pitch angle and yaw angle of the robot at the previous time, respectively
[0049] The robot gait stability reward function is R stable = R z + R fall + R rpy ;
[0050] (4) Early termination round conditions:
[0051] ① When the center of mass height is lower than 0.15m, it is determined that the robot is in a falling state; ② More than 5000 steps in a round; ③ The distance of advancing towards the target direction exceeds 5m; ④ Lateral deviation exceeds 2m; ⑤ Roll angle and pitch angle exceed 60 degrees, and yaw angle exceeds 30 degrees
[0052] In summary, the overall reward function of the robot is:
[0053] R = R x + R e + R stable (10).
[0054] Further, in step 4, the specific method for pre-training by using multi-process parallel data collection is as follows:
[0055] The pre-training of the DT model is performed on the edge device, the multi-core CPU resources of the edge device are utilized, historical data is collected in parallel by using multi-process, the historical data collection work is distributed to multiple twin agents, each twin agent independently interacts with the environment and collects data, and then the data is summarized in an experience buffer, the twin decision model samples in batches in the experience buffer, the policy is updated in the GPU, and finally the updated network parameters are distributed to each twin agent, and the cycle is repeated.
[0056] Further, in step 5, the specific method for deploying the twin decision model in the edge device and using the communication module to transmit information between the twin physical entity for real-time training of the crocodile robot is as follows:
[0057] In the real-time training stage, the crocodile robot physical entity loads a high-definition sensor to collect state information, and the edge device updates the twin system model in real time at each time step with the perception data from the physical entity. The twin decision model extracts the state data required for gait control from the twin system model, generates decision output, and transmits it to the twin physical entity through the communication module. The twin physical entity receives the decision output and performs the gait action. The cycle is repeated, and the twin system model will continuously obtain real-time trajectory sample data (r, s, s_next, d, a) from the physical entity and store it in the experience buffer of the edge device for real-time training of the twin decision model.
[0058] Compared with the existing technology, the present application has the following beneficial effects:
[0059] (1) The crocodile robot twin system model is used for pre-training, so that the edge device learns the gait characteristics and generates the DRL network decision value of the crocodile robot. This value can be directly applied to the crocodile robot, reducing the number of real-time training and accelerating the generation of gait control strategy.
[0060] (2) The edge device performs all operations of data fusion, analysis and calculation in the real-time training process, the physical entity is responsible for the specific execution of the gait action, and the state information collected by the built-in sensor is used to update the DT model in real time. This method not only increases the depth and breadth of the training model, but also shortens the data training period. At the same time, training the twin decision model on the edge device makes up for the limited computing and storage capacity of the crocodile robot, saves computing resources, and reduces the delay of gait control. BRIEF DESCRIPTION OF DRAWINGS
[0061] Figure 1 is a flow chart of the crocodile robot gait control training process assisted by digital twin.
[0062] Figure 2 is a pre-training stage flow chart of the present application.
[0063] Figure 3 is a real-time training stage flow chart of the present application. Embodiments
[0064] The specific embodiments of the present application are described in detail below with reference to the accompanying drawings, but it should be understood that the scope of protection of the application is not limited by the specific embodiments.
[0065] Reference Figure 1 The application discloses a digital twin assisted crocodile-like robot gait control deep reinforcement learning method, which is divided into pre-training and real-time training two stages, and the specific implementation steps are as follows:
[0066] Step 1: Use simulation and modeling technology to construct a crocodile-like robot twin system model, and store it in an edge device;
[0067] DT technology creates a virtual mirror image of a physical entity in a digital way, which can accurately reflect the state of the physical entity based on the collected data, so that the environment of the DT model is closer to the real environment. Therefore, integrating digital twin technology into deep reinforcement learning to improve training efficiency has become a new idea. At the same time, the edge device has strong computing power and storage resources, which can be used as a good resource supplement for the crocodile-like robot.
[0068] Specifically, the crocodile-like robot twin system model includes a crocodile-like robot twin model and a terrain environment model, wherein the crocodile-like robot twin model is a digital replication of the physical entity of the crocodile-like robot, and the terrain environment model is a description of the terrain environment collected by the on-board sensor of the crocodile-like robot physical entity and built. The edge device refers to a device that processes and stores data at the edge of the network and transmits information to and from the twin physical entity through a communication module, which is composed of a data processing unit, a data storage unit and a communication module, wherein the communication module adopts a wireless communication mode such as Wi-Fi, 5G, etc.;
[0069] Further, the role of the edge device includes two aspects. The first aspect is to establish a crocodile-like robot twin system model and pre-train it as the basis for real-time training, which speeds up the generation of gait control strategy. The second aspect is to perform all operations of data fusion, analysis and calculation in the real-time training process, which makes up for the limited computing and storage capacity of the crocodile-like robot, saves computing resources, and reduces the delay of gait control.
[0070] Further, the specific process of establishing the twin system model is as follows: a Solidworks three-dimensional model of the quadruped crocodile robot is designed, which has 8 degrees of freedom, each leg has two degrees of freedom of hip joint and knee joint, the coordinate system and rotation axis of each joint are set, and the URDF file format is derived to obtain the twin model of the crocodile robot; the crocodile robot physical entity is obtained by 3D printing and installing sensors, and the point cloud data of the surrounding terrain is obtained by the laser radar loaded therein, the point cloud data is processed to generate a three-dimensional terrain environment model; finally, the twin system model of the crocodile robot is stored in the edge device.
[0071] Further, the specific method of generating a three-dimensional terrain environment model using point cloud data is as follows: after the point cloud data is collected, first, denoising and downsampling are performed to reduce the data volume. Then, the point cloud data is adjusted to the coordinate system of the robot through coordinate transformation, and algorithms such as ICP are used for point cloud fusion, thereby forming a complete environment model. Subsequently, a three-dimensional grid or surface model is generated using point cloud processing software, visualization and feature extraction are performed to analyze terrain features such as terrain height and obstacle distance.
[0072] Step 2: Construct the state space and action space of the crocodile robot gait control;
[0073] Specifically, the state space is the state information of the crocodile robot twin system, including the state information of the crocodile robot and the terrain environment information, and the action space is the rotation angle of each joint motor of the crocodile robot;
[0074] Further, the specific method of designing the state space and action space is as follows:
[0075] The state space is composed of crocodile robot state information and terrain environment information, with a total of 19 dimensions; the first part is the crocodile robot state information, with a total of 17 dimensions, including the rotation angles of the hip joints and knee joints of the four legs (s h1 ,s h2 ,s h3 ,s h4 ,s k1 ,s k2 ,s k3 ,s k4 ), the roll, pitch and yaw angles of the body (s r ,s p ,s y ), the roll, pitch and yaw angular velocities of the body The displacements of the body mass center in the x, y and z axis directions (s x ,s y ,s z ); the second part is the terrain environment state information, with a total of 2 dimensions, which are the obstacle height and obstacle distance (s h ,ss ) ; the state space is expressed as follows:
[0076]
[0077] In formula (1), s h1 ,s h2 ,s h3 ,s h4 ,s k1 ,s k2 ,s k3 ,s k4 respectively represent the hip joint angle and the knee joint angle of the left front leg, the right front leg, the left rear leg and the right rear leg;
[0078] In the state space, the value range of each state information is different, which is not conducive to the learning of the policy network. All state information is normalized, and state normalization can accelerate the training process of the network. The normalization formula is as follows:
[0079]
[0080] In formula (2), x is the state value before normalization, and x ∈ (x min ,x max ), x' is the normalized state value, and x' ∈ [-1, 1];
[0081] The action space is the rotation angle of the hip joint and the knee joint of the four legs of the crocodile robot, and the continuous action space and the position control mode are adopted. For the action space, formula (2) is also used for normalization processing, so that the output is a uniform type of value, and after normalization, x ′ ∈ [-1, 1]; the action space is expressed as follows:
[0082] A = [s h1 ,s h2 ,s h3 ,s h4 ,s k1 ,s k2 ,s k3 ,s k4 ] (3).
[0083] Step 3: design a reward function considering forward motion gait speed, stability and low energy consumption:
[0084] Specifically, during the training process of the quadruped crocodile robot, the learning goal is to encourage the robot to learn a fast, stable, and low-energy forward motion gait through exploration and trial and error. Therefore, the reward function includes a forward motion reward, an energy consumption penalty, and a stability reward; the forward motion reward includes a forward speed reward and a forward direction reward; the energy consumption only considers the torque and speed of each joint motor; the stability reward includes the crocodile robot's center of mass height fluctuation, falls, and posture angle changes; the final reward function is a linear superposition of the above reward functions;
[0085] Furthermore, the specific method for designing the reward function is:
[0086] (1) Forward motion reward: used to guide the crocodile-like robot to move towards the target direction, including forward speed reward and forward direction reward. The target direction is the x-axis of the world coordinate system;
[0087] ① Forward speed reward: Encourages the robot to move towards the target direction, and the distance the robot's center of mass coordinates move towards the target direction in the world coordinate system at each moment is as large as possible;
[0088] R x1 =k x1 (p x -p′ x ) / Δt (4)
[0089] In formula (4), p x is the position of the robot's center of mass on the x-axis in the world coordinate system at the current moment, p′ x is the x-axis position of the robot body at the previous moment, k x1 is the weight coefficient, Δt represents the time step;
[0090] ② Forward direction reward: encourages the robot to move towards the target direction and punishes deviation;
[0091]
[0092] In formula (5), are the expressions of the current direction vector and the expected direction vector in the world coordinate system respectively. The angle between the two vectors indicates the degree of deviation of the current direction from the expected direction. ori is the weight parameter;
[0093] The robot's forward motion reward is R x =R x1 +R ori ;
[0094] (2) Energy consumption penalty: The robot is encouraged to achieve gait motion with the lowest possible energy consumption; the energy consumed by all motors in one time step is:
[0095]
[0096] In formula (6), i is the motor number, k e is the weight parameter, τ i is the motor torque, ω i is the motor speed;
[0097] (3) Stability reward: encourage the robot to walk with stable gait, including the center of mass height floating penalty, fall penalty and posture angle change penalty;
[0098] ① Center of mass height floating penalty: encourage the robot to move stably without overturning, so it is necessary to control the height error of the robot's gravity center coordinate in the z direction relative to the height when the robot stands upright;
[0099] R z = -k z |p z -p z_ref | (7)
[0100] In formula (7), p z_ref is the height of the robot's center of mass when standing still, p z is the height of the robot's center of mass at the current time, k z is the weight parameter of this term;
[0101] ② Fall penalty: penalize the robot for falling behavior during training;
[0102] R z = R fall = -k fall ·r fall (8)
[0103] In formula (8), k fall is the weight of the fall penalty term, r fall is a constant, representing a fixed fall penalty value;
[0104] ③ Posture angle change penalty: penalize the robot for too large posture angle change when walking, increase the stability of walking;
[0105] R rpy = -k rpy (k roll |α roll -α′ roll |+k pitch |β pitch -β′ pitch |+k yaw |γ yaw -γ′ yaw |) (9)
[0106] In formula (9), k rpy is a weight of the attitude angle change penalty term, k roll is a weight coefficient of the roll angle change, k pitch is a weight coefficient of the pitch angle change, k yaw is a weight coefficient of the yaw angle change, a roll , b pitch , g yaw are respectively a roll angle, a pitch angle and a yaw angle of the robot at a current moment, a' roll , b' pitch , g' yaw are respectively a roll angle, a pitch angle and a yaw angle of the robot at a previous moment;
[0107] The robot gait stability reward function is R stable = R z + R fall + R rpy ;
[0108] (4) Early termination round condition:
[0109] In the reinforcement learning training process, the crocodile robot needs to try and error through repeated interaction with the environment, and the design of the termination condition as a condition for judging a round has an important influence on the learning efficiency of the crocodile robot. When the current state of the robot deviates far from the set target state, the environment and the agent state should be reset to improve the learning efficiency of the agent.
[0110] When the roll angle and the pitch angle of the robot are large, it may directly lead to the robot falling or losing balance; when the center of mass is below a certain height, it can be judged that the robot is in a falling state; when the yaw angle is too large, it can be judged that the robot has deviated from the target direction; when the number of steps in a round is too long, the control strategy is updated slowly, and the learning effect is not significant. Therefore, the early termination round condition designed by the present application is as follows:
[0111] ① When the height of the center of mass is lower than 0.15m, it is determined that the robot is in a falling state; ② The number of steps in a round exceeds 5000; ③ The distance of advancing towards the target direction exceeds 5m; ④ The lateral offset exceeds 2m; ⑤ The roll angle and the pitch angle exceed 60 degrees, and the yaw angle exceeds 30 degrees;
[0112] In summary, the overall reward function of the robot is:
[0113] R = R x + R e + R stable (10).
[0114] The designed reward mechanism fuses the goals of gait speed, stability and energy consumption, and designs reward and penalty terms, and increases the weight coefficient in the combination process. By adjusting the weight of each reward term, different task goals can be encouraged, and finally the expected control effect of the quadruped crocodile robot is achieved.
[0115] Step 4: Pre-training is performed by collecting data in multiple processes in parallel to obtain a twin decision model;
[0116] Specifically, each sub-process in the multi-process is responsible for exploring the environment and collecting historical data using CPU resources, and the main process updates the strategy in the GPU; the DRL algorithm is deployed in the twin decision model, and an Actor-Critic network architecture is adopted to provide decision output for gait control;
[0117] Reference Figure 2 Further, the pre-training of the DT model is performed on the edge device, the multi-core CPU resources of the edge device are used, historical data are collected in multiple processes in parallel, the historical data collection work is distributed to multiple twin agents, each twin agent independently interacts with the environment and collects data, and then the data are summarized in an experience buffer, the twin decision model samples in batches in the experience buffer, updates the strategy in the GPU, and finally the updated network parameters are distributed to each twin agent, and the cycle is repeated; multi-process calculation is used to improve the efficiency of pre-training data collection and speed up the training speed of the decision model.
[0118] Step 5: The twin decision model is deployed in the edge device, and information is transmitted between the twin physical entity and the communication module to train the crocodile robot in real time;
[0119] In order to capture the characteristics of the dynamic environment and obtain an accurate DT model of the current environment, the DT model of the crocodile robot system is established from the perspective of execution in the edge device, and the DRL decision network of the crocodile robot is updated in real time.
[0120] Reference Figure 3 Specifically, in the real-time training stage, the physical entity of the crocodile robot loads a high-definition sensor to collect state information, and the edge device updates the twin system model in real time with the perception data from the physical entity at each time step. The twin decision model extracts the state data required for gait control from the twin system model, generates decision output, and transmits the decision output to the twin physical entity through the communication module. The twin physical entity receives the decision output and performs gait action. The cycle is repeated, the twin system model will continuously obtain real-time trajectory sample data (r, s, s_next, d, a) from the physical entity, and store it in the experience buffer of the edge device, for real-time training of the twin decision model.
[0121] In summary, the present application adopts digital twin technology to establish a crocodile robot twin system model and store it in the edge device. By pre-training the DT model first, then sampling in the real environment by the physical entity, and communicating with the DT model for real-time training, the resource-intensive utilization rate in the crocodile robot system and the training efficiency of DRL for gait control are improved.
[0122] The crocodile robot twin system model is pre-trained to make the edge device learn the gait characteristics and generate the crocodile robot DRL network decision value. This value can be directly applied to the crocodile robot, reducing the number of real-time training and speeding up the generation of gait control strategy.
[0123] The edge device performs all operations of data fusion, analysis and calculation in the real-time training process, the physical entity is responsible for the specific execution of gait action, and the state information collected by the on-board sensor is used to update the DT model in real time. This method not only increases the depth and breadth of the training model, but also shortens the data training cycle. At the same time, training the twin decision model on the edge device makes up for the limited computing and storage capacity of the crocodile robot, saves computing resources, and reduces the delay of gait control.
Claims
1. A digital twin assisted crocodile-like robot gait control deep reinforcement learning method, the method is divided into pre-training and real-time training two stages, characterized in that, Comprising the following steps: Step 1: Constructing a crocodile robot twin system model using simulation and modeling techniques and storing it in an edge device; The crocodile robot twin system model includes a crocodile robot twin model and a terrain environment model, the crocodile robot twin model is a digital replication of the crocodile robot physical entity, and the terrain environment model is a description of the terrain environment collected by the on-board sensor in the crocodile robot physical entity and built, the edge device refers to a device that processes and stores data at the network edge and transmits information with the twin physical entity through a communication module, which is composed of a data processing unit, a data storage unit and a communication module; Step 2: Constructing the state space and action space of the crocodile robot gait control; The state space is the state information of the crocodile robot twin system, including crocodile robot state information and terrain environment information, and the action space is the rotation angle of each joint motor of the crocodile robot; Step 3: Designing a reward function considering forward motion gait speed, stability and low energy consumption; The reward function includes forward motion reward, energy consumption penalty and stability reward; The forward motion reward includes forward speed reward and forward direction reward; The energy consumption only considers the torque and speed of each joint motor; The stability reward includes the centroid height floating, falling and posture angle change of the crocodile robot; The final reward function is the linear superposition of the above reward functions; Step 4: Pre-training by collecting data in multiple processes in parallel to obtain a twin decision model; Each sub-process in the multiple processes is responsible for exploring the environment and collecting historical data using CPU resources, while the main process updates the strategy in GPU; The DRL algorithm is deployed in the twin decision model, using the Actor-Critic network architecture; Step 5: Deploying the twin decision model in the edge device, using the communication module to transmit information with the twin physical entity, and training the crocodile robot in real time; The specific method for training the crocodile robot in real time is: in the real-time training phase, the crocodile robot physical entity on-board high-definition sensor collects state information, and the edge device updates the twin system model in real time at each time step with perception data from the physical entity; The twin decision model extracts the state data required for gait control from the twin system model, generates decision output, and transmits it to the twin physical entity through the communication module; The twin physical entity receives the decision output and executes the gait action; In turn, the twin system model will continue to obtain real-time trajectory sample data (r, s, s_next, d, a) from the physical entity and store it in the experience buffer of the edge device for real-time training of the twin decision model.
2. The digital twin assisted crocodile-inspired robot gait control deep reinforcement learning method according to claim 1, wherein, The specific method of step 1 is to construct a crocodile robot twin system model using simulation and modeling techniques and store it in an edge device: A Solidworks three-dimensional model of the quadruped crocodile robot is designed, which has a total of 8 degrees of freedom, each leg has two degrees of freedom of hip joint and knee joint, the coordinate system and rotation axis of each joint are set, and a URDF file format is derived to obtain a twin model of the crocodile robot; through 3D printing and installation of sensors, a physical entity of the crocodile robot is obtained, and point cloud data of the surrounding terrain is obtained through the laser radar loaded therein, and a three-dimensional terrain environment model is obtained by processing the point cloud data; finally, the twin system model of the crocodile robot is stored in the edge device.
3. The digital twin assisted crocodile-inspired robot gait control deep reinforcement learning method according to claim 1, wherein, In step 2, the specific method for constructing the state space and action space of the crocodile robot gait control is: The state space is composed of two parts, i.e., the state information of the crocodile robot and the terrain environment information, and has a total of 19 dimensions. The first part is the state information of the crocodile robot, which has a total of 17 dimensions, including the rotation angles of the hip joints and knee joints of the four legs (s h1 ,s h2 ,s h3 ,s h4 ,s k1 ,s k2 ,s k3 ,s k4 ), the roll, pitch and yaw angles of the body (s r ,s p ,s y ), the roll, pitch and yaw angular velocities of the body , and the displacements of the body mass center in the x, y and z axial directions (s x ,s y ,s z ). The second part is the state information of the terrain environment, which has a total of 2 dimensions, i.e., the obstacle height and the obstacle distance (s h ,s s ). The state space is expressed as follows: In formula (1), s h1 s h2 s h3 s h4 s k1 s k2 s k3 s k4 respectively represent the hip joint angle and the knee joint angle of the left front leg, the right front leg, the left rear leg, and the right rear leg; In the state space, the value range of each state information is different, which is not conducive to the learning of the strategy network, and all state information is normalized; The normalization formula is as follows: In formula (2), x is a state value before normalization, and x e (x min ,x max ); x' is a state value after normalization, and x' e [-1, 1]. The action space is the rotation angle of the hip joint and knee joint of the crocodile robot, and a continuous action space and position control mode are adopted; for the action space, formula (2) is also used for normalization processing, so that the output is of a uniform type of value, and after normalization, x' is in [-1, 1]; the action space is expressed as follows: A = [s h1 ,s h2 ,s h3 ,s h4 ,s k1 ,s k2 ,s k3 ,s k4 ] (3).
4. The digital twin assisted crocodile-inspired robot gait control deep reinforcement learning method according to claim 1, wherein, In step 3, the specific method for designing a reward function considering forward motion gait speed, stability and low energy consumption is: (1) Forward motion reward: used to guide the crocodile robot to move towards the target direction, including forward speed reward and forward direction reward, the target direction being the x-axis of the world coordinate system; ①Forward speed reward: encourages the robot to move towards the target direction, and the distance of the robot's center of mass coordinates in the world coordinate system towards the target direction before and after is as large as possible; R x1 = k x1 (p x -p′ x ) / Δt (4) In formula (4), p x is the position of the robot centroid in the world coordinate system x-axis at the current time, p′ x is the position of the robot body x-axis at the previous time, k x1 is the weight coefficient, and Δt represents the time step. ②Forward direction reward: encourages the robot to move in the desired direction, and punishes yaw; In formula (5), respectively, the angle between the two vectors represents the deviation of the current direction from the desired direction, k ori is a weight parameter; Robot forward motion reward is R x = R x1 + R ori ; (2) Energy consumption penalty: encourages the robot to achieve gait motion with as low energy consumption as possible; the energy consumed by all motors in one time step is: In formula (6), i is the motor number, k e is a weight parameter, τ i is the motor torque, ω i is the motor speed; (3) Stability reward: encourages the robot to walk with stable gait, including center of mass height floating penalty, falling penalty and attitude angle change penalty; ①Center of mass height floating penalty: encourages the robot to achieve stable motion, so that the center of mass height is near the height when standing still; R z = -k z |p z -p z_ref | (7) In formula (7), p z_ref is the height of the center of mass when the robot is standing still, p z is the height of the center of mass of the robot at the current time, k z is a weight parameter; ②Falling penalty: punishes the robot for falling behavior during training; R fall = -k fall ·r fall (8) In formula (8), k fall is a weight of the fall penalty term, r fall is a constant, representing a fixed fall penalty value; ③Attitude angle change penalty: punishes the robot for large attitude angle change when walking, increasing the stability of walking; R rpy = -k rpy (k roll |α roll -α′ roll |+k pitch |β pitch -β′ pitch |+k yaw |γ yaw -γ′ yaw |) (9) In formula (9), k rpy is a weight of the attitude angle change penalty term, k roll is a weight coefficient of the roll angle change, k pitch is a weight coefficient of the pitch angle change, k yaw is a weight coefficient of the yaw angle change, a roll , b pitch , g yaw are respectively a roll angle, a pitch angle and a yaw angle of the robot at a current time, a′ roll , b′ pitch , g′ yaw are respectively a roll angle, a pitch angle and a yaw angle of the robot at a previous time. Robot gait stability reward function is R stable = R z + R fall + R rpy ; (4) Early termination round conditions: ①When the center of mass height is lower than 0.15m, the robot is determined to be in a falling state; ②One round exceeds 5000 steps; ③The distance of forward movement towards the target direction exceeds 5m; ④The lateral offset exceeds 2m; ⑤The roll angle and pitch angle exceed 60 degrees, and the yaw angle exceeds 30 degrees; In summary, the overall reward function of the robot is: R = R x + R e + R stable (10).
5. The digital twin assisted crocodile-inspired robot gait control deep reinforcement learning method according to claim 1, wherein, In step 4, the specific method for pre-training the twin decision model by using multi-process parallel data collection is: The pre-training of the DT model is performed at the edge device, multi-core CPU resources of the edge device are used, historical data is collected in parallel by using multiple processes, the historical data collection work is distributed to multiple twin agents, each twin agent independently interacts with the environment and collects data, and then the data is summarized in an experience buffer, the twin decision model samples in batches in the experience buffer, policy updating is performed in the GPU, and finally the updated network parameters are distributed to each twin agent, and the cycle is repeated in turn.
Citation Information
Patent Citations
Robot motion control method and system based on deep reinforcement learning
CN113093779A
Quadruped robot adaptive gait autonomous generation method based on deep reinforcement learning
CN113190029A