Digital twinborn assisted alligator-imitated robot gait control deep reinforcement learning method

Abstract

The invention discloses a deep reinforcement learning method for gait control of a digital twinning-assisted crocodile-imitating robot. The method is divided into a pre-training stage and a real-time training stage. The method comprises the following specific steps: step 1, constructing a twin system model of the crocodile-imitating robot by utilizing simulation and modeling technologies, and storing the twin system model in edge equipment; step 2, constructing a state space and an action space of gait control of the crocodile-imitating robot; step 3, designing a reward function considering forward movement gait speed, stability and low energy consumption; step 4, collecting data in parallel by adopting multiple processes for pre-training to obtain a twin decision model; and step 5, deploying the twinborn decision model in edge equipment, and carrying out real-time training on the crocodile-imitating robot by utilizing mutual information transmission between a communication module and a twinborn physical entity. According to the method, the pre-training is carried out by establishing the digital twin model, and the training efficiency of the gait control of the crocodile-imitating robot is improved by utilizing the strong computing power and storage resources of the edge equipment.

Description

Technical Field

[0001] The present invention belongs to the field of gait control of crocodile-like robots, and in particular relates to a deep reinforcement learning method for gait control of crocodile-like robots assisted by digital twins. Background Art

[0002] The design of the crocodile robot is inspired by crocodiles in nature. Compared with other mobile robots such as crawler and wheeled robots, the landing point of the crocodile robot is discrete and can choose its own landing point. Its legs can cross obstacles and it has low requirements for the flatness of the ground environment. Therefore, the crocodile robot has broad application prospects in space exploration, power inspection, search and rescue, and other dangerous and space-restricted complex terrains.

[0003] Gait control of the crocodile robot is an important prerequisite for realizing its motion control. Traditional control methods such as virtual model control, model predictive control, central pattern generator, etc. rely on precise kinematic and dynamic models, and are only targeted at specific scenarios and require a lot of time for debugging and optimization. They are difficult to implement and have insufficient control accuracy.

[0004] The machine learning (ML) paradigm provides new ideas for the gait control of crocodile-like robots, especially reinforcement learning (RL) for sequential decision-making problems and deep learning (DL) for high-dimensional feature expression. The gait control of crocodile-like robots is also a sequential decision-making problem. The robot learns the optimal strategy for gait control through continuous interaction with the environment state, and deep learning can effectively solve the problem of feature expression in high-dimensional continuous state-action space in complex environment states. However, the current deep reinforcement learning (DRL) method still has shortcomings in gait control, such as low sample collection efficiency, slow training speed, simulation strategies cannot be directly applied to real robots, there is a gap between simulation and reality, and the robot's own computing and storage capabilities are limited, there is a response delay, etc.

[0005] In view of this, the present invention considers three factors: digital twin (DT) technology, pre-training and edge devices, and proposes a digital twin-assisted deep reinforcement learning method for gait control of a crocodile robot to solve the above technical problems. Summary of the invention

[0006] The purpose of the present invention is to provide a digital twin-assisted deep reinforcement learning method for gait control of a crocodile robot, which is used to improve the resource-intensive utilization rate and DRL training efficiency of gait control in the crocodile robot system. In order to achieve this purpose, the present invention discloses a digital twin-assisted deep reinforcement learning method for gait control of a crocodile robot, which is divided into two stages: pre-training and real-time training, and includes the following steps:

[0007] Step 1: Use simulation and modeling technology to build a twin system model of the crocodile robot and store it in the edge device;

[0008] The crocodile robot twin system model includes a crocodile robot twin model and a terrain environment model. The crocodile robot twin model is a digital reproduction of the crocodile robot physical entity. The terrain environment model is a terrain environment description collected and constructed by sensors carried in the crocodile robot physical entity. The edge device refers to a device that processes and stores data at the edge of the network and transmits information to and from the twin physical entity through a communication module. It is composed of a data processing unit, a data storage unit and a communication module.

[0009] Step 2: Construct the state space and action space of the gait control of the crocodile robot:

[0010] The state space is the state information of the crocodile robot twin system, including the crocodile robot state information and terrain environment information, and the action space is the rotation angle of each joint motor of the crocodile robot;

[0011] Step 3: Design a reward function that takes into account forward movement gait speed, stability, and low energy consumption:

[0012] The reward function includes forward motion reward, energy consumption penalty, and stability reward; the forward motion reward includes forward speed reward and forward direction reward; the energy consumption only considers the torque and speed factors of each joint motor; the stability reward includes the height fluctuation, fall and posture angle change of the crocodile robot's center of mass; the final reward function is the linear superposition of the above reward functions;

[0013] Step 4: Use multiple processes to collect data in parallel for pre-training to obtain the twin decision model:

[0014] Each sub-process in the multi-process is responsible for exploring the environment and collecting historical data using CPU resources, while the main process performs strategy updates in the GPU; the DRL algorithm is deployed in the twin decision model, using the Actor-Critic network architecture;

[0015] Step 5: Deploy the twin decision model in the edge device, use the communication module to transmit information with the twin physical entity, and perform real-time training on the crocodile robot.

[0016] Furthermore, in step 1, the specific method of using simulation and modeling technology to construct a crocodile robot twin system model and storing it in the edge device is:

[0017] A Solidworks 3D model of a four-legged crocodile robot is designed with a total of 8 degrees of freedom. Each leg has two degrees of freedom at the hip and knee joints. The coordinate system and rotation axis of each joint are set, and the URDF file format is exported to obtain the crocodile robot twin model. The physical entity of the crocodile robot is obtained by 3D printing and installing sensors, and the point cloud data of the surrounding terrain is obtained through its built-in laser radar. The point cloud data is processed to obtain a three-dimensional terrain environment model. Finally, the crocodile robot twin system model is stored in the edge device.

[0018] Furthermore, the specific method of step 2, constructing the state space and action space of the gait control of the crocodile-like robot is:

[0019] The state space consists of two parts: the state information of the crocodile robot and the terrain environment information, with a total of 19 dimensions. The first part is the state information of the crocodile robot, with a total of 17 dimensions, including the rotation angles of the hip and knee joints of the four legs (s h1 ,s h2 ,s h3 ,s h4 ,s k1 ,s k2 ,s k3 ,s k4 ), fuselage roll, pitch and yaw angles (s r ,s p ,s y ), fuselage roll, pitch, yaw angular velocity Displacement of the center of mass of the fuselage in the x, y, and z directions (s x ,s y ,s z ); The second part is the terrain environment status information, which has two dimensions, namely obstacle height and obstacle distance (s h ,s s ). The state space is expressed as follows:

[0020]

[0021] In formula (1), s h1 ,s h2 ,s h3 ,s h4 ,s k1 ,s k2 ,s k3 ,s k4 Represent the hip joint angles and knee joint angles of the left front leg, right front leg, left hind leg, and right hind leg respectively.

[0022] In the state space, the value ranges of various state information are different, which is not conducive to the learning of the policy network. The present invention normalizes all state information, and state normalization can accelerate the training process of the network. The normalization formula is as follows:

[0023]

[0024] In formula (2), x is the state value before normalization, and x∈(x min , x max ), x′ is the normalized state value, and x′∈[-1, 1].

[0025] The action space is the rotation angle of the hip and knee joints of the four legs of the crocodile robot, and a continuous action space and position control method is adopted. The action space is also normalized according to formula (2) to output a value of the same type. After normalization, x′∈[-1, 1]. The action space is expressed as follows:

[0026] a=[s h1 ,s h2 ,s h3 ,s h4 ,s k1 ,s k2 ,s k3 ,s k4 ] (3)

[0027] Furthermore, in step 3, the specific method of designing a reward function that takes into account forward motion gait speed, stability and low energy consumption is:

[0028] (1) Forward motion reward: This reward is used to guide the crocodile robot to move in the target direction (the x-axis of the world coordinate system), including forward speed reward and forward direction reward.

[0029] ① Forward speed reward: This reward encourages the robot to move towards the target direction, and the distance that the robot's center of mass coordinates move in the target direction in the world coordinate system at the previous and next moments is as large as possible.

[0030] R x1 =k x1 (p x -p′ x ) / Δt (4)

[0031] In formula (4), p x is the position of the robot's center of mass on the x-axis in the world coordinate system at the current moment, p′ x is the x-axis position of the robot body at the previous moment, k x1 is the weight coefficient, and Δt represents the time step.

[0032] ② Forward direction reward: encourages the robot to move in the target direction and punishes deviation.

[0033]

[0034] In formula (5), are the expressions of the current direction vector and the expected direction vector in the world coordinate system respectively. The angle between the two vectors indicates the degree of deviation of the current direction from the expected direction. ori is the weight parameter.

[0035] The robot's forward motion reward is R x =R x1 +R ori

[0036] (2) Energy consumption penalty, which encourages the robot to achieve gait motion with the lowest possible energy consumption. The energy consumed by all motors in one time step is:

[0037]

[0038] In formula (6), i is the motor number, k is e is the weight parameter, τ i is the motor torque, ω i is the motor speed.

[0039] (3) Stability reward: This reward encourages the robot to walk with a stable gait, which includes penalties for center of mass height fluctuation, fall, and posture angle change.

[0040] ① Center of mass height floating penalty: This item encourages the robot to achieve stable movement so that the center of mass height is close to the height of standing still.

[0041] R z =-k z |p z -p z_ref | (7)

[0042] In formula (7), p z_ref is the center of mass height of the robot when it is standing still, p z is the height of the robot's center of mass at the current moment, k z is the weight parameter of this item.

[0043] ②Fall penalty: This penalty is for the robot to fall during training.

[0044] R z =R fall =-k fall ·r fall (8)

[0045] In formula (8), kfall is the weight of the fall penalty term, r fall is a constant, indicating a fixed fall penalty value.

[0046] ③ Attitude angle change penalty: This penalty is for excessive changes in the attitude angle of the robot while walking, thereby increasing walking stability.

[0047] R rpy =-k rpy (k rpll |α roll -α′ roll |+k pitch |β pitch -β′ pitch |+k yaw |γ yaw -γ′ yaw |) (9)

[0048] In formula (9), k rpy is the weight of the attitude angle change penalty term, k rpll is the weight coefficient of the roll angle change, k pitch is the weight coefficient of pitch angle change, k yaw Weight coefficient for yaw angle change, α roll , β pitch , γ yaw are the roll angle, pitch angle and yaw angle of the robot at the current moment, α′ roll , β′ pitch , γ′ yaw They are the roll angle, pitch angle and yaw angle of the robot at the previous moment respectively.

[0049] The robot gait stability reward function is R stable =R z +R fall +R rpy

[0050] (4) Conditions for early termination of a round:

[0051] ① When the center of mass height is lower than 0.15m, the robot is judged to be in a falling state; ② One round exceeds 5000 steps; ③ The distance in the target direction exceeds 5m; ④ The lateral deviation exceeds 2m; ⑤ The roll angle and pitch angle exceed 60 degrees, and the yaw angle exceeds 30 degrees.

[0052] In summary, the overall reward function of the robot is:

[0053] R=R x +R e +R stable (10)

[0054] Furthermore, in step 4, multiple processes are used to collect data in parallel for pre-training, and the specific method for obtaining the twin decision model is:

[0055] The DT model is pre-trained on the edge device, using the multi-core CPU resources of the edge device and using multiple processes to collect historical data in parallel. The historical data collection work is distributed to multiple twin agents. Each twin agent interacts with the environment independently and collects data, which are then aggregated into an experience buffer. The twin decision model samples the experience buffer in batches, updates the strategy in the GPU, and finally distributes the updated network parameters to each twin agent, in a cycle.

[0056] Furthermore, in step 5, the twin decision model is deployed in the edge device, and the communication module is used to transmit information to the twin physical entity to perform real-time training on the crocodile-like robot.

[0057] During the real-time training phase, the physical entity of the crocodile robot is equipped with high-definition sensors to collect status information, and the edge device uses the perception data from the physical entity to update the twin system model in real time at each time step. The twin decision model extracts the status data required for gait control from the twin system model, generates a decision output, and transmits it to the twin physical entity through the communication module. The twin physical entity receives the decision output and performs gait actions. In a sequential cycle, the twin system model will continue to obtain real-time trajectory sample data (r, s, s_next, d, a) from the physical entity and store it in the experience buffer of the edge device for real-time training of the twin decision model.

[0058] Compared with the existing technology, the present invention has the following beneficial effects:

[0059] (1) The crocodile robot twin system model is used for pre-training, so that the edge device learns the gait characteristics and generates the DRL network decision value of the crocodile robot. This value can be directly applied to the crocodile robot, reducing the number of real-time training times and speeding up the generation of gait control strategies.

[0060] (2) The edge device performs all operations of data fusion, analysis, and calculation during real-time training. The physical entity is responsible for the specific execution of gait movements and uses built-in sensors to collect status information to update the DT model in real time. This method not only increases the depth and breadth of the training model, but also shortens the data training cycle. At the same time, training the twin decision model on the edge device makes up for the limited computing and storage capabilities of the crocodile robot, saves computing resources, and reduces the delay of gait control. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 It is a flowchart of the gait control training process of the crocodile robot assisted by digital twins.

[0062] Figure 2 It is a flowchart of the pre-training phase of the present invention.

[0063] Figure 3 It is a flowchart of the real-time training phase of the present invention. Specific embodiments

[0064] The specific implementation modes of the present invention are described in detail below in conjunction with the accompanying drawings, but it should be understood that the protection scope of the invention is not limited by the specific implementation modes.

[0065] refer to Figure 1 The present invention discloses a digital twin-assisted deep reinforcement learning method for gait control of a crocodile-like robot. The method is divided into two stages: pre-training and real-time training. The specific implementation steps are as follows:

[0066] Step 1: Use simulation and modeling technology to build a twin system model of the crocodile robot and store it in the edge device;

[0067] DT technology creates a virtual image of a physical entity in a digital way. The image can accurately reflect the state of the physical entity based on the collected data, making the environment of the DT model closer to the real environment. Therefore, integrating digital twin technology into deep reinforcement learning to improve training efficiency has become a new idea. At the same time, edge devices have powerful computing power and storage resources, which can serve as an excellent resource supplement for crocodile-like robots.

[0068] Specifically, the crocodile robot twin system model includes a crocodile robot twin model and a terrain environment model, wherein the crocodile robot twin model is a digital reproduction of the crocodile robot physical entity, and the terrain environment model is a description of the terrain environment collected and constructed by the sensors carried in the crocodile robot physical entity. Edge devices refer to devices that process and store data at the edge of the network and transmit information to and from the twin physical entity through a communication module. They are composed of a data processing unit, a data storage unit, and a communication module, wherein the communication module uses wireless communication methods such as Wi-Fi, 5G, etc.

[0069] Furthermore, the role of the edge device includes two aspects. The first aspect is to establish a twin system model of the crocodile robot and perform pre-training, which is used as the basis for real-time training to accelerate the generation of gait control strategies. The second aspect is to perform all operations of data fusion, analysis and calculation during real-time training, which makes up for the limited computing and storage capabilities of the crocodile robot, saves computing resources, and reduces the delay of gait control.

[0070] Furthermore, the specific process of establishing the twin system model is as follows: design a Solidworks three-dimensional model of a four-legged crocodile robot with a total of 8 degrees of freedom. Each leg has two degrees of freedom at the hip joint and the knee joint. Set the coordinate system and rotation axis of each joint, and export the URDF file format to obtain the crocodile robot twin model; obtain the physical entity of the crocodile robot through 3D printing and installation of sensors, and obtain the point cloud data of the surrounding terrain through its built-in laser radar, and process the point cloud data to generate a three-dimensional terrain environment model; finally, store the crocodile robot twin system model in the edge device.

[0071] Furthermore, the specific method of using point cloud data to generate a three-dimensional terrain environment model is as follows: after the point cloud data is collected, it is first denoised and downsampled to reduce the amount of data. Then, the point cloud data is adjusted to the robot's coordinate system through coordinate conversion, and point cloud fusion is performed using algorithms such as ICP to form a complete environment model. Subsequently, a three-dimensional mesh or surface model is generated using point cloud processing software for visualization and feature extraction to analyze terrain features such as terrain height and obstacle distance.

[0072] Step 2: Construct the state space and action space of the gait control of the crocodile robot:

[0073] Specifically, the state space is the state information of the crocodile robot twin system, including the crocodile robot state information and terrain environment information, and the action space is the rotation angle of each joint motor of the crocodile robot;

[0074] Furthermore, the specific method for designing the state space and action space is:

[0075] The state space consists of two parts: the state information of the crocodile robot and the terrain environment information, with a total of 19 dimensions. The first part is the state information of the crocodile robot, with a total of 17 dimensions, including the rotation angles of the hip and knee joints of the four legs (s h1 ,s h2 ,s h3 ,s h4 ,s k1 ,s k2 ,s k3 ,s k4 ), fuselage roll, pitch and yaw angles (s r ,s p ,s y ), fuselage roll, pitch, yaw angular velocity , the displacement of the center of mass of the fuselage in the x, y, and z directions (s x ,s y ,s z ); The second part is the terrain environment status information, which has two dimensions, namely obstacle height and obstacle distance (s h ,ss ). The state space is expressed as follows:

[0076]

[0077] In formula (1), s h1 ,s h2 ,s h3 ,s h4 ,s k1 ,s k2 ,s k3 ,s k4 Represent the hip joint angles and knee joint angles of the left front leg, right front leg, left hind leg, and right hind leg respectively.

[0078] In the state space, the value ranges of various state information are different, which is not conducive to the learning of the policy network. The present invention normalizes all state information, and state normalization can accelerate the training process of the network. The normalization formula is as follows:

[0079]

[0080] In formula (2), x is the state value before normalization, and x∈(x min , x max ), x′ is the normalized state value, and x′∈[-1, 1].

[0081] The action space is the rotation angle of the hip and knee joints of the four legs of the crocodile robot, and a continuous action space and position control method is adopted. The action space is also normalized according to formula (2) to output a value of the same type. After normalization, x′∈[-1, 1]. The action space is expressed as follows:

[0082] a=[s h1 ,s h2 ,s h3 ,s h4 ,s k1 ,s k2 ,s k3 ,s k4 ] (3)

[0083] Step 3: Design a reward function that takes into account forward movement gait speed, stability, and low energy consumption:

[0084] Specifically, during the training of the four-legged crocodile robot, the learning goal is to encourage the robot to learn fast, stable and low-energy forward motion gait through exploration and trial and error. Therefore, the reward function includes forward motion reward, energy consumption penalty and stability reward; the forward motion reward includes forward speed reward and forward direction reward; the energy consumption only considers the torque and speed of each joint motor; the stability reward includes the floating center of mass height, falling and posture angle change of the crocodile robot; the final reward function is the linear superposition of the above reward functions;

[0085] Furthermore, the specific method for designing the reward function is:

[0086] (1) Forward motion reward: This reward is used to guide the crocodile robot to move in the target direction (the x-axis of the world coordinate system), including forward speed reward and forward direction reward.

[0087] ① Forward speed reward: This reward encourages the robot to move towards the target direction, and the distance that the robot's center of mass coordinates move in the target direction in the world coordinate system at the previous and next moments is as large as possible.

[0088] R x1 =k x1 (p x -p′ x ) / Δt (4)

[0089] In formula (4), p x is the position of the robot's center of mass on the x-axis in the world coordinate system at the current moment, p′ x is the x-axis position of the robot body at the previous moment, k x1 is the weight coefficient, and Δt represents the time step.

[0090] ② Forward direction reward: encourages the robot to move in the target direction and punishes deviation.

[0091]

[0092] In formula (5), are the expressions of the current direction vector and the expected direction vector in the world coordinate system respectively. The angle between the two vectors indicates the degree of deviation of the current direction from the expected direction. ori is the weight parameter.

[0093] The robot's forward motion reward is R x =R x1 +R ori

[0094] (2) Energy consumption penalty, which encourages the robot to achieve gait motion with the lowest possible energy consumption. The energy consumed by all motors in one time step is:

[0095]

[0096] In formula (6), i is the motor number, k is e is the weight parameter, τ i is the motor torque, ω i is the motor speed.

[0097] (3) Stability reward: This reward encourages the robot to walk with a stable gait, which includes penalties for center of mass height fluctuation, fall, and posture angle change.

[0098] ① Center of mass height floating penalty: This item encourages the robot to move stably without tipping over. Therefore, it is necessary to control the height error of the robot's center of mass coordinates in the z direction relative to the body when it is standing.

[0099] R z =-k z |p z -p z_ref | (7)

[0100] In formula (7), p z_ref is the center of mass height of the robot when it is standing still, p z is the height of the robot's center of mass at the current moment, k z is the weight parameter of this item.

[0101] ②Fall penalty: This penalty is for the robot to fall during training.

[0102] R z =R fall =-k fall ·r fall (8)

[0103] In formula (8), k fall is the weight of the fall penalty term, r fall is a constant, indicating a fixed fall penalty value.

[0104] ③ Attitude angle change penalty: This penalty is for excessive changes in the attitude angle of the robot while walking, thereby increasing walking stability.

[0105] R rpy =-k rpy (k roll |α roll -α′ roll |+k pitch |β pitch -β′ pitch |+k yaw |γ yaw -γ′ yaw |) (9)

[0106] In formula (9), k rpy is the weight of the attitude angle change penalty term, k roll is the weight coefficient of the roll angle change, k pitch is the weight coefficient of pitch angle change, k yaw Weight coefficient for yaw angle change, α roll , β pitch , γ yaw are the roll angle, pitch angle and yaw angle of the robot at the current moment, α′ roll , β′ pitch , γ′ yaw They are the roll angle, pitch angle and yaw angle of the robot at the previous moment respectively.

[0107] The robot gait stability reward function is R stable =R z +R fall +R rpy

[0108] (4) Conditions for early termination of a round:

[0109] During the reinforcement learning training process, the crocodile robot needs to conduct trial and error through repeated interactions with the environment. The design of the termination condition as a condition for judging a round has an important impact on the learning efficiency of the crocodile robot. When the current state of the robot deviates far from the set target state, the environment and agent state should be reset to improve the learning efficiency of the agent.

[0110] When the robot's roll angle and pitch angle are large, it may directly cause the robot to fall or lose balance; when the center of mass is lower than a certain height, it can be judged that the robot is in a falling state; when the yaw angle is too large, it can be judged that the robot has deviated from the target direction; when the number of steps in a round is too long, the control strategy is updated slowly and the learning effect is not significant. Therefore, the present invention proposes the following termination round conditions:

[0111] ① When the center of mass height is lower than 0.15m, the robot is judged to be in a falling state; ② One round exceeds 5000 steps; ③ The distance in the target direction exceeds 5m; ④ The lateral deviation exceeds 2m; ⑤ The roll angle and pitch angle exceed 60 degrees, and the yaw angle exceeds 30 degrees.

[0112] In summary, the overall reward function of the robot is:

[0113] R=R x +R e +R stable (10)

[0114] The designed reward mechanism integrates the goals of gait speed, stability and energy consumption, designs reward and penalty items, and adds weight coefficients in the combination process. By adjusting the weights of each reward item, different task goals can be encouraged, and finally the desired control effect of the quadruped crocodile robot can be achieved.

[0115] Step 4: Use multiple processes to collect data in parallel for pre-training to obtain the twin decision model:

[0116] Specifically, each sub-process in the multi-process is responsible for using CPU resources to explore the environment and collect historical data, while the main process performs strategy updates in the GPU; the DRL algorithm is deployed in the twin decision model, using the Actor-Critic network architecture to provide decision output for gait control;

[0117] refer to Figure 2 Furthermore, the pre-training of the DT model is carried out on the edge device, utilizing the multi-core CPU resources of the edge device and adopting multi-process parallel collection of historical data. The historical data collection work is distributed to multiple twin agents. Each twin agent interacts with the environment independently and collects data, which are then aggregated into an experience buffer. The twin decision model samples in batches in the experience buffer, performs strategy updates in the GPU, and finally distributes the updated network parameters to each twin agent, in turn; multi-process computing is used to improve the efficiency of pre-training data collection and speed up the training of the decision model.

[0118] Step 5: Deploy the twin decision model in the edge device, use the communication module to transmit information with the twin physical entity, and perform real-time training on the crocodile robot.

[0119] In order to capture the characteristics of the dynamic environment and obtain an accurate DT model of the current environment, the present invention establishes a DT model of the crocodile robot system from the perspective of execution within the edge device and updates the crocodile robot DRL decision network in real time.

[0120] refer to Figure 3 Specifically, in the real-time training stage, the physical entity of the crocodile robot is equipped with high-definition sensors to collect state information, and the edge device uses the perception data from the physical entity to update the twin system model in real time at each time step. The twin decision model extracts the state data required for gait control from the twin system model, generates a decision output, and transmits it to the twin physical entity through the communication module. The twin physical entity receives the decision output and performs gait actions. In a sequential cycle, the twin system model will continue to obtain real-time trajectory sample data (r, s, s_next, d, a) from the physical entity and store it in the experience buffer of the edge device for real-time training of the twin decision model.

[0121] In summary, the present invention uses digital twin technology to establish a twin system model of the crocodile robot and store it in the edge device. By pre-training the DT model first, then sampling by the physical entity in the real environment, and communicating with the DT model for real-time training, the resource-intensive utilization rate in the crocodile robot system and the training efficiency of DRL for gait control are improved.

[0122] The crocodile robot twin system model is used for pre-training, so that the edge device can learn the gait characteristics and generate the crocodile robot DRL network decision value. This value can be directly applied to the crocodile robot, reducing the number of real-time training times and speeding up the generation of gait control strategies.

[0123] The edge device performs all operations of data fusion, analysis, and calculation during real-time training. The physical entity is responsible for the specific execution of gait movements and uses built-in sensors to collect status information to update the DT model in real time. This method not only increases the depth and breadth of the training model, but also shortens the data training cycle. At the same time, training the twin decision model on the edge device makes up for the limited computing and storage capabilities of the crocodile robot, saves computing resources, and reduces the delay of gait control.

Claims

1. A digital twin-assisted deep reinforcement learning method for gait control of a crocodile robot, which is divided into two stages: pre-training and real-time training, and is characterized in that: The steps include: Step 1: Use simulation and modeling technology to build a twin system model of the crocodile robot and store it in the edge device; The crocodile robot twin system model includes a crocodile robot twin model and a terrain environment model. The crocodile robot twin model is a digital reproduction of the crocodile robot physical entity. The terrain environment model is a terrain environment description collected and constructed by sensors carried in the crocodile robot physical entity. The edge device refers to a device that processes and stores data at the edge of the network and transmits information to and from the twin physical entity through a communication module. It is composed of a data processing unit, a data storage unit and a communication module. Step 2: Construct the state space and action space of the crocodile robot's gait control: The state space is the state information of the crocodile robot twin system, including the crocodile robot state information and terrain environment information, and the action space is the rotation angle of each joint motor of the crocodile robot; Step 3: Design a reward function that takes into account forward movement gait speed, stability, and low energy consumption: The reward function includes forward motion reward, energy consumption penalty and stability reward; the forward motion reward includes forward speed reward and forward direction reward; the energy consumption only considers the torque and speed of each joint motor; the stability reward includes the height fluctuation, fall and posture angle change of the crocodile robot's center of mass; the final reward function is the linear superposition of the above reward functions; Step 4: Use multiple processes to collect data in parallel for pre-training to obtain the twin decision model: Each sub-process in the multi-process is responsible for exploring the environment and collecting historical data using CPU resources, while the main process performs strategy updates in the GPU; the DRL algorithm is deployed in the twin decision model, using the Actor-Critic network architecture; Step 5: Deploy the twin decision model in the edge device, use the communication module to transmit information with the twin physical entity, and perform real-time training on the crocodile robot.

2. According to claim 1, a digital twin-assisted crocodile robot gait control deep reinforcement learning method is characterized in that: The specific method of step 1, using simulation and modeling technology to build a twin system model of the crocodile robot and store it in the edge device is: A Solidworks 3D model of a four-legged crocodile robot is designed with a total of 8 degrees of freedom. Each leg has two degrees of freedom at the hip and knee joints. The coordinate system and rotation axis of each joint are set, and the URDF file format is exported to obtain the crocodile robot twin model. The physical entity of the crocodile robot is obtained by 3D printing and installing sensors, and the point cloud data of the surrounding terrain is obtained through its built-in laser radar. The point cloud data is processed to obtain a three-dimensional terrain environment model. Finally, the crocodile robot twin system model is stored in the edge device.

3. According to claim 1, a digital twin-assisted crocodile robot gait control deep reinforcement learning method is characterized in that: In step 2, the specific method for constructing the state space and action space of the gait control of the crocodile-like robot is: The state space consists of two parts: the state information of the crocodile robot and the terrain environment information, with a total of 19 dimensions. The first part is the state information of the crocodile robot, with a total of 17 dimensions, including the rotation angles of the hip and knee joints of the four legs (s h1 ,s h2 ,s h3 ,s h4 ,s k1 ,s k2 ,s k3 ,s k4 ), fuselage roll, pitch and yaw angles (s r ,s p ,s y ), fuselage roll, pitch, yaw angular velocity Displacement of the center of mass of the fuselage in the x, y, and z directions (s x ,s y ,s z ); The second part is the terrain environment status information, which has two dimensions, namely obstacle height and obstacle distance (s h ,s s ). The state space is expressed as follows: In formula (1), s h1 ,s h2 ,s h3 ,s h4 ,s k1 ,s k2 ,s k3 ,s k4 Represent the hip joint angles and knee joint angles of the left front leg, right front leg, left hind leg, and right hind leg respectively. In the state space, the value ranges of various state information are different, which is not conducive to the learning of the policy network. The present invention normalizes all state information. The normalization formula is as follows: In formula (2), x is the state value before normalization, and x∈(x min ,x max ), x′ is the normalized state value, and x′∈[-1,1]. The action space is the rotation angle of the hip and knee joints of the four legs of the crocodile robot, and a continuous action space and position control method is adopted. The action space is also normalized according to formula (2) to output a value of the same type. After normalization, x′∈[-1,1]. The action space is expressed as follows: a=[s h1 ,s h2 ,s h3 ,s h4 ,s k1 ,s k2 ,s k3 ,s k4 ] (3)。 4. According to claim 1, a digital twin-assisted crocodile robot gait control deep reinforcement learning method is characterized in that: In step 3, the specific method for designing a reward function that takes into account forward motion gait speed, stability, and low energy consumption is: (1) Forward motion reward: This reward is used to guide the crocodile robot to move in the target direction (the x-axis of the world coordinate system), including forward speed reward and forward direction reward. ① Forward speed reward: This reward encourages the robot to move in the target direction, and the distance that the robot's center of mass coordinates move in the target direction in the world coordinate system at the previous and next moments is as large as possible. R x1 =k x1 (p x -p′ x ) / Δt (4) In formula (4), p x is the position of the robot's center of mass on the x-axis of the world coordinate system at the current moment, p′ x is the x-axis position of the robot body at the previous moment, k x1 is the weight coefficient, and Δt represents the time step. ② Forward direction reward: Encourage the robot to move in the desired direction and punish deviation. In formula (5), are the expressions of the current direction vector and the expected direction vector in the world coordinate system respectively. The angle between the two vectors indicates the degree of deviation of the current direction from the expected direction. ori is the weight parameter. The robot's forward motion reward is R x =R x1 +R ori (2) Energy consumption penalty, which encourages the robot to achieve gait motion with the lowest possible energy consumption. The energy consumed by all motors in one time step is: In formula (6), i is the motor number, k is e is the weight parameter, τ i is the motor torque, ω i is the motor speed. (3) Stability reward: This reward encourages the robot to walk with a stable gait, which includes penalties for center of mass height fluctuation, fall, and posture angle change. ① Center of mass height floating penalty: This item encourages the robot to achieve stable movement so that the center of mass height is close to the height when standing still. R z =-k z |p z -p z_ref | (7) In formula (7), p z_ref is the center of mass height of the robot when it is standing still, p z is the height of the robot's center of mass at the current moment, k z is the weight parameter of this item. ②Fall penalty: This penalty is for the robot to fall during training. R fall =-k fall ·r fall (8) In formula (8), k fall is the weight of the fall penalty term, r fall is a constant, indicating a fixed fall penalty value. ③ Attitude angle change penalty: This penalty is for excessive changes in the attitude angle of the robot while walking, thereby increasing walking stability. R rpy =-k rpy (k roll |a roll -a' roll |+k pitch |b pitch -b′ pitch |+k yaw |c yaw -c′ yaw |) (9) In formula (9), k rpy is the weight of the attitude angle change penalty term, k roll is the weight coefficient of the roll angle change, k pitch is the weight coefficient of pitch angle change, k yaw Weight coefficient for yaw angle change, α roll , β pitch , γ yaw are the roll angle, pitch angle and yaw angle of the robot at the current moment, α′ roll , β′ pitch , γ′ yaw They are the roll angle, pitch angle and yaw angle of the robot at the previous moment respectively. The robot gait stability reward function is R stable =R z +R fall +R rpy (4) Conditions for early termination of a round: ① When the center of mass height is lower than 0.15m, the robot is judged to be in a falling state; ② One round exceeds 5000 steps; ③ The distance in the target direction exceeds 5m; The lateral deviation of t exceeds 2m; ⑤ The roll angle and pitch angle exceed 60 degrees, and the yaw angle exceeds 30 degrees. In summary, the overall reward function of the robot is: R=R x +R e +R stable (10)。 5. According to claim 1, a digital twin-assisted crocodile robot gait control deep reinforcement learning method is characterized in that: In step 4, multiple processes are used to collect data in parallel for pre-training, and the specific method for obtaining the twin decision model is: The pre-training of the DT model is carried out on the edge device, utilizing the multi-core CPU resources of the edge device and adopting multi-process parallel collection of historical data. The historical data collection work is distributed to multiple twin agents. Each twin agent interacts with the environment and collects data independently, and then aggregates them into an experience buffer. The twin decision model samples in batches in the experience buffer, performs strategy updates in the GPU, and finally distributes the updated network parameters to each twin agent, in a cycle.

6. The method of deep reinforcement learning for gait control of a crocodile robot assisted by a digital twin according to claim 1, characterized in that: In step 5, the twin decision model is deployed in the edge device, and the communication module is used to transmit information to the twin physical entity. The specific method for real-time training of the crocodile-like robot is as follows: During the real-time training phase, the physical entity of the crocodile robot is equipped with high-definition sensors to collect status information, and the edge device uses the perception data from the physical entity to update the twin system model in real time at each time step. The twin decision model extracts the status data required for gait control from the twin system model, generates a decision output, and transmits it to the twin physical entity through the communication module. The twin physical entity receives the decision output and performs gait actions. In a sequential cycle, the twin system model will continue to obtain real-time trajectory sample data (r, s, s_next, d, a) from the physical entity and store it in the experience buffer of the edge device for real-time training of the twin decision model.

Images