Robot motion control strategy network training method based on deep reinforcement learning, robot motion control method, device, equipment, robot and storage medium

Abstract

This disclosure provides a robot motion control policy network training method, robot motion control method, device, equipment, robot, and storage medium based on deep reinforcement learning, relating to the field of robotics technology. This disclosure predicts future hidden states by utilizing body perception data and terrain perception data from the robot's historical state data. This enables the motion control policy network to learn the temporal patterns of the robot's body perception data and terrain perception data. Based on these temporal patterns, the motion control policy network can combine the currently perceived and processed terrain states to probabilistically predict unobserved terrain states ahead. This allows for dynamic adjustments to gait parameters, body posture, and other control quantities in advance during subsequent movement, improving the continuity and stability of the entire movement trajectory, avoiding problems such as falls, and significantly improving the robot's control performance in challenging tasks such as running and climbing.

Description

Technical Field

[0001] This disclosure relates to the field of robotics technology, and in particular to a method for training a robot motion control strategy network based on deep reinforcement learning, a robot motion control method, a device, an equipment, a robot, and a storage medium. Background Technology

[0002] Currently, in high-speed or highly dynamic and difficult tasks such as running and climbing, robots not only need to rely on body perception data, but may also need to rely on high-dimensional observation data, including terrain perception data. The motion control strategy network needs to process both body perception data and terrain perception data simultaneously during real-time control.

[0003] However, it takes time for a robot to grasp and process environmental perception information. In high-speed or high-dynamic motion scenarios, the robot may have already stepped out of or is about to step out of the current map perception range before the terrain perception data processing is completed. This time lag in perception and decision-making will cause the motion control strategy network to fail to obtain accurate feedback on the terrain ahead in a timely manner, resulting in unstable robot motion control, a tendency to fall, or poor performance.

[0004] Current related technologies, such as the deep reinforcement learning algorithm disclosed in Chinese patent application CN119830943A, which combines auxiliary tasks, empirical double sampling, and dynamic correction of value functions to control the actions of intelligent agents, have not specifically addressed the problem of perception lag under high-speed motion. While introducing auxiliary tasks improves the quality of feature extraction and enhances the model's understanding of the state, and empirical double sampling improves sample utilization efficiency and accelerates the learning process, they do not establish an effective prediction mechanism specifically for the problem of perception lag under high-speed motion. This makes it difficult for policy networks to predict future terrain and states when facing highly dynamic tasks that require rapid responses, resulting in unstable robot motion control, susceptibility to falls, or poor action performance.

[0005] Therefore, the current robot motion control strategy network still needs further optimization. Summary of the Invention

[0006] To address the problems existing in the aforementioned related technologies, this disclosure provides a robot motion control strategy network training method, robot motion control method, device, equipment, robot, and storage medium based on deep reinforcement learning. When the robot actually performs high-speed or high-dynamic motion tasks, the motion control strategy network can also predict the trend of the terrain ahead based on the temporal patterns learned internally. It can complete reasonable action preparation and compensation before entering unknown areas, improve the continuity and stability of the entire motion trajectory, avoid problems such as falling, and significantly improve the robot's control performance in high-difficulty tasks such as running and climbing.

[0007] The first aspect of this disclosure provides a method for training a robot motion control policy network based on deep reinforcement learning, including:

[0008] The robot's historical state data is acquired and input into the encoder to obtain the historical hidden state. The historical state data includes the robot's body perception data and terrain perception data.

[0009] The historical hidden states are input into the motion control policy network to generate action policies, and the historical hidden states are input into the dynamic transition model to generate predicted hidden states for multiple future time steps.

[0010] The first loss is calculated based on the difference between the predicted hidden state and the actual hidden state at multiple future time steps, and the second loss is calculated based on the action policy output by the motion control policy network. The parameters of the encoder, motion control policy network and dynamic transfer model are updated according to the first loss and the second loss.

[0011] Optionally, the above-mentioned acquisition of the robot's historical state data includes:

[0012] Acquire the robot's ontological perception data and terrain perception data at multiple historical time steps;

[0013] Align the terrain perception data with the ontology perception data by time steps to obtain state data for multiple historical time steps.

[0014] The robot's historical state data is obtained by sorting the state data of multiple historical time steps in chronological order.

[0015] Optionally, the above-mentioned input of historical state data into the encoder to obtain historical hidden states includes:

[0016] The latent state of the ontology is obtained by extracting the temporal features of ontology-aware data from historical state data based on the encoder.

[0017] The terrain latent state is obtained by extracting the temporal features of terrain sensing data from historical state data based on the encoder.

[0018] The historical hidden state is obtained by fusing the ontological hidden state and the terrain hidden state.

[0019] Optionally, the above-mentioned inputting historical hidden states into the motion control policy network to generate action policies includes:

[0020] The historical hidden states are input into the motion control policy network. For the hidden state of each historical time step in the historical hidden state, the motion control policy network performs action reasoning based on the hidden state and outputs the action policy corresponding to the hidden state of the historical time step.

[0021] Optionally, the above-mentioned inputting historical hidden states into the dynamic transition model to generate predicted hidden states for multiple future time steps includes:

[0022] The first hidden state of the first time step in the historical hidden state is input into the dynamic transition model;

[0023] Based on the dynamic transition model, the state prediction of the first hidden state is performed for multiple future time steps to obtain the predicted hidden state for multiple future time steps.

[0024] Optionally, the above-mentioned prediction of the first hidden state based on the dynamic transition model for multiple future time steps yields predicted hidden states for multiple future time steps, including:

[0025] Based on the dynamic transition model, the state prediction of the first hidden state at the next time step is performed to obtain the predicted hidden state at the second time step.

[0026] The predicted hidden state of the second time step is input into the dynamic transition model. Based on the dynamic transition model, the predicted hidden state of the second time step is used to predict the state of the next time step, thus obtaining the predicted hidden state of the third time step.

[0027] The predicted hidden state of the third time step is input into the dynamic transition model, and the dynamic transition model is repeatedly called to predict the state, and the predicted hidden states of multiple future time steps are obtained by combining them.

[0028] Optionally, the calculation of the first loss based on the difference between the predicted hidden state and the actual hidden state at multiple future time steps includes:

[0029] For each future time step, determine the target historical time step corresponding to the future time step, and determine the hidden state of the target historical time step in the historical hidden state as the true hidden state of the future time step.

[0030] For each future time step, calculate the difference between the predicted hidden state and the actual hidden state of the future time step to obtain the state difference data corresponding to the future time step.

[0031] The first loss is obtained by weighted summation of the state difference data corresponding to each future time step.

[0032] Optionally, the above calculation of the difference between the predicted hidden state at the future time step and the actual hidden state at the future time step yields the state difference data corresponding to the future time step, including:

[0033] Calculate the first mean squared error between the predicted hidden state at the future time step and the actual hidden state at the future time step, and use the first mean squared error as the state difference data corresponding to the future time step.

[0034] Optionally, the calculation of the first mean squared error between the predicted hidden state at a future time step and the actual hidden state at a future time step includes:

[0035] Calculate the first difference vector between the predicted hidden state at the future time step and the actual hidden state at the future time step;

[0036] Squaring the dimension value of each dimension in the first difference vector yields the first squared value of each dimension. The first squared values ​​of each dimension are then summed to obtain the first total sum of squares.

[0037] Divide the first total sum of squares by the dimension of the first difference vector to obtain the first mean square error.

[0038] Optionally, the calculation of the second loss based on the action policy output by the motion control policy network includes:

[0039] For each historical time step in the historical state data and the corresponding action strategy, the robot is controlled to execute the action strategy under the state data in the simulation environment. The simulation environment returns the target action taken by the robot when executing the action strategy, as well as the reward value and target state data of the robot after executing the target action.

[0040] The second loss is calculated based on the reward value corresponding to the historical state data and the target state data.

[0041] Optionally, the second loss is calculated based on the reward value corresponding to the historical state data and the target state data, including:

[0042] For each historical time step in the historical state data, based on the state data, the corresponding reward value, and the target state data, determine the advantage value for the robot to perform the target action corresponding to the state data.

[0043] Based on the advantage value corresponding to the state data and the execution probability of the target action in the action policy corresponding to the state data, calculate the sub-loss corresponding to the state data;

[0044] The second loss is obtained by summing the sub-losses corresponding to the state data at each historical time step in the historical state data.

[0045] Optionally, the above-mentioned determination of the robot's advantage value for performing the target action based on the state data, the reward value corresponding to the state data, and the target state data includes:

[0046] The current state value is obtained by evaluating the value of state data based on the value network, and the target state value is obtained by evaluating the value of target state data based on the value network.

[0047] The reward value and the target state value are summed to obtain the target sum value;

[0048] Calculate the difference between the target value and the current state value to obtain the advantage value corresponding to the state data.

[0049] Optionally, the parameters of the encoder, motion control policy network, and dynamic transfer model are updated based on the first loss and the second loss, including:

[0050] Obtain the state loss weight corresponding to the first loss and the policy loss weight corresponding to the second loss;

[0051] Based on the state loss weight and policy loss weight, the first loss and the second loss are weighted and summed to obtain the joint loss.

[0052] Based on the joint loss, the parameters of the encoder, motion control policy network, and dynamic transfer model are updated.

[0053] Optionally, the parameters of the encoder, motion control policy network, and dynamic transfer model, based on the joint loss, include:

[0054] The encoder, motion control policy network, and dynamic transfer model are trained separately based on the joint loss until the joint loss reaches its minimum value. Then, the parameters of the encoder, motion control policy network, and dynamic transfer model are updated.

[0055] Optionally, the encoder, motion control policy network, and dynamic transfer model are trained separately based on the joint loss until the joint loss reaches its minimum value. Then, the parameters of the encoder, motion control policy network, and dynamic transfer model are updated, including:

[0056] With the goal of minimizing the joint loss, backpropagation calculations are performed on the encoder, motion control policy network, and dynamic transfer model based on the joint loss to obtain the gradient values ​​of each parameter in the encoder, motion control policy network, and dynamic transfer model.

[0057] Obtain the first preset learning rate of the encoder, the second preset learning rate of the motion control strategy network, and the third preset learning rate of the dynamic transfer model;

[0058] The parameters of the encoder, motion control strategy network, and dynamic transfer model are updated based on the gradient value, the first preset learning rate, the second preset learning rate, and the third preset learning rate.

[0059] Optionally, the parameters of the encoder, motion control policy network, and dynamic transfer model are updated based on the gradient value, the first preset learning rate, the second preset learning rate, and the third preset learning rate, including:

[0060] The product of the gradient value of each parameter in the encoder and the first preset learning rate is calculated to obtain the update increment of each parameter in the encoder; the product of the gradient value of each parameter in the motion control policy network and the second preset learning rate is calculated to obtain the update increment of each parameter in the motion control policy network; the product of the gradient value of each parameter in the dynamic transfer model and the third preset learning rate is calculated to obtain the update increment of each parameter in the dynamic transfer model.

[0061] For each parameter, sum the parameter and its update increment to generate the updated value of the parameter;

[0062] The parameters in the encoder, motion control strategy network, and dynamic transfer model are replaced with their updated values ​​to complete the parameter update for the encoder, motion control strategy network, and dynamic transfer model.

[0063] The second aspect of this disclosure provides a robot motion control method based on deep reinforcement learning, comprising:

[0064] Acquire the robot's current state data, which includes the robot's body perception data and terrain perception data;

[0065] The current state data is input into a pre-trained encoder to generate the target hidden state;

[0066] The target hidden state is input into a pre-trained motion control policy network, and the motion control policy network is used to output the action policy for controlling the robot's movement.

[0067] The encoder and motion control policy network are obtained according to the deep reinforcement learning-based robot motion control policy network training method described in the first aspect above.

[0068] A third aspect of this disclosure provides a training device for a robot motion control policy network based on deep reinforcement learning, comprising:

[0069] The first acquisition module is used to acquire the robot's historical state data and input the historical state data into the encoder to obtain the historical hidden state. The historical state data includes the robot's body perception data and terrain perception data.

[0070] The generation module is used to input historical hidden states into the motion control policy network to generate action policies, and to input historical hidden states into the dynamic transition model to generate predicted hidden states for multiple future time steps.

[0071] The update module is used to calculate the first loss based on the difference between the predicted hidden state and the actual hidden state at multiple future time steps, and to calculate the second loss based on the action policy output by the motion control policy network. Based on the first loss and the second loss, the parameters of the encoder, the motion control policy network and the dynamic transfer model are updated.

[0072] A fourth aspect of this disclosure provides a robot motion control device based on deep reinforcement learning, comprising:

[0073] The second acquisition module is used to acquire the robot's current state data, which includes the robot's body perception data and terrain perception data.

[0074] The hidden state generation module is used to input the current state data into the pre-trained encoder to generate the target hidden state.

[0075] The strategy generation module is used to input the target hidden state into the pre-trained motion control policy network, and output the action policy to control the robot's movement based on the motion control policy network;

[0076] The encoder and motion control policy network are obtained according to the deep reinforcement learning-based robot motion control policy network training method described in the first aspect above.

[0077] The fifth aspect of this disclosure provides an electronic device including a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements either the deep reinforcement learning-based robot motion control policy network training method of the first aspect or the deep reinforcement learning-based robot motion control method of the second aspect.

[0078] The sixth aspect of this disclosure provides a robot, including a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the deep reinforcement learning-based robot motion control method of the second aspect.

[0079] Optionally, the aforementioned robots include any one of legged robots, quadruped robots, bipedal robots, wheeled robots, wheel-legged robots, quadrupedal robots, humanoid robots, and mobile robots.

[0080] The seventh aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements either the deep reinforcement learning-based robot motion control policy network training method of the first aspect or the deep reinforcement learning-based robot motion control method of the second aspect.

[0081] The technical solution provided in this disclosure has the following advantages compared with the prior art:

[0082] In this embodiment of the disclosure, historical state data of the robot is acquired and input into the encoder to obtain historical hidden states. The historical state data includes the robot's body perception data and terrain perception data. The historical hidden states are input into the motion control policy network to generate action policies, and the historical hidden states are input into the dynamic transfer model to generate predicted hidden states for multiple future time steps. A first loss is calculated based on the difference between the predicted hidden states and the actual hidden states for multiple future time steps, and a second loss is calculated based on the action policies output by the motion control policy network. The parameters of the encoder, the motion control policy network, and the dynamic transfer model are updated according to the first loss and the second loss. Therefore, by simultaneously introducing an encoder, a motion control strategy network, and a dynamic transfer model during training, and utilizing the robot's historical state data, including proprioceptive and terrain perception data, to predict future hidden states, the motion control strategy network can learn the temporal patterns of these data—that is, the changing patterns of proprioceptive and terrain perception data over time. Based on these temporal patterns, the motion control strategy network can combine the currently perceived and processed terrain states to probabilistically predict unobserved terrain states ahead. This allows for dynamic adjustments to gait parameters, body posture, and other control variables in advance during subsequent movements, rather than passively waiting for new terrain observations. In this way, even when the robot is performing high-speed or high-dynamic tasks, the motion control strategy network can predict the terrain's trajectory based on the pre-learned temporal patterns, completing reasonable action preparation and compensation before entering unknown areas. This improves the continuity and stability of the entire movement trajectory, avoids problems such as falls, and significantly enhances the robot's control performance in challenging tasks such as running and climbing.

[0083] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0084] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0085] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0086] Figure 1 This is a schematic diagram of the perception range of a robot provided in an embodiment of this disclosure;

[0087] Figure 2 This is a system architecture diagram provided in an embodiment of the present disclosure;

[0088] Figure 3 This is a flowchart of a robot motion control policy network training method based on deep reinforcement learning provided in an embodiment of this disclosure;

[0089] Figure 4 This is a flowchart of a robot motion control policy network training method based on deep reinforcement learning provided in an embodiment of this disclosure;

[0090] Figure 5 This is a flowchart of a robot motion control policy network training method based on deep reinforcement learning provided in an embodiment of this disclosure;

[0091] Figure 6 This is a flowchart of a robot motion control policy network training method based on deep reinforcement learning provided in an embodiment of this disclosure;

[0092] Figure 7 This is a flowchart of a robot motion control policy network training method based on deep reinforcement learning provided in an embodiment of this disclosure;

[0093] Figure 8 This is a flowchart of a robot motion control policy network training method based on deep reinforcement learning provided in an embodiment of this disclosure;

[0094] Figure 9 This is a flowchart of a robot motion control policy network training method based on deep reinforcement learning provided in an embodiment of this disclosure;

[0095] Figure 10 This is a flowchart of a robot motion control policy network training method based on deep reinforcement learning provided in an embodiment of this disclosure;

[0096] Figure 11 This is a flowchart of a robot motion control method based on deep reinforcement learning provided in an embodiment of this disclosure;

[0097] Figure 12 This is a schematic diagram of the structure of a robot motion control strategy network training device based on deep reinforcement learning, provided in an embodiment of this disclosure.

[0098] Figure 13 This is a schematic diagram of the structure of a robot motion control device based on deep reinforcement learning provided in an embodiment of this disclosure;

[0099] Figure 14 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure;

[0100] Figure 15 This is a schematic diagram of a robot provided in an embodiment of this disclosure;

[0101] Figure 16 This is a schematic diagram of another robot provided in an embodiment of this disclosure;

[0102] Figure 17 This is a schematic diagram of yet another robot provided in an embodiment of this disclosure. Detailed Implementation

[0103] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0104] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0105] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0106] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0107] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0108] To better understand the inventive concept of the embodiments of this disclosure, the technical solutions of the embodiments of this disclosure will be described below in conjunction with exemplary embodiments.

[0109] First, the relevant terms involved in the embodiments of this disclosure will be explained.

[0110] Deep Reinforcement Learning (DRL) can be understood as a combination of deep learning (DL) and reinforcement learning (RL), integrating the feature extraction capabilities of deep learning and the decision-making capabilities of reinforcement learning. It is a learning method that allows an agent to interact with the environment, continuously try and learn in the environment, and adjust its own parameters to obtain the optimal policy.

[0111] A robot can be understood as a programmable mechanism with a certain degree of autonomy, capable of performing motion, manipulation, or positioning.

[0112] Historical state data can include the robot's body perception data and terrain perception data.

[0113] Robot proprioception data can be understood as the observational information that a robot directly acquires through its own sensors. For example, robot proprioception data may include, but is not limited to, data such as the robot's joint angles, joint velocities, walking speed, and posture information.

[0114] A joint can be understood as a mechanical component that connects two rigid bodies and constrains the relative motion between them.

[0115] A robot's terrain perception data can be understood as the terrain data surrounding the robot acquired through sensors. Terrain data can be understood as geographic information data describing the shape and undulation features of the ground.

[0116] Motion control policy networks can be understood as a type of neural network based on deep reinforcement learning algorithms.

[0117] The dynamic transfer model can be understood as a forward prediction model.

[0118] A simulation environment can be understood as a virtual operating platform used to simulate robot dynamics, sensor feedback, and interaction mechanisms with the outside world.

[0119] A value network can be understood as a neural network that predicts the total future revenue and is used to evaluate the state value corresponding to the robot's state data, or simply state value.

[0120] The gradient value can be understood as a numerical value used to characterize the direction of updating model parameters.

[0121] The learning rate (LR) can be understood as the magnitude or step size of the update of model parameters.

[0122] Currently, in high-speed or highly dynamic and difficult tasks such as running and climbing, robots not only need to rely on body perception data, but may also need to rely on high-dimensional observation data, including terrain perception data. The motion control strategy network needs to process both body perception data and terrain perception data simultaneously during real-time control.

[0123] However, it takes time for a robot to grasp and process environmental perception information. In high-speed or high-dynamic motion scenarios, the robot may have already stepped out of or is about to step out of the current map perception range before the terrain perception data processing is completed. This time lag in perception and decision-making will cause the motion control strategy network to fail to obtain accurate feedback on the terrain ahead in a timely manner, resulting in unstable robot motion control, a tendency to fall, or poor performance.

[0124] For example, Figure 1 A schematic diagram of a robot's perception range is provided, such as... Figure 1 As shown, location A is the visual sensor installed on the robot, which allows the robot to perceive terrain data. Area ACD represents the robot's visual perception range 101, i.e., the area that the robot can perceive through the visual sensor; area ABC represents the robot's non-visual perception area 102, i.e., the area that the robot cannot perceive through the visual sensor; area CD represents the ground 103, and area BC represents the ground 104. Figure 1 In the above, ground 103 is located in the robot's visual perception area 101, and the robot can perceive the terrain data of ground 103; ground 104 is located in the robot's non-visual perception area 102, and the robot cannot perceive the terrain data of ground 104. In high-speed or high-dynamic motion scenarios, the robot may not have finished processing the terrain perception data of ground 103 before its feet have entered ground 104. Since the robot cannot perceive the terrain data of ground 104, the motion control strategy network will not be able to obtain the terrain state of ground 104 in time, which will lead to unstable motion control, easy falls, or poor movement performance when the robot moves on ground 104.

[0125] Therefore, the current robot motion control strategy network still needs further optimization.

[0126] Figure 2 The diagram illustrates a system architecture for a robot motion control policy network training method and a robot motion control method based on deep reinforcement learning, which can be applied to embodiments of this disclosure.

[0127] like Figure 2 As shown, the system architecture 200 may include a terminal device 201, a robot 202, a network 203, and a server 204. The terminal device 201 includes, but is not limited to, desktop computers, laptops, smartphones, and tablets. The terminal device 201 is equipped with a graphical user interface, which provides visualization functions to display the robot 202's operating status, motion trajectory, etc. It can also be used to configure training parameters, control the training process, monitor the model training status, and support sending motion control commands to the robot 202.

[0128] Robot 202 is equipped with a variety of sensors. Robot 202 can collect body perception data during its movement through the sensors; Robot 202 can also collect terrain perception data through the sensors, which is the terrain data around the robot.

[0129] Server 204 is equipped with a training module for implementing a deep reinforcement learning-based robot motion control policy network training method. This module acquires historical state data of the robot and, based on this data, trains an encoder, a motion control policy network, and a dynamic transfer model using deep reinforcement learning. After training, server 204 can deploy the trained encoder and motion control policy network to robot 202, enabling robot 202 to output motion control policies based on these technologies.

[0130] Network 203 serves as the medium for providing a communication link between terminal device 201, robot 202, and server 204. Network 203 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc. It should be understood that... Figure 2 The number and type of terminal devices, robots, networks, and servers shown are merely illustrative. Depending on implementation needs, any number and type of terminal devices, robots, networks, and servers can be included.

[0131] Through the coordinated operation of various parts in the system architecture 200, the encoder, the robot's motion control strategy network, and the dynamic transfer model can be trained efficiently. By simultaneously introducing the encoder, motion control strategy network, and dynamic transfer model during the training process, and using the body perception data and terrain perception data in the robot's historical state data to predict future hidden states, the motion control strategy network can learn the temporal patterns of the robot's body perception data and terrain perception data, that is, the changing patterns of body perception data and terrain perception data in the time dimension. Based on these temporal patterns, the motion control strategy network can combine the currently perceived and processed terrain state to make probabilistic predictions of the unobserved terrain state ahead, thereby making dynamic adjustments to gait parameters, body posture, and other control quantities in advance during subsequent movements, rather than passively waiting for new terrain observation results before responding. In this way, even when the robot is actually performing high-speed or high-dynamic motion tasks, the motion control strategy network can predict the trend of the terrain ahead based on the pre-learned temporal patterns, complete reasonable action preparation and compensation before entering unknown areas, improve the continuity and stability of the entire motion trajectory, avoid problems such as falling, and significantly improve the robot's control performance in high-difficulty tasks such as running and climbing.

[0132] This disclosure provides a method for training a robot motion control policy network based on deep reinforcement learning, such as... Figure 3 The flowchart shown illustrates a method for training a robot motion control policy network based on deep reinforcement learning. This method may include the following steps 310 to 330:

[0133] Step 310: Obtain the robot's historical state data and input the historical state data into the encoder to obtain the historical hidden state. The historical state data includes the robot's body perception data and terrain perception data.

[0134] Step 320: Input the historical hidden state into the motion control policy network to generate the action policy, and input the historical hidden state into the dynamic transition model to generate the predicted hidden state for multiple future time steps.

[0135] Step 330: Calculate the first loss based on the difference between the predicted hidden state and the actual hidden state based on multiple future time steps, and calculate the second loss based on the action policy output by the motion control policy network. Then, update the parameters of the encoder, the motion control policy network, and the dynamic transfer model according to the first loss and the second loss.

[0136] The robot motion control policy network training method based on deep reinforcement learning provided in this embodiment can be implemented by acquiring historical state data of the robot and inputting the historical state data into the encoder to obtain historical hidden states, the historical state data including the robot's body perception data and terrain perception data; inputting the historical hidden states into the motion control policy network to generate action policies, and inputting the historical hidden states into the dynamic transition model to generate predicted hidden states for multiple future time steps; calculating a first loss based on the difference between the predicted hidden states and the actual hidden states for multiple future time steps, and calculating a second loss based on the action policies output by the motion control policy network; and updating the parameters of the encoder, the motion control policy network, and the dynamic transition model according to the first loss and the second loss. Therefore, by simultaneously introducing an encoder, a motion control strategy network, and a dynamic transfer model during training, and utilizing the robot's historical state data, including proprioceptive and terrain perception data, to predict future hidden states, the motion control strategy network can learn the temporal patterns of these data—that is, the changing patterns of proprioceptive and terrain perception data over time. Based on these temporal patterns, the motion control strategy network can combine the currently perceived and processed terrain states to probabilistically predict unobserved terrain states ahead. This allows for dynamic adjustments to gait parameters, body posture, and other control variables in advance during subsequent movements, rather than passively waiting for new terrain observations. In this way, even when the robot is performing high-speed or high-dynamic tasks, the motion control strategy network can predict the terrain's trajectory based on the pre-learned temporal patterns, completing reasonable action preparation and compensation before entering unknown areas. This improves the continuity and stability of the entire movement trajectory, avoids problems such as falls, and significantly enhances the robot's control performance in challenging tasks such as running and climbing.

[0137] For example, continue with Figure 1 For example, Figure 1 As shown above, through the above Figure 3 In the embodiments described, the motion control strategy network can learn the temporal patterns of the robot's body perception data and terrain perception data. Based on these temporal patterns, the motion control strategy network can combine the terrain state of the ground 103 that has already been perceived and processed to make a probabilistic prediction of the terrain state of the ground 104 that has not yet been observed. Thus, before reaching the ground 104, the network can make dynamic adjustments to the gait parameters, body posture and other control quantities to adapt to the terrain state of the ground 104, complete reasonable action preparation and compensation, improve the continuity and stability of the entire motion trajectory, avoid problems such as the robot falling after entering the ground 104, and significantly improve the robot's control performance in high-difficulty tasks such as running and climbing.

[0138] The following will provide a detailed description of the robot motion control policy network training method based on deep reinforcement learning in the embodiments of this disclosure.

[0139] In step 310, the robot's historical state data is acquired and input into the encoder to obtain the historical hidden state. The historical state data includes the robot's body perception data and terrain perception data.

[0140] Historical state data can include the robot's body perception data and terrain perception data.

[0141] Robot body perception data can be understood as the state data of the robot body acquired by the robot through sensors. For example, robot body perception data may include, but is not limited to, body data such as robot joint angles, robot joint velocities, robot walking speed, and robot posture information.

[0142] Robot body perception data can be collected through inertial measurement units (IMUs), joint position sensors, velocity sensors, angle sensors, etc. installed on the robot.

[0143] A robot's terrain perception data can be understood as the terrain data surrounding the robot acquired through sensors. Terrain data can be understood as geographic information data describing the shape and undulation features of the ground.

[0144] The robot's terrain perception data can be collected through radar sensors, vision sensors, and other sensors installed on the robot.

[0145] During its movement, the robot uses sensors to collect body perception data and terrain perception data, and then stores the body perception data and terrain perception data as historical state data in local storage space or in cloud server.

[0146] In this embodiment of the disclosure, the robot's historical state data can be obtained from the robot's local storage space or cloud server, and then the robot's historical state data can be input into the encoder. The encoder encodes the historical state data, that is, the historical state data is feature extracted to obtain the historical hidden state corresponding to the historical state data.

[0147] Hidden states can be understood as state characteristics.

[0148] In some example implementations, the above-described acquisition of the robot's historical state data can be performed by... Figure 4 The flowchart shown is a training method for a robot motion control policy network based on deep reinforcement learning. Figure 4 As shown, steps 410-430 are included:

[0149] Step 410: Obtain the robot's body perception data and terrain perception data at multiple historical time steps.

[0150] A time step can be understood as a process in which a robot performs an action and receives feedback.

[0151] Step 420: Align the terrain perception data and the ontology perception data by time steps to obtain state data for multiple historical time steps.

[0152] For example, multiple historical time steps include three historical time steps: t1, t2, and t3. The terrain perception data of t1 can be aligned with the ontology perception data of t1 to obtain the state data of t1; the terrain perception data of t2 can be aligned with the ontology perception data of t2 to obtain the state data of t2; and the terrain perception data of t3 can be aligned with the ontology perception data of t3 to obtain the state data of t3.

[0153] Step 430: Sort the state data of multiple historical time steps according to the chronological order of the time steps to obtain the robot's historical state data.

[0154] For example, if multiple historical time steps include three historical time steps t1, t2, and t3, and the order of these three time steps is t1, t2, and t3, then the state data of t1, t2, and t3 are sorted sequentially to obtain the robot's historical state data.

[0155] In some example implementations, the historical state data is input into the encoder to obtain the historical hidden state, which can then be executed. Figure 5 The flowchart shown is a training method for a robot motion control policy network based on deep reinforcement learning. Figure 5 As shown, steps 510-530 are included:

[0156] Step 510: Extract the temporal features of the ontology-aware data from the historical state data based on the encoder to obtain the ontology hidden state.

[0157] Step 520: Extract the temporal features of terrain sensing data from historical state data based on the encoder to obtain the terrain latent state.

[0158] Step 530: Merge the hidden state of the ontology and the hidden state of the terrain to obtain the hidden state of history.

[0159] Therefore, the historical hidden state can be obtained by fusing the body's hidden state and the terrain's hidden state, which can improve the accuracy of the robot's historical hidden state.

[0160] In step 320, the historical hidden states are input into the motion control policy network to generate the action policy, and the historical hidden states are input into the dynamic transition model to generate the predicted hidden states for multiple future time steps.

[0161] The motion control policy network can be understood as a neural network based on deep reinforcement learning algorithms. For example, the motion control policy network can be a multi-layer fully connected perceptron, a Transformer structure, etc.

[0162] The dynamic transfer model can be understood as a forward prediction model.

[0163] In some example implementations, the historical hidden states can be input into the motion control policy network. For the hidden state of each historical time step in the historical hidden states, the motion control policy network performs action reasoning based on the hidden state and outputs the action policy corresponding to the hidden state of the historical time step, thereby obtaining the action policy corresponding to the hidden state of each historical time step.

[0164] Action strategy can be understood as the probability distribution of actions under state data, that is, the execution probability of each action under state data.

[0165] In some example implementations, the above-described method of inputting historical hidden states into a dynamic transition model to generate predicted hidden states for multiple future time steps can be executed. Figure 6 The flowchart shown is a training method for a robot motion control policy network based on deep reinforcement learning. Figure 6 As shown, steps 610-620 are included:

[0166] Step 610: Input the first hidden state of the first time step in the historical hidden state into the dynamic transition model.

[0167] The historical hidden states include the robot's hidden states at multiple historical time steps. The hidden state at the first time step in the historical hidden states can be determined as the first hidden state, and the first hidden state can be input into the dynamic transition model.

[0168] Step 620: Based on the dynamic transition model, predict the state of the first hidden state for multiple future time steps to obtain the predicted hidden state for multiple future time steps.

[0169] In some example implementations, the first hidden state is predicted for multiple future time steps based on a dynamic transition model to obtain predicted hidden states for multiple future time steps, which may include steps 6201-6203:

[0170] Step 6201: Based on the dynamic transition model, predict the state of the first hidden state for the next time step to obtain the predicted hidden state for the second time step.

[0171] For example, if the first time step in the historical hidden state is t1, the state prediction for the next time step is made based on the dynamic transition model of the first hidden state of t1, resulting in the predicted hidden state Y1 for the second time step t2.

[0172] Step 6202: Input the predicted hidden state of the second time step into the dynamic transition model, and predict the state of the next time step based on the predicted hidden state of the second time step to obtain the predicted hidden state of the third time step.

[0173] For example, the predicted hidden state Y1 at the second time step t2 is input into the dynamic transition model, and the state prediction for the next time step is performed based on the predicted hidden state Y1 at the dynamic transition model to obtain the predicted hidden state Y2 at the third time step t3.

[0174] Step 6203: Input the predicted hidden state of the third time step into the dynamic transition model, repeatedly call the dynamic transition model to predict the state, and synthesize the predicted hidden states of multiple future time steps.

[0175] For example, if the first time step t1 includes five future time steps: t2, t3, t4, t5, and t6, repeatedly calling the dynamic transition model for state prediction can synthesize the predicted hidden states of multiple future time steps: Y1 for t2, Y2 for t3, Y3 for t4, Y4 for t5, and Y5 for t6.

[0176] Therefore, historical hidden states can be input into the dynamic transition model, and the predicted hidden states for multiple future time steps can be generated accurately and quickly based on the dynamic transition model.

[0177] In step 330 above, a first loss is calculated based on the difference between the predicted hidden state and the actual hidden state at multiple future time steps, and a second loss is calculated based on the action policy output by the motion control policy network. The parameters of the encoder, motion control policy network and dynamic transfer model are updated according to the first loss and the second loss.

[0178] The first loss can be understood as an indicator used to measure the difference between the predicted hidden state at a future time step and the actual hidden state at a future time step.

[0179] The second loss can be understood as an indicator used to measure the difference between the action policy output by the motion control policy network and the actual action executed by the robot.

[0180] In some example implementations, the calculation of the first loss based on the difference between the predicted hidden state and the actual hidden state at multiple future time steps can be performed. Figure 7The flowchart shown is a training method for a robot motion control policy network based on deep reinforcement learning. Figure 7 As shown, steps 710-730 are included:

[0181] Step 710: For each future time step, determine the target historical time step corresponding to the future time step, and determine the hidden state of the target historical time step in the historical hidden state as the true hidden state of the future time step.

[0182] The robot's historical hidden states include hidden states from multiple historical time steps, which are obtained based on the robot's historical actual state data. Future time steps correspond one-to-one with historical time steps. Therefore, for each future time step, the target historical time step can be determined, and the hidden state of the target historical time step in the historical hidden states can be identified as the true hidden state of that future time step.

[0183] Step 720: For each future time step, calculate the difference between the predicted hidden state and the actual hidden state of the future time step to obtain the state difference data corresponding to the future time step.

[0184] In one example implementation, a first mean squared error can be calculated between the predicted hidden state at a future time step and the actual hidden state at a future time step, and the first mean squared error can be used as the state difference data corresponding to the future time step.

[0185] The first mean squared error can be understood as an indicator used to measure the difference between the predicted hidden state at a future time step and the actual hidden state at that future time step.

[0186] For example, the calculation of the first mean square error between the predicted hidden state at a future time step and the actual hidden state at a future time step can include S11-S13:

[0187] S11. Calculate the first difference vector between the predicted hidden state at the future time step and the actual hidden state at the future time step.

[0188] S12. Squaring the dimension value of each dimension in the first difference vector to obtain the first square value of each dimension, and summing the first square values ​​of each dimension to obtain the first total square sum.

[0189] S13. Divide the first total sum of squares by the dimension of the first difference vector to obtain the first mean square error.

[0190] Therefore, the first mean square error between the predicted hidden state at the future time step and the actual hidden state at the future time step can be accurately calculated.

[0191] Step 730: Weight the state difference data corresponding to each future time step and sum them to obtain the first loss.

[0192] Therefore, the first loss can be accurately and quickly calculated based on the difference between the predicted hidden state and the actual hidden state at multiple future time steps.

[0193] In some example implementations, the second loss calculated based on the action policy output by the motion control policy network can be executed. Figure 8 The flowchart shown is a training method for a robot motion control policy network based on deep reinforcement learning. Figure 8 As shown, steps 810-820 are included:

[0194] Step 810: For the state data of each historical time step in the historical state data and the corresponding action strategy, control the robot to execute the action strategy under the state data in the simulation environment, and receive the target action taken by the robot when executing the action strategy, the reward value of the robot after executing the target action, and the target state data of the robot returned by the simulation environment.

[0195] A simulation environment can be understood as a virtual operating platform used to simulate robot dynamics, sensor feedback, and interaction mechanisms with the outside world.

[0196] Controlling a robot to execute an action strategy under this state data in a simulation environment can be understood as the virtual robot corresponding to the robot in the simulation environment executing an action strategy under this state data.

[0197] When a robot executes a motion strategy corresponding to its state data, it determines the target motion based on that strategy. The reward value can be understood as a numerical value generated by the simulation environment based on preset motion evaluation rules after the robot performs the target motion. This value characterizes the quality of the robot's execution of the target motion. The reward value can be used to measure the quality of the robot's execution of the target motion.

[0198] A higher reward value indicates that the robot performs the target action better; a lower reward value indicates that the robot performs the target action worse.

[0199] The simulation environment can generate corresponding reward values ​​based on the robot's stability, posture changes, and energy consumption when performing target actions in the simulation environment.

[0200] The robot's target state data can be understood as the robot's state data returned by the simulation environment after performing the target action. This target state data can include the robot's body perception data and terrain perception data returned by the simulation environment after performing the target action.

[0201] Step 820: Calculate the second loss based on the reward value corresponding to the historical state data and the target state data.

[0202] In some example implementations, the calculation of the second loss based on the reward value corresponding to the historical state data and the target state data may include steps 8201-8203:

[0203] Step 8201: For the state data of each historical time step in the historical state data, based on the state data, the reward value corresponding to the state data, and the target state data, determine the advantage value of the robot to perform the target action corresponding to the state data.

[0204] The advantage value can be understood as a metric that measures how much better a robot performs a target action compared to the average action given the given state data.

[0205] A dominance value greater than 0 indicates that the target action is more effective than the average action.

[0206] A dominance value less than 0 indicates that the target action is less effective than the average action.

[0207] When the advantage value is 0, it means that the effect of the target action is the same as the effect of the average action.

[0208] In one example implementation, the above-mentioned determination of the advantage value of the robot to perform the target action based on the state data, the reward value corresponding to the state data, and the target state data may include S21-S23:

[0209] S21. The value of the state data is evaluated based on the value network to obtain the current state value, and the value of the target state data corresponding to the state data is evaluated based on the value network to obtain the target state value.

[0210] A value network can be understood as a neural network that predicts the total future revenue, used to evaluate the state value corresponding to the robot's state data, or simply the state value. The state value can be understood as the magnitude of the robot's future revenue derived from its state data.

[0211] The current state value can be understood as the state value of the robot's state data.

[0212] The target state value can be understood as the state value of the target state data corresponding to the robot's current state data.

[0213] S22. Sum the reward value and the target state value to obtain the target sum value.

[0214] The target and the value can be understood as the actual benefit the robot gains from performing the target action under the given state data.

[0215] S23. Calculate the difference between the target value and the current state value to obtain the advantage value corresponding to the state data.

[0216] Step 8202: Based on the advantage value corresponding to the state data and the execution probability of the target action in the action strategy corresponding to the state data, calculate the sub-loss corresponding to the state data.

[0217] The probability of executing a target action can be understood as the likelihood that the robot will execute the target action under given state data.

[0218] For example, the advantage value corresponding to the state data can be multiplied by the execution probability of the target action in the action policy corresponding to the state data to obtain the target product, and then the negative value of the target product can be determined as the sub-loss corresponding to the state data.

[0219] Step 8203: Sum the sub-losses corresponding to the state data of each historical time step in the historical state data to obtain the second loss.

[0220] Therefore, a second loss can be calculated based on deep reinforcement learning algorithms, improving the accuracy and speed of the second loss calculation. By minimizing the second loss, the motion control policy network can adjust the action probability distribution in the action policy according to the magnitude of the advantage value, thereby increasing the selection probability of actions with larger advantage values ​​and decreasing the selection probability of actions with smaller advantage values. This achieves continuous optimization of the motion control policy network, enabling it to maximize the expected policy reward.

[0221] In some example implementations, updating the parameters of the encoder, motion control policy network, and dynamic transfer model based on the first loss and the second loss can be performed. Figure 9 The flowchart shown is a training method for a robot motion control policy network based on deep reinforcement learning. Figure 9 As shown, steps 910-930 are included:

[0222] Step 910: Obtain the state loss weight corresponding to the first loss and the policy loss weight corresponding to the second loss.

[0223] The state loss weight and policy loss weight can be preset as needed, and are not limited here.

[0224] Step 920: Based on the state loss weight and policy loss weight, the first loss and the second loss are weighted and summed to obtain the joint loss.

[0225] Step 930: Based on the joint loss, update the parameters of the encoder, motion control policy network, and dynamic transfer model.

[0226] Therefore, the parameters of the encoder, motion control policy network, and dynamic transfer model can be updated by combining the state loss of the dynamic transfer model and the policy loss of the motion control policy network, resulting in a well-trained encoder, motion control policy network, and dynamic transfer model. By simultaneously introducing the encoder, motion control policy network, and dynamic transfer model during training, and using the body perception data and terrain perception data from the robot's historical state data to predict future hidden states, the motion control policy network can learn the temporal patterns of the robot's body perception data and terrain perception data, that is, the changing patterns of the body perception data and terrain perception data in the time dimension. Based on these temporal patterns, the motion control policy network can combine the currently perceived and processed terrain states to make probabilistic predictions of the unobserved terrain states ahead, thereby dynamically adjusting gait parameters, body posture, and other control variables in advance during subsequent movement, rather than passively waiting for new terrain observation results to respond. In this way, even when the robot is actually performing high-speed or high-dynamic motion tasks, the motion control strategy network can predict the trend of the terrain ahead based on the pre-learned temporal patterns, complete reasonable action preparation and compensation before entering unknown areas, improve the continuity and stability of the entire motion trajectory, avoid problems such as falling, and significantly improve the robot's control performance in high-difficulty tasks such as running and climbing.

[0227] In one example implementation, the encoder, motion control policy network, and dynamic transfer model can be trained separately based on the joint loss until the joint loss reaches its minimum value, and then the parameters of the encoder, motion control policy network, and dynamic transfer model can be updated.

[0228] In one example implementation, the encoder, motion control policy network, and dynamic transfer model are trained separately based on the joint loss until the joint loss reaches its minimum value. Then, the parameters of the encoder, motion control policy network, and dynamic transfer model are updated, enabling execution. Figure 10 The flowchart shown is a training method for a robot motion control policy network based on deep reinforcement learning. Figure 10 As shown, steps 1010-1030 are included:

[0229] Step 1010: With the goal of minimizing the joint loss, backpropagation calculations are performed on the encoder, motion control policy network, and dynamic transfer model based on the joint loss to obtain the gradient values ​​of each parameter in the encoder, motion control policy network, and dynamic transfer model.

[0230] The gradient value can be understood as a numerical value used to characterize the direction of updating model parameters.

[0231] The gradient value can be understood as the numerical value of the direction of change of model parameters that reduces the model's loss.

[0232] Step 1020: Obtain the first preset learning rate of the encoder, the second preset learning rate of the motion control strategy network, and the third preset learning rate of the dynamic transfer model.

[0233] The learning rate (LR) can be understood as the magnitude or step size of the update of model parameters.

[0234] The first, second, and third preset learning rates can be set in advance as needed, and are not limited here.

[0235] Step 1030: Update the parameters of the encoder, motion control strategy network and dynamic transfer model based on the gradient value, the first preset learning rate, the second preset learning rate and the third preset learning rate.

[0236] For example, updating the parameters of the encoder, motion control policy network, and dynamic transfer model based on the gradient value, the first preset learning rate, the second preset learning rate, and the third preset learning rate may include steps 1031-1033:

[0237] Step 1031: Calculate the product of the gradient value of each parameter in the encoder and the first preset learning rate to obtain the update increment of each parameter in the encoder; calculate the product of the gradient value of each parameter in the motion control strategy network and the second preset learning rate to obtain the update increment of each parameter in the motion control strategy network; calculate the product of the gradient value of each parameter in the dynamic transfer model and the third preset learning rate to obtain the update increment of each parameter in the dynamic transfer model.

[0238] Step 1032: For each parameter, sum the parameter and the parameter update increment to generate the parameter update value.

[0239] Step 1033: Replace each parameter in the encoder, motion control strategy network, and dynamic transfer model with the updated parameter values ​​to complete the parameter update of the encoder, motion control strategy network, and dynamic transfer model.

[0240] Therefore, by updating the parameters of the encoder, motion control policy network, and dynamic transfer model through gradient values ​​and learning rate, a well-trained encoder, motion control policy network, and dynamic transfer model can be obtained, improving the convergence speed of the joint loss and thus improving the training efficiency and accuracy of the encoder, motion control policy network, and dynamic transfer model.

[0241] This disclosure provides a robot motion control method based on deep reinforcement learning, such as... Figure 11The flowchart shown illustrates a robot motion control method based on deep reinforcement learning, which may include the following steps 1110-1130:

[0242] Step 1110: Obtain the robot's current state data, which includes the robot's body perception data and terrain perception data.

[0243] In this embodiment of the disclosure, the robot can acquire its current state data. The current state data may include the robot's body perception data and terrain perception data.

[0244] Step 1120: Input the current state data into the pre-trained encoder to generate the target hidden state.

[0245] In this embodiment of the disclosure, the robot can input the current state data into a pre-trained encoder, and encode the current state data based on the pre-trained encoder to obtain the robot's target hidden state.

[0246] Step 1130: Input the target hidden state into the pre-trained motion control policy network, and output the action policy for controlling the robot's motion based on the motion control policy network. The encoder and the motion control policy network are obtained according to the deep reinforcement learning-based robot motion control policy network training method described in any of the above embodiments.

[0247] In this embodiment of the present disclosure, the robot can input the target hidden state into a pre-trained motion control strategy network, and output an action strategy to control the robot's movement based on the motion control strategy network.

[0248] Therefore, a motion strategy for controlling the robot's movement can be generated using a pre-trained encoder and motion control strategy network. Based on this motion strategy, the robot's movement can be controlled. Since the motion control strategy network can learn the temporal patterns of the robot's body perception data and terrain perception data—that is, the changing patterns of these data over time—it can, based on these temporal patterns, combine the currently perceived and processed terrain state to probabilistically predict the unobserved terrain state ahead. This allows for dynamic adjustments to gait parameters, body posture, and other control variables in advance during subsequent movement, rather than passively waiting for new terrain observations. In this way, even when the robot is performing high-speed or high-dynamic tasks, the motion control strategy network can predict the terrain's trajectory based on the pre-learned temporal patterns, completing reasonable action preparation and compensation before entering unknown areas. This improves the continuity and stability of the entire movement trajectory, avoids problems such as falls, and significantly enhances the robot's control performance in challenging tasks such as running and climbing.

[0249] This disclosure provides a robot motion control policy network training device based on deep reinforcement learning, such as... Figure 12 The diagram shows a structural schematic of a robot motion control policy network training device based on deep reinforcement learning. Figure 12 As shown, the robot motion control policy network training device 1200 based on deep reinforcement learning includes:

[0250] The first acquisition module 1210 is used to acquire the robot's historical state data and input the historical state data into the encoder to obtain the historical hidden state. The historical state data includes the robot's body perception data and terrain perception data.

[0251] The generation module 1220 is used to input historical hidden states into the motion control policy network to generate action policies, and to input historical hidden states into the dynamic transition model to generate predicted hidden states for multiple future time steps.

[0252] The update module 1230 is used to calculate the first loss based on the difference between the predicted hidden state and the actual hidden state based on multiple future time steps, and to calculate the second loss based on the action policy output by the motion control policy network. Based on the first loss and the second loss, the parameters of the encoder, the motion control policy network and the dynamic transfer model are updated.

[0253] The robot motion control policy network training device based on deep reinforcement learning provided in this disclosure can implement the robot motion control policy network training method based on deep reinforcement learning in any of the above embodiments. Its execution method and beneficial effects are similar, and will not be described again here.

[0254] This disclosure provides a robot motion control device based on deep reinforcement learning, such as... Figure 13 The diagram shows a structural schematic of a robot motion control device based on deep reinforcement learning. Figure 13 As shown, the robot motion control device 1300 based on deep reinforcement learning includes:

[0255] The second acquisition module 1310 is used to acquire the robot's current state data, which includes the robot's body perception data and terrain perception data.

[0256] The hidden state generation module 1320 is used to input the current state data into the pre-trained encoder to generate the target hidden state;

[0257] The strategy generation module 1330 is used to input the target hidden state into the pre-trained motion control strategy network and output the action strategy for controlling the robot's motion based on the motion control strategy network.

[0258] The encoder and motion control policy network are obtained according to the deep reinforcement learning-based robot motion control policy network training method of any of the above embodiments.

[0259] The robot motion control device based on deep reinforcement learning provided in this disclosure can implement the robot motion control method based on deep reinforcement learning in any of the above embodiments. Its execution method and beneficial effects are similar, and will not be described again here.

[0260] This disclosure also provides an electronic device, which includes a processor and a memory, wherein the memory stores a computer program. When the computer program is executed by the processor, it can implement the methods of any of the above embodiments. The execution method and beneficial effects are similar, and will not be described again here.

[0261] The electronic device in this disclosure can be understood as any device with processing and computing capabilities, including but not limited to electronic devices such as smartphones, laptops, personal digital assistants (PDAs), tablet computers (PADs), portable multimedia players (PMPs), in-vehicle terminals, digital TVs, and desktop computers.

[0262] Figure 14 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure, such as... Figure 14 As shown, the electronic device 1400 may include a processor 1410 and a memory 1420. The memory 1420 stores a computer program 1421. When the computer program 1421 is executed by the processor 1410, it can implement the method provided in any of the above embodiments. The execution mode and beneficial effects are similar and will not be described again here.

[0263] Of course, for the sake of simplicity, Figure 14 Only some of the components of the electronic device 1400 relevant to the present invention are shown in this illustration; components such as buses, input / output interfaces, input devices, and output devices are omitted. In addition, the electronic device 1400 may include any other suitable components depending on the specific application.

[0264] This disclosure also provides a robot, which includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, it can implement the above-mentioned robot motion control method based on deep reinforcement learning. Its execution method and beneficial effects are similar and will not be described again here.

[0265] The robots include any one of the following: legged robots, quadruped robots, bipedal robots, wheeled robots, wheel-legged robots, quadrupedal robots, humanoid robots, and mobile robots. (Reference) Figure 15 , Figure 16 and Figure 17 As shown, Figure 15 A schematic diagram of a robot is shown. Figure 16 A schematic diagram of another type of robot is shown. Figure 17 A schematic diagram of yet another type of robot is shown.

[0266] This disclosure provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement the methods of any of the above embodiments. The execution method and beneficial effects are similar, and will not be described again here.

[0267] The aforementioned computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0268] The computer program described above can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer device, partially on the user's device, as a standalone software package, partially on the user's computer device and partially on a remote computer device, or entirely on a remote computer device or server.

[0269] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0270] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0271] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for training a robot motion control policy network based on deep reinforcement learning, characterized in that, include: The robot's historical state data is acquired and input into the encoder to obtain the historical hidden state. The historical state data includes the robot's body perception data and terrain perception data. The historical hidden states are input into the motion control policy network to generate action policies, and the historical hidden states are input into the dynamic transition model to generate predicted hidden states for multiple future time steps. The first loss is calculated based on the difference between the predicted hidden state and the actual hidden state at the multiple future time steps, and the second loss is calculated based on the action policy output by the motion control policy network. The parameters of the encoder, the motion control policy network and the dynamic transfer model are updated according to the first loss and the second loss. Wherein, the true hidden state is the hidden state of the target historical time step corresponding to the future time step in the historical hidden state.

2. The method for training a robot motion control policy network based on deep reinforcement learning according to claim 1, characterized in that, The acquisition of the robot's historical state data includes: Acquire the robot's body perception data and terrain perception data at multiple historical time steps; The terrain perception data and the ontology perception data are aligned by time steps to obtain state data for multiple historical time steps. The state data of the robot are sorted according to the chronological order of the time steps to obtain the robot's historical state data.

3. The method for training a robot motion control policy network based on deep reinforcement learning according to claim 1, characterized in that, The step of inputting the historical state data into the encoder to obtain the historical hidden state includes: Based on the encoder, the temporal features of the ontology-aware data in the historical state data are extracted to obtain the ontology latent state. Based on the encoder, the temporal features of the terrain sensing data in the historical state data are extracted to obtain the terrain latent state; The historical hidden state is obtained by fusing the ontology hidden state and the terrain hidden state.

4. The method for training a robot motion control policy network based on deep reinforcement learning according to claim 1, characterized in that, The step of inputting the historical hidden state into the motion control policy network to generate an action policy includes: The historical hidden states are input into the motion control policy network. For the hidden state of each historical time step in the historical hidden states, the motion control policy network performs action reasoning based on the hidden states and outputs the action policy corresponding to the hidden state of the historical time step.

5. The method for training a robot motion control policy network based on deep reinforcement learning according to claim 1, characterized in that, The step of inputting the historical hidden states into the dynamic transition model to generate predicted hidden states for multiple future time steps includes: The first hidden state of the first time step in the historical hidden state is input into the dynamic transition model; Based on the dynamic transition model, the state prediction of the first hidden state at multiple future time steps is performed to obtain the predicted hidden states at multiple future time steps.

6. The method for training a robot motion control policy network based on deep reinforcement learning according to claim 5, characterized in that, The process of predicting the state of the first hidden state at multiple future time steps based on the dynamic transition model to obtain multiple predicted hidden states at the future time steps includes: Based on the dynamic transition model, the state prediction of the first hidden state at the next time step is performed to obtain the predicted hidden state at the second time step. The predicted hidden state of the second time step is input into the dynamic transition model, and the state prediction of the next time step is performed based on the predicted hidden state of the second time step to obtain the predicted hidden state of the third time step. The predicted hidden state of the third time step is input into the dynamic transition model, and the dynamic transition model is repeatedly called to predict the state, so as to obtain the predicted hidden states of multiple future time steps.

7. The method for training a robot motion control policy network based on deep reinforcement learning according to claim 1, characterized in that, The calculation of the first loss based on the difference between the predicted hidden state and the actual hidden state at the multiple future time steps includes: For each future time step, determine the target historical time step corresponding to the future time step, and determine the hidden state of the target historical time step in the historical hidden state as the true hidden state of the future time step. For each future time step, calculate the difference between the predicted hidden state of the future time step and the actual hidden state of the future time step to obtain the state difference data corresponding to the future time step; The first loss is obtained by weighted summation of the state difference data corresponding to each future time step.

8. The method for training a robot motion control policy network based on deep reinforcement learning according to claim 7, characterized in that, The step of calculating the difference between the predicted hidden state at the future time step and the actual hidden state at the future time step to obtain the state difference data corresponding to the future time step includes: Calculate the first mean squared error between the predicted hidden state at the future time step and the actual hidden state at the future time step, and use the first mean squared error as the state difference data corresponding to the future time step.

9. The method for training a robot motion control policy network based on deep reinforcement learning according to claim 8, characterized in that, The calculation of the first mean squared error between the predicted hidden state at the future time step and the actual hidden state at the future time step includes: Calculate the first difference vector between the predicted hidden state at the future time step and the actual hidden state at the future time step; The dimension value of each dimension in the first difference vector is squared to obtain the first square value of each dimension, and the first square values ​​of each dimension are summed to obtain the first total sum of squares; Divide the first total sum of squares by the number of dimensions of the first difference vector to obtain the first mean square error.

10. The method for training a robot motion control policy network based on deep reinforcement learning according to claim 1, characterized in that, The calculation of the second loss based on the action policy output by the motion control policy network includes: For each historical time step in the historical state data and the corresponding action strategy, the robot is controlled to execute the action strategy under the state data in the simulation environment. The simulation environment returns the target action taken by the robot when executing the action strategy, the reward value of the robot after executing the target action, and the target state data of the robot. The second loss is calculated based on the reward value corresponding to the historical state data and the target state data.

11. The method for training a robot motion control policy network based on deep reinforcement learning according to claim 10, characterized in that, The calculation of the second loss based on the reward value corresponding to the historical state data and the target state data includes: For each historical time step in the historical state data, based on the state data, the reward value corresponding to the state data, and the target state data, the advantage value of the robot performing the target action corresponding to the state data is determined; Based on the advantage value corresponding to the state data and the execution probability of the target action in the action strategy corresponding to the state data, calculate the sub-loss corresponding to the state data; The second loss is obtained by summing the sub-losses corresponding to the state data of each historical time step in the historical state data.

12. The method for training a robot motion control policy network based on deep reinforcement learning according to claim 11, characterized in that, The step of determining the advantage value of the robot performing the target action corresponding to the state data based on the state data, the reward value corresponding to the state data, and the target state data includes: The value of the state data is evaluated based on the value network to obtain the current state value, and the value of the target state data is evaluated based on the value network to obtain the target state value. The reward value and the target state value are summed to obtain the target sum value; The difference between the target value and the current state value is calculated to obtain the advantage value corresponding to the state data.

13. The method for training a robot motion control policy network based on deep reinforcement learning according to claim 1, characterized in that, The step of updating the parameters of the encoder, the motion control policy network, and the dynamic transfer model based on the first loss and the second loss includes: Obtain the state loss weight corresponding to the first loss and the policy loss weight corresponding to the second loss; Based on the state loss weight and the policy loss weight, the first loss and the second loss are weighted and summed to obtain the joint loss; Based on the joint loss, the parameters of the encoder, the motion control policy network, and the dynamic transfer model are updated.

14. The method for training a robot motion control policy network based on deep reinforcement learning according to claim 13, characterized in that, The step of updating the parameters of the encoder, the motion control policy network, and the dynamic transfer model based on the joint loss includes: The encoder, the motion control policy network, and the dynamic transfer model are trained based on the joint loss until the joint loss reaches its minimum value, and then the parameters of the encoder, the motion control policy network, and the dynamic transfer model are updated.

15. The method for training a robot motion control policy network based on deep reinforcement learning according to claim 14, characterized in that, The process of training the encoder, motion control policy network, and dynamic transfer model based on the joint loss until the joint loss reaches its minimum value, and then updating the parameters of the encoder, motion control policy network, and dynamic transfer model, includes: With the goal of minimizing the joint loss, backpropagation calculations are performed on the encoder, the motion control strategy network, and the dynamic transfer model based on the joint loss to obtain the gradient values ​​of each parameter in the encoder, the motion control strategy network, and the dynamic transfer model. Obtain the first preset learning rate of the encoder, the second preset learning rate of the motion control strategy network, and the third preset learning rate of the dynamic transfer model; Based on the gradient value, the first preset learning rate, the second preset learning rate, and the third preset learning rate, the parameters of the encoder, the motion control strategy network, and the dynamic transfer model are updated.

16. The method for training a robot motion control policy network based on deep reinforcement learning according to claim 15, characterized in that, The step of updating the parameters of the encoder, the motion control policy network, and the dynamic transfer model based on the gradient value, the first preset learning rate, the second preset learning rate, and the third preset learning rate includes: The product of the gradient value of each parameter in the encoder and the first preset learning rate is calculated to obtain the update increment of each parameter in the encoder; the product of the gradient value of each parameter in the motion control policy network and the second preset learning rate is calculated to obtain the update increment of each parameter in the motion control policy network; the product of the gradient value of each parameter in the dynamic transfer model and the third preset learning rate is calculated to obtain the update increment of each parameter in the dynamic transfer model. For each parameter, the update value of the parameter is generated by summing the parameter and the update increment of the parameter. The parameters in the encoder, the motion control strategy network, and the dynamic transfer model are replaced with their updated values ​​to complete the parameter update for the encoder, the motion control strategy network, and the dynamic transfer model.

17. A robot motion control method based on deep reinforcement learning, characterized in that, include: Acquire the robot's current state data, which includes the robot's body perception data and terrain perception data; The current state data is input into a pre-trained encoder to generate the target hidden state; The target hidden state is input into a pre-trained motion control policy network, and an action policy for controlling the robot's motion is output based on the motion control policy network. The encoder and the motion control policy network are obtained by the robot motion control policy network training method based on deep reinforcement learning according to any one of claims 1 to 16.

18. A robot motion control policy network training device based on deep reinforcement learning, characterized in that, include: The first acquisition module is used to acquire the robot's historical state data and input the historical state data into the encoder to obtain the historical hidden state. The historical state data includes the robot's body perception data and terrain perception data. The generation module is used to input the historical hidden state into the motion control policy network to generate an action policy, and to input the historical hidden state into the dynamic transition model to generate predicted hidden states for multiple future time steps. The update module is used to calculate a first loss based on the difference between the predicted hidden state and the actual hidden state at the multiple future time steps, and to calculate a second loss based on the action policy output by the motion control policy network, and to update the parameters of the encoder, the motion control policy network and the dynamic transfer model according to the first loss and the second loss. Wherein, the true hidden state is the hidden state of the target historical time step corresponding to the future time step in the historical hidden state.

19. A robot motion control device based on deep reinforcement learning, characterized in that, include: The second acquisition module is used to acquire the robot's current state data, which includes the robot's body perception data and terrain perception data. The hidden state generation module is used to input the current state data into a pre-trained encoder to generate the target hidden state. The strategy generation module is used to input the target hidden state into a pre-trained motion control strategy network, and output an action strategy to control the robot's motion based on the motion control strategy network. The encoder and the motion control policy network are obtained by the robot motion control policy network training method based on deep reinforcement learning according to any one of claims 1 to 16.

20. An electronic device, characterized in that, include: A memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the method as described in any one of claims 1-17.

21. A robot, characterized in that, include: A memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the method of claim 17.

22. The robot according to claim 21, characterized in that, The robots include mobile robots.

23. The robot according to claim 21, characterized in that, The robots include humanoid robots.

24. The robot according to claim 21, characterized in that, The robot includes either a legged robot or a wheeled robot.

25. The robot according to claim 21, characterized in that, The robot includes a wheeled robot.

26. The robot according to claim 21, characterized in that, The robot includes any one of quadruped robots, bipedal robots, and four-wheeled robots.

27. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-17.

Images