Robot motion control strategy network training method based on deep reinforcement learning, robot motion control method, device, equipment, robot and storage medium

Abstract

The present disclosure provides a robot motion control strategy network training method based on deep reinforcement learning, a robot motion control method, device, equipment, robot and storage medium, relating to the technical field of robots. The present disclosure introduces a forward prediction model and shares the encoder output features with the motion control strategy network. The parameters of the encoder, the motion control strategy network and the forward prediction model are updated by the comprehensive loss of the state loss of the forward prediction model and the policy loss of the motion control strategy network, so that the motion control strategy network can autonomously learn and adapt to changes in the environment during the training process, accelerate the policy convergence speed of the policy network, improve the utilization rate, training efficiency and generalization ability of the sample, so that the robot can generate good coherent and smooth posture transition actions when executing the action sequence, reduce the probability of action mutation or posture switching stiffness, and improve the naturalness and continuity of the action in the robot motion process.

Description

Technical Field

[0001] This disclosure relates to the field of robotics technology, and in particular to a method for training a robot motion control strategy network based on deep reinforcement learning, a robot motion control method, a device, an equipment, a robot, and a storage medium. Background Technology

[0002] Currently, when training the motion control policy network (PCP) of robots, the PCP struggles to learn the temporal relationships between consecutive actions, resulting in low sample utilization, slow policy convergence, and long training cycles. This is particularly problematic in tasks involving continuous posture adjustments, where performance is unstable. During the policy network deployment phase, the lack of predictive ability for the robot's future states by the PCP can lead to abrupt changes in actions or stiff posture transitions during execution, failing to meet the requirements for natural and continuous movements in highly dynamic tasks.

[0003] For example, Chinese patent application CN119830943A discloses a deep reinforcement learning algorithm that combines auxiliary tasks, empirical double sampling, and dynamic correction of value functions when controlling the actions of an intelligent agent. Although the introduction of auxiliary tasks improves the quality of feature extraction and enhances the model's understanding of the state, and empirical double sampling improves the efficiency of sample utilization and accelerates the learning process, it still suffers from problems such as low sample utilization, slow policy convergence speed, long training cycle, high probability of sudden action changes or stiff posture switching, and poor naturalness and continuity of actions.

[0004] Therefore, the current training methods for robot motion control strategy networks still need further optimization. Summary of the Invention

[0005] To address the problems of low sample utilization, slow policy convergence, long training cycles, high probability of sudden action changes or stiff posture transitions, and poor naturalness and continuity of actions in the aforementioned related technologies, this disclosure provides a robot motion control policy network training method, robot motion control method, device, equipment, robot, and storage medium based on deep reinforcement learning. This enables the motion control policy network to learn autonomously and adapt to changes in the environment during training, accelerates the policy convergence speed of the motion control policy network, improves sample utilization, and enhances the training efficiency and generalization ability of the motion control policy network. It also reduces the probability of sudden action changes or stiff posture transitions in the robot, and improves the naturalness and continuity of actions during robot movement.

[0006] The first aspect of this disclosure provides a method for training a robot motion control policy network based on deep reinforcement learning, including:

[0007] The robot's historical state data is acquired and input into the encoder to obtain the first representation feature. The historical state data is the robot's body perception data.

[0008] Historical state data is sampled to obtain sampled state data, and the sampled state data is then enhanced and input into the encoder to obtain the second representation feature.

[0009] The first representation feature is input into the motion control policy network to generate the action policy, and the second representation feature is input into the forward prediction model to generate predicted state data for multiple future time steps.

[0010] The first loss is calculated based on the difference between the predicted state data and the actual state data at multiple future time steps, and the second loss is calculated based on the action policy output by the motion control policy network. The parameters of the encoder, motion control policy network and forward prediction model are updated according to the first loss and the second loss.

[0011] Optionally, the above-mentioned acquisition of the robot's historical state data and input into the encoder to obtain the first representation feature includes:

[0012] Acquire the robot's ontological perception data at multiple historical time steps;

[0013] The robot's historical state data is obtained by sorting multiple body perception data according to the chronological order of time steps.

[0014] Historical state data is input into the encoder, and feature extraction is performed on the ontology perception data of multiple historical time steps in the historical state data based on the encoder to obtain the first representation feature corresponding to the historical state data.

[0015] Optionally, the above sampling of historical state data yields sampled state data, including:

[0016] Read status data from multiple time steps in historical status data;

[0017] In the state data, multiple state data are selected as candidate samples according to a preset time step interval;

[0018] The candidate samples are sorted according to the order of their corresponding time steps to obtain the sampling state data.

[0019] Optionally, the sampled state data, after undergoing enhancement processing, is input into the encoder to obtain the second representation feature, including:

[0020] The sampled state data is enhanced based on a preset perturbation strategy to obtain enhanced state data. The preset perturbation strategy includes one or more of noise perturbation strategy and mask perturbation strategy.

[0021] The enhanced state data is input into the encoder, and features are extracted from the enhanced state data based on the encoder to obtain the second representation features.

[0022] Optionally, when the preset perturbation strategy includes a noise perturbation strategy, the sampled state data is enhanced based on the preset perturbation strategy to obtain enhanced state data, including:

[0023] Gaussian noise is added to the sampled state data to obtain enhanced state data.

[0024] Optionally, when the preset perturbation strategy includes a noise perturbation strategy, the sampled state data is enhanced based on the preset perturbation strategy to obtain enhanced state data, including:

[0025] The target noise data is uniformly added to the sampled state data to obtain the enhanced state data.

[0026] Optionally, when the preset perturbation strategy includes a mask perturbation strategy, the sampled state data is enhanced based on the preset perturbation strategy to obtain enhanced state data, including:

[0027] Add a random mask to the sampled state data to obtain enhanced state data.

[0028] Optionally, the above-mentioned addition of a random mask to the sampled state data to obtain enhanced state data includes:

[0029] Randomly select multiple target locations from the sampled state data;

[0030] The data corresponding to each target position in the sampled state data is replaced with a mask to obtain the enhanced state data.

[0031] Optionally, the above-mentioned inputting the first representation feature into the motion control policy network to generate an action policy includes:

[0032] The first representation feature is input into the motion control policy network. For each historical time step in the first representation feature, the motion control policy network performs action reasoning based on the representation feature and outputs the action policy corresponding to the state data of the historical time step.

[0033] Optionally, the above-mentioned inputting the second representation feature into the forward prediction model to generate predicted state data for multiple future time steps includes:

[0034] The first state feature corresponding to the first time step in the second representation feature is input into the forward prediction model;

[0035] Based on the forward prediction model, the state of the first state feature is predicted for multiple future time steps, resulting in predicted state data for multiple future time steps.

[0036] Optionally, the above-mentioned forward prediction model is used to predict the state of the first state features at multiple future time steps, resulting in predicted state data for multiple future time steps, including:

[0037] Based on the forward prediction model, the state of the first state feature is predicted for the next time step to obtain the predicted state data for the second time step.

[0038] The predicted state data of the second time step is input into the forward prediction model. Based on the predicted state data of the second time step, the state prediction of the next time step is performed to obtain the predicted state data of the third time step.

[0039] The predicted state data of the third time step is input into the forward prediction model, and the forward prediction model is repeatedly called to predict the state, and the predicted state data of multiple future time steps are obtained by combining them.

[0040] Optionally, the calculation of the first loss based on the difference between the predicted state data and the actual state data at multiple future time steps includes:

[0041] For each future time step, the historical state data of the future time step is determined as the actual state data of the future time step;

[0042] For each future time step, calculate the difference between the predicted state data and the actual state data of the future time step to obtain the state difference data corresponding to the future time step;

[0043] The first loss is obtained by weighted summation of the state difference data corresponding to each future time step.

[0044] Optionally, for each future time step, the difference between the predicted state data and the actual state data of the future time step is calculated to obtain the state difference data corresponding to the future time step, including:

[0045] For each future time step, calculate the first mean squared error between the predicted state data and the actual state data of the future time step, and use the first mean squared error as the state difference data corresponding to the future time step.

[0046] Optionally, the calculation of the first mean squared error between the predicted state data and the actual state data at future time steps includes:

[0047] Calculate the first difference between the predicted state data at the future time step and the actual state data at the future time step;

[0048] Squaring the data in each dimension of the first difference data yields the first squared value for each dimension. The first squared values ​​of each dimension are then summed to obtain the first total sum of squares.

[0049] Divide the first total sum of squares by the number of dimensions of the first difference data to obtain the first mean square error.

[0050] Optionally, the calculation of the second loss based on the action policy output by the motion control policy network includes:

[0051] For each historical time step in the historical state data and the corresponding action strategy, the robot is controlled to execute the action strategy under the state data in the simulation environment. The simulation environment returns the target action taken by the robot when executing the action strategy, as well as the reward value and target state data of the robot after executing the target action.

[0052] The second loss is calculated based on the reward value corresponding to the historical state data and the target state data.

[0053] Optionally, the second loss is calculated based on the reward value corresponding to the historical state data and the target state data, including:

[0054] For each historical time step in the historical state data, based on the state data, the corresponding reward value, and the target state data, determine the advantage value for the robot to perform the target action corresponding to the state data.

[0055] Based on the advantage value corresponding to the state data and the execution probability of the target action in the action policy corresponding to the state data, calculate the sub-loss corresponding to the state data;

[0056] The second loss is obtained by summing the sub-losses corresponding to the state data at each historical time step in the historical state data.

[0057] Optionally, the above-mentioned determination of the robot's advantage value for performing the target action based on the state data, the reward value corresponding to the state data, and the target state data includes:

[0058] The current state value is obtained by evaluating the value of state data based on the value network, and the target state value is obtained by evaluating the value of target state data based on the value network.

[0059] The reward value and the target state value are summed to obtain the target sum value;

[0060] Calculate the difference between the target value and the current state value to obtain the advantage value of the robot in performing the target action corresponding to the state data.

[0061] Optionally, the parameters of the encoder, motion control policy network, and forward prediction model are updated based on the first loss and the second loss, including:

[0062] Obtain the state loss weight corresponding to the first loss and the policy loss weight corresponding to the second loss;

[0063] Based on the state loss weight and policy loss weight, the first loss and the second loss are weighted and summed to obtain the joint loss.

[0064] Based on the joint loss, the parameters of the encoder, motion control policy network, and forward prediction model are updated.

[0065] Optionally, the parameters of the encoder, motion control policy network, and forward prediction model are updated based on the joint loss, including:

[0066] The encoder, motion control policy network, and forward prediction model are trained separately based on the joint loss until the joint loss reaches its minimum value. Then, the parameters of the encoder, motion control policy network, and forward prediction model are updated.

[0067] Optionally, the encoder, motion control policy network, and forward prediction model are trained separately based on the joint loss until the joint loss reaches its minimum value. Then, the parameters of the encoder, motion control policy network, and forward prediction model are updated, including:

[0068] With the goal of minimizing the joint loss, backpropagation calculations are performed on the encoder, motion control policy network, and forward prediction model based on the joint loss to obtain the gradient values ​​of each parameter in the encoder, motion control policy network, and forward prediction model.

[0069] Obtain the first preset learning rate of the encoder, the second preset learning rate of the motion control strategy network, and the third preset learning rate of the forward prediction model;

[0070] The parameters of the encoder, motion control strategy network, and forward prediction model are updated based on the gradient value, the first preset learning rate, the second preset learning rate, and the third preset learning rate.

[0071] Optionally, the parameters of the encoder, motion control policy network, and forward prediction model are updated based on the gradient value, the first preset learning rate, the second preset learning rate, and the third preset learning rate, including:

[0072] The product of the gradient value of each parameter in the encoder and the first preset learning rate is calculated to obtain the update increment of each parameter in the encoder; the product of the gradient value of each parameter in the motion control policy network and the second preset learning rate is calculated to obtain the update increment of each parameter in the motion control policy network; the product of the gradient value of each parameter in the forward prediction model and the third preset learning rate is calculated to obtain the update increment of each parameter in the forward prediction model.

[0073] For each parameter, sum the parameter and its update increment to generate the updated value of the parameter;

[0074] The parameters in the encoder, motion control strategy network, and forward prediction model are replaced with their updated values ​​to complete the parameter update for the encoder, motion control strategy network, and forward prediction model.

[0075] The second aspect of this disclosure provides a robot motion control method based on deep reinforcement learning, comprising:

[0076] Acquire the robot's current state data, which includes the robot's body perception data;

[0077] The current state data is input into a pre-trained encoder to generate target representation features;

[0078] The target representation features are input into a pre-trained motion control policy network, and the motion control policy network outputs the action policy to control the robot's motion.

[0079] The encoder and motion control policy network are obtained according to any one of the deep reinforcement learning-based robot motion control policy network training methods in the first aspect.

[0080] A third aspect of this disclosure provides a training device for a robot motion control policy network based on deep reinforcement learning, comprising:

[0081] The first acquisition module is used to acquire the robot's historical state data and input it into the encoder to obtain the first representation feature. The historical state data is the robot's body perception data.

[0082] The enhancement module is used to sample historical state data to obtain sampled state data, and after enhancing the sampled state data, it is input into the encoder to obtain the second representation feature.

[0083] The generation module is used to input the first representation feature into the motion control policy network to generate the action policy, and to input the second representation feature into the forward prediction model to generate predicted state data for multiple future time steps.

[0084] The update module is used to calculate the first loss based on the difference between the predicted state data and the actual state data at multiple future time steps, and to calculate the second loss based on the action policy output by the motion control policy network. Based on the first loss and the second loss, the parameters of the encoder, the motion control policy network and the forward prediction model are updated.

[0085] A fourth aspect of this disclosure provides a robot motion control device based on deep reinforcement learning, comprising:

[0086] The second acquisition module acquires the robot's current state data, which includes the robot's body perception data.

[0087] The feature generation module is used to input the current state data into the pre-trained encoder to generate target representation features;

[0088] The strategy generation module is used to input the target representation features into a pre-trained motion control policy network, and output the action policy for controlling the robot's motion based on the motion control policy network.

[0089] The encoder and motion control policy network are obtained according to any one of the deep reinforcement learning-based robot motion control policy network training methods in the first aspect.

[0090] The fifth aspect of this disclosure provides an electronic device including a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements either the deep reinforcement learning-based robot motion control policy network training method of the first aspect or the deep reinforcement learning-based robot motion control method of the second aspect.

[0091] The sixth aspect of this disclosure provides a robot including a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the deep reinforcement learning-based robot motion control method of the second aspect.

[0092] Optionally, the aforementioned robots include any one of legged robots, quadruped robots, bipedal robots, wheeled robots, wheel-legged robots, quadrupedal robots, humanoid robots, and mobile robots.

[0093] The seventh aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements either the deep reinforcement learning-based robot motion control policy network training method of the first aspect or the deep reinforcement learning-based robot motion control method of the second aspect.

[0094] The technical solution provided in this disclosure has the following advantages compared with the prior art:

[0095] In this embodiment, by introducing a forward prediction model and sharing encoder output features with the motion control policy network, the state data of future time steps is predicted based on the forward prediction model. The parameters of the encoder, motion control policy network, and forward prediction model are updated by the combined loss of the state loss of the forward prediction model and the policy loss of the motion control policy network. This enables the motion control policy network to not only learn the mapping from the current state to the action policy during training, but also implicitly learn the dynamic evolution law from the current state to the future state. This improves the motion control policy network's ability to model continuous action structures, allowing it to learn autonomously and adapt to changes in the environment during training. This accelerates the policy convergence speed of the policy network, improves sample utilization, and enhances the training efficiency and generalization ability of the motion control policy network. At the same time, since the motion control policy network can predict future state changes, the robot can generate good, coherent, and smooth posture transitions when executing action sequences, reducing the probability of sudden action changes or stiff posture switching, and improving the naturalness and continuity of the robot's movements.

[0096] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0097] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0098] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0099] Figure 1 This is a system architecture diagram provided in an embodiment of the present disclosure;

[0100] Figure 2 This is a flowchart of a robot motion control policy network training method based on deep reinforcement learning provided in an embodiment of this disclosure;

[0101] Figure 3 This is a flowchart of a robot motion control policy network training method based on deep reinforcement learning provided in an embodiment of this disclosure;

[0102] Figure 4 This is a flowchart of a robot motion control policy network training method based on deep reinforcement learning provided in an embodiment of this disclosure;

[0103] Figure 5 This is a flowchart of a robot motion control policy network training method based on deep reinforcement learning provided in an embodiment of this disclosure;

[0104] Figure 6 This is a flowchart of a robot motion control policy network training method based on deep reinforcement learning provided in an embodiment of this disclosure;

[0105] Figure 7 This is a flowchart of a robot motion control policy network training method based on deep reinforcement learning provided in an embodiment of this disclosure;

[0106] Figure 8 This is a flowchart of a robot motion control policy network training method based on deep reinforcement learning provided in an embodiment of this disclosure;

[0107] Figure 9 This is a flowchart of a robot motion control policy network training method based on deep reinforcement learning provided in an embodiment of this disclosure;

[0108] Figure 10 This is a flowchart of a robot motion control policy network training method based on deep reinforcement learning provided in an embodiment of this disclosure;

[0109] Figure 11 This is a flowchart of a robot motion control method based on deep reinforcement learning provided in an embodiment of this disclosure;

[0110] Figure 12 This is a schematic diagram of the structure of a robot motion control strategy network training device based on deep reinforcement learning, provided in an embodiment of this disclosure.

[0111] Figure 13 This is a schematic diagram of the structure of a robot motion control device based on deep reinforcement learning provided in an embodiment of this disclosure;

[0112] Figure 14 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure;

[0113] Figure 15 This is a schematic diagram of a robot provided in an embodiment of this disclosure;

[0114] Figure 16 This is a schematic diagram of a robot provided in an embodiment of this disclosure;

[0115] Figure 17 This is a schematic diagram of a robot provided in an embodiment of this disclosure. Detailed Implementation

[0116] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0117] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0118] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0119] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0120] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0121] To better understand the inventive concept of the embodiments of this disclosure, the technical solutions of the embodiments of this disclosure will be described below in conjunction with exemplary embodiments.

[0122] First, the relevant terms involved in the embodiments of this disclosure will be explained.

[0123] A robot can be understood as a programmable mechanism with a certain degree of autonomy, capable of performing motion, manipulation, or positioning.

[0124] A joint can be understood as a mechanical component that connects two rigid bodies and constrains the relative motion between them.

[0125] Robot proprioception data can be understood as the observational information that a robot directly acquires through its own sensors. For example, robot proprioception data may include, but is not limited to, data such as the robot's joint angles, joint velocities, walking speed, and posture information.

[0126] The noise perturbation strategy can be understood as a perturbation strategy that adds noise data to the sampled state data.

[0127] The mask perturbation strategy can be understood as a perturbation strategy that adds a mask to the sampled state data.

[0128] Gaussian noise data can be understood as a type of noise data whose probability density function follows a Gaussian distribution (i.e., a normal distribution).

[0129] Motion control policy networks can be understood as a type of neural network based on deep reinforcement learning algorithms.

[0130] A forward prediction model can be understood as a model that predicts forward.

[0131] A simulation environment can be understood as a virtual operating platform used to simulate robot dynamics, sensor feedback, and interaction mechanisms with the outside world.

[0132] A value network can be understood as a neural network that predicts the total future revenue and is used to evaluate the state value corresponding to the robot's state data, or simply state value.

[0133] The gradient value can be understood as a numerical value used to characterize the direction of updating model parameters.

[0134] The learning rate (LR) can be understood as the magnitude or step size of the update of model parameters.

[0135] Figure 1 The diagram illustrates a system architecture for a robot motion control policy network training method and a robot motion control method based on deep reinforcement learning, which can be applied to embodiments of this disclosure.

[0136] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a robot 102, a network 103, and a server 104. The terminal device 101 includes, but is not limited to, desktop computers, laptops, smartphones, and tablets. The terminal device 101 is equipped with a graphical user interface, which can provide visualization functions to display the robot 102's operating status, motion trajectory, etc. It can also be used to configure training parameters, control the training process, monitor the model training status, and support sending motion control commands to the robot 102.

[0137] Robot 102 is equipped with a variety of sensors, such as inertial measurement units, joint position sensors, speed sensors, and angle sensors, to collect its own state data in real time during movement.

[0138] Server 104 is equipped with a training module for implementing a deep reinforcement learning-based robot motion control policy network training method. This module acquires historical state data of the robot and, based on this data, trains an encoder and motion control policy network using deep reinforcement learning. After training, server 104 deploys the trained encoder and motion control policy network to robot 102, enabling robot 102 to output motion control strategies based on these networks.

[0139] Network 103 serves as the medium for providing a communication link between terminal device 101, robot 102, and server 104. Network 103 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc. It should be understood that... Figure 1 The number and type of terminal devices, robots, networks, and servers shown are merely illustrative. Depending on implementation needs, any number and type of terminal devices, robots, networks, and servers can be included.

[0140] Through the coordinated operation of various parts in system architecture 100, the motion control policy network of the encoder and the robot can be trained efficiently. During the training process, the motion control policy network not only learns the mapping from the current state to the action policy, but also implicitly learns the dynamic evolution law from the current state to the future state. This improves the motion control policy network's ability to model continuous action structures, enabling it to learn and adapt to changes in the environment autonomously during training. This accelerates the policy convergence speed of the motion control policy network, improves sample utilization, and enhances the training efficiency and generalization ability of the motion control policy network. At the same time, because the motion control policy network can predict future state changes, the robot can generate good, coherent, and smooth posture transitions when executing action sequences, reducing the probability of sudden action changes or stiff posture switching, and improving the naturalness and continuity of the robot's movements.

[0141] The Deep Reinforcement Learning (DRL) disclosed herein can be understood as a combination of Deep Learning (DL) and Reinforcement Learning (RL), integrating the feature extraction capabilities of Deep Learning and the decision-making capabilities of Reinforcement Learning. It is a learning method that allows an agent to interact with the environment, continuously try and learn in the environment, and adjust its own parameters to obtain the optimal policy.

[0142] This disclosure provides a method for training a robot motion control policy network based on deep reinforcement learning, such as... Figure 2 The flowchart shown illustrates a method for training a robot motion control policy network based on deep reinforcement learning. This method may include the following steps 210 to 240:

[0143] Step 210: Obtain the robot's historical state data and input it into the encoder to obtain the first representation feature. The historical state data is the robot's body perception data.

[0144] Step 220: Sample the historical state data to obtain sampled state data, and after enhancing the sampled state data, input it into the encoder to obtain the second representation feature.

[0145] Step 230: Input the first representation feature into the motion control policy network to generate an action policy, and input the second representation feature into the forward prediction model to generate predicted state data for multiple future time steps.

[0146] Step 240: Calculate the first loss based on the difference between the predicted state data and the actual state data at multiple future time steps, and calculate the second loss based on the action policy output by the motion control policy network. Then, update the parameters of the encoder, the motion control policy network, and the forward prediction model according to the first loss and the second loss.

[0147] The robot motion control policy network training method based on deep reinforcement learning provided in the exemplary embodiments of this disclosure can introduce a forward prediction model and share encoder output features with the motion control policy network. Based on the forward prediction model, it predicts the state data of future time steps. The parameters of the encoder, motion control policy network, and forward prediction model are updated through the combined loss of the state loss of the forward prediction model and the policy loss of the motion control policy network. This allows the motion control policy network to not only learn the mapping from the current state to the action policy during training, but also implicitly learn the dynamic evolution law from the current state to the future state. This improves the motion control policy network's ability to model continuous action structures, enabling it to autonomously learn and adapt to environmental changes during training. This accelerates the policy convergence speed of the policy network, improves sample utilization, and enhances the training efficiency and generalization ability of the motion control policy network. Furthermore, because the motion control policy network can predict future state changes, the robot can generate smooth and coherent posture transitions when executing action sequences, reducing the probability of sudden action changes or stiff posture switching, and improving the naturalness and continuity of the robot's movements.

[0148] The following will provide a detailed description of the robot motion control policy network training method based on deep reinforcement learning in the embodiments of this disclosure.

[0149] In step 210, the robot's historical state data is acquired and input into the encoder to obtain the first representation feature. The historical state data is the robot's body perception data.

[0150] Historical state data can be understood as the robot's proprioceptive perception data. This data refers to the observational information directly acquired by the robot through its own sensors. For example, robot proprioceptive perception data may include, but is not limited to, data such as joint angles, joint velocities, walking speed, and posture information.

[0151] The robot's proprioception data can be collected through sensors installed on the robot. For example, sensors such as inertial measurement units (IMUs) can collect the robot's proprioception data. During movement, the robot uses sensors to collect proprioception data and then stores the data in local storage or on a cloud server.

[0152] In this embodiment of the disclosure, the robot's historical state data can be obtained from the robot's local storage space or cloud server, and then the robot's historical state data can be input into the encoder. The encoder encodes the historical state data to obtain the first representation feature corresponding to the historical state data.

[0153] In some example implementations, the above-described acquisition of the robot's historical state data and input into the encoder to obtain the first representation feature can be executed. Figure 3 The flowchart shown is a training method for a robot motion control policy network based on deep reinforcement learning. Figure 3 As shown, steps 310-330 are included:

[0154] Step 310: Obtain the robot's ontological perception data at multiple historical time steps.

[0155] A time step can be understood as a process in which a robot performs an action and receives feedback.

[0156] Step 320: Sort the multiple body perception data according to the chronological order of the time steps to obtain the robot's historical state data.

[0157] For example, if multiple historical time steps include three historical time steps t1, t2, and t3, and the order of these three time steps is t1, t2, and t3, then the body perception data of t1, t2, and t3 are sorted in sequence to obtain the robot's historical state data.

[0158] Step 330: Input the historical state data into the encoder, and extract features from the ontology perception data of multiple historical time steps in the historical state data based on the encoder to obtain the first representation feature corresponding to the historical state data.

[0159] The first representation feature can be understood as a high-dimensional feature vector.

[0160] Therefore, multiple ontological perception data can be sorted according to the chronological order of time steps to construct the robot's historical state data. Features can be extracted from the historical state data to obtain the first representation feature corresponding to the historical state data. This can ensure the temporal sequence of the historical state data and the first representation feature and improve the accuracy of the first representation feature corresponding to the historical state data.

[0161] In step 220, historical state data is sampled to obtain sampled state data, and the sampled state data is enhanced and then input into the encoder to obtain the second representation feature.

[0162] This involves sampling historical state data to obtain sampled state data. In other words, the number of state data points is increased based on the historical state data, and the increased state data is used as the sampled state data.

[0163] In some example implementations, the above-described sampling of historical state data yields sampled state data, which can then be executed. Figure 4 The flowchart shown is a training method for a robot motion control policy network based on deep reinforcement learning. Figure 4 As shown, steps 410-430 are included:

[0164] Step 410: Read the status data of multiple time steps in the historical status data.

[0165] Step 420: Select multiple state data as candidate samples from the state data according to a preset time step interval.

[0166] The preset time step interval can be set as needed, and is not limited here.

[0167] For example, multiple time steps include six time steps: t1, t2, t3, t4, t5, t6, and t7. If the preset time step interval is 2, then the state data of time step t1, the state data of time step t4, and the state data of time step t7 are selected as candidate samples.

[0168] Step 430: Sort the candidate samples according to the order of their corresponding time steps to obtain the sampling state data.

[0169] For example, if the candidate samples include state data at time step t1, time step t4, and time step t7, and the order of the time steps is t1, t4, and t7, then the candidate samples are sorted according to the order of the time steps corresponding to the candidate samples, and the resulting sampled state data are the state data at time step t1, time step t4, and time step t7.

[0170] Therefore, state data can be sampled at preset time step intervals to obtain candidate samples. Then, the candidate samples can be sorted according to the order of the time steps to obtain sampled state data. This can ensure the orderliness of state data sampling and improve the representativeness of the sampled state data.

[0171] In some example implementations, the sampled state data, after undergoing the enhancement process described above, is input to the encoder to obtain a second representation feature, which can then be executed. Figure 5 The flowchart shown is a training method for a robot motion control policy network based on deep reinforcement learning. Figure 5 As shown, steps 510-520 are included:

[0172] Step 510: Enhance the sampled state data based on a preset perturbation strategy to obtain enhanced state data. The preset perturbation strategy includes one or more of noise perturbation strategy and mask perturbation strategy.

[0173] Among them, the noise perturbation strategy can be understood as a perturbation strategy that adds noise data to the sampled state data.

[0174] The mask perturbation strategy can be understood as a perturbation strategy that adds a mask to the sampled state data.

[0175] In one example implementation, when the preset perturbation strategy includes a noise perturbation strategy, enhancing the sampled state data based on the preset perturbation strategy to obtain enhanced state data may include:

[0176] Gaussian noise is added to the sampled state data to obtain enhanced state data.

[0177] Gaussian noise data can be understood as a type of noise data whose probability density function follows a Gaussian distribution (i.e., a normal distribution).

[0178] For example, the enhanced state data y can be represented as:

[0179] y = x + N(0,1);

[0180] Where y represents the augmented state data; x represents the sampled state data; and N(0,1) represents the standard normal distribution noise data with a mean of 0 and a standard deviation of 1.

[0181] In one example implementation, when the preset perturbation strategy includes a noise perturbation strategy, enhancing the sampled state data based on the preset perturbation strategy to obtain enhanced state data may include:

[0182] The target noise data is uniformly added to the sampled state data to obtain the enhanced state data.

[0183] The target noise data can be set as needed; there are no restrictions here.

[0184] In one example implementation, when the preset perturbation strategy includes a mask perturbation strategy, enhancing the sampled state data based on the preset perturbation strategy to obtain enhanced state data may include:

[0185] Add a random mask to the sampled state data to obtain enhanced state data.

[0186] Adding a random mask to the sampled state data can be understood as randomly selecting a portion of the sampled state data for masking (zeroing, replacing, or blocking).

[0187] A mask can be understood as a string of binary code that masks the input bits by performing a bitwise AND operation on the target field.

[0188] For example, multiple target locations can be randomly selected from the sampled state data; the data corresponding to each target location in the sampled state data can be replaced with a mask to obtain enhanced state data.

[0189] Step 520: Input the enhanced state data into the encoder, and extract features from the enhanced state data based on the encoder to obtain the second representation features.

[0190] Therefore, by enhancing the sampled state data based on a preset perturbation strategy, enhanced state data is obtained, enabling the encoder to generate stable second representation features even when the state data is perturbed. This processing not only enhances the robustness of the second representation features but also improves the behavioral stability of the control policy network to uncertain inputs and enhances the continuous prediction capability of the subsequent forward prediction model.

[0191] In step 230, the first representation feature is input into the motion control policy network to generate an action policy, and the second representation feature is input into the forward prediction model to generate predicted state data for multiple future time steps.

[0192] The motion control policy network can be understood as a neural network based on deep reinforcement learning algorithms. For example, the motion control policy network can be a multi-layer fully connected perceptron, a Transformer structure, etc.

[0193] A forward prediction model can be understood as a model that predicts forward.

[0194] In some example implementations, a first representation feature can be input into a motion control policy network. For each historical time step in the first representation feature, the motion control policy network performs action reasoning based on the representation feature and outputs the action policy corresponding to the state data of that historical time step.

[0195] Action strategy can be understood as the probability distribution of actions under state data, that is, the execution probability of each action under state data.

[0196] In some example implementations, the above-described input of the second representation feature into the forward prediction model to generate predicted state data for multiple future time steps can be performed. Figure 5 The flowchart shown is a training method for a robot motion control policy network based on deep reinforcement learning. Figure 5 As shown, steps 610-620 are included:

[0197] Step 610: Input the first state feature corresponding to the first time step in the second representation feature into the forward prediction model.

[0198] The second representation feature corresponding to the enhanced state data includes robot state data for multiple time steps. The first state feature corresponding to the first time step in the second representation feature can be input into the forward prediction model.

[0199] Step 620: Based on the forward prediction model, perform state prediction for the first state feature at multiple future time steps to obtain predicted state data for multiple future time steps.

[0200] The future time step can be understood as the time step after the first time step in the second representation feature.

[0201] In some example implementations, the state prediction of the first state features at multiple future time steps is performed based on the forward prediction model to obtain predicted state data for multiple future time steps, which may include steps 6201-6203:

[0202] Step 6201: Based on the forward prediction model, predict the state of the first state feature for the next time step to obtain the predicted state data for the second time step.

[0203] For example, at a time step K1, the state prediction for the next time step is performed based on the first state feature corresponding to K1 using the forward prediction model, resulting in the predicted state data Y1 for the second time step K2.

[0204] Step 6202: Input the first predicted state data into the forward prediction model, and based on the forward prediction model, predict the state of the next time step based on the predicted state data of the second time step to obtain the predicted state data corresponding to the third time step.

[0205] For example, the predicted state data Y1 of the second time step K2 is input into the forward prediction model, and the state prediction of the next time step is performed based on the predicted state data Y1 of the forward prediction model to obtain the predicted state data Y2 of the third time step K3.

[0206] Step 6203: Input the predicted state data of the third time step into the forward prediction model, repeatedly call the forward prediction model to predict the state, and synthesize the predicted state data of multiple future time steps.

[0207] For example, if the first time step K1 includes five future time steps K2, K3, K4, K5, and K6, then by repeatedly calling the forward prediction model to predict the state, the predicted state data for the five future time steps Y1 (K2), Y2 (K3), Y3 (K4), Y4 (K5), and Y5 (K6) can be obtained.

[0208] Therefore, the second representation feature corresponding to the enhanced state data can be input into the forward prediction model, and the predicted state data for multiple future time steps can be generated based on the forward prediction model.

[0209] In step 240, a first loss is calculated based on the difference between the predicted state data and the actual state data at multiple future time steps, and a second loss is calculated based on the action policy output by the motion control policy network. The parameters of the encoder, the motion control policy network, and the forward prediction model are updated according to the first loss and the second loss.

[0210] The first loss can be understood as an indicator used to measure the difference between the predicted state data at future time steps and the actual state data at future time steps.

[0211] The second loss can be understood as an indicator used to measure the difference between the action policy output by the motion control policy network and the actual action executed by the robot.

[0212] In some example implementations, the calculation of the first loss based on the difference between predicted state data and actual state data at multiple future time steps can be performed. Figure 7The flowchart shown is a training method for a robot motion control policy network based on deep reinforcement learning. Figure 7 As shown, steps 710-730 are included:

[0213] Step 710: For each future time step, determine the historical state data of that future time step as the actual state data of that future time step.

[0214] Step 720: For each future time step, calculate the difference between the predicted state data and the actual state data of that future time step to obtain the state difference data corresponding to the future time step.

[0215] In one example implementation, for each future time step, a first mean squared error between the predicted state data and the actual state data of that future time step can be calculated, and the first mean squared error can be used as the state difference data corresponding to that future time step.

[0216] The first mean squared error can be understood as an indicator used to measure the difference between the predicted state data at a future time step and the actual state data at that future time step.

[0217] For example, calculating the first mean squared error between the predicted state data and the actual state data at the future time step can include steps S11-S13:

[0218] S11. Calculate the first difference between the predicted state data of the future time step and the actual state data of the future time step.

[0219] S12. Squaring the data in each dimension of the first difference data to obtain the first square value of each dimension, and summing the first square values ​​of each dimension to obtain the first total square sum.

[0220] S13. Divide the first total sum of squares by the number of dimensions of the first difference data to obtain the first mean square error.

[0221] Step 730: Weight the state difference data corresponding to each future time step and sum them to obtain the first loss.

[0222] Therefore, the first loss can be determined based on the state difference data corresponding to each future time step, thereby improving the accuracy of the first loss.

[0223] In some example implementations, the second loss calculated based on the action policy output by the motion control policy network can be executed. Figure 8 The flowchart shown is a training method for a robot motion control policy network based on deep reinforcement learning. Figure 8 As shown, steps 810-820 are included:

[0224] Step 810: For the state data of each historical time step in the historical state data and the corresponding action strategy, control the robot to execute the action strategy under the state data in the simulation environment, and receive the target action taken by the robot when executing the action strategy, the reward value of the robot after executing the target action, and the target state data of the robot returned by the simulation environment.

[0225] A simulation environment can be understood as a virtual operating platform used to simulate robot dynamics, sensor feedback, and interaction mechanisms with the outside world.

[0226] Controlling a robot to execute an action strategy under this state data in a simulation environment can be understood as the virtual robot corresponding to the robot in the simulation environment executing an action strategy under this state data.

[0227] When a robot executes a motion strategy corresponding to its state data, it determines the target motion based on that strategy. The reward value can be understood as a numerical value generated by the simulation environment based on preset motion evaluation rules after the robot performs the target motion. This value characterizes the quality of the robot's execution of the target motion. The reward value can be used to measure the quality of the robot's execution of the target motion.

[0228] A higher reward value indicates that the robot performs the target action better; a lower reward value indicates that the robot performs the target action worse.

[0229] The simulation environment can generate corresponding reward values ​​based on the robot's stability, posture changes, and energy consumption when performing target actions in the simulation environment.

[0230] The robot's target state data is the robot's target state data returned by the simulation environment after the robot performs the target action. In other words, it is the robot's state data returned by the simulation environment after the robot performs the target action, which is the robot's body perception data returned by the simulation environment after the robot performs the target action.

[0231] Step 820: Calculate the second loss based on the state data of each historical time step in the historical state data, the reward value corresponding to each state data, and the target state data.

[0232] In some example implementations, calculating the second loss based on the state data at each historical time step in the historical state data, the reward value corresponding to each state data, and the target state data may include steps 8201-8202:

[0233] Step 8201: For the state data of each historical time step in the historical state data, based on the state data, the reward value corresponding to the state data, and the target state data, determine the advantage value of the robot to perform the target action corresponding to the state data.

[0234] The advantage value can be understood as a metric that measures how much better a robot performs a target action compared to the average action given the given state data.

[0235] A dominance value greater than 0 indicates that the target action is more effective than the average action.

[0236] A dominance value less than 0 indicates that the target action is less effective than the average action.

[0237] When the advantage value is 0, it means that the effect of the target action is the same as the effect of the average action.

[0238] In one example implementation, determining the advantage value of the robot performing the target action corresponding to the state data based on the state data, the reward value corresponding to the state data, and the target state data may include S21-S23:

[0239] S21. The value of the state data is evaluated based on the value network to obtain the current state value, and the value of the target state data corresponding to the state data is evaluated based on the value network to obtain the target state value.

[0240] A value network can be understood as a neural network that predicts the total future revenue, used to evaluate the state value corresponding to the robot's state data, or simply the state value. The state value can be understood as the magnitude of the robot's future revenue derived from its state data.

[0241] The current state value can be understood as the state value of the robot's state data.

[0242] The target state value can be understood as the state value of the target state data corresponding to the robot's current state data.

[0243] S22. Sum the reward value and the target state value to obtain the target sum value.

[0244] The target and the value can be understood as the actual benefit the robot gains from performing the target action under the given state data.

[0245] S23. Calculate the difference between the target value and the current state value to obtain the advantage value of the robot in performing the target action corresponding to the state data.

[0246] Step 8202: Based on the advantage value corresponding to the state data and the execution probability of the target action in the action strategy corresponding to the state data, calculate the sub-loss corresponding to the state data.

[0247] The probability of executing a target action can be understood as the likelihood that the robot will execute the target action under given state data.

[0248] For example, the advantage value corresponding to the state data can be multiplied by the execution probability of the target action in the action policy corresponding to the state data to obtain the target product, and then the negative value of the target product can be determined as the sub-loss corresponding to the state data.

[0249] Step 8203: Sum the sub-losses corresponding to the state data of each historical time step in the historical state data to obtain the second loss.

[0250] By minimizing the second loss, the motion control policy network can adjust the action probability distribution in the action policy according to the size of the advantage value, thereby increasing the selection probability of actions with larger advantage values ​​and decreasing the selection probability of actions with smaller advantage values. This enables the motion control policy network to continuously optimize and maximize the expected policy return.

[0251] Therefore, the second loss can be calculated based on deep reinforcement learning algorithms, thereby improving the accuracy of the second loss calculation.

[0252] In some example implementations, updating the parameters of the encoder, motion control policy network, and forward prediction model based on the first loss and the second loss can be performed. Figure 9 The flowchart shown is a training method for a robot motion control policy network based on deep reinforcement learning. Figure 9 As shown, steps 910-930 are included:

[0253] Step 910: Obtain the state loss weight corresponding to the first loss and the policy loss weight corresponding to the second loss.

[0254] The state loss weight and policy loss weight can be preset as needed.

[0255] Step 920: Based on the state loss weight and policy loss weight, the first loss and the second loss are weighted and summed to obtain the joint loss.

[0256] Step 930: Based on the joint loss, update the parameters of the encoder, motion control policy network, and forward prediction model.

[0257] In one example implementation, the encoder, motion control policy network, and forward prediction model can be trained separately based on the joint loss until the joint loss reaches its minimum value, and then the parameters of the encoder, motion control policy network, and forward prediction model can be updated.

[0258] In one example implementation, the encoder, motion control policy network, and forward prediction model are trained separately based on the joint loss until the joint loss reaches its minimum value. The parameters of the encoder, motion control policy network, and forward prediction model are then updated, enabling execution... Figure 10 The flowchart shown is a training method for a robot motion control policy network based on deep reinforcement learning. Figure 10 As shown, steps 1010-1030 are included:

[0259] Step 1010: With the goal of minimizing the joint loss, backpropagation calculations are performed on the encoder, motion control policy network, and forward prediction model based on the joint loss to obtain the gradient values ​​of each parameter in the encoder, motion control policy network, and forward prediction model.

[0260] The gradient value can be understood as a numerical value used to characterize the direction of updating model parameters.

[0261] The gradient value can be understood as the numerical value of the direction of change of model parameters that reduces the model's loss.

[0262] Step 1020: Obtain the first preset learning rate of the encoder, the second preset learning rate of the motion control strategy network, and the third preset learning rate of the forward prediction model.

[0263] The learning rate (LR) can be understood as the magnitude or step size of the update of model parameters.

[0264] The first, second, and third preset learning rates can be set in advance as needed, and are not limited here.

[0265] Step 1030: Update the parameters of the encoder, motion control strategy network, and forward prediction model based on the gradient value, the first preset learning rate, the second preset learning rate, and the third preset learning rate.

[0266] For example, updating the parameters of the encoder, motion control policy network, and forward prediction model based on the gradient value, the first preset learning rate, the second preset learning rate, and the third preset learning rate may include steps 1031-1033:

[0267] Step 1031: Calculate the product of the gradient value of each parameter in the encoder and the first preset learning rate to obtain the update increment of each parameter in the encoder; calculate the product of the gradient value of each parameter in the motion control strategy network and the second preset learning rate to obtain the update increment of each parameter in the motion control strategy network; calculate the product of the gradient value of each parameter in the forward prediction model and the third preset learning rate to obtain the update increment of each parameter in the forward prediction model.

[0268] Step 1032: For each parameter, sum the parameter and the parameter update increment to generate the parameter update value.

[0269] Step 1033: Replace each parameter in the encoder, motion control strategy network, and forward prediction model with the updated parameter values ​​to complete the parameter update of the encoder, motion control strategy network, and forward prediction model.

[0270] Therefore, the parameters of the encoder, motion control policy network, and forward prediction model can be updated by combining the state loss of the forward prediction model and the policy loss of the motion control policy network. This allows the motion control policy network to not only learn the mapping from the current state to the action policy during training, but also implicitly learn the dynamic evolution law from the current state to the future state. This improves the motion control policy network's ability to model continuous action structures, enabling it to learn autonomously and adapt to changes in the environment during training. Consequently, it accelerates the policy convergence speed of the policy network, improves the utilization rate of samples, and enhances the training efficiency and generalization ability of the motion control policy network.

[0271] This disclosure provides a robot motion control method based on deep reinforcement learning, such as... Figure 11 The flowchart shown illustrates a robot motion control method based on deep reinforcement learning, which may include the following steps 1110 to 1140:

[0272] Step 1110: Obtain the robot's current state data, which includes the robot's body perception data.

[0273] Step 1120: Input the current state data into the pre-trained encoder to generate target representation features.

[0274] Step 1130: Input the target representation features into the pre-trained motion control policy network, and output the action policy for controlling the robot's motion based on the motion control policy network. The encoder and motion control policy network can be obtained according to the deep reinforcement learning-based robot motion control policy network training method of any of the above embodiments.

[0275] Therefore, the parameters of the encoder, motion control policy network, and forward prediction model can be updated by combining the state loss of the forward prediction model and the policy loss of the motion control policy network. This allows the motion control policy network to not only learn the mapping from the current state to the action policy during training, but also implicitly learn the dynamic evolution law from the current state to the future state. Since the motion control policy network can predict future state changes, the robot can generate good, coherent, and smooth posture transitions when executing action sequences, reducing the probability of sudden action changes or stiff posture switching, and improving the naturalness and continuity of the robot's movements.

[0276] This disclosure provides a robot motion control policy network training device based on deep reinforcement learning, such as... Figure 12 The diagram shows a structural schematic of a robot motion control policy network training device based on deep reinforcement learning. Figure 12 As shown, the robot motion control policy network training device 1200 based on deep reinforcement learning includes:

[0277] The first acquisition module 1210 is used to acquire the robot's historical state data and input it into the encoder to obtain the first representation feature. The historical state data is the robot's body perception data.

[0278] The enhancement module 1220 is used to sample historical state data to obtain sampled state data, and after enhancing the sampled state data, it is input to the encoder to obtain the second representation feature.

[0279] The generation module 1230 is used to input the first representation feature into the motion control policy network to generate the action policy, and to input the second representation feature into the forward prediction model to generate the predicted state data for multiple future time steps.

[0280] The update module 1240 is used to calculate a first loss based on the difference between the predicted state data and the actual state data at multiple future time steps, and to calculate a second loss based on the action policy output by the motion control policy network. Based on the first loss and the second loss, the parameters of the encoder, the motion control policy network and the forward prediction model are updated.

[0281] The robot motion control policy network training device based on deep reinforcement learning provided in this disclosure can implement the robot motion control policy network training method based on deep reinforcement learning in any of the above embodiments. Its execution method and beneficial effects are similar, and will not be described again here.

[0282] This disclosure provides a robot motion control device based on deep reinforcement learning, such as... Figure 13The diagram shows a structural schematic of a robot motion control device based on deep reinforcement learning. Figure 13 As shown, the robot motion control device 1300 based on deep reinforcement learning includes:

[0283] The second acquisition module 1310 acquires the robot's current state data, which includes the robot's body perception data.

[0284] The feature generation module 1320 is used to input the current state data into the pre-trained encoder to generate target representation features;

[0285] The strategy generation module 1330 is used to input the target representation features into a pre-trained motion control strategy network and output an action strategy for controlling the robot's motion based on the motion control strategy network; wherein, the encoder and the motion control strategy network are obtained according to the robot motion control strategy network training method based on deep reinforcement learning in any of the above embodiments.

[0286] The robot motion control device based on deep reinforcement learning provided in this disclosure can implement the robot motion control method based on deep reinforcement learning in any of the above embodiments. Its execution method and beneficial effects are similar, and will not be described again here.

[0287] This disclosure also provides an electronic device, which includes a processor and a memory, wherein the memory stores a computer program. When the computer program is executed by the processor, it can implement the methods of any of the above embodiments. The execution method and beneficial effects are similar, and will not be described again here.

[0288] The electronic device in this disclosure can be understood as any device with processing and computing capabilities. This device may include, but is not limited to, mobile terminals such as smartphones, laptops, personal digital assistants (PDAs), tablet computers (PADs), portable multimedia players (PMPs), in-vehicle terminals (e.g., in-vehicle navigation terminals), and wearable devices, as well as fixed electronic devices such as digital TVs, desktop computers, and smart home devices.

[0289] Figure 14 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure, such as... Figure 14 As shown, the electronic device 1400 may include a processor 1410 and a memory 1420. The memory 1420 stores a computer program 1421. When the computer program 1421 is executed by the processor 1410, it can implement the method provided in any of the above embodiments. The execution mode and beneficial effects are similar and will not be described again here.

[0290] Of course, for the sake of simplicity, Figure 14Only some of the components of the electronic device 1400 relevant to the present invention are shown in this illustration; components such as buses, input / output interfaces, input devices, and output devices are omitted. In addition, the electronic device 1400 may include any other suitable components depending on the specific application.

[0291] This disclosure also provides a robot, which includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, it can implement the above-mentioned robot motion control method based on deep reinforcement learning. Its execution method and beneficial effects are similar and will not be described again here.

[0292] The robots include any one of the following: legged robots, quadruped robots, bipedal robots, wheeled robots, wheel-legged robots, quadrupedal robots, humanoid robots, and mobile robots. (Reference) Figure 15-17 The diagram shows schematics of three different robots.

[0293] This disclosure provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement the methods of any of the above embodiments. The execution method and beneficial effects are similar, and will not be described again here.

[0294] The aforementioned computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0295] The computer program described above can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's electronic device, partially on the user's device, as a standalone software package, partially on the user's electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server.

[0296] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0297] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0298] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for training a robot motion control policy network based on deep reinforcement learning, characterized in that, The method comprises: obtaining historical state data of a robot and inputting the historical state data into an encoder to obtain first representation features, the historical state data being proprioceptive data of the robot; sampling the historical state data to obtain sampled state data, and inputting the sampled state data into the encoder after enhancement processing to obtain second representation features; inputting the first representation features into a motion control policy network to generate an action policy, and inputting the second representation features into a forward prediction model to generate predicted state data of a plurality of future time steps; calculating a first loss based on a difference between the predicted state data of the plurality of future time steps and real state data, and calculating a second loss based on the action policy output by the motion control policy network, and updating parameters of the encoder, the motion control policy network and the forward prediction model according to the first loss and the second loss. 2.The method of claim 1, wherein, The method comprises: obtaining proprioceptive data of the robot at a plurality of historical time steps; sorting the plurality of proprioceptive data in a time step order to obtain the historical state data of the robot; inputting the historical state data into the encoder to extract features of the proprioceptive data of the plurality of historical time steps in the historical state data based on the encoder, to obtain first representation features corresponding to the historical state data. 3.The method of claim 1, wherein, The method comprises: reading state data of a plurality of time steps in the historical state data; selecting a plurality of state data as candidate samples in the state data at a preset time step interval; sorting the candidate samples in a time step order corresponding to the candidate samples to obtain sampled state data. 4.The method of claim 1, wherein, The method comprises: enhancing the sampled state data based on a preset perturbation strategy to obtain enhanced state data, the preset perturbation strategy comprising one or more of a noise perturbation strategy and a mask perturbation strategy; inputting the enhanced state data into the encoder to extract features of the enhanced state data based on the encoder, to obtain second representation features.

5. The method of claim 4, wherein the method further comprises: When the preset perturbation strategy comprises a noise perturbation strategy, the method comprises: adding Gaussian noise data to the sampled state data to obtain enhanced state data. 6.The method of claim 4, wherein, When the preset perturbation strategy comprises a noise perturbation strategy, the method comprises: uniformly adding target noise data to the sampled state data to obtain enhanced state data.

7. The method of claim 4, wherein the method further comprises: When the preset perturbation strategy comprises a mask perturbation strategy, the method comprises: Adding a random mask to the sampled state data to obtain enhanced state data.

8. The method of claim 7, wherein the method further comprises: The adding a random mask to the sampled state data to obtain enhanced state data comprises: Randomly selecting a plurality of target positions of the sampled state data; Replacing data corresponding to each of the target positions in the sampled state data with a mask to obtain enhanced state data. 9.The method of claim 1, wherein, The inputting the first representation feature into the motion control policy network to generate an action policy comprises: The inputting the first representation feature into the motion control policy network comprises: for each representation feature of a historical time step in the first representation feature, performing action inference on the representation feature based on the motion control policy network to output an action policy corresponding to state data of the historical time step. 10.The method of claim 1, wherein, The inputting the second representation feature into the forward prediction model to generate predicted state data of a plurality of future time steps comprises: Inputting a first state feature corresponding to a first time step in the second representation feature into the forward prediction model; Performing state prediction on the first state feature for a plurality of future time steps based on the forward prediction model to obtain predicted state data of the plurality of future time steps.

11. The method of claim 10, wherein the method further comprises: The performing state prediction on the first state feature for a plurality of future time steps based on the forward prediction model to obtain predicted state data of the plurality of future time steps comprises: Performing state prediction on the first state feature for a next time step based on the forward prediction model to obtain predicted state data of a second time step; Inputting the predicted state data of the second time step into the forward prediction model, performing state prediction on the predicted state data of the second time step for a next time step based on the forward prediction model to obtain predicted state data of a third time step; Inputting the predicted state data of the third time step into the forward prediction model, repeatedly calling the forward prediction model to perform state prediction, and comprehensively obtaining predicted state data of the plurality of future time steps.

12. The method of claim 1, wherein, The calculating a first loss based on differences between the predicted state data of the plurality of future time steps and real state data comprises: For each future time step, determining historical state data of the future time step as real state data of the future time step; For each future time step, calculating a difference between predicted state data of the future time step and real state data of the future time step to obtain state difference data corresponding to the future time step; Summing the state difference data corresponding to each future time step to obtain a first loss.

13. The method of claim 12, wherein the method further comprises: The calculating a difference between predicted state data of the future time step and real state data of the future time step to obtain state difference data corresponding to the future time step comprises: For each future time step, calculating a first mean square difference value between predicted state data of the future time step and real state data of the future time step, and taking the first mean square difference value as state difference data corresponding to the future time step.

14. The method of claim 13, wherein the method further comprises: The first mean square error value between the predicted state data of the future time step and the real state data of the future time step is calculated, comprising: calculating the first difference value data between the predicted state data of the future time step and the real state data of the future time step; squaring the data of each dimension in the first difference value data to obtain the first square value of each dimension, and summing the first square value of each dimension to obtain the first total square sum; divide the first total square sum by the number of dimensions of the first difference value data to obtain the first mean square error value.

15. The method of claim 1, wherein, The second loss is calculated based on the action strategy output by the motion control strategy network, comprising: for the state data of each historical time step in the historical state data and the action strategy corresponding to the state data, control the robot to execute the action strategy under the state data in the simulation environment, receive the target action taken by the robot when executing the action strategy and the reward value and the target state data of the robot after executing the target action returned by the simulation environment; based on the reward value and the target state data corresponding to the historical state data, calculate the second loss.

16. The method of claim 15, wherein the method further comprises: The second loss is calculated based on the reward value and the target state data corresponding to the historical state data, comprising: for the state data of each historical time step in the historical state data, based on the state data, the reward value and the target state data corresponding to the state data, determine the advantage value of the robot executing the target action corresponding to the state data; based on the advantage value corresponding to the state data and the execution probability of the target action in the action strategy corresponding to the state data, calculate the sub-loss corresponding to the state data; sum the sub-losses corresponding to the state data of each historical time step in the historical state data to obtain the second loss.

17. The method of claim 16, wherein the method further comprises: The advantage value of the robot executing the target action corresponding to the state data is determined based on the state data, the reward value and the target state data corresponding to the state data, comprising: value evaluation of the state data based on the value network to obtain the current state value, value evaluation of the target state data based on the value network to obtain the target state value; sum the reward value and the target state value to obtain the target sum value; calculate the difference value between the target sum value and the current state value to obtain the advantage value of the robot executing the target action corresponding to the state data.

18. The method of claim 1, wherein, According to the first loss and the second loss, the parameters of the encoder, the motion control strategy network and the forward prediction model are updated, comprising: obtain the state loss weight corresponding to the first loss and the strategy loss weight corresponding to the second loss; based on the state loss weight and the strategy loss weight, weighted sum the first loss and the second loss to obtain the joint loss; based on the joint loss, update the parameters of the encoder, the motion control strategy network and the forward prediction model.

19. The method of claim 18, wherein the method further comprises: The parameters of the encoder, the motion control strategy network and the forward prediction model are updated based on the joint loss, including: The parameters of the encoder, the motion control strategy network and the forward prediction model are trained based on the joint loss respectively until the joint loss reaches a minimum value, and the parameters of the encoder, the motion control strategy network and the forward prediction model are updated.

20. The method of claim 19, wherein the method further comprises: The parameters of the encoder, the motion control strategy network and the forward prediction model are trained based on the joint loss respectively until the joint loss reaches a minimum value, and the parameters of the encoder, the motion control strategy network and the forward prediction model are updated, including: The parameters of the encoder, the motion control strategy network and the forward prediction model are trained based on the joint loss respectively until the joint loss reaches a minimum value, and the parameters of the encoder, the motion control strategy network and the forward prediction model are updated. The gradient values of each parameter in the encoder, the motion control strategy network and the forward prediction model are calculated based on the joint loss, and the first preset learning rate, the second preset learning rate and the third preset learning rate of the encoder, the motion control strategy network and the forward prediction model are obtained. The parameters of the encoder, the motion control strategy network and the forward prediction model are updated based on the gradient values, the first preset learning rate, the second preset learning rate and the third preset learning rate.

21. The method of claim 20, wherein the method further comprises: The parameters of the encoder, the motion control strategy network and the forward prediction model are updated based on the gradient values, the first preset learning rate, the second preset learning rate and the third preset learning rate, including: The product of the gradient value of each parameter in the encoder and the first preset learning rate is calculated to obtain the update increment of each parameter in the encoder; the product of the gradient value of each parameter in the motion control strategy network and the second preset learning rate is calculated to obtain the update increment of each parameter in the motion control strategy network; the product of the gradient value of each parameter in the forward prediction model and the third preset learning rate is calculated to obtain the update increment of each parameter in the forward prediction model. For each parameter, the parameter and the update increment of the parameter are summed to generate the update value of the parameter. The parameters in the encoder, the motion control strategy network and the forward prediction model are replaced by the update value of the parameter to complete the parameter update of the encoder, the motion control strategy network and the forward prediction model.

22. A method for robot motion control based on deep reinforcement learning, the method comprising: Including: Obtain current state data of the robot, the current state data including body perception data of the robot; Input the current state data into the pre-trained encoder to generate target feature representation; Input the target feature representation into the pre-trained motion control strategy network to output an action strategy for controlling the motion of the robot based on the motion control strategy network. The encoder and the motion control strategy network are obtained according to the robot motion control strategy network training method based on deep reinforcement learning in any one of claims 1-21.

23. A deep reinforcement learning based robot motion control policy network training apparatus, characterized by, The method comprises: a first obtaining module configured to obtain historical state data of a robot and input the historical state data to an encoder to obtain a first representation feature, the historical state data being proprioceptive data of the robot; an enhancement module configured to sample the historical state data to obtain sampled state data, and input the sampled state data to the encoder after enhancement processing to obtain a second representation feature; a generation module configured to input the first representation feature to a motion control policy network to generate an action policy, and input the second representation feature to a forward prediction model to generate predicted state data of a plurality of future time steps; an updating module configured to calculate a first loss based on a difference between the predicted state data of the plurality of future time steps and real state data, and calculate a second loss based on the action policy output by the motion control policy network, and update parameters of the encoder, the motion control policy network and the forward prediction model according to the first loss and the second loss.

24. A robot motion control apparatus based on deep reinforcement learning, characterized by, The method comprises: a second obtaining module configured to obtain current state data of a robot, the current state data including proprioceptive data of the robot; a feature generation module configured to input the current state data to a pre-trained encoder to generate a target representation feature; a policy generation module configured to input the target representation feature to a pre-trained motion control policy network, and output an action policy for controlling motion of the robot based on the motion control policy network. The encoder and the motion control policy network are obtained according to the method for training a robot motion control policy network based on deep reinforcement learning according to any one of claims 1-21.

25. An electronic device, comprising: The method comprises: a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the method according to any one of claims 1-22 is implemented.

26. A robot characterized by The method comprises: a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the method according to claim 22 is implemented.

27. The robot of claim 26, wherein, The robot comprises any one of a foot-type robot, a quadruped robot, a biped robot, a wheeled robot, a wheel-foot robot, a four-wheel-foot robot, a humanoid robot and a mobile robot.

28. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and when the computer program is executed by the processor, the method according to any one of claims 1-22 is implemented.

Images