Robot motion control strategy network training method and device based on deep reinforcement learning, robot motion control method and device, equipment, robot and storage medium

Abstract

The invention provides a robot motion control strategy network training method and device based on deep reinforcement learning, a robot motion control method and device, equipment, a robot and a storage medium, and relates to the technical field of robots. According to the invention, the forward prediction model is introduced and shares the output characteristics of the encoder with the motion control strategy network, and the parameters of the encoder, the motion control strategy network and the forward prediction model are updated through the comprehensive loss of the state loss of the forward prediction model and the strategy loss of the motion control strategy network. The motion control strategy network can autonomously learn and adapt to environment changes in the training process, the strategy convergence speed of the strategy network is increased, the sample utilization rate, the training efficiency and the generalization ability are improved, a robot can generate good coherent and smooth posture transition actions when executing an action sequence, and the training efficiency is improved. The occurrence probability of action sudden change or posture switching stiffness is reduced, and the naturalness and continuity of actions in the movement process of the robot are improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of robots, and particularly relates to a robot motion control strategy network training method based on deep reinforcement learning, a robot motion control method, a device, an apparatus, a robot and a storage medium. BACKGROUND

[0002] Currently, when training a robot motion control strategy network, the motion control strategy network is difficult to learn the timing relationship between continuous actions, resulting in low sample utilization rate, slow policy convergence speed, long training period, and unstable performance in tasks involving continuous adjustment of posture. In the strategy network deployment stage, due to the lack of prediction ability of the motion control strategy network for the future state of the robot, the output action strategy is prone to cause action mutation or posture switching stiffness in the execution process, and it is difficult to meet the requirements of action naturalness and continuity in high dynamic tasks.

[0003] For example, in the Chinese patent application with publication number CN119830943A, a deep reinforcement learning algorithm combining auxiliary tasks, experience double sampling and dynamic correction of value function is disclosed when controlling the actions of an agent. Although the quality of feature extraction is improved and the model's understanding of the state is enhanced by introducing auxiliary tasks, and the sample utilization efficiency is improved and the learning process is accelerated by experience double sampling, there are still problems of low sample utilization rate, slow policy convergence speed, long training period, high probability of action mutation or posture switching stiffness, and poor action naturalness and continuity.

[0004] Therefore, the current robot motion control strategy network training method still needs to be further optimized. SUMMARY

[0005] To solve the problems of low sample utilization rate, slow policy convergence speed, long training period, high probability of action mutation or posture switching stiffness, and poor action naturalness and continuity in the related art, the present disclosure provides a robot motion control strategy network training method based on deep reinforcement learning, a robot motion control method, a device, an apparatus, a robot and a storage medium, which enables the motion control strategy network to learn and adapt to changes in the environment during training, accelerates the policy convergence speed of the motion control strategy network, improves the sample utilization rate, training efficiency and generalization ability of the motion control strategy network, reduces the probability of action mutation or posture switching stiffness of the robot, and improves the naturalness and continuity of the robot's motion.

[0006] A first aspect of the present disclosure provides a robot motion control strategy network training method based on deep reinforcement learning, comprising: The historical state data of the robot is acquired and input into an encoder to obtain a first representation feature, and the historical state data is body perception data of the robot; The historical state data is sampled to obtain sampled state data, and the sampled state data is input into the encoder after enhancement processing to obtain a second representation feature; The first representation feature is input into a motion control policy network to generate an action policy, and the second representation feature is input into a forward prediction model to generate predicted state data of a plurality of future time steps; A first loss is calculated based on a difference between the predicted state data of the plurality of future time steps and real state data, and a second loss is calculated based on the action policy output by the motion control policy network, and parameters of the encoder, the motion control policy network and the forward prediction model are updated according to the first loss and the second loss.

[0007] Optionally, the above-mentioned acquiring the historical state data of the robot and inputting the historical state data into the encoder to obtain the first representation feature comprises: Acquiring body perception data of the robot at a plurality of historical time steps; The plurality of body perception data are sorted in a time step order to obtain the historical state data of the robot; The historical state data is input into the encoder, and the encoder extracts features of the body perception data of the plurality of historical time steps in the historical state data to obtain the first representation feature corresponding to the historical state data.

[0008] Optionally, the above-mentioned sampling the historical state data to obtain the sampled state data comprises: Reading state data of a plurality of time steps in the historical state data; In the state data, a plurality of state data are selected as candidate samples at a preset time step interval; The candidate samples are sorted in a time step order corresponding to the candidate samples to obtain the sampled state data.

[0009] Optionally, the above-mentioned inputting the sampled state data into the encoder after enhancement processing to obtain the second representation feature comprises: The sampled state data is enhanced based on a preset perturbation strategy to obtain enhanced state data, and the preset perturbation strategy comprises one or more of a noise perturbation strategy and a mask perturbation strategy; The enhanced state data is input into the encoder, and the encoder extracts features of the enhanced state data to obtain the second representation feature.

[0010] Optionally, when the preset perturbation strategy comprises a noise perturbation strategy, the above-mentioned enhancing the sampled state data based on the preset perturbation strategy to obtain the enhanced state data comprises: Gaussian noise data is added to the sampled state data to obtain enhanced state data.

[0011] Optionally, when the preset perturbation strategy includes the noise perturbation strategy, the enhanced processing is performed on the sampled state data based on the preset perturbation strategy to obtain the enhanced state data, including: Target noise data is added to the sampled state data to obtain enhanced state data.

[0012] Optionally, when the preset perturbation strategy includes the mask perturbation strategy, the enhanced processing is performed on the sampled state data based on the preset perturbation strategy to obtain the enhanced state data, including: Random mask is added to the sampled state data to obtain enhanced state data.

[0013] Optionally, the random mask is added to the sampled state data to obtain the enhanced state data, including: Randomly selecting a plurality of target positions of the sampled state data; Replacing the data corresponding to each target position in the sampled state data with a mask to obtain enhanced state data.

[0014] Optionally, the first representation feature is input into the motion control strategy network to generate the action strategy, including: The first representation feature is input into the motion control strategy network, and for each representation feature of the historical time step in the first representation feature, the motion control strategy network performs action reasoning based on the representation feature to output the action strategy corresponding to the state data of the historical time step.

[0015] Optionally, the second representation feature is input into the forward prediction model to generate the predicted state data of the plurality of future time steps, including: The first state feature corresponding to the first time step in the second representation feature is input into the forward prediction model; The forward prediction model is used to perform state prediction of the plurality of future time steps on the first state feature to obtain the predicted state data of the plurality of future time steps.

[0016] Optionally, the forward prediction model is used to perform state prediction of the plurality of future time steps on the first state feature to obtain the predicted state data of the plurality of future time steps, including: The forward prediction model is used to perform state prediction of the next time step on the first state feature to obtain the predicted state data of the second time step; The predicted state data of the second time step is input into the forward prediction model, and the forward prediction model is used to perform state prediction of the next time step on the predicted state data of the second time step to obtain the predicted state data of the third time step; The prediction state data of the third time step is input into the forward prediction model, the forward prediction model is repeatedly called for state prediction, and the prediction state data of multiple future time steps is obtained by integration.

[0017] Optionally, the first loss is calculated based on the difference between the prediction state data of the multiple future time steps and the real state data. For each future time step, the historical state data of the future time step is determined as the real state data of the future time step. For each future time step, the difference between the prediction state data of the future time step and the real state data of the future time step is calculated to obtain state difference data corresponding to the future time step. The state difference data corresponding to each future time step is weighted and summed to obtain the first loss.

[0018] Optionally, the difference between the prediction state data of the future time step and the real state data of the future time step is calculated for each future time step to obtain state difference data corresponding to the future time step, comprising: For each future time step, a first mean square error value between the prediction state data of the future time step and the real state data of the future time step is calculated, and the first mean square error value is taken as the state difference data corresponding to the future time step.

[0019] Optionally, the first mean square error value between the prediction state data of the future time step and the real state data of the future time step is calculated, comprising: The first difference value data between the prediction state data of the future time step and the real state data of the future time step is calculated. Each dimension of the first difference value data is squared to obtain a first square value of each dimension, and the first square values of each dimension are summed to obtain a first total square sum. The first total square sum is divided by the number of dimensions of the first difference value data to obtain the first mean square error value.

[0020] Optionally, the second loss is calculated based on the action strategy output by the motion control strategy network, comprising: For each historical time step of the historical state data and the state data corresponding to the action strategy, the robot is controlled to execute the action strategy under the state data in the simulation environment, the target action taken by the robot when executing the action strategy and the reward value and the target state data of the robot after executing the target action are received from the simulation environment. The second loss is calculated based on the reward value and the target state data corresponding to the historical state data.

[0021] Optionally, the reward value corresponding to the historical state data and the target state data are used to calculate a second loss, including: For each historical state data in the historical state data, an advantage value of the robot performing the target action corresponding to the state data is determined based on the state data, the reward value corresponding to the state data, and the target state data; A sub-loss corresponding to the state data is calculated based on the advantage value corresponding to the state data and an execution probability of the target action in the action policy corresponding to the state data; The sub-losses corresponding to the state data of each historical time step in the historical state data are summed to obtain the second loss.

[0022] Optionally, the advantage value of the robot performing the target action corresponding to the state data is determined based on the state data, the reward value corresponding to the state data, and the target state data, including: The state data is evaluated by the value network to obtain a current state value, and the target state data is evaluated by the value network to obtain a target state value; The reward value and the target state value are summed to obtain a target sum value; The difference between the target sum value and the current state value is calculated to obtain the advantage value of the robot performing the target action corresponding to the state data.

[0023] Optionally, the parameters of the encoder, the motion control policy network, and the forward prediction model are updated according to the first loss and the second loss, including: The state loss weight corresponding to the first loss and the policy loss weight corresponding to the second loss are obtained; The first loss and the second loss are weighted and summed based on the state loss weight and the policy loss weight to obtain a joint loss; The parameters of the encoder, the motion control policy network, and the forward prediction model are updated based on the joint loss.

[0024] Optionally, the parameters of the encoder, the motion control policy network, and the forward prediction model are updated based on the joint loss, including: The encoder, the motion control policy network, and the forward prediction model are trained based on the joint loss respectively until the joint loss reaches a minimum value, and the parameters of the encoder, the motion control policy network, and the forward prediction model are updated.

[0025] Optionally, the parameters of the encoder, the motion control policy network, and the forward prediction model are updated based on the joint loss, including: The gradients of the parameters in the encoder, the motion control strategy network and the forward prediction model are obtained by performing back propagation calculation on the encoder, the motion control strategy network and the forward prediction model based on the joint loss. The first preset learning rate of the encoder, the second preset learning rate of the motion control strategy network and the third preset learning rate of the forward prediction model are obtained. The parameters of the encoder, the motion control strategy network and the forward prediction model are updated based on the gradient values, the first preset learning rate, the second preset learning rate and the third preset learning rate.

[0026] Optionally, the updating of the parameters of the encoder, the motion control strategy network and the forward prediction model based on the gradient values, the first preset learning rate, the second preset learning rate and the third preset learning rate comprises: The product of the gradient value of each parameter in the encoder and the first preset learning rate is calculated to obtain the update increment of each parameter in the encoder; the product of the gradient value of each parameter in the motion control strategy network and the second preset learning rate is calculated to obtain the update increment of each parameter in the motion control strategy network; and the product of the gradient value of each parameter in the forward prediction model and the third preset learning rate is calculated to obtain the update increment of each parameter in the forward prediction model. For each parameter, the parameter and the update increment of the parameter are summed to generate an updated value of the parameter. The parameters in the encoder, the motion control strategy network and the forward prediction model are replaced by the updated values of the parameters, and the parameter updating of the encoder, the motion control strategy network and the forward prediction model is completed.

[0027] The second aspect of the present disclosure provides a robot motion control method based on deep reinforcement learning, comprising: The current state data of the robot is obtained, and the current state data includes the body perception data of the robot. The current state data is input into the pre-trained encoder to generate target representation features. The target representation features are input into the pre-trained motion control strategy network, and the motion control strategy network outputs an action strategy for controlling the motion of the robot. The encoder and the motion control strategy network are obtained according to the robot motion control strategy network training method based on deep reinforcement learning of any one of the first aspect.

[0028] The third aspect of the present disclosure provides a robot motion control strategy network training device based on deep reinforcement learning, comprising: The first acquisition module is configured to obtain the historical state data of the robot and input the historical state data into the encoder to obtain first representation features, and the historical state data is the body perception data of the robot. an enhancement module, configured to sample the historical state data to obtain sampled state data, and input the sampled state data into the encoder after enhancement processing to obtain second representation features; a generation module, configured to input the first representation features into the motion control policy network to generate action policies, and input the second representation features into the forward prediction model to generate predicted state data of a plurality of future time steps; an update module, configured to calculate a first loss based on a difference between the predicted state data of the plurality of future time steps and the real state data, and calculate a second loss based on the action policies output by the motion control policy network, and update parameters of the encoder, the motion control policy network and the forward prediction model according to the first loss and the second loss.

[0029] A fourth aspect of the present disclosure provides a robot motion control device based on deep reinforcement learning, comprising: a second acquisition module, configured to acquire current state data of the robot, the current state data comprising body perception data of the robot; a feature generation module, configured to input the current state data into a pre-trained encoder to generate target representation features; a policy generation module, configured to input the target representation features into a pre-trained motion control policy network, and output action policies for controlling the motion of the robot based on the motion control policy network; The encoder and the motion control policy network are obtained according to the robot motion control policy network training method based on deep reinforcement learning of any one of the first aspect.

[0030] A fifth aspect of the present disclosure provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the robot motion control policy network training method based on deep reinforcement learning of the first aspect or the robot motion control method based on deep reinforcement learning of the second aspect is implemented.

[0031] A sixth aspect of the present disclosure provides a robot comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the robot motion control method based on deep reinforcement learning of the second aspect is implemented.

[0032] Optionally, the robot comprises any one of a foot-type robot, a quadruped robot, a biped robot, a wheeled robot, a wheel-foot robot, a four-wheel-foot robot, a humanoid robot and a mobile robot.

[0033] The seventh aspect of the present disclosure provides a computer-readable storage medium, and the computer program is stored in the storage medium, and when the computer program is executed by a processor, the training method of the robot motion control strategy network based on deep reinforcement learning of the first aspect or the robot motion control method based on deep reinforcement learning of the second aspect is realized.

[0034] Compared with the prior art, the technical solutions provided by the present disclosure have the following beneficial effects: In the embodiments of the present disclosure, by introducing a forward prediction model and sharing the encoder output features with the motion control strategy network, the state data of the future time step is predicted based on the forward prediction model, and the parameters of the encoder, the motion control strategy network and the forward prediction model are updated by the comprehensive loss of the state loss of the forward prediction model and the policy loss of the motion control strategy network, so that the motion control strategy network not only learns the mapping from the current state to the action policy during the training process, but also implicitly learns the dynamic evolution rule from the current state to the future state, improves the modeling ability of the motion control strategy network for continuous action structure, and enables the motion control strategy network to learn and adapt to the changes of the environment during the training process, thereby speeding up the policy convergence speed of the policy network, improving the utilization rate of samples, and improving the training efficiency and generalization ability of the motion control strategy network. At the same time, since the motion control strategy network can predict the change of the future state, the robot can generate good smooth posture transition actions when executing the action sequence, thereby reducing the probability of action mutation or posture switching stiffness and improving the naturalness and continuity of the action during the robot motion.

[0035] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0036] The drawings incorporated into the specification and forming a part of the specification, show embodiments consistent with the present disclosure, and together with the specification, serve to explain the principles of the present disclosure.

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the drawings needed in the embodiment or prior art description will be briefly introduced as follows. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0038] Figure 1 is a system architecture diagram provided by the embodiments of the present disclosure; Figure 2 is a flowchart of a training method of a robot motion control strategy network based on deep reinforcement learning provided by the embodiments of the present disclosure; Figure 3is a flowchart of a robot motion control strategy network training method based on deep reinforcement learning provided by an embodiment of the present disclosure; Figure 4 is a flowchart of a robot motion control strategy network training method based on deep reinforcement learning provided by an embodiment of the present disclosure; Figure 5 is a flowchart of a robot motion control strategy network training method based on deep reinforcement learning provided by an embodiment of the present disclosure; Figure 6 is a flowchart of a robot motion control strategy network training method based on deep reinforcement learning provided by an embodiment of the present disclosure; Figure 7 is a flowchart of a robot motion control strategy network training method based on deep reinforcement learning provided by an embodiment of the present disclosure; Figure 8 is a flowchart of a robot motion control strategy network training method based on deep reinforcement learning provided by an embodiment of the present disclosure; Figure 9 is a flowchart of a robot motion control strategy network training method based on deep reinforcement learning provided by an embodiment of the present disclosure; Figure 10 is a flowchart of a robot motion control strategy network training method based on deep reinforcement learning provided by an embodiment of the present disclosure; Figure 11 is a flowchart of a robot motion control method based on deep reinforcement learning provided by an embodiment of the present disclosure; Figure 12 is a structural schematic diagram of a robot motion control strategy network training device based on deep reinforcement learning provided by an embodiment of the present disclosure; Figure 13 is a structural schematic diagram of a robot motion control device based on deep reinforcement learning provided by an embodiment of the present disclosure; Figure 14 is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure; Figure 15 is a schematic diagram of a robot provided by an embodiment of the present disclosure; Figure 16 is a schematic diagram of a robot provided by an embodiment of the present disclosure; Figure 17 is a schematic diagram of a robot provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0039] In order to more clearly understand the above-mentioned purposes, features and advantages of the present disclosure, the solutions of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0040] In the following description, a large number of specific details are set forth in order to facilitate a thorough understanding of the present disclosure, but the present disclosure can also be implemented in other different manners from those described herein; obviously, the embodiments described in the specification are only a part of the embodiments of the present disclosure, and not all the embodiments.

[0041] It should be understood that each step recorded in the method embodiments of the present disclosure can be executed in different orders and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the execution of the steps shown. The scope of the present disclosure is not limited in this respect.

[0042] It should be noted that, in this document, relational terms such as “first” and “second”, and the like, are used solely to distinguish one entity or action from another entity or action, without necessarily requiring or implying any actual such relationship or order between or among the entities or actions. Moreover, the terms “comprises”, “comprising”, or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitations, an element defined by the phrase “comprising a...” does not exclude the existence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0043] It should be noted that the modification of “one” and “multiple” mentioned in the present disclosure is illustrative rather than limiting, and those skilled in the art should understand that, unless otherwise explicitly indicated in the context, it should be understood as “one or more”.

[0044] In order to better understand the inventive concept of the embodiments of the present disclosure, the technical solutions of the embodiments of the present disclosure will be described below in conjunction with exemplary embodiments.

[0045] First, the related terms involved in the embodiments of the present disclosure are explained.

[0046] The robot can be understood as a programmable mechanism with certain autonomous ability to perform movement, operation or positioning.

[0047] The joint can be understood as a mechanical component connecting two rigid bodies and capable of constraining the relative motion between the two rigid bodies.

[0048] The body perception data of the robot can be understood as observation information directly obtained by the robot through its own sensors. For example, the body perception data of the robot can include, but is not limited to, joint angles of the robot, joint speeds of the robot, walking speeds of the robot, posture information of the robot, and the like.

[0049] The noise disturbance strategy can be understood as a disturbance strategy of adding noise data in the sampled state data.

[0050] The mask disturbance strategy can be understood as a disturbance strategy of adding a mask in the sampled state data.

[0051] The Gaussian noise data can be understood as a type of noise data whose probability density function obeys a Gaussian distribution (i.e., a normal distribution).

[0052] The motion control strategy network can be understood as a neural network based on a deep reinforcement learning algorithm.

[0053] The forward prediction model can be understood as a model for forward prediction.

[0054] The simulation environment can be understood as a virtual running platform for simulating robot dynamics, sensor feedback, and interaction mechanisms with the outside world.

[0055] The value network can be understood as a neural network for predicting the sum of future returns, used to evaluate the state value of the state data of the robot, referred to as state value.

[0056] The gradient value (Gradient) can be understood as a numerical value for representing the update direction of the model parameters.

[0057] The learning rate (Learning Rate, LR) can be understood as a value for representing the update amplitude or update step length of the model parameters.

[0058] Figure 1 A system architecture diagram of the robot motion control strategy network training method based on deep reinforcement learning and the robot motion control method based on deep reinforcement learning in the embodiments of the present disclosure is shown.

[0059] As shown in Figure 1 The system architecture 100 can include a terminal device 101, a robot 102, a network 103, and a server 104. The terminal device 101 includes, but is not limited to, a desktop computer, a portable computer, a smartphone, a tablet computer, and the like. The terminal device 101 is configured with a graphical user interface and can provide a visualization function to display the running state, motion trajectory, and the like of the robot 102, and can also be used to configure training parameters, control the training process, and monitor the model training state, and support sending motion control instructions to the robot 102.

[0060] Robot 102 is equipped with a variety of sensors, such as inertial measurement units, joint position sensors, speed sensors, and angle sensors, to collect its own state data in real time during movement.

[0061] Server 104 is equipped with a training module for implementing a deep reinforcement learning-based robot motion control policy network training method. This module acquires historical state data of the robot and, based on this data, trains an encoder and motion control policy network using deep reinforcement learning. After training, server 104 deploys the trained encoder and motion control policy network to robot 102, enabling robot 102 to output motion control strategies based on these networks.

[0062] Network 103 serves as the medium for providing a communication link between terminal device 101, robot 102, and server 104. Network 103 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc. It should be understood that... Figure 1 The number and type of terminal devices, robots, networks, and servers shown are merely illustrative. Depending on implementation needs, any number and type of terminal devices, robots, networks, and servers can be included.

[0063] Through the coordinated operation of various parts in system architecture 100, the motion control policy network of the encoder and the robot can be trained efficiently. During the training process, the motion control policy network not only learns the mapping from the current state to the action policy, but also implicitly learns the dynamic evolution law from the current state to the future state. This improves the motion control policy network's ability to model continuous action structures, enabling it to learn and adapt to changes in the environment autonomously during training. This accelerates the policy convergence speed of the motion control policy network, improves sample utilization, and enhances the training efficiency and generalization ability of the motion control policy network. At the same time, because the motion control policy network can predict future state changes, the robot can generate good, coherent, and smooth posture transitions when executing action sequences, reducing the probability of sudden action changes or stiff posture switching, and improving the naturalness and continuity of the robot's movements.

[0064] The Deep Reinforcement Learning (DRL) disclosed herein can be understood as a combination of Deep Learning (DL) and Reinforcement Learning (RL), integrating the feature extraction capabilities of Deep Learning and the decision-making capabilities of Reinforcement Learning. It is a learning method that allows an agent to interact with the environment, continuously try and learn in the environment, and adjust its own parameters to obtain the optimal policy.

[0065] The embodiment of the present disclosure provides a robot motion control strategy network training method based on deep reinforcement learning, which comprises the following steps: Figure 2 A flowchart of a robot motion control strategy network training method based on deep reinforcement learning is shown, which can comprise the following steps 210 to 240: In step 210, historical state data of the robot is acquired and input into an encoder to obtain first representation features, wherein the historical state data is body perception data of the robot.

[0066] In step 220, the historical state data is sampled to obtain sampled state data, and the sampled state data is input into the encoder after enhancement processing to obtain second representation features.

[0067] In step 230, the first representation features are input into a motion control strategy network to generate an action strategy, and the second representation features are input into a forward prediction model to generate predicted state data of multiple future time steps.

[0068] In step 240, a first loss is calculated based on the difference between the predicted state data of the multiple future time steps and real state data, a second loss is calculated based on the action strategy output by the motion control strategy network, and the parameters of the encoder, the motion control strategy network and the forward prediction model are updated according to the first loss and the second loss.

[0069] By performing the robot motion control strategy network training method based on deep reinforcement learning provided by the example embodiment of the present disclosure, the encoder output features are shared by introducing a forward prediction model and the motion control strategy network, the state data of the future time steps is predicted based on the forward prediction model, the parameters of the encoder, the motion control strategy network and the forward prediction model are updated by the comprehensive loss of the state loss of the forward prediction model and the policy loss of the motion control strategy network, so that the motion control strategy network not only learns the mapping from the current state to the action strategy during the training process, but also implicitly learns the dynamic evolution rule from the current state to the future state, which improves the modeling ability of the motion control strategy network for continuous action structure, enables the motion control strategy network to learn and adapt to the changes of the environment during the training process, thereby speeding up the policy convergence speed of the policy network, improving the utilization rate of samples, and improving the training efficiency and generalization ability of the motion control strategy network. In addition, since the motion control strategy network can predict the future state change, the robot can generate good smooth posture transition actions when executing the action sequence, which reduces the probability of action mutation or posture switching stiffness and improves the naturalness and continuity of the action during the robot motion.

[0070] In the following, the robot motion control strategy network training method based on deep reinforcement learning in the embodiments of the present disclosure will be described in detail.

[0071] In step 210, the historical state data of the robot is obtained and input into the encoder to obtain the first representation feature. The historical state data is the proprioceptive data of the robot.

[0072] The historical state data can be understood as the proprioceptive data of the robot. The proprioceptive data of the robot can be understood as the observation information directly obtained by the robot through its own sensors. For example, the proprioceptive data of the robot can include but is not limited to the joint angle of the robot, the joint speed of the robot, the walking speed of the robot, the attitude information of the robot, and the like.

[0073] The proprioceptive data of the robot can be collected through sensors arranged on the robot. For example, the proprioceptive data of the robot can be collected through an inertial measurement unit (IMU) sensor or the like. During the movement of the robot, the proprioceptive data is collected by the sensor, and then stored in the local storage space or stored in the cloud server.

[0074] In the embodiments of the present disclosure, the historical state data of the robot can be obtained from the local storage space or the cloud server of the robot, and then the historical state data of the robot is input into the encoder. The historical state data is encoded based on the encoder to obtain the first representation feature corresponding to the historical state data.

[0075] In some example embodiments, the above-mentioned obtaining the historical state data of the robot and inputting into the encoder to obtain the first representation feature can be performed Figure 3 A flowchart of a robot motion control strategy network training method based on deep reinforcement learning is shown, as shown in Figure 3 As shown in the figure, it includes steps 310-330: Step 310, obtaining the proprioceptive data of the robot at multiple historical time steps.

[0076] The time step can be understood as a process in which the robot performs an action and obtains feedback.

[0077] Step 320, sorting the multiple proprioceptive data according to the chronological order to obtain the historical state data of the robot.

[0078] For example, the plurality of historical time steps include three historical time steps t1, t2, t3, and the order of the three time steps t1, t2, t3 is t1, t2, t3. The ontological perception data of t1, the ontological perception data of t2, and the ontological perception data of t3 are sequentially sorted to obtain the historical state data of the robot.

[0079] In step 330, the historical state data is input to the encoder, and the encoder extracts features from the ontological perception data of the plurality of historical time steps in the historical state data to obtain first representation features corresponding to the historical state data.

[0080] The first representation features can be understood as a high-dimensional feature vector.

[0081] Therefore, the plurality of ontological perception data can be sorted according to the order of the time steps, the historical state data of the robot is constructed, and the historical state data is feature-extracted to obtain first representation features corresponding to the historical state data, which can ensure the time sequence of the historical state data and the first representation features and improve the accuracy of the first representation features corresponding to the historical state data.

[0082] In step 220, the historical state data is sampled to obtain sampled state data, and the sampled state data is input to the encoder after being enhanced to obtain second representation features.

[0083] The historical state data is sampled to obtain the sampled state data, that is, the number of state data is increased based on the historical state data, and the increased state data is used as the sampled state data.

[0084] In some example embodiments, the above sampling of the historical state data to obtain the sampled state data can be performed Figure 4 A flowchart of a method for training a robot motion control strategy network based on deep reinforcement learning is shown, as shown in Figure 4 As shown, the method includes steps 410-430: Step 410, reading state data of a plurality of time steps in historical state data.

[0085] Step 420, selecting a plurality of state data as candidate samples in the state data according to a preset time step interval.

[0086] The preset time step interval can be set as needed and is not limited here.

[0087] For example, the plurality of time steps include six time steps t1, t2, t3, t4, t5, t6, t7. If the preset time step interval is 2, the state data of t1 time step, the state data of t4 time step, and the state data of t7 time step are selected as candidate samples.

[0088] Step 430, sort the candidate samples according to the chronological order of the time steps corresponding to the candidate samples, to obtain the sampled state data.

[0089] For example, the candidate samples include state data of the t1 time step, state data of the t4 time step, and state data of the t7 time step, and the chronological order of the time steps is t1, t4, and t7. According to the chronological order of the time steps corresponding to the candidate samples, the sampled state data obtained is the state data of the t1 time step, the state data of the t4 time step, and the state data of the t7 time step.

[0090] Therefore, the state data can be sampled according to the preset time step interval to obtain the candidate samples, and then the candidate samples are sorted according to the chronological order of the time steps to obtain the sampled state data, which can ensure the order of the state data sampling and improve the representativeness of the sampled state data.

[0091] In some example embodiments, the above-mentioned sampled state data is input to the encoder after the enhancement processing to obtain the second representation feature, which can be performed Figure 5 The flowchart of a robot motion control strategy network training method based on deep reinforcement learning is shown, as shown in Figure 5 As shown, it includes steps 510-520: Step 510, enhancing the sampled state data based on a preset perturbation strategy to obtain enhanced state data, the preset perturbation strategy including one or more of a noise perturbation strategy and a mask perturbation strategy.

[0092] The noise perturbation strategy can be understood as a perturbation strategy of adding noise data in the sampled state data.

[0093] The mask perturbation strategy can be understood as a perturbation strategy of adding a mask in the sampled state data.

[0094] In an example embodiment, when the preset perturbation strategy includes the noise perturbation strategy, enhancing the sampled state data based on the preset perturbation strategy to obtain the enhanced state data can include: Adding Gaussian noise data in the sampled state data to obtain the enhanced state data.

[0095] The Gaussian noise data can be understood as a class of noise data with a probability density function subject to Gaussian distribution (i.e., normal distribution).

[0096] For example, the enhanced state data y can be represented as: y = x + N(0, 1); Where y represents the enhanced state data; x represents the sampled state data; N(0, 1) represents standard normal distribution noise data with a mean of 0 and a standard deviation of 1.

[0097] In an example embodiment, when the preset perturbation strategy includes a noise perturbation strategy, the enhancement processing of the sampled state data based on the preset perturbation strategy to obtain the enhanced state data can include: uniformly adding target noise data in the sampled state data to obtain the enhanced state data.

[0098] The target noise data can be set as needed, which is not limited here.

[0099] In an example embodiment, when the preset perturbation strategy includes a mask perturbation strategy, the enhancement processing of the sampled state data based on the preset perturbation strategy to obtain the enhanced state data can include: adding a random mask to the sampled state data to obtain the enhanced state data.

[0100] Wherein, adding a random mask to the sampled state data can be understood as randomly selecting part of the data in the sampled state data for masking (zeroing, replacing, masking).

[0101] The mask can be understood as a string of binary code, which realizes the masking of the input bits by performing a bitwise AND operation on the target field.

[0102] For example, a plurality of target positions of the sampled state data can be randomly selected; the data corresponding to each target position in the sampled state data is replaced with a mask to obtain the enhanced state data.

[0103] Step 520, input the enhanced state data into the encoder, and perform feature extraction on the enhanced state data based on the encoder to obtain the second representation feature.

[0104] Thus, the enhancement processing of the sampled state data based on the preset perturbation strategy to obtain the enhanced state data enables the encoder to still generate stable second representation features in the case of state data perturbation. This processing not only enhances the robustness of the second representation features, but also improves the behavior stability of the control policy network for uncertain inputs and improves the continuous prediction ability of the subsequent forward prediction model.

[0105] In step 230, the first representation feature is input into the motion control policy network to generate an action policy, and the second representation feature is input into the forward prediction model to generate predicted state data of a plurality of future time steps, Wherein, the motion control policy network can be understood as a neural network based on a deep reinforcement learning algorithm. For example, the motion control policy network can be a multi-layer fully connected perceptron, a Transformer structure, etc.

[0106] The forward prediction model can be understood as a forward prediction model.

[0107] In some example implementations, a first representation feature can be input into a motion control policy network. For each historical time step in the first representation feature, the motion control policy network performs action reasoning based on the representation feature and outputs the action policy corresponding to the state data of that historical time step.

[0108] Action strategy can be understood as the probability distribution of actions under state data, that is, the execution probability of each action under state data.

[0109] In some example implementations, the above-described input of the second representation feature into the forward prediction model to generate predicted state data for multiple future time steps can be performed. Figure 5 The flowchart shown is a training method for a robot motion control policy network based on deep reinforcement learning. Figure 5 As shown, steps 610-620 are included: Step 610: Input the first state feature corresponding to the first time step in the second representation feature into the forward prediction model.

[0110] The second representation feature corresponding to the enhanced state data includes robot state data for multiple time steps. The first state feature corresponding to the first time step in the second representation feature can be input into the forward prediction model.

[0111] Step 620: Based on the forward prediction model, perform state prediction for the first state features at multiple future time steps to obtain predicted state data for multiple future time steps.

[0112] The future time step can be understood as the time step after the first time step in the second representation feature.

[0113] In some example implementations, the state prediction of the first state features at multiple future time steps is performed based on the forward prediction model to obtain predicted state data for multiple future time steps, which may include steps 6201-6203: Step 6201: Based on the forward prediction model, predict the state of the first state feature for the next time step to obtain the predicted state data for the second time step.

[0114] For example, at a time step K1, the state prediction for the next time step is performed based on the first state feature corresponding to K1 using the forward prediction model, resulting in the predicted state data Y1 for the second time step K2.

[0115] Step 6202: Input the first predicted state data into the forward prediction model, and based on the forward prediction model, predict the state of the next time step based on the predicted state data of the second time step to obtain the predicted state data corresponding to the third time step.

[0116] For example, the prediction state data Y1 of the second time step K2 is input into the forward prediction model, and the forward prediction model is called repeatedly to perform state prediction based on the prediction state data Y1 to obtain the prediction state data Y2 of the third time step K3.

[0117] Step 6203, input the prediction state data of the third time step into the forward prediction model, and repeatedly call the forward prediction model to perform state prediction to obtain the prediction state data of multiple future time steps.

[0118] For example, if the multiple future time steps of the first time step K1 include K2, K3, K4, K5, and K6, the forward prediction model is called repeatedly to perform state prediction, and the prediction state data Y1 of K2, the prediction state data Y2 of K3, the prediction state data Y3 of K4, the prediction state data Y4 of K5, and the prediction state data Y5 of K6 can be obtained.

[0119] Therefore, the second feature corresponding to the enhanced state data can be input into the forward prediction model, and the prediction state data of multiple future time steps can be generated based on the forward prediction model.

[0120] In step 240, a first loss is calculated based on the difference between the prediction state data of multiple future time steps and the real state data, and a second loss is calculated based on the action strategy output by the motion control strategy network, and the parameters of the encoder, the motion control strategy network, and the forward prediction model are updated according to the first loss and the second loss.

[0121] The first loss can be understood as an index for measuring the difference between the prediction state data of the future time step and the real state data of the future time step.

[0122] The second loss can be understood as an index for measuring the difference between the action strategy output by the motion control strategy network and the actual execution action of the robot.

[0123] In some example embodiments, the first loss can be calculated based on the difference between the prediction state data of multiple future time steps and the real state data, which can be performed by Figure 7 A flowchart of a method for training a robot motion control strategy network based on deep reinforcement learning is shown, as shown in Figure 7 As shown, the method includes steps 710-730: Step 710, for each future time step, the historical state data of the future time step is determined as the real state data of the future time step.

[0124] Step 720, for each future time step, calculate the difference between the predicted state data of the future time step and the real state data of the future time step to obtain the state difference data corresponding to the future time step.

[0125] In an example embodiment, for each future time step, a first mean square error value between the predicted state data of the future time step and the real state data of the future time step can be calculated, and the first mean square error value is taken as the state difference data corresponding to the future time step.

[0126] The first mean square error value can be understood as an index for measuring the difference between the predicted state data of the future time step and the real state data of the future time step.

[0127] For example, the above calculation of the first mean square error value between the predicted state data of the future time step and the real state data of the future time step can include S11-S13: S11, calculate the first difference value data between the predicted state data of the future time step and the real state data of the future time step.

[0128] S12, square the data of each dimension in the first difference value data to obtain the first square value of each dimension, and sum the first square value of each dimension to obtain the first total square sum.

[0129] S13, divide the first total square sum by the number of dimensions of the first difference value data to obtain the first mean square error value.

[0130] Step 730, weighted sum the state difference data corresponding to each future time step to obtain the first loss.

[0131] Thus, the first loss can be determined based on the state difference data corresponding to each future time step, improving the accuracy of the first loss.

[0132] In some example embodiments, the above calculation of the second loss based on the action policy output by the motion control strategy network can be performed Figure 8 A flowchart of a robot motion control strategy network training method based on deep reinforcement learning is shown, as shown in Figure 8 As shown, the method includes steps 810-820: Step 810, for each state data in the historical state data and the action policy corresponding to the state data, control the robot to execute the action policy under the state data in the simulation environment, receive the target action taken by the robot when executing the action policy and the reward value and the target state data of the robot after the robot executes the target action returned by the simulation environment.

[0133] The simulation environment can be understood as a virtual running platform for simulating robot dynamics, sensory feedback, and interaction mechanisms with the outside world.

[0134] Controlling the robot to execute the action policy under the state data in the simulation environment can be understood as a virtual robot corresponding to the robot executing the action policy under the state data in the simulation environment.

[0135] When the robot executes the action policy corresponding to the state data, the robot determines a target action according to the action policy. The reward value can be understood as a numerical value generated by the simulation environment according to a preset action evaluation rule to represent the good or bad of the robot executing the target action after the robot executes the target action in the simulation environment. The reward value can be used to measure the good or bad of the robot executing the target action.

[0136] The higher the reward value, the better the robot executes the target action; the lower the reward value, the worse the robot executes the target action.

[0137] The simulation environment can generate a corresponding reward value according to the stability, attitude change, and energy consumption of the robot executing the target action in the simulation environment.

[0138] The target state data of the robot is the target state data of the robot returned by the simulation environment after the robot executes the target action, that is, the state data of the robot returned by the simulation environment after the robot executes the target action, that is, the proprioceptive data of the robot returned by the simulation environment after the robot executes the target action.

[0139] Step 820, based on the state data of each historical time step in the historical state data, the reward value corresponding to each state data, and the target state data, calculate the second loss.

[0140] In some example embodiments, based on the state data of each historical time step in the historical state data, the reward value corresponding to each state data, and the target state data, the second loss can be calculated, which can include steps 8201-8202: Step 8201, for each state data of each historical time step in the historical state data, based on the state data, the reward value corresponding to the state data, and the target state data, determine the advantage value of the robot executing the target action corresponding to the state data.

[0141] The advantage value can be understood as an index for measuring how much better the effect of the robot executing the target action under the state data is than the effect of executing the average action.

[0142] The advantage value is greater than 0, indicating that the effect of the target action is better than that of the average action; The advantage value is less than 0, indicating that the effect of the target action is worse than that of the average action. When the advantage value is 0, it indicates that the effect of the target action is the same as the effect of the average action.

[0143] In an example embodiment, determining the advantage value of the robot performing the target action based on the state data, the reward value corresponding to the state data, and the target state data can include S21-S23: S21, value evaluation is performed on the state data based on the value network to obtain a current state value, and value evaluation is performed on the target state data corresponding to the state data based on the value network to obtain a target state value.

[0144] The value network can be understood as a neural network for predicting the sum of future benefits, which is used to evaluate the state value corresponding to the state data of the robot. The state value can be understood as the size of the future benefit of the robot corresponding to the state data of the robot.

[0145] The current state value can be understood as the state value of the state data of the robot.

[0146] The target state value can be understood as the state value of the target state data corresponding to the state data of the robot.

[0147] S22, sum the reward value and the target state value to obtain a target sum value.

[0148] The target sum value can be understood as the actual benefit of the robot performing the target action under the state data.

[0149] S23, calculate the difference between the target sum value and the current state value to obtain the advantage value of the robot performing the target action corresponding to the state data.

[0150] Step 8202, based on the advantage value corresponding to the state data and the execution probability of the target action in the action strategy corresponding to the state data, calculate the sub-loss corresponding to the state data.

[0151] The execution probability of the target action can be understood as the possibility of the robot performing the target action under the state data.

[0152] For example, the advantage value corresponding to the state data and the execution probability of the target action in the action strategy corresponding to the state data can be multiplied to obtain a target product, and then the negative value of the target product is determined as the sub-loss corresponding to the state data.

[0153] Step 8203, sum the sub-losses corresponding to the state data of each historical time step in the historical state data to obtain a second loss.

[0154] By minimizing the second loss, the motion control policy network can adjust the action probability distribution in the action policy according to the size of the advantage value, so as to increase the selection probability of the action with a larger advantage value and reduce the selection probability of the action with a smaller advantage value, thereby realizing continuous optimization of the motion control policy network and enabling the motion control policy network to maximize the policy expected return.

[0155] Therefore, the second loss can be calculated based on the deep reinforcement learning algorithm, and the accuracy of the second loss calculation is improved.

[0156] In some example embodiments, the above-mentioned updating of the parameters of the encoder, the motion control policy network, and the forward prediction model according to the first loss and the second loss can be performed by Figure 9 A flowchart of a robot motion control policy network training method based on deep reinforcement learning is shown as follows: Figure 9 As shown, the method includes steps 910-930: Step 910: Obtain a state loss weight corresponding to the first loss and a policy loss weight corresponding to the second loss.

[0157] The state loss weight and the policy loss weight can be set in advance as needed.

[0158] Step 920: Weighted sum the first loss and the second loss based on the state loss weight and the policy loss weight to obtain a joint loss.

[0159] Step 930: Update the parameters of the encoder, the motion control policy network, and the forward prediction model based on the joint loss.

[0160] In an example embodiment, the encoder, the motion control policy network, and the forward prediction model can be trained based on the joint loss respectively until the joint loss reaches a minimum value, and the parameters of the encoder, the motion control policy network, and the forward prediction model are updated.

[0161] In an example embodiment, the above-mentioned training of the encoder, the motion control policy network, and the forward prediction model based on the joint loss respectively until the joint loss reaches a minimum value, and the updating of the parameters of the encoder, the motion control policy network, and the forward prediction model can be performed by Figure 10 A flowchart of a robot motion control policy network training method based on deep reinforcement learning is shown as follows: Figure 10 As shown, the method includes steps 1010-1030: Step 1010: Based on the joint loss, perform backpropagation calculation on the encoder, the motion control policy network, and the forward prediction model respectively to obtain gradient values of the parameters in the encoder, the motion control policy network, and the forward prediction model, with the goal of minimizing the joint loss.

[0162] The gradient value can be understood as a numerical value for representing the update direction of the model parameters.

[0163] The gradient value can be understood as a numerical value for representing the change direction of the model parameters that makes the loss of the model smaller.

[0164] Step 1020, obtaining a first preset learning rate of the encoder, a second preset learning rate of the motion control strategy network, and a third preset learning rate of the forward prediction model.

[0165] The learning rate (LR) can be understood as a numerical value for representing the update amplitude or update step of the model parameters.

[0166] The first preset learning rate, the second preset learning rate, and the third preset learning rate can be set in advance as needed, which is not limited here.

[0167] Step 1030, updating the parameters of the encoder, the motion control strategy network, and the forward prediction model based on the gradient value, the first preset learning rate, the second preset learning rate, and the third preset learning rate.

[0168] For example, the above updating the parameters of the encoder, the motion control strategy network, and the forward prediction model based on the gradient value, the first preset learning rate, the second preset learning rate, and the third preset learning rate can include steps 1031-1033: Step 1031, calculating the product of the gradient value of each parameter in the encoder and the first preset learning rate to obtain the update increment of each parameter in the encoder; calculating the product of the gradient value of each parameter in the motion control strategy network and the second preset learning rate to obtain the update increment of each parameter in the motion control strategy network; calculating the product of the gradient value of each parameter in the forward prediction model and the third preset learning rate to obtain the update increment of each parameter in the forward prediction model.

[0169] Step 1032, for each parameter, summing the parameter and the update increment of the parameter to generate the update value of the parameter.

[0170] Step 1033, replacing each parameter in the encoder, the motion control strategy network, and the forward prediction model with the update value of the parameter to complete the parameter update of the encoder, the motion control strategy network, and the forward prediction model.

[0171] Therefore, the parameters of the encoder, the motion control policy network and the forward prediction model can be updated by the comprehensive loss of the state loss of the forward prediction model and the policy loss of the motion control policy network, so that the motion control policy network not only learns the mapping from the current state to the action policy during the training process, but also implicitly learns the dynamic evolution rule from the current state to the future state, improves the modeling capability of the motion control policy network for the continuous action structure, and enables the motion control policy network to autonomously learn and adapt to the changes of the environment during the training process, thereby accelerating the policy convergence speed of the policy network, improving the utilization rate of samples, and improving the training efficiency and generalization capability of the motion control policy network.

[0172] The embodiment of the present disclosure provides a robot motion control method based on deep reinforcement learning, which comprises the following steps: Figure 11 A flowchart of a robot motion control method based on deep reinforcement learning is shown, which can comprise the following steps 1110 to 1140: In step 1110, current state data of the robot is acquired, and the current state data comprises body perception data of the robot.

[0173] In step 1120, the current state data is input into a pre-trained encoder to generate target feature representation.

[0174] In step 1130, the target feature representation is input into a pre-trained motion control policy network, and an action policy for controlling the motion of the robot is output based on the motion control policy network, wherein the encoder and the motion control policy network can be obtained according to the robot motion control policy network training method based on deep reinforcement learning of any of the above embodiments.

[0175] Therefore, the parameters of the encoder, the motion control policy network and the forward prediction model can be updated by the comprehensive loss of the state loss of the forward prediction model and the policy loss of the motion control policy network, so that the motion control policy network not only learns the mapping from the current state to the action policy during the training process, but also implicitly learns the dynamic evolution rule from the current state to the future state, improves the modeling capability of the motion control policy network for the continuous action structure, and enables the motion control policy network to autonomously learn and adapt to the changes of the environment during the training process, thereby accelerating the policy convergence speed of the policy network, improving the utilization rate of samples, and improving the training efficiency and generalization capability of the motion control policy network.

[0176] The embodiment of the present disclosure provides a robot motion control policy network training device based on deep reinforcement learning, which comprises the following steps: Figure 12 A structural schematic diagram of a robot motion control policy network training device based on deep reinforcement learning is shown. As shown in the figure, Figure 12 As shown, the robot motion control policy network training device 1200 based on deep reinforcement learning comprises: The first obtaining module 1210 is configured to obtain historical state data of the robot and input the historical state data to the encoder to obtain a first representation feature, the historical state data being body perception data of the robot. The enhancement module 1220 is configured to sample the historical state data to obtain sampled state data, and input the sampled state data to the encoder after enhancement processing to obtain a second representation feature. The generation module 1230 is configured to input the first representation feature to the motion control policy network to generate an action policy, and input the second representation feature to the forward prediction model to generate predicted state data of a plurality of future time steps. The update module 1240 is configured to calculate a first loss based on a difference between the predicted state data of the plurality of future time steps and real state data, and calculate a second loss based on the action policy output by the motion control policy network, and update parameters of the encoder, the motion control policy network and the forward prediction model according to the first loss and the second loss.

[0177] The robot motion control policy network training apparatus based on deep reinforcement learning provided by the embodiments of the present disclosure can implement the robot motion control policy network training method based on deep reinforcement learning of any of the above embodiments, and has similar implementation manners and beneficial effects, which will not be described here again.

[0178] The embodiments of the present disclosure provide a robot motion control apparatus based on deep reinforcement learning, which comprises: Figure 13 A structural schematic diagram of a robot motion control apparatus based on deep reinforcement learning is shown. As shown in the figure, Figure 13 The robot motion control apparatus 1300 based on deep reinforcement learning comprises: The second obtaining module 1310 is configured to obtain current state data of the robot, the current state data comprising body perception data of the robot. The feature generation module 1320 is configured to input the current state data to the pre-trained encoder to generate a target representation feature. The policy generation module 1330 is configured to input the target representation feature to the pre-trained motion control policy network, and output an action policy for controlling the motion of the robot based on the motion control policy network; wherein the encoder and the motion control policy network are obtained according to the robot motion control policy network training method based on deep reinforcement learning of any of the above embodiments.

[0179] The robot motion control apparatus based on deep reinforcement learning provided by the embodiments of the present disclosure can implement the robot motion control method based on deep reinforcement learning of any of the above embodiments, and has similar implementation manners and beneficial effects, which will not be described here again.

[0180] The electronic device provided by the embodiments of the present disclosure can be understood as any device with processing and computing capabilities, which can include but is not limited to a mobile terminal such as a smart phone, a notebook computer, a personal digital assistant (PDA), a tablet computer (PAD), a portable multimedia player (PMP), a vehicle terminal (for example, a vehicle navigation terminal), a wearable device, and the like, and a fixed electronic device such as a digital TV, a desktop computer, a smart home device, and the like.

[0181] The electronic device in the embodiments of the present disclosure can be understood as any device with processing and computing capabilities, which can include but is not limited to a mobile terminal such as a smart phone, a notebook computer, a personal digital assistant (PDA), a tablet computer (PAD), a portable multimedia player (PMP), a vehicle terminal (for example, a vehicle navigation terminal), a wearable device, and the like, and a fixed electronic device such as a digital TV, a desktop computer, a smart home device, and the like.

[0182] Figure 14 is a structural schematic diagram of an electronic device provided by the embodiments of the present disclosure, as Figure 14 shown, the electronic device 1400 can include a processor 1410 and a memory 1420, wherein the memory 1420 stores a computer program 1421, and when the computer program 1421 is executed by the processor 1410, the method provided by any of the above embodiments can be implemented, and the execution manner and beneficial effects are similar, which will not be repeated here.

[0183] Of course, in order to simplify, Figure 14 only some of the components related to the present disclosure in the electronic device 1400 are shown, and components such as buses, input / output interfaces, input devices, and output devices are omitted. In addition, according to specific application circumstances, the electronic device 1400 can also include any other appropriate components.

[0184] The embodiments of the present disclosure also provide a robot, which includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the robot motion control method based on deep reinforcement learning can be implemented, and the execution manner and beneficial effects are similar, which will not be repeated here.

[0185] Among them, the robot includes any one of a foot type robot, a four-legged robot, a biped robot, a wheeled robot, a wheel-foot robot, a four-wheel-foot robot, a humanoid robot and a mobile robot. Referring to Figures 15-17 shown, three different robots are shown respectively.

[0186] The embodiments of the present disclosure provide a computer readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the method of any of the above embodiments can be implemented, and the execution manner and beneficial effects are similar, which will not be repeated here.

[0187] The computer readable storage medium can be a combination of one or more computer readable media. The computer readable media can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium can include, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0188] The computer program can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's electronic device, partly on the user's electronic device, as a stand-alone software package, partly on the user's electronic device and partly on a remote electronic device or entirely on the remote electronic device or server.

[0189] The above description is merely illustrative of the exemplary embodiments of this disclosure and the principles thereof. It will be readily understood to those skilled in the art that the disclosed embodiments are not limited to the specific configurations described herein and that the scope of the disclosure is not limited to the specific embodiments described herein but extends to other embodiments that will be apparent to those skilled in the art. For example, the features of the described embodiments can be combined in any combination or order without departing from the scope of the disclosure.

[0190] In addition, while operations are depicted in a particular, chronological sequence in this disclosure, this should not be understood as requiring that such operations be performed in the order shown or that all of the operations be performed, as some of the operations can be performed in different orders or not at all, depending on the circumstances. Similarly, while several specific implementation details are discussed in the above context, these should not be understood as limiting the scope of the disclosure, as other implementations can be employed. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub-combination.

[0191] The foregoing is merely illustrative of the various implementations of the present disclosure and the general principles thereof. Numerous modifications can be made to these illustrations, and equivalents can be substituted therefor, without departing from the scope of the present disclosure. The specific embodiments commensurate with the specific application are intended to be illustrative only and not limiting of the scope of the application as set forth in the following claims.

Claims

1. A method for training a robot motion control policy network based on deep reinforcement learning, characterized in that, The method comprises: obtaining historical state data of a robot and inputting the historical state data into an encoder to obtain first representation features, the historical state data being proprioceptive data of the robot; sampling the historical state data to obtain sampled state data, and inputting the sampled state data into the encoder after enhancement processing to obtain second representation features; inputting the first representation features into a motion control policy network to generate an action policy, and inputting the second representation features into a forward prediction model to generate predicted state data of a plurality of future time steps; calculating a first loss based on a difference between the predicted state data of the plurality of future time steps and real state data, and calculating a second loss based on the action policy output by the motion control policy network, and updating parameters of the encoder, the motion control policy network and the forward prediction model according to the first loss and the second loss. 2.The method of claim 1, wherein, The method comprises: obtaining proprioceptive data of the robot at a plurality of historical time steps; sorting the plurality of proprioceptive data in a time step order to obtain the historical state data of the robot; inputting the historical state data into the encoder to extract features of the proprioceptive data of the plurality of historical time steps in the historical state data based on the encoder, to obtain first representation features corresponding to the historical state data. 3.The method of claim 1, wherein, The method comprises: reading state data of a plurality of time steps in the historical state data; selecting a plurality of state data as candidate samples in the state data at a preset time step interval; sorting the candidate samples in a time step order corresponding to the candidate samples to obtain sampled state data.

4. The method of claim 1, wherein the method further comprises: The method comprises: enhancing the sampled state data based on a preset perturbation strategy to obtain enhanced state data, the preset perturbation strategy comprising one or more of a noise perturbation strategy and a mask perturbation strategy; inputting the enhanced state data into the encoder to extract features of the enhanced state data based on the encoder, to obtain second representation features.

5. The method of claim 4, wherein the method further comprises: When the preset perturbation strategy comprises a noise perturbation strategy, the method comprises: adding Gaussian noise data to the sampled state data to obtain enhanced state data. 6.The method of claim 4, wherein, When the preset perturbation strategy comprises a noise perturbation strategy, the method comprises: uniformly adding target noise data to the sampled state data to obtain enhanced state data.

7. The method of claim 4, wherein the method further comprises: When the preset perturbation strategy comprises a mask perturbation strategy, the method comprises: Adding a random mask to the sampled state data to obtain enhanced state data.

8. The method of claim 7, wherein the method further comprises: The adding a random mask to the sampled state data to obtain enhanced state data comprises: Randomly selecting a plurality of target positions of the sampled state data; Replacing data corresponding to each of the target positions in the sampled state data with a mask to obtain enhanced state data. 9.The method of claim 1, wherein, The inputting the first representation feature into the motion control policy network to generate an action policy comprises: The inputting the first representation feature into the motion control policy network comprises: for each representation feature of a historical time step in the first representation feature, performing action inference on the representation feature based on the motion control policy network to output an action policy corresponding to state data of the historical time step. 10.The method of claim 1, wherein, The inputting the second representation feature into the forward prediction model to generate predicted state data of a plurality of future time steps comprises: Inputting a first state feature corresponding to a first time step in the second representation feature into the forward prediction model; Performing state prediction on the first state feature for a plurality of future time steps based on the forward prediction model to obtain predicted state data of the plurality of future time steps.

11. The method of claim 10, wherein the method further comprises: The performing state prediction on the first state feature for a plurality of future time steps based on the forward prediction model to obtain predicted state data of the plurality of future time steps comprises: Performing state prediction on the first state feature for a next time step based on the forward prediction model to obtain predicted state data of a second time step; Inputting the predicted state data of the second time step into the forward prediction model, performing state prediction on the predicted state data of the second time step for a next time step based on the forward prediction model to obtain predicted state data of a third time step; Inputting the predicted state data of the third time step into the forward prediction model, repeatedly calling the forward prediction model to perform state prediction, and comprehensively obtaining predicted state data of the plurality of future time steps.

12. The method of claim 1, wherein, The calculating a first loss based on differences between the predicted state data of the plurality of future time steps and real state data comprises: For each future time step, determining historical state data of the future time step as real state data of the future time step; For each future time step, calculating a difference between predicted state data of the future time step and real state data of the future time step to obtain state difference data corresponding to the future time step; Summing the state difference data corresponding to each future time step to obtain a first loss.

13. The method of claim 12, wherein the method further comprises: The calculating a difference between predicted state data of the future time step and real state data of the future time step to obtain state difference data corresponding to the future time step comprises: For each future time step, calculating a first mean square difference value between predicted state data of the future time step and real state data of the future time step, and taking the first mean square difference value as state difference data corresponding to the future time step.

14. The method of claim 13, wherein the method further comprises: The first mean square error value between the predicted state data of the future time step and the real state data of the future time step is calculated, comprising: calculating the first difference value data between the predicted state data of the future time step and the real state data of the future time step; squaring the data of each dimension in the first difference value data to obtain the first square value of each dimension, and summing the first square value of each dimension to obtain the first total square sum; divide the first total square sum by the number of dimensions of the first difference value data to obtain the first mean square error value.

15. The method of claim 1, wherein, The second loss is calculated based on the action strategy output by the motion control strategy network, comprising: for the state data of each historical time step in the historical state data and the action strategy corresponding to the state data, control the robot to execute the action strategy under the state data in the simulation environment, receive the target action taken by the robot when executing the action strategy and the reward value and the target state data of the robot after executing the target action returned by the simulation environment; based on the reward value and the target state data corresponding to the historical state data, calculate the second loss.

16. The method of claim 15, wherein the method further comprises: The second loss is calculated based on the reward value and the target state data corresponding to the historical state data, comprising: for the state data of each historical time step in the historical state data, based on the state data, the reward value and the target state data corresponding to the state data, determine the advantage value of the robot executing the target action corresponding to the state data; based on the advantage value corresponding to the state data and the execution probability of the target action in the action strategy corresponding to the state data, calculate the sub-loss corresponding to the state data; sum the sub-losses corresponding to the state data of each historical time step in the historical state data to obtain the second loss.

17. The method of claim 16, wherein the method further comprises: The advantage value of the robot executing the target action corresponding to the state data is determined based on the state data, the reward value and the target state data corresponding to the state data, comprising: value evaluation of the state data based on the value network to obtain the current state value, value evaluation of the target state data based on the value network to obtain the target state value; sum the reward value and the target state value to obtain the target sum value; calculate the difference value between the target sum value and the current state value to obtain the advantage value of the robot executing the target action corresponding to the state data.

18. The method of claim 1, wherein, According to the first loss and the second loss, the parameters of the encoder, the motion control strategy network and the forward prediction model are updated, comprising: obtain the state loss weight corresponding to the first loss and the strategy loss weight corresponding to the second loss; based on the state loss weight and the strategy loss weight, weighted sum the first loss and the second loss to obtain the joint loss; based on the joint loss, update the parameters of the encoder, the motion control strategy network and the forward prediction model.

19. The method of claim 18, wherein the method further comprises: The parameters of the encoder, the motion control strategy network and the forward prediction model are updated based on the joint loss, including: The parameters of the encoder, the motion control strategy network and the forward prediction model are trained based on the joint loss respectively until the joint loss reaches a minimum value, and the parameters of the encoder, the motion control strategy network and the forward prediction model are updated.

20. The method of claim 19, wherein the method further comprises: The parameters of the encoder, the motion control strategy network and the forward prediction model are trained based on the joint loss respectively until the joint loss reaches a minimum value, and the parameters of the encoder, the motion control strategy network and the forward prediction model are updated, including: The parameters of the encoder, the motion control strategy network and the forward prediction model are trained based on the joint loss respectively until the joint loss reaches a minimum value, and the parameters of the encoder, the motion control strategy network and the forward prediction model are updated. The gradient values of each parameter in the encoder, the motion control strategy network and the forward prediction model are calculated based on the joint loss, and the first preset learning rate, the second preset learning rate and the third preset learning rate of the encoder, the motion control strategy network and the forward prediction model are obtained. The parameters of the encoder, the motion control strategy network and the forward prediction model are updated based on the gradient values, the first preset learning rate, the second preset learning rate and the third preset learning rate.

21. The method of claim 20, wherein the method further comprises: The parameters of the encoder, the motion control strategy network and the forward prediction model are updated based on the gradient values, the first preset learning rate, the second preset learning rate and the third preset learning rate, including: The product of the gradient value of each parameter in the encoder and the first preset learning rate is calculated to obtain the update increment of each parameter in the encoder; the product of the gradient value of each parameter in the motion control strategy network and the second preset learning rate is calculated to obtain the update increment of each parameter in the motion control strategy network; the product of the gradient value of each parameter in the forward prediction model and the third preset learning rate is calculated to obtain the update increment of each parameter in the forward prediction model. For each parameter, the parameter and the update increment of the parameter are summed to generate the update value of the parameter. The parameters in the encoder, the motion control strategy network and the forward prediction model are replaced by the update value of the parameter to complete the parameter update of the encoder, the motion control strategy network and the forward prediction model.

22. A method for robot motion control based on deep reinforcement learning, the method comprising: Including: Obtain the current state data of the robot, the current state data including the body perception data of the robot; Input the current state data into the pre-trained encoder to generate target feature representation; Input the target feature representation into the pre-trained motion control strategy network, and output the action strategy for controlling the motion of the robot based on the motion control strategy network. The encoder and the motion control strategy network are obtained according to the robot motion control strategy network training method based on deep reinforcement learning in any one of claims 1-21.

23. A deep reinforcement learning based robot motion control policy network training apparatus, characterized by, The method comprises: a first obtaining module configured to obtain historical state data of a robot and input the historical state data to an encoder to obtain a first representation feature, the historical state data being proprioceptive data of the robot; an enhancement module configured to sample the historical state data to obtain sampled state data, and input the sampled state data to the encoder after enhancement processing to obtain a second representation feature; a generation module configured to input the first representation feature to a motion control policy network to generate an action policy, and input the second representation feature to a forward prediction model to generate predicted state data of a plurality of future time steps; an updating module configured to calculate a first loss based on a difference between the predicted state data of the plurality of future time steps and real state data, and calculate a second loss based on the action policy output by the motion control policy network, and update parameters of the encoder, the motion control policy network and the forward prediction model according to the first loss and the second loss.

24. A robot motion control apparatus based on deep reinforcement learning, characterized by, The method comprises: a second obtaining module configured to obtain current state data of a robot, the current state data including proprioceptive data of the robot; a feature generation module configured to input the current state data to a pre-trained encoder to generate a target representation feature; a policy generation module configured to input the target representation feature to a pre-trained motion control policy network, and output an action policy for controlling motion of the robot based on the motion control policy network. The encoder and the motion control policy network are obtained according to the method for training a robot motion control policy network based on deep reinforcement learning according to any one of claims 1-21.

25. An electronic device, comprising: The method comprises: a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the method according to any one of claims 1-22 is implemented.

26. A robot characterized by The method comprises: a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the method according to claim 22 is implemented.

27. The robot of claim 26, wherein, The robot comprises any one of a foot-type robot, a quadruped robot, a biped robot, a wheeled robot, a wheel-foot robot, a four-wheel-foot robot, a humanoid robot and a mobile robot.

28. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and when the computer program is executed by the processor, the method according to any one of claims 1-22 is implemented.

Images