Robot motion control model training method and device based on deep reinforcement learning

Abstract

This disclosure provides a method and apparatus for training a robot motion control model based on deep reinforcement learning, relating to the fields of sensors and robotics. The method includes utilizing a teacher-student model framework to perform deep reinforcement learning-based training on a policy network that outputs action strategies for controlling robot motion. The policy network training process also includes encoding the robot's historical linear velocity information via a linear velocity encoder to generate a first latent vector, which is then input into the policy network for auxiliary training. This disclosure comprehensively considers the robot's motion trends, state change trajectories, and surrounding terrain structure when making action decisions, achieving more robust and precise control strategy output.

Description

Technical Field

[0001] The present disclosure relates to the field of sensors and robotics technology, and in particular to a method and device for training a robot motion control model based on deep reinforcement learning. Background Art

[0002] In the field of robotic motion control, traditional control methods are typically based on model predictive control (MPC), proportional-integral-derivative control (PID), or classic optimization algorithms. However, these methods often struggle to achieve stable and precise control in complex dynamic environments, scenarios with drastic speed changes, or high-dimensional control tasks. In particular, traditional methods lack adaptability in the presence of unknown disturbances or partial observation information, making it difficult to quickly adjust strategies, which in turn affects the robot's motion performance in complex environments.

[0003] In recent years, control methods based on deep reinforcement learning (DRL) have become an important research direction in the field of robotic control. Deep reinforcement learning uses neural networks to extract features from high-dimensional input information and learns optimal control policies through policy optimization algorithms (such as PPO and DDPG).

[0004] For example, the paper "CTS Concurrent Teacher-Student Reinforcement Learning for Legged Locomotion" (arXiv:2405.10830v2 [cs.RO], 20240901) employs a teacher-student model to improve training efficiency and policy stability. Similar teacher-student models are also disclosed in Chinese patent applications with publication numbers CN119458315A, CN116931475A, and KR1020250009137A. In this model architecture, the teacher policy is trained in an environment with complete information, while the student policy is learned in an environment with limited information and gradually approximates the teacher policy through knowledge distillation.

[0005] However, because the student model typically has access to limited observational information during deployment, and the privileged information relied upon by the teacher model is unavailable during deployment, the student model's learning effectiveness is limited, making it difficult for it to reach the level of the teacher policy. For example, the inventors discovered that the control policy output by the policy network is prone to lag or instability in robot control under conditions of large speed variations or high-speed motion.

[0006] Therefore, although the above method improves the training efficiency and stability of the policy network, the accuracy and stability of the policy network in speed changes or high dynamic scenarios still need to be further optimized. Summary of the Invention

[0007] The present disclosure provides a robot motion control model training method and device based on deep reinforcement learning, which are used to improve the control accuracy and stability of the robot in speed-varying or highly dynamic scenarios.

[0008] Additional aspects and advantages of the disclosure will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the disclosure.

[0009] According to a first aspect of the present disclosure, a robot motion control model training method based on deep reinforcement learning is provided, comprising:

[0010] Using a teacher-student model framework, a policy network that outputs action policies for controlling robot motion is trained using deep reinforcement learning. The policy network training process includes:

[0011] Encoding the robot's historical linear velocity information through a linear velocity encoder to generate a first latent vector, and inputting the first latent vector into a policy network;

[0012] Based on the value estimation of the current state output by the value network in the teacher-student model framework and the action strategy output by the policy network combined with the first latent vector, the parameters of the policy network are updated using a deep reinforcement learning algorithm.

[0013] In an exemplary embodiment of the present disclosure, a teacher-student model framework includes a student encoder and a teacher encoder;

[0014] Using the teacher-student model framework, the policy network used to output the action policy for controlling the robot's motion is trained based on deep reinforcement learning, including:

[0015] The robot's own historical motion state information is encoded by the student encoder and a second latent vector is generated, and the robot's privileged state information is encoded by the teacher encoder and a third latent vector is generated;

[0016] Selecting a second latent vector or a third latent vector according to a preset strategy and inputting the vector into the strategy network;

[0017] Outputting, through the policy network, an action strategy for controlling the movement of the robot based on the received first latent vector, the second latent vector or the third latent vector selected by the preset policy, and the current motion state information;

[0018] Utilize privileged state information through the value network to output a value estimate of the current state;

[0019] Based on the action strategy and value estimation, the parameters of the policy network are updated using the deep reinforcement learning algorithm.

[0020] In an exemplary embodiment of the present disclosure, the value network utilizes privileged state information to output a value estimate of the current state, including:

[0021] The value network utilizes privileged state information and combines it with the first latent vector, the second latent vector selected by the preset strategy, or the third latent vector to output a value estimate of the current state.

[0022] In an exemplary embodiment of the present disclosure, a linear velocity encoder comprises:

[0023] The input layer is used to receive the robot's linear velocity information at multiple historical moments and preprocess the linear velocity information;

[0024] The hidden layer is used to extract time series features from the pre-processed linear velocity information using a neural network model;

[0025] The output layer is used to generate a first latent vector based on the temporal features extracted by the hidden layer.

[0026] In an exemplary embodiment of the present disclosure, the neural network model in the hidden layer includes a long short-term memory neural network or a gated recurrent unit neural network.

[0027] In an exemplary embodiment of the present disclosure, a policy network outputs an action policy for controlling the motion of a robot based on a received first latent vector, a second latent vector or a third latent vector selected by a preset policy, and current motion state information, including:

[0028] Strategic Network Based on:

[0029]

[0030] Output action strategies to control robot motion;

[0031] in, Indicates the current state Take action The probability distribution of MLP represents a multi-layer perceptron. represents the second latent vector or the third latent vector selected according to the preset strategy, represents the first latent vector.

[0032] In an exemplary embodiment of the present disclosure, the training process of the policy network further includes:

[0033] The robot's historical elevation map information is encoded through a terrain map encoder to generate a fourth latent vector, which is then input into the policy network for auxiliary training.

[0034] In an exemplary embodiment of the present disclosure, a teacher-student model framework includes a student encoder and a teacher encoder;

[0035] Using the teacher-student model framework, the policy network used to output the action policy for controlling the robot's motion is trained based on deep reinforcement learning, including:

[0036] The robot's own historical motion state information is encoded by the student encoder and a second latent vector is generated, and the robot's privileged state information is encoded by the teacher encoder and a third latent vector is generated;

[0037] Selecting a second latent vector or a third latent vector according to a preset strategy and inputting the vector into the strategy network;

[0038] Outputting an action strategy for controlling the movement of the robot by using a strategy network based on the received first latent vector, the fourth latent vector, the preset strategy, selecting the second latent vector or the third latent vector, and the current motion state information;

[0039] Utilize privileged state information through the value network to output a value estimate of the current state;

[0040] Based on the action strategy and value estimation, the parameters of the policy network are updated using the deep reinforcement learning algorithm.

[0041] In an exemplary embodiment of the present disclosure, the value network utilizes privileged state information to output a value estimate of the current state, including:

[0042] The value network utilizes the privileged state information and combines the first latent vector, the fourth latent vector, the second latent vector selected by the preset strategy, or the third latent vector to output a value estimate of the current state.

[0043] In an exemplary embodiment of the present disclosure, a policy network is used to select a second latent vector or a third latent vector based on a received first latent vector, a fourth latent vector, a preset policy, and current motion state information, and output an action policy for controlling the motion of the robot, including:

[0044] Strategic Network Based on:

[0045]

[0046] Output action strategies to control robot motion;

[0047] in, Indicates the current state Take action The probability distribution of MLP represents a multi-layer perceptron. represents the second latent vector or the third latent vector selected according to the preset strategy, represents the first latent vector, represents the fourth latent vector.

[0048] In an exemplary embodiment of the present disclosure, selecting the second latent vector or the third latent vector according to a preset strategy includes:

[0049] Assigning a first probability of selection to the second latent vector and a second probability of selection to the third latent vector;

[0050] Randomly select from the second latent vector and the third latent vector according to the first selection probability and the second selection probability.

[0051] In an exemplary embodiment of the present disclosure, the method further includes:

[0052] According to the increase of training time, the ratio of the first selection probability to the second selection probability is dynamically increased; wherein, in the initial stage of training, the ratio of the first selection probability to the second selection probability is less than 1.

[0053] In an exemplary embodiment of the present disclosure, a teacher-student model framework is used to train a policy network for outputting an action policy for controlling robot motion based on deep reinforcement learning, further comprising:

[0054] Based on the difference between the second latent vector and the third latent vector, the parameters of the student encoder are optimized.

[0055] In an exemplary embodiment of the present disclosure, optimizing parameters of a student encoder based on a difference between the second latent vector and the third latent vector includes:

[0056] Calculate the difference between the second latent vector and the third latent vector through a supervised learning module;

[0057] Construct a loss function based on the difference;

[0058] The parameters of the student encoder are updated through back propagation according to the loss function, so that the second latent vector output by the student encoder gradually approaches the third latent vector output by the teacher encoder.

[0059] In an exemplary embodiment of the present disclosure, based on the action strategy and value estimation, the parameters of the policy network are updated using a deep reinforcement learning algorithm, including:

[0060] Obtain the reward signal generated by the interaction with the environment and the current state value output by the value network, and construct the advantage function based on the reward signal and the current value estimate ;

[0061] The ratio of the current action strategy output by the policy network to the old strategy And the advantage function , construct the objective function of proximal strategy optimization :

[0062]

[0063] in, It is a preset hyperparameter used to limit the amplitude of strategy update; Indicates that Limited to the range Inside, Indicates expected value;

[0064] The parameters of the policy network are gradient updated through the objective function of proximal policy optimization.

[0065] In an exemplary embodiment of the present disclosure, based on the action strategy and the value estimation, the parameters of the policy network are updated using a deep reinforcement learning algorithm, further comprising:

[0066] Set the entropy coefficient for the objective function optimized by the proximal strategy;

[0067] After each preset time interval, the entropy coefficient is decayed.

[0068] In an exemplary embodiment of the present disclosure, setting an entropy coefficient for the objective function of proximal strategy optimization includes:

[0069] according to:

[0070]

[0071] Calculate the total loss;

[0072] in, is the total loss, is the entropy coefficient that decays over time, Indicates the current state Next, the current strategy The entropy of the action distribution.

[0073] According to a second aspect of the present disclosure, a robot motion control method based on deep reinforcement learning is provided, comprising:

[0074] Encoding the robot's historical linear velocity information through a linear velocity encoder to generate a first latent vector, and inputting the first latent vector into a policy network;

[0075] The robot's own historical motion state information is encoded through the student encoder to generate a second latent vector, which is then input into the policy network.

[0076] The strategy network outputs the action strategy for controlling the robot's motion based on the received potential vectors and the current motion state information;

[0077] Among them, the student encoder, linear velocity encoder and strategy network are obtained according to the robot motion control model training method based on deep reinforcement learning in the above embodiment.

[0078] In an exemplary embodiment of the present disclosure, the method further includes:

[0079] encoding the robot's historical elevation map information through a terrain map encoder to generate a fourth latent vector, and inputting the fourth latent vector into a policy network;

[0080] Among them, the student encoder, linear velocity encoder, topographic map encoder and strategy network are obtained according to the robot motion control model training method based on deep reinforcement learning in the above embodiment.

[0081] According to a third aspect of the present disclosure, a robot motion control model training device based on deep reinforcement learning is provided, comprising:

[0082] The policy network training module is used to train the policy network used to output the action policy for controlling the robot's motion using the teacher-student model framework. The policy network training process includes:

[0083] The robot's historical linear velocity information is encoded through a linear velocity encoder to generate a first latent vector, which is then input into the policy network for auxiliary training.

[0084] According to a fourth aspect of the present disclosure, a robot motion control device based on deep reinforcement learning is provided, comprising:

[0085] a linear velocity feature encoding module, configured to encode historical linear velocity information of the robot to generate a first latent vector, and input the first latent vector into a policy network;

[0086] The student encoder encoding module is used to encode the robot's own historical motion state information to generate a second latent vector and input it into the policy network;

[0087] The policy network inference module is used to output the action strategy for controlling the robot's motion based on the received potential vectors and the current motion state information;

[0088] Among them, the student encoder, linear velocity encoder and strategy network are obtained according to the robot motion control model training method based on deep reinforcement learning in the above embodiment.

[0089] According to a fifth aspect of the present disclosure, there is provided an electronic device, including:

[0090] processor; and

[0091] The memory stores computer-readable instructions, which implement the method in the above embodiment when executed by the processor.

[0092] According to a sixth aspect of the present disclosure, there is provided a robot, comprising:

[0093] processor; and

[0094] The memory stores computer-readable instructions, which implement the method in the above embodiment when executed by the processor.

[0095] In an exemplary embodiment of the present disclosure, the robot includes any one of a legged robot, a quadruped robot, a bipedal robot, a wheeled robot, a wheel-legged robot, a quadrupedal robot, a humanoid robot, a cleaning robot, a transport robot, a mobile robot and a robotic arm.

[0096] According to a seventh aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program code instructions are stored. When the computer program code instructions are called by a processor of a robot, the robot executes the method as described in the above embodiment.

[0097] It can be seen from the above technical solutions that the present disclosure has at least one of the following advantages and positive effects:

[0098] The robot motion control model training method based on deep reinforcement learning provided in the exemplary embodiment of the present disclosure encodes the robot's own historical motion state information through a student encoder to generate a first latent vector, and simultaneously encodes the robot's privileged state information through a teacher encoder to generate a second latent vector. The first latent vector or the second latent vector is selected according to a preset strategy and input into the policy network. In addition, the robot's historical linear velocity information is encoded through a linear velocity encoder to generate a third latent vector, which is input into the policy network, enabling the policy network to optimize control decisions based on the historical velocity information. By introducing a linear velocity encoder to encode the robot's historical linear velocity information and through the synergistic effect of the student encoder, teacher encoder, and linear velocity encoder, as well as adding motion trend information in the time dimension to the policy network, the control strategy can more accurately predict the robot's future motion state and make control decisions that are more consistent with the laws of motion. Therefore, compared to the traditional reinforcement learning method that relies solely on the current state for decision-making, the present disclosure can improve the policy network's sensitivity to speed changes by supplementing the state representation with linear velocity information, thereby reducing control lag in high-speed motion or environments with drastic speed changes, improving the response speed and stability of the control strategy, and improving the robot's adaptability in high-speed or high-dynamic environments. Furthermore, since linear velocity information directly reflects the changing trends in the robot's motion state, the present disclosure provides additional information support during the training phase through the linear velocity encoder, enabling the policy network to more quickly learn control strategies that adapt to speed changes, thereby reducing randomness during the exploration process and improving training efficiency. Compared to traditional methods that require large amounts of data for training to adapt to different speed scenarios, the present disclosure enables the control strategy to adapt to different speed states more quickly, improves the learning convergence speed, and reduces the demand for training data. BRIEF DESCRIPTION OF THE DRAWINGS

[0099] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0100] Figure 1 A system architecture diagram is shown to which the robot motion control model training method based on deep reinforcement learning and the robot motion control method based on deep reinforcement learning in the embodiments of the present disclosure can be applied.

[0101] Figure 2 A flow chart of a robot motion control model training method based on deep reinforcement learning in an embodiment of the present disclosure is shown.

[0102] Figure 3 A schematic diagram of a two-stage training-inference framework of a policy network in an embodiment of the present disclosure is shown.

[0103] Figure 4 A schematic diagram of a process of pre-training a strategy network in an embodiment of the present disclosure is shown.

[0104] Figure 5 A schematic diagram of a process for updating student encoder parameters in an embodiment of the present disclosure is shown.

[0105] Figure 6 A schematic diagram of a two-stage training-inference framework of another policy network in an embodiment of the present disclosure is shown.

[0106] Figure 7 A schematic diagram of a two-stage training-inference framework of another policy network in an embodiment of the present disclosure is shown.

[0107] Figure 8 A flow chart of another pre-training strategy network in an embodiment of the present disclosure is shown.

[0108] Figure 9 A schematic diagram of a two-stage training-inference framework of another policy network in an embodiment of the present disclosure is shown.

[0109] Figure 10 A flow chart of a robot motion control method based on deep reinforcement learning in an embodiment of the present disclosure is shown.

[0110] Figure 11 A block diagram of a robot motion control model training device based on deep reinforcement learning in an embodiment of the present disclosure is shown.

[0111] Figure 12 A block diagram of a robot motion control device based on deep reinforcement learning in an embodiment of the present disclosure is shown.

[0112] Figure 13 A schematic diagram of a robot in an embodiment of the present disclosure is shown.

[0113] Figure 14 A schematic diagram of another robot in an embodiment of the present disclosure is shown.

[0114] Figure 15 A schematic diagram of another robot in an embodiment of the present disclosure is shown.

[0115] Figure 16 A schematic structural diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0116] Figure 17 A schematic structural diagram of a program product in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0117] In this disclosure, the terms "first" and "second" are used for descriptive purposes only and do not indicate relative importance or imply the number of technical features. Therefore, "first" and "second" may explicitly or implicitly include at least one of these features. "Multiple" means at least two, unless expressly limited otherwise.

[0118] Figure 1 A system architecture diagram is shown to which the robot motion control model training method based on deep reinforcement learning in the embodiments of the present disclosure can be applied.

[0119] like Figure 1 As shown, system architecture 100 may include a terminal device 101, a robot 102, a network 103, and a server 104. Terminal device 101 includes, but is not limited to, desktop computers, portable computers, smartphones, and tablet computers. Terminal device 101 can serve as an interactive interface, providing visualization capabilities to display the operating status and motion trajectory of robot 102, and supporting the sending of motion control commands to robot 102.

[0120] The robot 102 is equipped with a variety of sensors to collect its own motion state information under different tasks, and can execute action strategies under the control of the server 104 to complete the corresponding training actions or tasks. The server 104 is deployed with a training framework based on deep reinforcement learning, including modules such as a teacher-student model structure, a policy network, a value network, and a linear velocity encoder. During the training process, the server 104 receives historical linear velocity information from the robot 102, and generates a first latent vector through the linear velocity encoder to input into the policy network. The policy network parameters are optimized using a deep reinforcement learning algorithm by combining the policy output with the current state value estimated by the value network. The server 104 can also undertake supervised learning tasks in the teacher-student model, aligning the latent vectors of the teacher encoder and the student encoder, and guiding the student encoder to learn effective state representations.

[0121] After the training is completed, the server 104 can deploy the trained strategy network to the robot 102 to enable its autonomous reasoning and execution of action strategies in actual operation. The terminal device 101 can also be used to call the deployment strategy, monitor the robot's motion status or remote scheduling control.

[0122] The network 103 is used to provide a medium for communication links between the terminal device 101, the robot 102 and the server 104. The network 103 may include various connection types, such as wired, wireless communication links or fiber optic cables, etc. It should be understood that Figure 1The number and types of terminal devices, robots, networks, and servers in the embodiment are merely illustrative. Depending on the implementation requirements, any number and type of terminal devices, robots, networks, and servers may be used.

[0123] The present disclosure provides a method for training a robot motion control model based on deep reinforcement learning. The method utilizes a teacher-student model framework to train a policy network that outputs action policies for controlling robot motion based on deep reinforcement learning.

[0124] Among them, the teacher-student model framework includes a student encoder and a teacher encoder. Through the two collaborative structures of the teacher encoder and the student encoder, the policy network is guided to obtain higher quality state representation and action output in the process of deep reinforcement learning. At the same time, the model framework provides a structured knowledge transfer path, so that the policy network can not only learn the optimal strategy from the environment, but also accelerate convergence and improve sample efficiency with the guidance of the teacher encoder. In the exemplary implementation of this disclosure, deep reinforcement learning includes but is not limited to algorithms such as PPO (Proximal Policy Optimization), DDPG (Deep Deterministic Policy Gradient), and SAC (Soft Actor-Critic). This disclosure does not limit the specific algorithm type of deep reinforcement learning.

[0125] In some example implementations, reference Figure 2 As shown, the training process of the policy network includes steps S201 and S202:

[0126] In step S201 , the robot's historical linear velocity information is encoded using a linear velocity encoder to generate a first latent vector, and the first latent vector is input into a policy network.

[0127] Among them, linear velocity is the distance that a point moves along a straight line or curved trajectory per unit time. In this example implementation, the linear velocity information is the displacement speed of the robot in three-dimensional space that changes with time. For example, the linear velocity information of the robot is represented by the spatial velocity of the robot's center of mass or torso center, which is used to characterize the overall motion trend and dynamic state of the robot. The linear velocity encoder is used to extract motion trend features with dynamic continuity and time dependence from the linear velocity information of the robot at multiple historical moments, and encode it into a potential representation suitable for policy network input. For example, the linear velocity encoder can be a one-dimensional convolutional neural network, a recurrent neural network, a multi-layer perceptron, etc. The present disclosure does not limit the specific type of the linear velocity encoder.

[0128] Exemplarily, the linear velocity encoder may include an input layer, a hidden layer, and an output layer. First, the input layer receives the linear velocity information of the robot at multiple historical moments and preprocesses the linear velocity information, such as normalization, dimension alignment, and other operations, to ensure the stability and effectiveness of subsequent network calculations. Next, the hidden layer uses a neural network model to extract time series features from the preprocessed linear velocity information, which can effectively capture the dynamic patterns and acceleration and deceleration trends during the speed change process. For example, the neural network model can be a long short-term memory neural network or a gated recurrent unit neural network, which can retain important information from past moments and model time correlation. Finally, the output layer maps the time series features extracted by the hidden layer into a latent vector of a fixed dimension, namely the first latent vector.

[0129] The first latent vector is fed into the policy network, which can provide auxiliary information in the time dimension for the policy network and enhance the policy's ability to perceive speed changes. This allows the policy network to not only rely on the current state but also perceive past motion trajectories and speed changes, helping to improve the stability and responsiveness of the control strategy in high-speed motion or dynamically changing scenarios.

[0130] Step S202 , based on the value estimate of the current state output by the value network in the teacher-student model framework and the action strategy output by the policy network combined with the first latent vector, the parameters of the policy network are updated using a deep reinforcement learning algorithm.

[0131] The value network is used to estimate the long-term reward of the current state, that is, starting from this state, if the current strategy is continuously executed, how much cumulative reward can be obtained in the future. Specifically, after receiving the first latent vector, the policy network outputs the current action strategy. At the same time, combined with the value network in the teacher-student model framework, it estimates the value of the current state. The loss function is constructed through the deep reinforcement learning algorithm, and the policy output and value estimate are used to jointly optimize the policy network. For example, through the policy gradient and backpropagation mechanism, the parameters of the policy network are updated, thereby continuously improving the control performance of the strategy in the current environment. Taking the PPO algorithm as an example, a "clip ratio" strategy is adopted to limit the amplitude of each update to improve training stability and sample utilization.

[0132] In this step, the teacher encoder provides stable feature supervision to the student encoder, guiding it to learn higher-quality state representations, enabling the policy network to achieve better convergence speed and generalization during training. Importantly, the introduction of the linear velocity encoder enables the policy network to model motion trends in the temporal dimension, thereby improving the accuracy of future state predictions and making action decisions more consistent with the dynamics of the robot itself. This further enhances the control strategy's responsiveness and stability in high-speed or dynamic environments. Combined with the global evaluation information provided by the value network and the guidance of the teacher model, this overall improves training efficiency, policy quality, and adaptability in complex task scenarios.

[0133] refer to Figure 3 Figure 2 shows a schematic diagram of a two-stage training and inference framework for a policy network. The training phase combines a teacher-student model framework comprising a teacher encoder 301 and a student encoder 302, a linear velocity encoder 303, and a deep reinforcement learning framework. The deep reinforcement learning framework is composed of a policy network 304, a value network 305, and a PPO algorithm. During the training phase, privileged state information and historical linear velocity information are introduced to guide policy learning. This allows the student encoder 302 to stably output high-quality control actions during the inference phase, relying solely on observable data. This observable data includes the robot's current motion state information.

[0134] It is important to note that the teacher encoder 301 is only used during the training phase to extract high-dimensional semantic representations from the privileged state information, which are subsequently used as policy input to guide learning. The student encoder 302 encodes the robot's own historical motion state information during training and inference, and the linear velocity encoder 303 encodes the robot's historical linear velocity information during training and inference to obtain dynamic features relevant to action decisions. The current motion state information is then fed into the policy network 304 to generate control actions. Furthermore, the student encoder 302 ultimately needs to mimic the representation output by the teacher encoder 301, so distillation training is performed during training by minimizing the mean squared error (MSE).

[0135] The policy network 304 is the core module for generating motion policies. During the training phase, the policy network 304 receives the latent vectors from the teacher encoder 301 or the student encoder 302, the latent vectors from the linear velocity encoder 303, and the current motion state information. In conjunction with the value network 305, the policy network 304 uses the PPO algorithm to update the policy. During the inference phase, the policy network 304 receives the latent vectors from the student encoder 302 and the linear velocity encoder 303, as well as the current motion state information, and outputs the motion policy for controlling the robot's motion.

[0136] based on Figure 3 The frame diagram shown is shown in Figure 4As shown, the training process of the policy network for outputting the action policy for controlling the robot motion based on deep reinforcement learning may include the following steps S401 to S405:

[0137] In step S401 , the robot's historical motion state information is encoded by the student encoder and a second latent vector is generated. The robot's privileged state information is encoded by the teacher encoder and a third latent vector is generated.

[0138] The robot's historical motion state information can include joint angles, joint velocities, foot contact states, center of gravity trajectory, and other information from the past several frames. For example, the past 10 frames of the historical motion state sequence are input into the student encoder 302 for encoding to generate a second latent vector. This second latent vector can capture the continuity and dynamic characteristics of the robot's motion.

[0139] Privileged state information can include the robot's actual contact force with the ground, an environmental height map, and disturbance information. This privileged state information is input into the teacher encoder 301 for encoding, generating a third latent vector. This third latent vector can highly condense key semantics such as terrain structure, obstacle distribution, and robot-environment interaction, helping to construct a more complete high-dimensional feature representation of the environmental state.

[0140] It should be noted that privileged state information can only be obtained during the training phase but cannot be observed during the testing or deployment phase. Therefore, the teacher encoder 301 can use the complete information to learn the best potential expression, thereby guiding the student encoder 302 to learn. Both the teacher encoder 301 and the student encoder 302 can compress high-dimensional, temporal state information into low-dimensional potential representations, providing behavioral semantic representations from different sources for the policy network 304. For example, the teacher encoder 301 and the student encoder 302 can be multi-layer perceptrons, temporal convolutional networks, or recurrent neural networks, capable of extracting temporal features and compressing them into fixed-length semantic vectors. In addition, the network architecture of the teacher encoder 301 and the network architecture of the student encoder 302 can be the same or different, and this disclosure does not limit this.

[0141] Step S402: Select a second latent vector or a third latent vector according to a preset strategy and input it into the strategy network.

[0142] According to a preset strategy, the second latent vector generated by the student encoder 302 or the third latent vector generated by the teacher encoder 301 is fed into the policy network 304 for decision-making. The preset strategy can be set based on the real-time environment state, the training stage, or specific performance indicators, such as the action execution error threshold, the difference between the simulation and the real environment, etc.

[0143] For example, a first probability of selection can be assigned to the second latent vector and a second probability of selection can be assigned to the third latent vector. Random selection can be made from the second and third latent vectors based on the first and second probabilities. Note that the ratio of the first probability of selection to the second probability of selection can be dynamically increased as training time increases. Initially, the ratio is less than 1.

[0144] For example, at the beginning of training, the high-quality third latent vector generated by the teacher encoder 301 is preferentially used to guide the strategy network 304 to quickly converge to a near-optimal solution. After the student encoder 302 is optimized through knowledge distillation, it gradually transitions to using only the second latent vector to reduce dependence on privileged information. Accordingly, the third latent vector can be selected according to the second selection probability p, and the second latent vector can be selected according to the first selection probability 1-p, and the value of p can be gradually reduced to dynamically adjust the ratio between the first selection probability 1-p and the second selection probability p. It can be understood that the third latent vector needs to be preferentially selected in the initial stage of training. Therefore, the second selection probability p is greater than the first selection probability 1-p, and the ratio of the first selection probability 1-p to the second selection probability p is less than 1.

[0145] Of course, the second latent vector and the third latent vector can also be concatenated and sent to the policy network 304, and dynamically weighted through an attention mechanism or a gating module, so that the policy network 304 can flexibly combine historical experience and privileged knowledge in complex scenarios.

[0146] The selective input mechanism can not only accelerate the training process by utilizing the ideal state prior knowledge provided by the teacher encoder 301, but also cope with sensor limitations or environmental disturbances in actual operation through the generalization ability of the student encoder 302. At the same time, the policy network 304 conditions the latent vector to achieve smooth switching and robust decision-making of motion control. For example, when the robot encounters unknown terrain, it preferentially adjusts the gait based on the second latent vector, while relying on the first latent vector to maintain efficiency during the stable walking stage. Finally, the closed-loop feedback optimizes the policy selection rules, so that the robot can balance motion performance and adaptability in different stages and environmental conditions.

[0147] In addition, in addition to the selected latent vector, the first latent vector and the current motion state information of the robot may also be input into the strategy network 304 for decision making.

[0148] Step S403: Outputting an action strategy for controlling the movement of the robot through the strategy network based on the received first latent vector, the second latent vector or the third latent vector selected by the preset strategy, and the current motion state information.

[0149] The strategy network 304 can be a multi-layer perceptron, a Transformer structure, etc. The strategy network 304 performs multimodal feature fusion on the second latent vector or the third latent vector, the first latent vector, and the current motion state information, and outputs an action strategy, which is recorded as .

[0150] For example, a policy network can be created based on:

[0151] (1)

[0152] Output action strategies to control robot motion;

[0153] in, Indicates the current state Take action The probability distribution of MLP represents a multi-layer perceptron. represents the second latent vector or the third latent vector selected according to the preset strategy, represents the first latent vector.

[0154] For another example, the second latent vector or the third latent vector, the first latent vector, and the current motion state information are spliced ​​or weighted interacted in the embedding space, and action strategies such as joint angle targets, torque commands, or gait phase parameters are generated through nonlinear transformation.

[0155] Step S404, using the privileged state information through the value network to output a value estimate of the current state;

[0156] During the training of the policy network 304 , the long-term reward of the current state is estimated through the value network 305 to optimize the policy network 304 .

[0157] The value network 305 can be a multi-layer fully connected perceptron. For example, when the value network 305 uses the privileged state information to estimate the value of the current state, the privileged state information is first encoded into a high-dimensional feature vector, such as extracting dynamic features through a convolution or fully connected layer, and then fused with the current motion state information in the latent space. Then, a multi-layer nonlinear transformation is performed to output the value estimate representing the current state, which is recorded as This value estimate is used for policy optimization in reinforcement learning to guide the policy network 304 to learn better behaviors.

[0158] Step S405: Based on the action strategy and value estimation, the parameters of the policy network are updated using a deep reinforcement learning algorithm.

[0159] Take the PPO algorithm as an example. It can obtain the reward signal generated by the interaction with the environment and the current state value output by the value network, and construct the advantage function based on the reward signal and the current value estimate. , advantage function Indicates the quality of the current action relative to the average strategy.

[0160] For example, during the training of the policy network 304, the current policy network 304 is used to interact with the environment, based on the current state Select an action strategy , returns the reward after executing the action and the next state , and the interaction trajectory sequence is obtained by sampling ( ), which is used for subsequent strategy optimization. Next, the value network 305 is introduced to evaluate the current value of each state .

[0161] Based on this, the advantage function constructed can be:

[0162] (2)

[0163] in, is a discount factor used to measure the importance of future rewards.

[0164] Furthermore, according to the ratio of the current action strategy output by the strategy network 304 to the old strategy And the advantage function , construct the objective function of proximal strategy optimization :

[0165] (3)

[0166] in, It is a preset hyperparameter used to limit the amplitude of strategy update; Indicates that Limited to the range Inside, Formula (3) ensures that the policy does not grow too fast when the advantage function is positive, and does not decrease too much when the advantage function is negative, thus making a "proximal" constraint on the policy update.

[0167] Finally, the parameters of the policy network 304 are gradient-updated using the objective function of the proximal policy optimization. For example, the loss value is calculated using the objective function shown in formula (3), and the gradient is calculated using the backpropagation algorithm. The parameters of the policy network 304 are iteratively updated in conjunction with the optimizer. This update process can guide the policy network 304 to gradually learn the action strategy that maximizes the long-term cumulative reward while maintaining stable output, thereby improving the robustness and execution efficiency of the control strategy in real environments.

[0168] In some example implementations, by introducing an entropy term into the objective function, the policy network 304 can be encouraged to output an action distribution with high uncertainty during the initial stages of training, thereby enhancing comprehensive exploration of the action space and preventing the policy from prematurely falling into a local optimum. However, if the entropy weight is always maintained high, the policy will not fully converge to a stable action. Therefore, during the training process of the policy network 304, an entropy coefficient can be set for the objective function of the proximal policy optimization, and the entropy coefficient can be decayed after a preset time interval.

[0169] For example, you can:

[0170] (4)

[0171] Calculate the total loss;

[0172] in, is the total loss, is the entropy coefficient that decays over time, Indicates the current state Next, the current strategy The entropy of the action distribution is used to measure the uncertainty of the strategy in selecting each action.

[0173] In this example, on the one hand, policy diversity and sample efficiency are improved in the early stages of training, allowing the model to more fully explore the environment state space. On the other hand, by reducing the entropy weight in the later stages of training, the policy becomes more deterministic, improving control stability and execution consistency. Overall, this helps the policy network achieve a dynamic balance between exploration and convergence during training, improving the final training results and the reliability of policy deployment.

[0174] In addition, based on Figure 4 The policy network used to output the action policy for controlling robot motion is trained using deep reinforcement learning. This training process also includes optimizing the parameters of the student encoder based on the difference between the second and third latent vectors. This step quantifies the difference in the distribution of the two in the latent space and backpropagates the gradient of this difference to update the network weights of the student encoder 302, thereby guiding the student encoder 302 to learn to generate a latent representation close to that of the teacher encoder 301, achieving teacher knowledge distillation.

[0175] Exemplarily, during the training phase, the parameters of the teacher encoder 301 are fixed, and the historical motion state information and the corresponding privileged state information are taken as parallel inputs. After generating latent vectors through the student encoder 302 and the teacher encoder 301 respectively, the distance between the two is minimized using a contrastive learning framework, or adversarial training is used to make the output distribution of the student encoder 302 approach the latent space characteristics of the teacher encoder 301. At the same time, noise injection or data enhancement is introduced to simulate sensor errors in actual deployment, forcing the student encoder 302 to extract feature expressions compatible with privileged information encoding under the condition of limited input information.

[0176] In some example implementations, reference Figure 5 As shown, based on the difference between the second latent vector and the third latent vector, the process of optimizing the parameters of the student encoder may include the following steps S501 to S503:

[0177] Step S501 : Calculate the difference between the second latent vector and the third latent vector through a supervised learning module.

[0178] The supervised learning module receives the second latent vector and the third latent vector and performs a calculation to represent the difference. It should be noted that the supervised learning module in step S501 is not a specific neural network structure, but a training mechanism for measuring the proximity between the output of the student encoder 302 and the output of the teacher encoder 301 in the feature space. For example, a vector distance metric such as mean square error, cosine distance, or L1 norm is used to quantify the difference between the two latent vectors.

[0179] This difference result will serve as the basis for the supervised loss function in step S502 and be used to train the student encoder 302 so that the student encoder 302 can generate a latent state representation similar to that of the teacher encoder 301 without privileged information, thereby achieving knowledge transfer of high-quality representations and providing effective support for the subsequent decision-making of the policy network 304.

[0180] Step S502: construct a loss function based on the difference.

[0181] Accordingly, the loss function can adopt mean square error loss, L1 loss, cosine similarity loss or contrast loss, etc., which is used to reflect the similarity between the representation generated by the student encoder 302 and the high-quality semantic features output by the teacher encoder 301. The smaller the difference, the more successfully the student encoder 302 has learned the feature extraction ability of the teacher encoder 301.

[0182] The constructed loss function will act on the parameter update process of the student encoder 302 through the error backpropagation mechanism in step S503, guiding the student encoder 302 to learn a latent state vector that is more task-relevant and more expressive without privileged information, thereby improving the input quality and overall performance of the policy network 304.

[0183] Step S503: Update the parameters of the student encoder through back propagation according to the loss function, so that the second latent vector output by the student encoder gradually approaches the third latent vector output by the teacher encoder.

[0184] Using the error backpropagation algorithm, the student encoder's parameters are gradient-updated according to the difference loss function, so that its output second latent vector gradually approaches the second latent vector output by the teacher encoder. Through continuous optimization, the student encoder learns to extract feature representations close to the privileged information from historical motion states, thereby enhancing the generalization ability of the policy network 304 and improving its decision-making quality in real-world environments. When deployed, the policy network 304 can also achieve near-teacher performance without relying on privileged information.

[0185] refer to Figure 6 Figure 2 shows another schematic diagram of a two-stage training and inference framework for a policy network. During the training phase, the policy network 304 receives the latent vector from the teacher encoder 301 or the student encoder 302, the first latent vector from the linear velocity encoder 303, and the current motion state information. In conjunction with the value network 305, the policy network 304 uses the PPO algorithm to update the policy. During the inference phase, the policy network 304 receives the second latent vector from the student encoder 302, the first latent vector from the linear velocity encoder 303, and the current motion state information, and outputs an action policy for controlling the robot's motion.

[0186] based on Figure 6 The framework diagram shown in Figure 2 shows the training process of the policy network. Figure 4 The training process is similar to that shown in FIG. 4 , except that the step of outputting the value estimate of the current state by using the privileged state information through the value network 305 is different from the implementation of step S404. Figure 6 The training phase shown is to output a value estimate of the current state by utilizing the privileged state information through the value network 305 in combination with the first latent vector, the second latent vector selected by the preset strategy, or the third latent vector.

[0187] In this example, the latent features are fused with privileged information and fed into the value network 305. This not only preserves the high credibility of the privileged information in understanding the environment, but also enhances the adaptability of the value estimate to the strategy's behavior trajectory and motion trends, better reflecting the expected long-term reward for the robot after taking actions in its current state. Furthermore, the fusion of latent vector inputs enhances the representational richness and generalization capabilities of the value network, making its output more closely aligned with the value distribution under real-world strategy behavior, thereby improving the stability and performance of overall strategy training.

[0188] In some example implementations, Figure 2 The training process of the policy network shown also includes encoding the robot's historical elevation map information through a terrain map encoder to generate a fourth latent vector, and inputting the fourth latent vector into the policy network for auxiliary training.

[0189] The historical elevation map information refers to the elevation map information collected by the robot at several historical moments. This elevation map information is derived from lidar, depth cameras, or other three-dimensional sensors and can reflect spatial features such as the undulations, slopes, and obstacle locations of the terrain on which the robot is located. The terrain map encoder is used to extract temporal evolution features of the terrain from the historical elevation map information to assist in the training of the policy network. For example, the terrain map encoder can be a convolutional neural network, a visual Transformer architecture, or other models suitable for image and point cloud processing, capable of encoding the original spatial information in the elevation map into a latent vector with a fixed dimension.

[0190] Exemplarily, the topographic map encoder may include an input layer, a hidden layer, and an output layer. First, the input layer receives elevation maps of multiple historical moments, such as two-dimensional or three-dimensional raster maps, and performs preprocessing operations such as size unification, pixel normalization, and occlusion area filling to ensure the standardization and stability of the network input. Next, the hidden layer extracts features from the preprocessed elevation map to capture information such as local structural changes, landform features, spatial gradients, and their temporal evolution in the terrain. For example, the convolution structure can extract spatial local features, and temporal convolution or stacking of input sequences can retain the dynamic evolution pattern of terrain changes, thereby obtaining terrain understanding capabilities with temporal correlation. Finally, the output layer maps the spatial-temporal features extracted from the hidden layer into a latent vector of fixed length, namely the fourth latent vector.

[0191] The fourth latent vector, an abstract representation of terrain cognitive information, is fed into the policy network and, along with other latent vectors, participates in policy generation and training. This allows the policy network to consider historical terrain trends and structural features when making movement decisions, resulting in greater environmental adaptability and path planning robustness. When faced with complex terrain, uneven surfaces, or local obstacles, the terrain latent vector provided by the terrain map encoder helps the policy network better understand the terrain's constraints and guidance on movement, improving the policy's generalization and responsiveness in real-world environments.

[0192] refer to Figure 7 As shown, a schematic diagram of a two-stage training-inference framework of another policy network is shown. Figure 7 In the

[15] , a terrain map encoder 306 is introduced. The terrain map encoder 306 encodes the robot's historical elevation map information to obtain terrain features relevant to action decisions. During the training phase, the policy network 304 receives the latent vector from the teacher encoder 301 or the student encoder 302, the first latent vector from the linear velocity encoder 303, the fourth latent vector from the terrain map encoder 306, and the current motion state information. In conjunction with the value network 305, the policy is updated using the PPO algorithm. During the inference phase, the policy network 304 receives the latent vector from the student encoder 302, the linear velocity encoder 303, the latent vector from the terrain map encoder 306, and the current motion state information, and outputs the action policy that controls the robot's motion.

[0193] based on Figure 7 The frame diagram shown is shown in Figure 8 As shown, the training process of the policy network based on deep reinforcement learning may include the following steps S801 to S805:

[0194] In step S801 , the robot's historical motion state information is encoded by the student encoder and a second latent vector is generated. The robot's privileged state information is encoded by the teacher encoder and a third latent vector is generated.

[0195] Step S802: Select a second latent vector or a third latent vector according to a preset strategy and input it into the strategy network.

[0196] Step S803: The strategy network selects the second latent vector or the third latent vector based on the received first latent vector, the fourth latent vector, the preset strategy, and the current motion state information, and outputs an action strategy for controlling the motion of the robot.

[0197] For example, the policy network can be based on:

[0198] (5)

[0199] Output action strategies to control robot motion;

[0200] in, Indicates the current state Take action The probability distribution of MLP represents a multi-layer perceptron. represents the second latent vector or the third latent vector selected according to the preset strategy, represents the first latent vector, represents the fourth latent vector.

[0201] Step S804, using the privileged state information through the value network to output the value estimate of the current state;

[0202] Step S805: Based on the action strategy and value estimation, the parameters of the policy network are updated using a deep reinforcement learning algorithm.

[0203] It should be noted that, except for step S803, the details of other steps can be found in Figure 4 The policy network training process and supplementary steps shown, such as the detailed description of the step of optimizing the student encoder parameters based on the difference between the second latent vector and the third latent vector, are not repeated here.

[0204] refer to Figure 9 As shown in FIG, a schematic diagram of a two-stage training-reasoning framework of another strategy network is shown. Figure 9 In the training phase shown, the value network 305 utilizes privileged state information and combines the latent vector from the teacher encoder 301 or the student encoder 302, the first latent vector from the linear velocity encoder 303, and the fourth latent vector from the topographic map encoder 306 to output a value estimate of the current state.

[0205] In this example, the fourth latent vector generated by the terrain map encoder 306 further incorporates semantic information such as the terrain structure and obstacle layout within the spatial environment, enhancing the value estimation's sensitivity to spatial context and terrain complexity. The fusion of multiple latent features enables the value network 305 to perform valuations based not only on the global privileged state but also on temporal features and environmental adaptation features that are strongly correlated with policy behavior. This improves the accuracy, stability, and generalization of the current state value assessment, enabling more precise guidance of policy updates in reinforcement learning optimization, while increasing sample efficiency and training convergence speed.

[0206] The present disclosure also provides a robot motion control method based on deep reinforcement learning. Figure 10 As shown, the method may include the following steps S1001 to S1003:

[0207] In step S1001 , the robot's historical linear velocity information is encoded using a linear velocity encoder to generate a first latent vector, and the first latent vector is input into a policy network.

[0208] The system uses a linear velocity encoder to encode the robot's linear velocity data at multiple historical moments, extracting the temporal characteristics of velocity changes during motion and generating a first latent vector. This latent vector contains dynamic trend information such as the robot's acceleration, deceleration, and direction changes within a short time window, helping the policy network understand the motion context of the current action.

[0209] Step S1002: Encode the robot's own historical motion state information through the student encoder to generate a second latent vector, and input it into the policy network.

[0210] The student encoder receives the robot's observable state information at multiple historical time steps, such as joint angles, angular velocities, IMU, foot contact status, etc., extracts structural motion semantics and generates a second latent vector, which reflects the robot's own state evolution trajectory and is an abstract expression of the internal motion process.

[0211] Step S1003: Outputting an action strategy for controlling the robot's motion based on the received potential vectors and the current motion state information through the strategy network.

[0212] The policy network simultaneously receives the first and second latent vectors, as well as the current motion state information, performs joint encoding and fusion inference, and outputs the current action policy used to control the robot's motion. By combining multi-source latent features with the current state, the policy network not only has the ability to perceive the immediate state but also understands the historical continuity and dynamic context of the action. This allows it to output a smoother, more responsive, and more environmentally adaptable control policy, demonstrating greater stability and intelligence in complex terrain or highly dynamic scenarios.

[0213] In some example implementations, historical elevation map information of the robot may be further encoded via a terrain map encoder to generate a fourth latent vector, and the fourth latent vector may be input into the policy network.

[0214] The terrain map encoder receives the elevation map data observed by the robot in the past several time steps. The terrain map encoder extracts the spatial distribution pattern of the elevation map data and generates a fourth latent vector, which encodes structural information such as terrain undulations, surface continuity, and environmental change trends on the historical path.

[0215] This fourth latent vector is input into the policy network together with the first latent vector from the linear velocity encoder, the second latent vector from the student encoder, and the current motion state information. The policy network can comprehensively consider the robot's own motion trend, state change trajectory and surrounding terrain structure when making action decisions, thereby achieving a more robust and terrain-adaptive control strategy output.

[0216] It can be understood that the student encoder, linear velocity encoder, topographic map encoder and strategy network are trained according to the robot motion control model training method based on deep reinforcement learning described in detail in other embodiments of the present disclosure, and will not be repeated here.

[0217] In an exemplary embodiment of the present disclosure, a robot motion control model training device based on deep reinforcement learning is also provided. Figure 11 As shown, the robot motion control model training device 1100 based on deep reinforcement learning includes a policy network training module 1101, which is used to train a policy network for outputting an action strategy for controlling robot motion using a teacher-student model framework; wherein the policy network training process includes:

[0218] The robot's historical linear velocity information is encoded through a linear velocity encoder to generate a first latent vector, which is then input into the policy network for auxiliary training.

[0219] The specific details of the above-mentioned strategy network training module 1101 have been described in detail in the corresponding robot motion control model training method based on deep reinforcement learning, so they will not be repeated here.

[0220] In an exemplary embodiment of the present disclosure, a robot motion control device based on deep reinforcement learning is also provided. Figure 12 As shown, the robot motion control device 1200 based on deep reinforcement learning includes a linear velocity feature encoding module 1201, a student encoder encoding module 1202 and a strategy network reasoning module 1203, wherein:

[0221] A linear velocity feature encoding module 1201 is configured to encode the robot's historical linear velocity information to generate a first latent vector, and input the first latent vector into the policy network;

[0222] The student encoder encoding module 1202 is used to encode the robot's own historical motion state information to generate a second latent vector and input it into the policy network;

[0223] The strategy network reasoning module 1203 is used to output the action strategy for controlling the robot motion based on the received potential vectors and the current motion state information;

[0224] Among them, the student encoder, linear velocity encoder and strategy network are obtained according to the robot motion control model training method based on deep reinforcement learning in the embodiment of the present disclosure.

[0225] The specific details of each module in the above-mentioned robot motion control device based on deep reinforcement learning have been described in detail in the corresponding robot motion control method based on deep reinforcement learning, so they will not be repeated here.

[0226] In an exemplary embodiment of the present disclosure, a robot is further provided, the robot comprising a processor and a memory, the memory storing computer-readable instructions, and the computer-readable instructions implementing the above method when executed by the processor. The robot comprises any one of a legged robot, a quadruped robot, a bipedal robot, a wheeled robot, a wheeled-legged robot, a quadrupedal robot, a humanoid robot, a cleaning robot, a transport robot, a mobile robot, and a robotic arm. Figures 13 to 15 As shown, three different robot schematics are shown respectively.

[0227] refer to Figure 16 As shown, an electronic device capable of implementing the above method is also provided. Among them, the electronic device 1600 includes a processor 1601 and a memory 1602. The memory 1602 stores computer-readable instructions. When the computer-readable instructions are executed by the processor 1601, the robot motion control model training method based on deep reinforcement learning in the embodiment of the present disclosure is implemented.

[0228] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is further provided, on which computer program code instructions are stored. When the computer program code instructions are called by a processor of a robot, the robot executes the method in the embodiment.

[0229] refer to Figure 17 As shown, a program product 1700 for implementing the above-mentioned deep reinforcement learning-based robot motion control model training method according to an embodiment of the present disclosure is described. The program product 1700 can be implemented in a portable compact disk read-only memory (CDROM) and includes program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0230] Through the description of the above embodiments, it will be readily understood by those skilled in the art that the example embodiments described herein can be implemented via software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, or mobile hard drive) or on a network and includes several instructions to enable a computing device (such as a personal computer, server, touch terminal, or network device) to execute the methods according to the embodiments of the present disclosure.

[0231] Finally, the above preferred embodiments are intended only to illustrate the technical solutions of this application and are not intended to be limiting. Although this application has been described in detail, those skilled in the art will appreciate that variations in form and detail may be made without departing from the scope of the claims. The dimensions of the drawings are not related to the actual objects and may be modified arbitrarily.

Claims

1. A robot motion control model training method based on deep reinforcement learning, characterized in that: include: Using a teacher-student model framework, a policy network for outputting action policies for controlling robot motion is trained using deep reinforcement learning. The training process of the policy network includes: Encoding the historical linear velocity information of the robot through a linear velocity encoder to generate a first latent vector, and inputting the first latent vector into the policy network; Based on the value estimate of the current state output by the value network in the teacher-student model framework and the action strategy output by the policy network in combination with the first latent vector, the parameters of the policy network are updated using a deep reinforcement learning algorithm.

2. The robot motion control model training method based on deep reinforcement learning according to claim 1 is characterized in that: The teacher-student model framework includes a student encoder and a teacher encoder; The teacher-student model framework is used to train a policy network for outputting an action policy for controlling robot motion based on deep reinforcement learning, including: Encoding the robot's own historical motion state information through the student encoder and generating a second latent vector, and encoding the robot's privileged state information through the teacher encoder and generating a third latent vector; Selecting a second latent vector or a third latent vector according to a preset strategy and inputting the vector into the strategy network; Outputting, by the strategy network, an action strategy for controlling the movement of the robot based on the received first latent vector, the second latent vector or the third latent vector selected by the preset strategy, and current motion state information; Utilizing the privileged state information through a value network, outputting a value estimate of the current state; Based on the action strategy and the value estimate, the parameters of the policy network are updated using a deep reinforcement learning algorithm.

3. The robot motion control model training method based on deep reinforcement learning according to claim 2 is characterized in that: Outputting a value estimate of the current state by utilizing the privileged state information through the value network includes: The value network utilizes privileged state information and combines the first latent vector and the second latent vector or the third latent vector selected by the preset strategy to output a value estimate of the current state.

4. The robot motion control model training method based on deep reinforcement learning according to claim 1 is characterized in that: The linear velocity encoder comprises: An input layer, configured to receive linear velocity information of the robot at multiple historical moments and pre-process the linear velocity information; The hidden layer is used to extract time series features from the pre-processed linear velocity information using a neural network model; The output layer is configured to generate the first latent vector according to the temporal features extracted by the hidden layer.

5. The robot motion control model training method based on deep reinforcement learning according to claim 4 is characterized in that: The neural network model in the hidden layer includes a long short-term memory neural network or a gated recurrent unit neural network.

6. The robot motion control model training method based on deep reinforcement learning according to claim 2 is characterized in that: Outputting, by the strategy network, an action strategy for controlling the movement of the robot based on the received first latent vector, the second latent vector or the third latent vector selected by the preset strategy, and current motion state information, includes: The policy network is based on: Outputting an action strategy for controlling the movement of the robot; in, Indicates the current state Take action The probability distribution of MLP represents a multi-layer perceptron. represents the second latent vector or the third latent vector selected according to the preset strategy, Denote the first latent vector.

7. The robot motion control model training method based on deep reinforcement learning according to claim 1 is characterized in that: The training process of the policy network also includes: The robot's historical elevation map information is encoded by a terrain map encoder to generate a fourth latent vector, and the fourth latent vector is input into the policy network for auxiliary training.

8. The robot motion control model training method based on deep reinforcement learning according to claim 7 is characterized in that: The teacher-student model framework includes a student encoder and a teacher encoder; The teacher-student model framework is used to train a policy network for outputting an action policy for controlling robot motion based on deep reinforcement learning, including: Encoding the robot's own historical motion state information through the student encoder and generating a second latent vector, and encoding the robot's privileged state information through the teacher encoder and generating a third latent vector; Selecting a second latent vector or a third latent vector according to a preset strategy and inputting the vector into the strategy network; Selecting, by the strategy network, the second latent vector or the third latent vector and current motion state information based on the received first latent vector, the fourth latent vector, and the preset strategy, and outputting an action strategy for controlling the motion of the robot; Utilizing the privileged state information through a value network, outputting a value estimate of the current state; Based on the action strategy and the value estimate, the parameters of the policy network are updated using a deep reinforcement learning algorithm.

9. The robot motion control model training method based on deep reinforcement learning according to claim 8 is characterized in that: Outputting a value estimate of the current state by utilizing the privileged state information through the value network includes: The value network utilizes privileged state information and combines the first latent vector, the fourth latent vector, and the second latent vector or the third latent vector selected by the preset strategy to output a value estimate of the current state.

10. The robot motion control model training method based on deep reinforcement learning according to claim 8, characterized in that: The step of selecting the second latent vector or the third latent vector and the current motion state information based on the received first latent vector, the fourth latent vector, and the preset strategy through the strategy network, and outputting an action strategy for controlling the motion of the robot includes: The policy network is based on: Outputting an action strategy for controlling the movement of the robot; in, Indicates the current state Take action The probability distribution of MLP represents a multi-layer perceptron. represents the second latent vector or the third latent vector selected according to the preset strategy, represents the first latent vector, Denotes the fourth latent vector.

11. The robot motion control model training method based on deep reinforcement learning according to claim 2 or 8, characterized in that: The selecting the second latent vector or the third latent vector according to a preset strategy includes: configuring a first probability of selection for the second latent vector and configuring a second probability of selection for the third latent vector; Randomly select from the second latent vector and the third latent vector according to the first selection probability and the second selection probability.

12. The robot motion control model training method based on deep reinforcement learning according to claim 10, characterized in that: The method further comprises: According to the increase of training time, the ratio of the first selection probability to the second selection probability is dynamically increased; wherein, in the initial stage of training, the ratio of the first selection probability to the second selection probability is less than 1.

13. The robot motion control model training method based on deep reinforcement learning according to claim 2 or 8, characterized in that: The method of using a teacher-student model framework to train a policy network for outputting an action policy for controlling robot motion based on deep reinforcement learning also includes: Parameters of the student encoder are optimized based on a difference between the second latent vector and the third latent vector.

14. The robot motion control model training method based on deep reinforcement learning according to claim 13 is characterized in that: The optimizing the parameters of the student encoder based on the difference between the second latent vector and the third latent vector comprises: calculating the difference between the second latent vector and the third latent vector by a supervised learning module; constructing a loss function based on the difference; The parameters of the student encoder are updated by back propagation according to the loss function, so that the second latent vector output by the student encoder gradually approaches the third latent vector output by the teacher encoder.

15. The robot motion control model training method based on deep reinforcement learning according to claim 2 or 8, characterized in that: The updating of the parameters of the policy network using a deep reinforcement learning algorithm based on the action strategy and the value estimate includes: Obtain the reward signal generated based on the interaction with the environment and the current state value output by the value network, and construct an advantage function based on the reward signal and the current value estimate ; The ratio of the current action strategy output by the policy network to the old strategy And the advantage function , construct the objective function of proximal strategy optimization : in, It is a preset hyperparameter used to limit the amplitude of strategy update; Indicates that Limited to the range Inside, Indicates expected value; The parameters of the policy network are gradient updated using the objective function of the proximal policy optimization.

16. The robot motion control model training method based on deep reinforcement learning according to claim 15, characterized in that: The updating of the parameters of the policy network using a deep reinforcement learning algorithm based on the action strategy and the value estimate further includes: Setting an entropy coefficient for the objective function optimized by the proximal strategy; After each preset time interval, the entropy coefficient is decayed.

17. The robot motion control model training method based on deep reinforcement learning according to claim 16, characterized in that: The step of setting an entropy coefficient for the objective function of the proximal strategy optimization includes: according to: Calculate the total loss; in, is the total loss, is the entropy coefficient that decays over time, Indicates the current state Next, the current strategy The entropy of the action distribution.

18. A robot motion control method based on deep reinforcement learning, characterized in that: include: Encoding the historical linear velocity information of the robot through a linear velocity encoder to generate a first latent vector, and inputting the first latent vector into a policy network; Encoding the robot's own historical motion state information through a student encoder to generate a second latent vector, and inputting the second latent vector into the policy network; Outputting, through the strategy network, an action strategy for controlling the movement of the robot based on the received potential vectors and current motion state information; The student encoder, linear velocity encoder and strategy network are obtained according to the robot motion control model training method based on deep reinforcement learning according to any one of claims 1 to 17.

19. The robot motion control method based on deep reinforcement learning according to claim 18, characterized in that: The method further comprises: encoding the historical elevation map information of the robot through a terrain map encoder to generate a fourth latent vector, and inputting the fourth latent vector into the policy network; Among them, the student encoder, linear velocity encoder, topographic map encoder and strategy network are obtained according to the robot motion control model training method based on deep reinforcement learning according to any one of claims 1 to 17.

20. A robot motion control model training device based on deep reinforcement learning, characterized in that: include: A policy network training module is used to train a policy network for outputting action strategies for controlling robot motion using a teacher-student model framework. The policy network training process includes: The historical linear velocity information of the robot is encoded by a linear velocity encoder to generate a first latent vector, and the first latent vector is input into the policy network for auxiliary training.

21. A robot motion control device based on deep reinforcement learning, characterized in that: include: a linear velocity feature encoding module, configured to encode historical linear velocity information of the robot to generate a first latent vector, and input the first latent vector into the policy network; a student encoder encoding module, configured to encode the robot's own historical motion state information to generate a second latent vector, and input the second latent vector into the policy network; a policy network reasoning module, configured to output an action strategy for controlling the robot's motion based on the received potential vectors and the current motion state information; The student encoder, linear velocity encoder and strategy network are obtained according to the robot motion control model training method based on deep reinforcement learning according to any one of claims 1 to 17.

22. An electronic device, characterized in that: include: processor; as well as A memory having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by the processor, implement the method according to any one of claims 1 to 19.

23. A robot, characterized in that: include: processor; as well as A memory having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by the processor, implement the method according to any one of claims 1 to 19.

24. The robot according to claim 23, characterized in that The robot includes any one of a legged robot, a quadruped robot, a bipedal robot, a wheeled robot, a wheel-legged robot, a quadrupedal robot, a humanoid robot, a cleaning robot, a transport robot, a mobile robot and a robotic arm.

25. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer program code instructions, and when the computer program code instructions are called by a processor of the robot, the robot executes the method according to any one of claims 1 to 19.

Images