Motion control method and device based on reinforcement learning

Abstract

The invention provides a motion control method and device based on reinforcement learning, electronic equipment and a storage medium, and relates to the technical field of robot motion control. The motion control method based on reinforcement learning comprises the following steps: training and generating a teacher network model corresponding to a robot according to a robot model description file, road surface detection sample data and motion performance information, and determining student network training initial parameters corresponding to the teacher network model; training and generating a corresponding student network model according to the student network training initial parameters and the road surface detection sample data; deploying the student network model to the robot, analyzing a real-time image collected by the robot based on the road surface type classification detection network model, and determining a corresponding road surface type; and determining the target joint position offset of the robot based on the road surface type corresponding to the real-time image, the current road surface detection data and the student network model, and determining the joint torque of the robot according to the target joint position offset. The effect of controlling robot motion based on reinforcement learning is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot motion control, for example to a motion control method and device based on reinforcement learning, an electronic device and a storage medium. BACKGROUND

[0002] Traditional robot control methods need to use sensors to obtain environmental and self-state information, then perform feature extraction, and finally the controller calculates the control output according to the external and internal state information of the robot, which acts on the controlled object. The traditional control algorithm has the following difficulties: an accurate controlled object model needs to be established, but this is often difficult to obtain; it lacks robustness in the face of complex terrain, perception failure and other uncertain factors in the real world; nonlinear controller design is difficult, with many constraints and complex calculations, and there are deficiencies in real-time environment, multi-type scene adaptive agile motion. In addition, due to the variety of contact forms between the foot end and the ground, the contact force has the characteristics of imbalance, strong impact, etc., which causes the failure of local linearization, the discontinuity of the gradient, and the problem of unstable solution of the classical algorithm, so it is often necessary to use the strategy of pre-setting the terrain-gait, which has a large amount of engineering implementation work. Reinforcement learning is an intelligent algorithm that learns control strategies through interaction between agents and the environment without data and labels. For complex control problems, deep learning algorithms are often introduced, and deep neural networks can be used to obtain data features.

[0003] In some related technologies, the method of reinforcement learning is introduced to robots to enable robots to autonomously learn complex control strategies with as little human design and intervention as possible, and to achieve flexible and stable motion. However, in this process, due to the complexity of the reward function design, the difficulty of reward coefficient tuning, the large inertia of large robots, and the presence of many nonlinear factors, it is difficult for the trained robot to have a good natural gait. Therefore, the control effect of the current motion control method based on reinforcement learning is poor. SUMMARY

[0004] The present application aims to provide a motion control method and device based on reinforcement learning, an electronic device and a storage medium, which can improve the control effect of reinforcement learning motion and enable the trained robot to have a good natural gait.

[0005] According to an aspect of the present application, a motion control method based on reinforcement learning is provided, comprising: training and generating a teacher network model corresponding to a robot according to a robot model description file, road surface detection sample data and motion performance information, and determining student network training initial parameters corresponding to the teacher network model; training and generating a student network model corresponding to the teacher network model according to the student network training initial parameters and the road surface detection sample data; deploying the student network model to the robot, and determining a road surface type corresponding to a real-time image collected by the robot based on a road surface type classification detection network model; determining a target joint position offset of the robot based on the road surface type corresponding to the real-time image, current road surface detection data and the student network model, and determining a joint torque of the robot according to the target joint position offset to control the robot to perform a corresponding action.

[0006] According to some embodiments, training and generating a teacher network model corresponding to a robot according to a robot model description file, road surface detection sample data and motion performance information comprises: importing the robot model description file in a training environment to establish a simulation model of the robot; training and generating the teacher network model of the robot based on the simulation model, road surface detection sample data and motion performance information using an action-evaluation framework and a proximal policy optimization algorithm.

[0007] According to some embodiments, training and generating a teacher network model of a robot based on road surface detection sample data and motion performance information using an action-evaluation framework and a proximal policy optimization algorithm comprises: constructing a teacher network model framework based on a multi-layer perception and a nonlinear activation function; inputting the road surface detection sample data and the motion performance information into the teacher network model framework, and taking an expected reward function threshold as a target to construct an action-evaluation framework; wherein the motion performance information includes a ground friction coefficient, a ground rebound coefficient and a weight; and iteratively updating model parameters of the teacher network model based on the action-evaluation framework through a gradient descent method to train and generate the teacher network model of the robot.

[0008] According to some embodiments, the reward function of the teacher network model comprises a style reward function, a task reward function and a safety energy efficiency reward function; wherein the method further comprises: training the teacher network model according to an original style data set to output a robot motion strategy; performing motion style similarity calculation based on the original style data set and the robot motion strategy to determine a style similarity; training the teacher network model according to motion task execution information to determine a task reward parameter and an energy efficiency reward parameter; and determining an expected reward function threshold based on the style similarity, the task reward parameter, the energy efficiency reward parameter, the motion task execution information and a current motion task.

[0009] According to some embodiments, the student network model comprises a student state evaluation network and a student action policy network; wherein the student network model corresponding to the teacher network model is trained and generated according to the initial parameters of the student network training and the road surface detection sample data, comprising: in the training environment, based on the initial parameters of the student network training, setting the initial values corresponding to the student state evaluation network and the student action policy network respectively; according to the initial values corresponding to the student state evaluation network and the student action policy network respectively and the road surface detection sample data, the student network model corresponding to the teacher network model is trained and generated by using a data aggregation method.

[0010] According to some embodiments, the teacher network model comprises a teacher information network and a teacher action policy network, and the student network model further comprises a long short-term memory network and a student distillation network, and the road surface detection sample data comprises proprioceptive information, terrain classification and height information of a movement direction; wherein the student network model corresponding to the teacher network model is trained and generated according to the initial values corresponding to the student state evaluation network and the student action policy network respectively and the road surface detection sample data by using a data aggregation method, comprising: constructing a student network model framework based on a multi-layer perception and a nonlinear activation function; embedding the proprioceptive information and the terrain classification information into the hidden state of the long short-term memory network to determine a first student network model; training the first student network model according to a supervised learning mode to minimize the reconstruction loss between the teacher information network and the student distillation network and the imitation loss between the teacher action policy network and the student action policy network to determine a second student network model; training the second student network model by using the data aggregation method based on the initial values corresponding to the student state evaluation network and the student action policy network respectively, the reconstruction loss and the imitation loss to generate the student network model corresponding to the teacher network model.

[0011] According to some embodiments, the above-mentioned motion control method based on reinforcement learning further comprises: labeling and classifying the pre-collected road surface information based on the terrain information to generate a road surface information dataset containing corresponding road surfaces and road surface types; and training and generating a road surface type classification detection network model based on the road surface information dataset.

[0012] According to an aspect of the present application, a motion control device based on reinforcement learning is provided, comprising:

[0013] The teacher network training module is configured to train and generate a teacher network model corresponding to a robot according to a robot model description file, road surface detection sample data and motion performance information, and determine initial parameters of the student network training corresponding to the teacher network model.

[0014] The student network training module is configured to train and generate a student network model corresponding to the teacher network model according to the initial parameters of the student network training and the road surface detection sample data.

[0015] a road surface type determination module configured to deploy the student network model to the robot, and analyze a real-time image collected by the robot based on a road surface type classification detection network model to determine a road surface type corresponding to the real-time image;

[0016] a joint torque determination module configured to determine a target joint position offset of the robot based on the road surface type corresponding to the real-time image, the current road surface detection data, and the student network model, and determine a joint torque of the robot according to the target joint position offset to control the robot to perform a corresponding action.

[0017] Optionally, the teacher network training module, when training and generating the teacher network model of the robot based on the robot model description file, the road surface detection sample data, and the motion performance information, is configured to:

[0018] In the training environment, the robot model description file is imported to establish a simulation model of the robot;

[0019] For the simulation model, the teacher network model of the robot is trained and generated based on the road surface detection sample data and the motion performance information by using an action-evaluation framework and a proximal policy optimization algorithm.

[0020] Optionally, when the teacher network training module trains and generates the teacher network model of the robot based on the road surface detection sample data and the motion performance information by using the action-evaluation framework and the proximal policy optimization algorithm, the teacher network training module is configured to:

[0021] The teacher network model framework is constructed based on a multi-layer perception and a nonlinear activation function;

[0022] The road surface detection sample data and the motion performance information are input to the teacher network model framework, and an action-evaluation framework is constructed with an expected reward function threshold as a target, wherein the motion performance information includes a ground friction coefficient, a ground rebound coefficient, and a weight;

[0023] Based on the action-evaluation framework, the model parameters of the teacher network model are iteratively updated by a gradient descent method to train and generate the teacher network model of the robot.

[0024] Optionally, the reward function of the teacher network model includes a style reward function, a task reward function, and a safety energy efficiency reward function; the motion control device based on reinforcement learning further includes an expected reward function threshold determination module configured to:

[0025] The teacher network model is trained according to the original style data set to output a robot motion strategy;

[0026] Based on the original style data set and the robot motion strategy, a motion style similarity calculation is performed to determine a style similarity;

[0027] training the teacher network model according to the motion task execution information to determine the task reward parameter and the energy efficiency reward parameter;

[0028] determining the expected reward function threshold value based on the style similarity, the task reward parameter, the energy efficiency reward parameter, the motion task execution information and the current motion task.

[0029] Optionally, the student network model comprises a student state evaluation network and a student action policy network; and the student network training module is configured to:

[0030] In the training environment, the student network training module is configured to set initial values corresponding to the student state evaluation network and the student action policy network based on the initial parameters of the student network.

[0031] The student network training module is configured to train and generate the student network model corresponding to the teacher network model by using a data aggregation method according to the initial values corresponding to the student state evaluation network and the student action policy network and the road surface detection sample data.

[0032] Optionally, the teacher network model comprises a teacher information network and a teacher action policy network, the student network model further comprises a long short-term memory network and a student distillation network, and the road surface detection sample data comprises proprioceptive information and terrain classification and height information of a motion direction; and the student network training module is configured to:

[0033] The student network training module is configured to construct a student network model framework based on a multi-layer perceptron and a nonlinear activation function.

[0034] The student network training module is configured to embed the proprioceptive information and the terrain classification information into a hidden state of the long short-term memory network to determine a first student network model.

[0035] The student network training module is configured to train the first student network model according to a supervised learning manner to minimize a reconstruction loss between the teacher information network and the student distillation network and an imitation loss between the teacher action policy network and the student action policy network to determine a second student network model.

[0036] The student network training module is configured to train the second student network model by using a data aggregation method based on the initial values corresponding to the student state evaluation network and the student action policy network, the reconstruction loss and the imitation loss to generate the student network model corresponding to the teacher network model.

[0037] Optionally, the motion control device based on reinforcement learning further comprises a road surface type classification detection network model generation module configured to:

[0038] The pre-acquired road surface information is labeled and classified based on terrain information to generate a road surface information dataset containing corresponding road surfaces and road surface types.

[0039] Based on the road surface information dataset, a road surface type classification detection network model is trained and generated.

[0040] According to an example embodiment, by utilizing the teacher network model for pre-training, a complex model with good performance can be quickly generated, providing an effective learning basis for subsequent student network models. The teacher network model is trained based on the robot model description file, road surface detection sample data, and motion performance information, enabling it to accurately output the motion control mode of the robot according to the input road surface detection sample data and motion performance information. The student network model is trained under the supervision of the teacher network, which can inherit the advantages of the teacher network model while reducing the risk of overfitting and improving the accuracy of prediction. The student network model calculates the target joint position offset and joint torque in real time based on the current road surface detection data, road surface type, and other information, ensuring that the robot can quickly respond to environmental changes and perform corresponding actions, thereby improving the accuracy of motion control based on reinforcement learning.

[0041] According to an aspect of the present application, an electronic device is provided, which includes a processor, a memory storing a computer program that, when executed by the processor, causes the processor to perform the motion control method based on reinforcement learning as described above.

[0042] According to an aspect of the present application, a non-transitory computer-readable medium is provided, which stores readable instructions that, when executed by a processor, cause the processor to perform the motion control method based on reinforcement learning as described above.

[0043] It should be understood that the above general description and the following detailed description are only exemplary and do not limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art based on these drawings without departing from the scope of the present application.

[0045] Figure 1 A flowchart of the motion control method based on reinforcement learning provided by the embodiments of the present application;

[0046] Figure 2 A training and deployment diagram of the motion control method based on reinforcement learning provided by the embodiments of the present application;

[0047] Figure 3 A block diagram of the motion control device based on reinforcement learning provided by the embodiment of the present application;

[0048] Figure 4 A structural schematic diagram of the electronic device provided by the embodiment of the present application.

[0049] Legend: S t , S t+1 , state; W t , W s , W d , network parameter; A t , A s , action; 300, motion control device based on reinforcement learning; 301, teacher network training module; 302, student network training module; 303, road surface type determination module; 304, joint torque determination module; 305, expected reward function threshold determination module; 306, road surface type classification detection network model generation module; 400, electronic device; 401, memory; 402, processor. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0051] The specific implementation can refer to the following embodiments.

[0052] Figure 1 A flowchart of the motion control method based on reinforcement learning provided by the embodiment of the present application. The method of the present embodiment can be applied to a control server. As shown in the figure, the method comprises: Figure 1

[0053] S101, training and generating a teacher network model corresponding to a robot according to a robot model description file, road surface detection sample data and motion performance information, and determining student network training initial parameters corresponding to the teacher network model.

[0054] The student network training initial parameters include weight parameters, bias parameters, etc., and can be used to represent the initial setting parameters when starting to train the student network model.

[0055] ​The robot model description file can include shape parameters and physical attributes of the robot, the shape parameters can include joint data, link data, etc., and the physical attributes can include weight, inertia, etc. of the robot. The road detection sample data can include terrain height information of different roads and body perception of the robot when moving on the corresponding road, and can also include road types. The body perception can be used to represent information obtained by sensors carried by the robot, such as robot angular velocity, attitude, joint position, etc.; the motion performance information can be used to represent related parameters involved when the robot moves on the road, and can include ground friction coefficient, ground rebound coefficient, weight, center of mass position, motor strength coefficient, motor proportion and differential coefficient, and remote control speed instruction, etc.

[0056] The teacher network model can be used to represent a complex model that has been trained and has good performance, and can pass the learned knowledge or representation to the corresponding student network model. The student network training initial parameters can be generated during the training of the teacher network model, and can include weight parameters of motion speed, etc., and can be used for subsequent training of the student network model.

[0057] Training the teacher network model of the robot based on the robot model description file, the road detection sample data and the motion performance information can enable the teacher network model to have the ability to output the robot motion control mode when inputting the road data.

[0058] S102, training and generating a student network model corresponding to the teacher network model according to the student network training initial parameters and the road detection sample data.

[0059] The student network model can be a relatively simple and low-cost model, which is usually used to replace the teacher network for deployment, and can imitate the output or intermediate representation of the teacher network, so as to learn the knowledge of the teacher network.

[0060] In some implementations, the road detection sample data can be used as input to train the student network model using the student network training initial parameters.

[0061] S103, deploying the student network model to the robot, and analyzing real-time images collected by the robot based on the road type classification detection network model to determine the road type corresponding to the real-time images.

[0062] The road type classification detection network model can have the ability to input road related information such as pictures or road data, and output the corresponding road type.

[0063] In some implementations, the student network model can be written into a control device of the robot, such as a CPU, a GPU, etc., to realize deployment. An image acquisition device is arranged on the robot body and can be used to acquire real-time images. The control server can input the real-time images acquired by the robot to the road type classification detection network model to directly output the road type corresponding to the real-time images.

[0064] S104, based on the road type corresponding to the real-time image, the current road detection data and the student network model, determine the target joint position offset of the robot, and determine the joint torque of the robot according to the target joint position offset, to control the robot to perform corresponding actions.

[0065] The current road detection data can include the body perception of the robot at the current time and the height information of the terrain.

[0066] The target joint position offset can be used to represent what kind of offset is needed to achieve the corresponding motion of the robot, and the joint torque can be used to represent the size of the torque or rotational force generated by the motor when pushing the joint of the robot to achieve the aforementioned target joint position offset.

[0067] In some implementations, the road type and the current road detection data can be input to the student network model to directly output the target joint position offset of the robot. Then the target joint position offset can be input to a proportional-differential controller, which processes the target joint position offset to obtain the joint torque and outputs it to the driver of the robot to perform corresponding operations and realize the corresponding motion control process of the robot.

[0068] In some implementations, the control server can be used to train the teacher network model and the student network model, and then the student network model can be deployed on the robot for real-time image analysis and joint torque calculation by the control device of the robot.

[0069] The present application can quickly generate a complex model with good performance by using the teacher network model for pre-training, providing an effective learning basis for the subsequent student network model. The teacher network model is trained based on the robot model description file, the road detection sample data and the motion performance information, so that it can accurately output the motion control mode of the robot according to the input road detection sample data and motion performance information; the student network model is trained under the supervision of the teacher network, which can inherit the advantages of the teacher network model, while reducing the risk of overfitting and improving the accuracy of prediction. The student network model calculates the target joint position offset and joint torque in real time according to the current road detection data, road type and other information, ensures that the robot can quickly respond to environmental changes and perform corresponding actions, and improves the accuracy of motion control based on reinforcement learning.

[0070] According to some embodiments, a robot model description file can be imported in a training environment to establish a simulation model of the robot; and a teacher network model of the robot can be trained and generated based on the road surface detection sample data and the motion performance information, using an action-evaluation framework and a proximal policy optimization algorithm for the simulation model.

[0071] The training environment can be a simulation environment pre-established for model training. The action-evaluation framework is a framework widely used in the fields of artificial intelligence, machine learning, and robotics, and its core idea is to continuously optimize and improve the performance of the system through two links of “action” and “evaluation”. The proximal policy optimization algorithm (PPO) is an important reinforcement learning algorithm, which introduces a parameter to limit the update amplitude of the strategy, ensuring that the new strategy will not deviate too much from the old strategy when updated, so that the update is more stable and reliable.

[0072] In some implementations, various parameters in the robot model description file can be used to construct a simulation model of the robot in the training environment. Then in the training environment, an action-evaluation framework can be constructed. The framework includes two parts of action issuing module and evaluation module. The action issuing module is responsible for generating the motion action (such as joint angle, etc.) of the robot according to the current road surface condition and the robot state. The evaluation module evaluates the action according to the performance (such as stability, efficiency, etc.) of the robot after executing the action, and gives the corresponding evaluation value. Then the teacher network model can be trained in the action-evaluation framework using the proximal policy optimization algorithm. First, a policy network (i.e. the teacher network model) can be initialized, which can output the corresponding action according to the input road surface information and robot state.

[0073] Then, the following steps can be performed in the training environment: a set of road surface samples and initial robot states are randomly selected and input into the teacher network model to obtain initial actions. The initial actions are executed, the motion performance of the robot is observed, and the corresponding reward value (i.e. evaluation value) is recorded. The parameters of the teacher network model are updated using the PPO algorithm, so that the output action can obtain a higher reward value. Repeat the above steps until the teacher network model converges or reaches a preset number of training rounds. Through continuous trial and error and learning, the teacher network model can gradually learn how to generate appropriate motion actions according to the road surface condition and the robot state to maximize the cumulative reward value.

[0074] The application trains and generates a teacher network model of the corresponding robot in a training environment by using an action-evaluation framework and a proximal policy optimization algorithm. The teacher network model can adaptively adjust the robot motion strategy according to different road conditions, thereby providing strong support for the practical application of the robot.

[0075] According to some embodiments, the teacher network model framework can be constructed based on a multi-layer perception and a nonlinear activation function; the road detection sample data and the motion performance information are input into the teacher network model framework, and an expected reward function threshold is taken as a target to construct an action-evaluation framework; wherein the motion performance information includes a ground friction coefficient, a ground rebound coefficient and a load; based on the action-evaluation framework, the model parameters of the teacher network model are iteratively updated by a gradient descent method to train and generate the teacher network model of the robot.

[0076] The motion performance information can be difficult to obtain in real time in a real environment, but can be used as additional guidance information in the training process. The motion performance information is used as part of the input features to construct the input layer of the teacher network model. In this way, the model can learn the correlation between the motion performance information and the robot motion strategy during the training process. The sample data containing the motion performance information is used for training, and the parameters of the teacher network model are iteratively optimized by the gradient descent method.

[0077] In some implementations, a multi-layer perception (MLP) framework can be constructed as an initial structure of the teacher network model according to the road detection sample data, including road types, textures, lighting conditions, etc. Then the teacher network model framework can be constructed based on the multi-layer perception (MLP) mechanism and the nonlinear activation function (such as ReLU, Leaky ReLU, Tanh, etc.). The input of the framework is the road detection sample data, including road type, road flatness and other feature information; the output is the motion strategy of the robot, such as joint angle, etc. Through the multi-layer perception, the model can learn the complex mapping relationship from the input features to the output strategy. In order to evaluate the performance of the robot after executing a certain motion strategy, an expected reward function can be designed in advance. The function is defined according to the success rate, stability and efficiency of the robot in executing the task under the current road conditions. For example, when the robot moves stably and efficiently on rough road, the expected reward function will give a higher value; on the contrary, when it falls or moves slowly on uneven road, the reward value will be lower.

[0078] Then the gradient descent method can be used to iteratively update the model parameters, although the PPO algorithm does not directly use the gradient descent method for optimization, but uses a similar idea to the gradient descent method when updating the strategy. The specific steps of iterative training can be:

[0079] (1) Initialize the parameters of the teacher network model, such as weights and bias terms.

[0080] (2) According to the training purpose, a certain number of samples are selected from a plurality of road surface detection sample data, and the certain number of samples and corresponding motion performance information are taken as training data.

[0081] (3) The training data is input into the teacher network model to obtain the motion strategy of the robot.

[0082] (4) The reward value after executing the motion strategy is calculated according to the expected reward function.

[0083] (5) The error between the reward value and the expected reward function threshold is calculated, and the gradient is calculated based on the error.

[0084] (6) The parameters of the teacher network model are updated using the gradient descent method to reduce the error.

[0085] (7) Steps (2) to (6) are repeated until the model converges or reaches a preset number of training rounds.

[0086] During the training process, the Proximal Policy Optimization (PPO) algorithm can be used to limit the amplitude of policy update, ensuring that the new policy does not deviate too much from the old policy during update, thereby maintaining the stability and reliability of the learning process.

[0087] After updating the model parameters of the teacher network model, the teacher network model can be evaluated. By testing the performance of the model in the actual environment or simulation environment, it can be determined whether it meets the expected requirements. If the model performs poorly, the model structure, activation function, expected reward function, etc. can be adjusted, and the training and optimization can be performed again.

[0088] The multi-layer perception structure used in the present application can learn and process complex road surface detection data. The introduction of the nonlinear activation function enables the model to capture the nonlinear relationship in the data, thereby improving the adaptability and generalization ability of the model. This enables the trained teacher network model to cope with various unknown or changing road conditions and provide accurate motion strategies for the robot. Training with the expected reward function threshold enables the model to continuously try various possible motion strategies during the training process and adjust based on the reward values they bring. In this way, the model can learn the optimal or near-optimal motion strategy under specific road conditions, thereby improving the motion efficiency and stability of the robot. Iterative training by gradient descent ensures that the model parameters are updated in the direction of the optimal solution at each iteration. At the same time, combining PPO can limit the magnitude of policy updates, avoiding performance degradation or instability due to excessive updates, and improving the stability and reliability of the entire training process, thereby generating a high-quality teacher network model. In addition, by introducing motion performance information, the teacher network model can learn more information about road conditions, load situations, etc. during the training process. This enables the model to more accurately evaluate the optimal motion strategy under different conditions, thereby improving its adaptability and generalization ability.

[0089] According to some embodiments, the reward function of the teacher network model includes a style reward function, a task reward function, and a safety-energy efficiency reward function. The teacher network model can also be trained based on the original style dataset to output a robot motion strategy; based on the original style dataset and the robot motion strategy, a motion style similarity calculation is performed to determine the style similarity; the teacher network model is trained based on the motion task execution information to determine the task reward parameter and the energy efficiency reward parameter; and based on the style similarity, the task reward parameter, the energy efficiency reward parameter, the motion task execution information, and the current motion task, the expected reward function threshold is determined.

[0090] The original style dataset can include a robust gait dataset formed by adopting rigid body dynamics trajectory optimization and an agile animal motion dataset formed by motion capture mapping. By performing style similarity calculation on the robot motion strategy network output and the motion dataset, the robot is trained to have the ability to learn robust motion strategies in challenging terrain and agile motion strategies in natural terrain.

[0091] The motion task execution information can be used to represent information about the execution of tracking-related instructions, which can include tracking instructions for linear velocity commands, angular velocity commands, etc. The current motion task can include tracking tasks for various commands issued to the robot at the current time.

[0092] In some implementations, a teacher network model framework can be constructed based on a multi-layer perceptron and a nonlinear activation function. This framework accepts road detection sample data and motion performance information as input and outputs a robot action policy. The teacher network model is trained using an original style dataset. The original style dataset contains a series of robot motion trajectories of different styles. Through training, the model can learn the motion characteristics of different styles and output a robot motion strategy that matches the given style. Based on the original style dataset and the robot output action policy, a motion style similarity calculation is performed. By comparing the differences between the actual output and the expected style, the style similarity is determined. This step is used to evaluate the model's ability to maintain a specific motion style. The teacher network model is trained using motion task execution information, which contains task objectives and constraint conditions under different road conditions. Through training, the model can learn the strategy for completing tasks under different road conditions and determine the task reward parameter and energy efficiency reward parameter. Based on the motion style similarity, the task reward parameter, and the energy efficiency reward parameter, the expected reward function threshold is determined. The expected reward function can be a comprehensive evaluation index that considers style, task, and energy efficiency, etc., to measure the overall performance of the robot when executing tasks. By adjusting the weights and thresholds of different reward functions based on the motion task execution information and the current motion task, fine control of the robot motion strategy can be achieved.

[0093] The present application introduces style reward functions, task reward functions, and safety and energy efficiency reward functions to achieve diversified evaluation of robot motion strategies, and can generate robot motion strategies that both conform to a specific motion style and efficiently complete tasks. Through training based on the original style dataset and motion style similarity calculation, the ability of the teacher network model to maintain a specific motion style is improved, which enables the robot to maintain a stable motion style when executing tasks.

[0094] According to some embodiments, the student network model includes a student state evaluation network and a student action policy network. The initial parameters of the student network can be trained in a training environment, and the student state evaluation network and the student action policy network each correspond to an initial value; based on the initial values of the student state evaluation network and the student action policy network and the road detection sample data, a data aggregation method is used to train and generate a student network model corresponding to the teacher network model.

[0095] The initial parameters of the student network can come from the training process of the teacher network model or prior knowledge. The road detection sample data can be pre-set, and these data contain sensor information, environmental data collected by the robot under different road conditions, and corresponding labels or target values. These data will be used to train the student network model.

[0096] The data aggregation method refers to collecting new trajectory data by using the current student strategy in each iteration, and then training the next student strategy by using a teacher strategy demonstration data set, and aggregating the newly obtained trajectory and the demonstrated trajectory into a data set for iterative training to improve the learning adaptability of the student network. The data aggregation method can include batch processing, stochastic gradient descent, and the like.

[0097] In some implementations, a training environment can be constructed, which can simulate real-world road conditions, robot actions, and other related factors. The training environment provides a repeatable and controllable experimental platform for training the student network model. In the training environment, the student network training initial parameters can be set based on the student state evaluation network and the student action strategy network respectively corresponding initial values. The student action strategy network is used to set the specific action to be performed by the robot under different road conditions. The student state evaluation network is used to evaluate the performance of the robot in the current state and provide a basis for subsequent action decision-making. The road detection sample data is input into the student network model, and the data aggregation method is used for training. During the training process, the robot has different motion states under the same input due to the student action strategy network, and the student state evaluation network can output an evaluation value according to the current motion state of the robot. This evaluation value is compared and optimized with the output of the student network model (i.e., action decision), so that the robot can make more appropriate action decisions in the future. Through continuous iteration and optimization, the student network model can gradually learn how to make reasonable action decisions based on the current state and environmental information, and gradually approach the performance of the teacher network model.

[0098] The present application sets the initial values of the student state evaluation network and the student action strategy network by using the initial parameters of the corresponding student network of the teacher network model. The student network model can have a good starting point at the beginning of training, thereby accelerating the learning speed and improving the training efficiency. The student state evaluation network can evaluate the performance of the robot in the current state in real time and adjust the action decision according to the evaluation result, so that the robot can make more reasonable and efficient action decisions under different road conditions, and improve the overall task completion efficiency and quality.

[0099] According to some embodiments, the teacher network model comprises a teacher information network and a teacher action policy network, and the student network model further comprises a long short-term memory network and a student distillation network; the road detection sample data comprises body perception information and terrain classification and height information of a movement direction. The student network model framework can be constructed based on a multi-layer perception and a nonlinear activation function; the body perception information and the terrain classification information are embedded in the hidden state of the long short-term memory network to determine a first student network model; the first student network model is trained according to a supervised learning manner to minimize the reconstruction loss between the teacher information network and the student distillation network and the imitation loss between the teacher action policy network and the student action policy network to determine a second student network model; and the second student network model is trained based on the initial values, the reconstruction loss and the imitation loss of the student state evaluation network and the student action policy network respectively by using a data aggregation method to generate the student network model corresponding to the teacher network model.

[0100] The reconstruction loss is used to measure the closeness of the student distillation network and the teacher information network in the student network model in the output, and the imitation loss is used to evaluate the closeness of the student action policy network and the teacher action policy network in the output.

[0101] In some implementations, the basic framework of the student network model can be constructed based on a multi-layer perception (MLP) and a nonlinear activation function first. This framework will serve as the basis for adding a long short-term memory (LSTM) network and a student distillation network later. The body perception information and the terrain classification information of the movement direction in the road detection sample data are embedded in the hidden state of the LSTM. The embedding process can be achieved by taking the body perception information and the terrain classification information as the input features of the LSTM, or converting them into a format that can be processed by the LSTM using appropriate feature extraction and encoding methods. Then the first student network model can be trained in a supervised learning manner. In this stage, the student network model will try to imitate the behavior of the teacher information network and the teacher action policy network, that is, to minimize the reconstruction loss between the teacher information network and the student distillation network and the imitation loss between the teacher action policy network and the student action policy network. Based on the initial values (i.e., the initial parameters of the student network corresponding to the teacher network model), the reconstruction loss and the imitation loss, the second student network model can be trained by using a data aggregation method. In this stage, the student network model will further adjust its parameters according to the road detection sample data in the training data to optimize its performance. Through continuous iteration and optimization, the student network model can gradually approach the performance of the teacher network model.

[0102] The application embeds the body perception information and the terrain classification information into the hidden state of the LSTM, so that the student network model can make full use of the useful information in the road surface detection sample data, improve the accuracy of the model, and enhance the adaptability of the model to complex environments. By using the teacher-student network framework and supervised learning method to train the student network model, the performance of the student network model can be ensured to be close to that of the teacher network model while maintaining a certain accuracy, so that the student network model can have stronger robustness and generalization ability in practical application. By using the data aggregation method and optimization algorithm to train the student network model, the calculation cost and time can be significantly reduced, and the efficiency and practicality of the training method in practical application are improved.

[0103] According to some embodiments, the pre-collected road surface information can be labeled and classified based on the terrain information to generate a road surface information dataset containing corresponding road surfaces and road surface types; and based on the road surface information dataset, a road surface type classification detection network model is trained and generated.

[0104] In some implementations, before pre-collecting the road surface information, the terrain information of the target area is first obtained using a terrain mapping device such as a laser radar, satellite remote sensing, etc. These terrain information includes altitude, slope, terrain undulation, etc. The road surface of the target area is pre-collected using a camera, sensor, etc. to obtain visual information such as road surface image, texture, color, and physical information such as road surface flatness, friction coefficient, etc. Based on the terrain information, the pre-collected road surface information is labeled and classified. For example, road surfaces with a slope greater than a degree and an altitude within a certain range are labeled as "mountain roads"; road surfaces with flat terrain and high friction coefficient are labeled as "urban trunk roads", etc. The labeled and classified road surface information is combined with the corresponding terrain information to construct a road surface information dataset containing road surfaces and corresponding road surface types. The dataset contains multiple road surface types and their corresponding feature information.

[0105] A deep neural network model can be pre-set as a road surface type classification detection network model. The model can use a convolutional neural network (CNN) as the basic structure, combined with a recurrent neural network (RNN) or a long short-term memory network (LSTM) network structure for processing sequence data to better capture the spatiotemporal features of road surface information. The constructed road surface information dataset is used to train the road surface type classification detection network model. During the training process, the model parameters are optimized through a backpropagation algorithm, so that the model can accurately map the input road surface information to the corresponding road surface type. The trained road surface type classification detection network model is evaluated using a test set, and the model is optimized and adjusted according to the evaluation results. The optimization methods include adjusting the network structure, increasing the amount of training data, using data augmentation techniques, etc.

[0106] The present application annotates and classifies pre-acquired road surface information based on terrain information, and combines a road surface information dataset to train a generated road surface type classification detection network model, which can more accurately identify different types of road surfaces. Since the road surface information dataset contains various road surface types and their corresponding feature information, the road surface type classification detection network model trained can better adapt to different types of road surface scenes. At the same time, through model evaluation and optimization, the generalization ability of the model can be further improved, making it more stable and reliable in actual application.

[0107] According to some embodiments, the reward function of the teacher network (teacher network model) includes a style reward function, a task reward function, and a safety-energy efficiency reward function. The style reward function adopts a double-prior style GAN discriminator training method, the task reward function includes rewards for linear velocity command and angular velocity command tracking, and encourages the robot to complete a complete task. The safety-energy efficiency reward function includes rewards for motion smoothness and penalties for collision and fall detection.

[0108] In some implementations, the reward function r t includes a style reward function a task reward function and a safety-energy efficiency reward function comprising:

[0109]

[0110] The style reward function adopts a double-prior style GAN discriminator training method, and calculates the style similarity of the network output of the robot motion strategy and the motion dataset to train the robot to have the ability to learn a robust motion strategy in challenging terrain and a agile motion strategy in natural terrain. The robust gait dataset in the above-mentioned original style dataset can be D1, and the agile motion dataset can be D2. The style reward function is defined as formula (2) as follows:

[0111]

[0112] Wherein, α is an adjustable parameter between 0 and 1, and in some implementations, α = 0.5.

[0113] Taking the robust gait dataset D1 as an example, the style GAN discriminator training and value function calculation process are described as follows:

[0114] Define a discriminator with a parameter neural network representation of the state s t t+1 The samples can be from the dataset D1 or from the agent-generated samples. t 31 (31-dimensional vector) including the body linear velocity, body angular velocity, joint position, joint velocity, body height relative to the terrain, and the robust gait dataset includes Trot, forward-backward walking, and turning trajectory data.

[0115]

[0116] If "+" is used as a separator, formula (3) can be divided into three terms, where the first two terms of formula (3) are the least square GAN formula, which is used to assist the aforementioned discriminator to distinguish whether the input state of a given point is from the robot A or from the robust gait dataset D1, and the third term of formula (3) is the gradient penalty. is defined as:

[0117]

[0118] wherein is normalized to the range [0-1], and the training and reward function calculation process of the agile motion dataset D2 can refer to the training of D1.

[0119] The task reward function includes the reward for tracking the linear velocity command and the angular velocity command. In some implementations, can be defined as:

[0120]

[0121] In formula (5), a and β are adjustable parameters, and in some implementations, a and β can be set to 1.1 and 0.4, respectively, to motivate the robot in the present application and improve the tracking accuracy of the command speed. is the commanded linear velocity, is the current linear velocity of the robot, is the commanded angular velocity, is the current angular velocity of the robot.

[0122] The safety energy efficiency reward function includes the reward for motion smoothness and the penalty for collision and fall detection, can be defined as:

[0123]

[0124] ​​If "+" is used as a separator, the data on the right side of formula (6) can be separated into four terms, where the first term in formula (6) is the 2-norm of the joint torque, the second term is the 2-norm of the joint acceleration, the third term is the 2-norm of the acceleration of the reinforcement learning action policy output value, and the fourth term is the current time of the robot and the ground collision detection value.

[0125] In some implementations, a = -2e-7, b = -2.5e-9, d = -2e-4, and l = 0.05.

[0126] According to other embodiments, the teacher network includes a teacher information network, a teacher action policy network, a teacher state evaluation network, and a double-prior style adversarial network, which are constituted by using a multi-layer perception and a nonlinear activation function, and are iteratively trained by using a gradient descent method to maximize the expected reward function value.

[0127] The teacher network uses motion performance information during training, which includes a ground friction coefficient, a ground bounce coefficient, a weight, a center of mass position, a motor strength coefficient, a motor proportion and differential coefficient, and a remote control speed instruction. The body perception includes the robot's Euler angle, linear velocity, angular velocity, joint position, and joint velocity of the current and past N periods. The terrain height information is the height of the M sampling points near the foot end. In some implementations, N can be 5 and M can be 183.

[0128] In some implementations, the number of input layers of the teacher information network (390) = body perception (34x5) + motion performance information (37) + terrain height information (183), the number of hidden layers is [256, 128], and the number of output layers is 256. The number of input layers of the teacher action policy network is 256, the number of hidden layers is [512, 256, 128, 64], and the number of output layers is 12. The number of input layers of the teacher state evaluation network is 256, the number of hidden layers is [512, 256, 128], and the number of output layers is 1. The input layer of the double-prior style adversarial network is 31, the hidden layer is [512, 256], and the number of output layers is 1.

[0129] According to other embodiments, in the student network (student network model), the long short-term memory network can fuse the embedding of the historical multiple periods of body perception information and the terrain classification information of the motion direction in the hidden state to improve the reconstruction and adaptation ability of the student distillation network. The student distillation network, the student action policy network, and the student state evaluation network are all constituted by using a multi-layer perception and a nonlinear activation function. The student network uses a supervised learning method to train by minimizing the reconstruction loss between the teacher information network and the student distillation network and the imitation loss between the teacher action policy network and the student action policy network.

[0130] In some implementations, the long short-term memory network encodes the proprioception Ot∈R34, the terrain classification information Terrainclass∈R16, the past hidden state ht-1 and the cell state ct-1 into a current hidden state ht∈R256, which is then passed to the student distillation network. The student distillation network has 439 input layers (256 hidden layers of the long short-term memory network + 183 terrain height information), [1024, 512] hidden layers, and 390 output layers. The input-hidden-output layer numbers of the student action policy network and the student state evaluation network are consistent with the corresponding numbers of the teacher network.

[0131] According to another embodiment, during the teacher-student network training process, a terrain curriculum from easy to difficult is defined, and when the robot adapts to the current difficult terrain, the walking speed and terrain difficulty (such as step height, road slope, obstacle number and dispersion degree, etc.) are automatically increased, so as to have the ability to track different speed commands in different terrains. For the coefficient of the penalty term in the safety energy efficiency reward function, the penalty term coefficient adopts a small value in the initial stage, and is automatically adjusted to increase to a preset value according to the number of iterations, encouraging more exploration in the initial training stage of the strategy and improving the training success rate.

[0132] In some implementations, during the teacher-student network training process, a domain parameter randomization method is used to randomly set the center of gravity deviation, the weight size, the leg and torso mass, the joint PD gain, the initial position, the external force size and position, etc., to improve the adaptability from training simulation to real machine deployment.

[0133] According to other embodiments, the road surface type classification detection network can use the YOLOV8 model, pre-collect image data, and through a supervised learning algorithm, can classify sixteen types of cement ground, grassland, sandy ground, gravel road, stairs, snow ground, slope and vegetation, etc. The road surface type classification detection network outputs the probability value of each terrain classification information.

[0134] According to other embodiments, the road surface type classification detection network model and the student network model can be loaded in CPU, GPU, NPU, a reinforcement learning motion type controller is added in the motion control algorithm, and the reinforcement learning motion state can be switched through the remote control keys. Proprioceptive data is obtained through IMU and encoder, and road surface images are obtained by using the front camera of the robot. The road surface type classification detection network model is called for inference to output the terrain classification information. The student action policy network model in the student network is called for inference to output the relative position offset of each joint of the robot, and then a PD controller is used to calculate the joint torque, which is sent to the motor driver module to drive the joint to reach the specified position.

[0135] Figure 2A training deployment diagram of motion control based on reinforcement learning is provided for an embodiment of the present application.

[0136] As shown in Figure 2 The training deployment diagram of motion control based on reinforcement learning can include three parts:

[0137] 1. Training a teacher-student network;

[0138] 2. Training a road surface type classification detection network;

[0139] 3. Deploying and running the student network and the road surface type classification detection network.

[0140] In stage 1 of training the teacher-student network, the proprioceptive perception, motion performance information, and terrain height information can be used as input to train the teacher information network, the teacher action policy network, and the teacher state evaluation network.

[0141] The double-prior style adversarial network includes the robust style GAN discriminator corresponding to the aforementioned dynamic trajectory optimization dataset and the agile style GAN discriminator corresponding to the aforementioned legged animal motion dataset, which is used to distinguish the style type during training. In the physical simulation environment, proximal policy optimization algorithm training, action execution, and reward function calculation can be performed.

[0142] In stage 2 of training the teacher-student network, the proprioceptive perception, terrain classification information, and terrain height information can be used as input to train the long short-term memory network in the student network, the student distillation network, the student action policy network, and the student state evaluation network. The action output by the student action policy network is applied to the aforementioned physical simulation environment for state updating and iterative training.

[0143] The road surface type classification detection network can be trained, and the pre-collected road terrain images can be used as input information and input into the road surface type classification detection network to output terrain classification information.

[0144] When deploying and running the student network and the road surface type classification detection network, the proprioceptive perception, terrain classification information corresponding to the image collected by the real-time camera, and terrain height information can be used as input information and input into the long short-term memory network and the student distillation network in the student network structure (student network model), so that the agent, i.e., the robot, can perform corresponding processing according to different actions output by the student action policy network, such as motion execution, determination of joint torque and position, and joint drive control, and then the state of the input information can be updated based on the execution of the action.

[0145] The device embodiment of the present application is described below, which can be used to execute the method embodiment of the present application. For details not disclosed in the device embodiment of the present application, reference can be made to the method embodiment of the present application.

[0146] Figure 3 A block diagram of the motion control device based on reinforcement learning provided for the embodiments of the present application. As shown in the figure, the motion control device based on reinforcement learning 300 includes a teacher network training module 301, a student network training module 302, a road surface type determination module 303, and a joint torque determination module 304. Figure 3

[0147] The teacher network training module 301 is configured to train and generate a teacher network model corresponding to the robot according to a robot model description file, road surface detection sample data, and motion performance information, and determine initial student network training parameters corresponding to the teacher network model.

[0148] The student network training module 302 is configured to train and generate a student network model corresponding to the teacher network model according to the initial student network training parameters and the road surface detection sample data.

[0149] The road surface type determination module 303 is configured to deploy the student network model to the robot, and analyze real-time images collected by the robot based on a road surface type classification detection network model to determine a road surface type corresponding to the real-time images.

[0150] The joint torque determination module 304 is configured to determine a target joint position offset of the robot based on the road surface type corresponding to the real-time images, current road surface detection data, and the student network model, and determine a joint torque of the robot according to the target joint position offset to control the robot to perform a corresponding action.

[0151] Optionally, the teacher network training module 301, when training and generating the teacher network model corresponding to the robot according to the robot model description file, the road surface detection sample data, and the motion performance information, is configured to:

[0152] import the robot model description file in a training environment to establish a simulation model of the robot;

[0153] train and generate the teacher network model of the robot based on the road surface detection sample data and the motion performance information by using an action-evaluation framework and a proximal policy optimization algorithm for the simulation model.

[0154] Optionally, the teacher network training module 301, when training and generating the teacher network model of the robot based on the road surface detection sample data and the motion performance information by using the action-evaluation framework and the proximal policy optimization algorithm, is configured to:

[0155] construct a teacher network model framework based on a multi-layer perception and a nonlinear activation function;

[0156] ​The road surface detection sample data and the motion performance information are input into the teacher network model framework, and an action-evaluation framework is constructed with a target of an expected reward function threshold value, wherein the motion performance information includes a ground friction coefficient, a ground rebound coefficient and a weight load;

[0157] Based on the action-evaluation framework, the model parameters of the teacher network model are iteratively updated by a gradient descent method to train and generate the teacher network model of the robot.

[0158] Optionally, the reward function of the teacher network model includes a style reward function, a task reward function and a safety energy efficiency reward function; the motion control device 300 based on reinforcement learning further includes an expected reward function threshold value determination module 305, configured to:

[0159] The teacher network model is trained according to the original style data set to output a robot motion strategy;

[0160] Based on the original style data set and the robot motion strategy, a motion style similarity calculation is performed to determine a style similarity;

[0161] The teacher network model is trained according to the motion task execution information to determine a task reward parameter and an energy efficiency reward parameter;

[0162] Based on the style similarity, the task reward parameter, the energy efficiency reward parameter, the motion task execution information and the current motion task, an expected reward function threshold value is determined.

[0163] Optionally, the student network model includes a student state evaluation network and a student action strategy network; the student network training module 302 is configured to:

[0164] In the training environment, based on the student network training initial parameters, the initial values corresponding to the student state evaluation network and the student action strategy network are set respectively;

[0165] According to the initial values corresponding to the student state evaluation network and the student action strategy network and the road surface detection sample data, the student network model corresponding to the teacher network model is trained and generated by using a data aggregation method.

[0166] Optionally, the teacher network model includes a teacher information network and a teacher action strategy network, the student network model further includes a long short-term memory network and a student distillation network, and the road surface detection sample data includes proprioceptive information and terrain classification and height information of a motion direction; when the student network training module 302 trains and generates the student network model corresponding to the teacher network model by using the data aggregation method according to the initial values corresponding to the student state evaluation network and the student action strategy network and the road surface detection sample data, the student network training module 302 is configured to:

[0167] The student network model framework is constructed based on a multilayer perceptron and a nonlinear activation function.

[0168] The ontology perception information and the terrain classification information are embedded into a hidden state of a long short-term memory network to determine a first student network model.

[0169] The first student network model is trained according to a supervised learning manner to minimize a reconstruction loss between a teacher information network and a student distillation network and an imitation loss between a teacher action policy network and a student action policy network to determine a second student network model.

[0170] The second student network model is trained based on initial values of the student state evaluation network and the student action policy network respectively, the reconstruction loss and the imitation loss by using a data aggregation method to generate a student network model corresponding to the teacher network model.

[0171] Optionally, the motion control device 300 based on reinforcement learning further includes a road surface type classification detection network model generation module 306 configured to:

[0172] The pre-collected road surface information is labeled and classified based on the terrain information to generate a road surface information dataset containing corresponding road surfaces and road surface types.

[0173] The road surface type classification detection network model is trained and generated based on the road surface information dataset.

[0174] The device performs similar functions to the method provided above, and other functions can be referred to the foregoing description, which will not be described here.

[0175] Figure 4 A structural schematic diagram of an electronic device provided in the embodiment of the present application is shown in FIG. 4. Figure 4 The electronic device 400 of the embodiment can include a memory 401 and a processor 402.

[0176] The memory 401 stores a computer program, and when the computer program is executed by the processor 402, the aforementioned processor 402 executes the method in the above embodiment.

[0177] The processor 402 and the memory 401 are connected, such as through a bus.

[0178] Optionally, the electronic device 400 can further include a transceiver. It should be noted that the transceiver in actual application is not limited to one, and the structure of the electronic device 400 does not constitute a limitation on the embodiments of the present application.

[0179] The processor 402 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic device, transistor logic device, hardware component, or any combination thereof. It can implement or execute various exemplary logical blocks, modules and circuits described in combination with the disclosure. The processor 402 can also be a combination of computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0180] The bus can include a path for transmitting information between the above-mentioned components. The bus can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is shown in the figure, but it does not mean that there is only one bus or only one type of bus.

[0181] The memory 401 can be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and instructions, an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, an optical disk storage (including a compact disk, a laser disk, an optical disk, a digital versatile disk, a Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program codes in the form of instructions or data structures and can be accessed by a computer, but not limited thereto.

[0182] The memory 401 is used to store application program codes for implementing the scheme of the present application, and is controlled by the processor 402 to execute. The processor 402 is used to execute the application program codes stored in the memory 401 to realize the content shown in the foregoing method embodiments.

[0183] The electronic device includes, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (Personal Digital Assistant), a PAD (Tablet Personal Computer), a PMP (Portable Multimedia Player), a car terminal (for example, a car navigation terminal), and the like, and a stationary terminal such as a digital TV, a desktop computer, and the like. It can also be a server or the like. Figure 4 The electronic device shown is merely an example and should not impose any limitation on the function and use range of the embodiments of the present application.

[0184] The electronic device of the embodiments can be used to execute the method of any of the above embodiments, and has similar implementation principles and technical effects, which will not be described here.

[0185] The present application also provides a non-transitory computer-readable storage medium having computer-readable instructions stored thereon, and when the instructions are executed by a processor, the processor executes the method in the above embodiments.

[0186] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instructions related to hardware. The foregoing program can be stored in a non-transitory computer-readable storage medium. The program, when executed, performs steps including the above-mentioned method embodiments; and the foregoing storage medium includes ROM, RAM, magnetic or optical disk, and various media that can store program codes.

[0187] The embodiments of the present application are described in detail above, and the specific examples are applied to the principles and implementation modes of the present application. The above description of the embodiments is only used to help understand the method of the present application and its core idea. Meanwhile, the changes or deformations made by the skilled in the art according to the idea of the present application, based on the specific implementation mode and application range of the present application, all belong to the scope of protection of the present application. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A motion control method based on reinforcement learning, characterized in that, include: Based on the robot model description file, road surface detection sample data, and motion performance information, a teacher network model for the corresponding robot is trained and generated, and the initial parameters for training the student network corresponding to the teacher network model are determined. Based on the initial parameters of the student network training and the road surface detection sample data, train and generate the student network model corresponding to the teacher network model; The student network model is deployed to the robot, and the real-time images collected by the robot are analyzed based on the road surface type classification and detection network model to determine the road surface type corresponding to the real-time images. Based on the road surface type corresponding to the real-time image, the current road surface detection data, and the student network model, the target joint position offset of the robot is determined, and the joint torque of the robot is determined according to the target joint position offset to control the robot to perform corresponding actions.

2. The method according to claim 1, characterized in that, The step of training and generating a teacher network model for the corresponding robot based on the robot model description file, road surface detection sample data, and motion performance information includes: In the training environment, import the robot model description file to build a simulation model of the robot; For the simulation model, based on the road surface detection sample data and the motion performance information, a teacher network model of the robot is trained and generated using an action-evaluation framework and a proximal policy optimization algorithm.

3. The method according to claim 2, characterized in that, The process of training and generating a teacher network model for the robot based on the road surface detection sample data and the motion performance information, using an action-evaluation framework and a proximal policy optimization algorithm, includes: A teacher network model framework is constructed based on multilayer perceptron and nonlinear activation function; The road surface detection sample data and the motion performance information are input into the teacher network model framework, and the action-evaluation framework is constructed with the expected reward function threshold as the objective; wherein, the motion performance information includes the ground friction coefficient, the ground rebound coefficient, and the load. Based on the action-evaluation framework, the model parameters of the teacher network model are iteratively updated using the gradient descent method to train and generate the teacher network model of the robot.

4. The method according to claim 3, characterized in that, The reward function of the teacher network model includes a style reward function, a task reward function, and a safety and efficiency reward function; the method further includes: The teacher network model is trained based on the original style dataset to output robot motion strategies; Based on the original style dataset and the robot motion strategy, motion style similarity is calculated to determine style similarity; The teacher network model is trained based on the exercise task execution information to determine the task reward parameters and energy efficiency reward parameters; The expected reward function threshold is determined based on the style similarity, the task reward parameter, the energy efficiency reward parameter, the exercise task execution information, and the current exercise task.

5. The method according to claim 3 or 4, characterized in that, The student network model includes a student state evaluation network and a student action strategy network; The step of training and generating the student network model corresponding to the teacher network model based on the initial parameters of the student network training and the road surface detection sample data includes: In the training environment, based on the initial parameters of the student network training, the initial values ​​corresponding to the student state evaluation network and the student action policy network are set respectively; Based on the initial values ​​of the student state evaluation network and the student action strategy network, and the road surface detection sample data, a student network model corresponding to the teacher network model is trained and generated using a data aggregation method.

6. The method according to claim 5, characterized in that, The teacher network model includes a teacher information network and a teacher action strategy network, and the student network model also includes a long short-term memory network and a student distillation network. The road surface detection sample data includes ontological perception information, terrain classification of movement direction, and height information. The step of training and generating the student network model corresponding to the teacher network model using a data aggregation method based on the initial values ​​of the student state evaluation network and the student action strategy network, and the road surface detection sample data, includes: A student network model framework is constructed based on the multilayer perceptron and the nonlinear activation function. The ontology perception information and the terrain classification information are embedded into the hidden state of the long short-term memory network to determine the first student network model; The first student network model is trained according to a supervised learning method to minimize the reconstruction loss between the teacher information network and the student distillation network, and the imitation loss between the teacher action policy network and the student action policy network, so as to determine the second student network model. Based on the initial values ​​of the student state evaluation network and the student action policy network, the reconstruction loss and the imitation loss, the second student network model is trained using a data aggregation method to generate the student network model corresponding to the teacher network model.

7. The method according to claim 1, characterized in that, Also includes: Based on terrain information, the pre-collected road surface information is labeled and classified to generate a road surface information dataset containing the corresponding road surface and road surface type. Based on the road surface information dataset, the road surface type classification and detection network model is trained and generated.

8. A motion control device based on reinforcement learning, characterized in that, include: The teacher network training module is used to train and generate a teacher network model for the corresponding robot based on the robot model description file, road surface detection sample data and motion performance information, and to determine the initial parameters for student network training corresponding to the teacher network model. The student network training module is used to train and generate the student network model corresponding to the teacher network model based on the initial parameters of the student network training and the road surface detection sample data. The road surface type determination module is used to deploy the student network model to the robot, and analyze the real-time images collected by the robot based on the road surface type classification and detection network model to determine the road surface type corresponding to the real-time images; The joint torque determination module is used to determine the target joint position offset of the robot based on the road surface type corresponding to the real-time image, the current road surface detection data, and the student network model, and to determine the joint torque of the robot based on the target joint position offset, so as to control the robot to perform corresponding actions.

9. An electronic device, characterized in that, include: processor; A memory storing a computer program that, when executed by the processor, causes the processor to perform the motion control method based on reinforcement learning as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium, characterized in that, It stores computer-readable instructions that, when executed by a processor, cause the processor to perform the motion control method based on reinforcement learning as described in any one of claims 1-7.

Images