Robot motion control model training method and robot motion control method

Abstract

A robot motion control model training method and a robot motion control method. The robot motion control model training method comprises: on the basis of the state of a training robot at the current moment, making a decision by using a policy network of a motion control model to be trained, so as to obtain training control instructions and training hidden variables of a future preset number of frames, wherein the preset number of frames is greater than 1, and the training hidden variables are random variables that are unobservable but play a role in the decision-making and prediction of said model (S101); on the basis of the training hidden variables, using a decoding network of said model to perform prediction, so as to obtain a predicted state of the robot at a future preset moment, wherein the preset moment is determined on the basis of a parameter update period of said model (S102); using a simulation engine to sequentially execute the training control instructions of the preset number of frames, so as to obtain the actual state of the robot (S103); and on the basis of the predicted state of the robot at the preset moment and the actual state of the robot, updating parameters of the policy network and the decoding network of said model, so as to obtain a trained motion control model (S104). The model predicts future robot states that are closer to actual robot states, thereby improving the stability of robot control.

Description

A method for training a robot motion control model and a method for robot motion control.

[0001] This application claims priority to Chinese Patent Application No. 2024114348589, filed on July 15, 2024, entitled "A Method for Training a Robot Motion Control Model and a Method for Controlling Robot Motion", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of intelligent control technology, and in particular to a robot motion control model training method and a robot motion control method. Background Technology

[0003] Currently, robot motion control is mainly based on dynamic model-based methods. These methods calculate based on robot and environment models, and with high-frequency control at the underlying level, the execution of actions is relatively stable and accurate in a defined environment. However, they cannot well cope with changes in the robot and environment models. Reinforcement learning model-based methods can make the generated motion control strategy adaptable to model changes by randomly generating robot and environment models during training. However, reinforcement learning-based control methods directly generate the underlying joint control commands at the current moment, resulting in actions that are often highly random and violent.

[0004] Therefore, current methods for controlling robot motion suffer from the technical problem of unstable robot motion.

[0005] Summary of the Invention

[0006] In view of this, the purpose of this application is to provide a robot motion control model training method and a robot motion control method, which solves the technical problem of insufficiently smooth robot motion in the prior art.

[0007] To address the aforementioned technical problems, this application provides a method for training a robot motion control model, comprising:

[0008] Based on the current state of the training robot, the policy network of the motion control model to be trained is used to make decisions, and training control instructions and training latent variables for a preset number of future frames are obtained; wherein, the preset number of frames is greater than 1, and the training latent variables are unobservable random variables that play a role in the model's decision-making and prediction.

[0009] Based on the training latent variables, the decoding network of the motion control model to be trained is used to predict the robot's predicted state at a future preset time; wherein, the preset time is determined according to the parameter update cycle of the motion control model to be trained.

[0010] The simulation engine is used to execute the training control commands for the preset number of frames in sequence to obtain the actual state of the robot;

[0011] Based on the robot's predicted state and actual state at the preset time, the parameters of the policy network and decoding network of the motion control model to be trained are updated to obtain the trained motion control model.

[0012] Optionally, the relationship between the training control command for the preset number of frames and the preset time includes:

[0013] When the preset number of frames is n, the n time points corresponding to the n-frame training control command are respectively t+Δt is the preset time; where t is the current time and Δt is the parameter update period of the motion control model to be trained.

[0014] Optionally, updating the parameters of the policy network and decoding network of the motion control model to be trained based on the robot's predicted state at the preset time and the robot's actual state includes:

[0015] The state prediction reward is determined based on the robot's predicted state at the preset time and the robot's actual state at the corresponding preset time.

[0016] The task reward is determined based on the actual state of the robot and the environmental state obtained by the simulation engine.

[0017] The parameters of the policy network and decoding network of the motion control model to be trained are updated based on the state prediction reward and the task reward.

[0018] Optionally, determining the state prediction reward based on the robot's predicted state at the preset time and its corresponding actual state at the preset time includes:

[0019] The robot's predicted state at the preset time is compared with the robot's actual state at the preset time to determine the state prediction reward; wherein, the closer the robot's predicted state at the preset time is to the robot's actual state at the preset time, the closer the state prediction reward is to 1, otherwise the state prediction reward is closer to 0.

[0020] Optionally, updating the parameters of the policy network and decoding network of the motion control model to be trained based on the state prediction reward and the task reward includes:

[0021] The sum of the state prediction reward and the task reward is used as the training reward;

[0022] Based on the training rewards, update the parameters of the policy network and decoding network of the motion control model to be trained.

[0023] Optionally, the training latent variables include at least one of environmental features, robot historical behavior patterns, and robot load capacity.

[0024] This application also provides a robot motion control method, including:

[0025] Get the current robot status;

[0026] Based on the current robot state, a policy network of a trained motion control model is used to make decisions to obtain control instructions and latent variables for a preset number of future frames; wherein, the motion control model is the aforementioned trained motion control model.

[0027] The preset frame control instruction is selected from the preset frame control instructions to control the robot, and no processing is performed on the other frame control instructions and the hidden variables.

[0028] This application also provides a robot motion control model training device, including:

[0029] The determination module is used to make decisions based on the current state of the training robot and the policy network of the motion control model to be trained, so as to obtain training control instructions and training latent variables for a preset number of future frames; wherein, the preset number of frames is greater than 1, and the training latent variables are unobservable random variables that play a role in the model's decision-making and prediction.

[0030] The prediction module is used to predict the robot's predicted state at a future preset time based on the training latent variables and the decoding network of the motion control model to be trained; wherein the preset time is determined according to the parameter update period of the motion control model to be trained.

[0031] The execution module is used to execute the training control instructions of the preset number of frames sequentially using the simulation engine to obtain the actual state of the robot;

[0032] The update module is used to update the parameters of the policy network and decoding network of the motion control model to be trained based on the robot's predicted state and the robot's actual state at the preset time, so as to obtain the trained motion control model.

[0033] This application also provides a robot motion control device, including:

[0034] The status acquisition module is used to acquire the current robot status;

[0035] The instruction generation module is used to make decisions based on the current robot state using the policy network of the trained motion control model to obtain control instructions and latent variables for a preset number of future frames; wherein, the motion control model is the aforementioned trained motion control model.

[0036] The instruction selection module is used to select a preset frame control instruction from the preset frame control instructions for controlling the robot, without processing the other frame control instructions or the hidden variables.

[0037] This application also provides an electronic device, including:

[0038] Memory, used to store computer programs;

[0039] A processor for executing the computer program to implement the steps of the method described above.

[0040] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described above.

[0041] This application also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0042] The beneficial effects of this application are as follows: During the training process, the robot motion control model outputs control instructions and latent variables for a preset number of future frames, predicts the future robot state based on the latent variables, and updates the model parameters according to the predicted robot state and the actual robot state, so that the model's prediction of the future robot state is closer to the actual robot state. The control instructions generated based on the model are more accurate, thereby improving the stability of subsequent robot control.

[0043] In addition, this application also provides a robot motion control model training device, equipment, and computer-readable storage medium, as well as a motion control method, device, equipment, and computer-readable storage medium, which also have the above-mentioned beneficial effects. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0045] Figure 1 is a flowchart of a robot motion control model training method provided in an embodiment of this application;

[0046] Figure 2 is a flowchart illustrating a robot motion control model training method provided in an embodiment of this application;

[0047] Figure 3 is a structural framework diagram of a robot motion control model training method provided in an embodiment of this application;

[0048] Figure 4 is a flowchart of a robot motion control method provided in an embodiment of this application;

[0049] Figure 5 is a flowchart illustrating a robot motion control method provided in an embodiment of this application;

[0050] Figure 6 is a structural framework diagram of a robot motion control method provided in an embodiment of this application;

[0051] Figure 7 is a schematic diagram of the structure of a robot motion control model training device provided in an embodiment of this application;

[0052] Figure 8 is a schematic diagram of a robot motion control device provided in an embodiment of this application;

[0053] Figure 9 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0055] Please refer to Figure 1, which is a flowchart of a robot motion control model training method provided in an embodiment of this application. The method may include:

[0056] S101, based on the current state of the training robot, the policy network of the motion control model to be trained is used to make decisions and obtain the training control instructions and training latent variables for a preset number of frames in the future; wherein, the preset number of frames is greater than 1, and the training latent variables are unobservable random variables that play a role in the model's decision-making and prediction.

[0057] The execution subject of this embodiment is an electronic device. For example, the electronic device in this embodiment may be a computer, etc.

[0058] This embodiment does not limit the specific type of motion control model to be trained. For example, the motion control model to be trained in this embodiment can be a reinforcement learning model, which includes a policy network and a decoding network, such as Q-Learning (a value function-based reinforcement learning method that learns the expected utility, i.e., the Q-value, of taking a specific action in a given state) or SARSA (an algorithm similar to Q-Learning, but which updates the current Q-value using the actual next state and reward, rather than using a method that maximizes future rewards). Because reinforcement learning models have strong adaptability and flexibility, they improve the adaptability and flexibility of the motion control model.

[0059] This embodiment does not limit the specific type of policy network. For example, the policy network in this embodiment can be a convolutional neural network (CNN); or it can also be a recurrent neural network. The structure of the policy network in this embodiment can be a multilayer perceptron network (MLP) structure. The input layer dimension of the policy network includes the robot's actual state, and the output layer dimension includes the robot control commands and latent variables. The multilayer perceptron network structure can learn complex feature representations through multilayer nonlinear transformations, making it suitable for handling decision problems in high-dimensional state spaces.

[0060] The robot state in this embodiment may include at least one of the following: the position of each joint of the robot, the velocity of each joint of the robot, and feedback data from the robot's inertial sensors. The feedback data from the robot's inertial sensors includes parameters such as acceleration, angular velocity, and attitude. This data is crucial for the robot's motion control. Through high-precision measurement and data processing, the robot can accurately perceive its own motion state in space. It is understood that by providing the states related to the robot's joints, subsequent accurate control of the robot's motion can be achieved.

[0061] In this embodiment, the control command for the preset frame number is the control command for the next n frames, where n can be a value such as 5, 6, 8, 10, etc., and the control command is the desired position command for each joint of the robot.

[0062] The latent variables in this embodiment include predictions of future states. These latent variables can represent factors not directly observed by the robot, such as the potential state of the environment. By introducing these variables, the model can more richly express the problem, thus gaining a deeper understanding of complex scenarios. By abstracting some unobservable factors, the model is not limited to specific observed data but can also make inferences and decisions; this generalization ability is key to handling unknown environments.

[0063] It should be further explained that the aforementioned latent variables may include at least one of environmental characteristics, robot historical behavior patterns, and robot payload capacity. In this embodiment, the environmental characteristics refer to the current environment in which the robot is located; the robot historical behavior patterns refer to the robot's historical operating states; and the robot payload capacity refers to the maximum load mass that the robot can handle when performing a task. It is understood that latent variables are random variables that cannot be directly observed but play an important role in the model's prediction and decision-making. For robot behavior, these latent variables can include various factors, such as environmental characteristics, robot historical behavior patterns, and robot physical and operational attributes, such as payload capacity. By comprehensively considering these factors, the model can better understand and predict the robot's behavior and performance in a specific environment.

[0064] S102, based on the training latent variables, the decoding network of the motion control model to be trained is used to predict the robot's predicted state at a future preset time; wherein, the preset time is determined according to the parameter update cycle of the motion control model to be trained.

[0065] In this embodiment, latent variables are used as inputs to the motion control model to be trained, resulting in the robot's predicted state at a preset time. When the preset number of frames is n, the n time points corresponding to the n training control commands are respectively... t+Δt is the preset time; where t is the current time and Δt is the parameter update period of the motion control model to be trained.

[0066] This embodiment does not limit the specific type of the decoding network. For example, the decoding network in this embodiment can be a recurrent neural network (RNN); or it can be a convolutional neural network (CNN). The structure of the decoding network in this embodiment can be a multilayer perceptron network (MLP) structure, where the input layer dimension of the decoding network includes latent variables, and the output layer dimension includes the robot's predicted state.

[0067] S103 uses the simulation engine to execute training control commands for a preset number of frames in sequence to obtain the actual state of the robot.

[0068] The actual state of the robot obtained using the simulation engine can be used as the state of the trained robot mentioned above.

[0069] By using a simulation engine to execute training control commands for a preset number of frames in sequence, not only can the actual state of the robot be obtained, but also the environmental state.

[0070] This embodiment does not limit the specific execution order of steps S103 and S102. For example, steps S103 and S102 can be executed simultaneously; or step S103 can be executed first and step S102 can be executed later; or step S102 can be executed first and step S103 can be executed later.

[0071] S104. Based on the robot's predicted state and actual state at a preset time, update the parameters of the policy network and decoding network of the motion control model to be trained, and obtain the trained motion control model.

[0072] This embodiment does not limit the specific process of updating the parameters of the policy network and decoding network of the motion control model to be trained based on the robot's predicted state and actual state at a preset time. For example, this embodiment can determine the accuracy of the prediction by comparing the robot's predicted state at a preset time with the robot's actual state at a preset time, obtain a state prediction reward, and then use this state prediction reward to update the parameters of the policy network and decoding network of the motion control model to be trained, thereby obtaining a trained motion control model. Alternatively, this embodiment can use the sum of the state prediction reward and the task reward as the total reward to update the parameters of the policy network and decoding network of the motion control model to be trained.

[0073] It should be further explained that, in order to improve the stability and reliability of the motion control model, the parameters of the policy network and decoding network of the motion control model to be trained, which are updated based on the robot's predicted state and actual state at preset times, may include:

[0074] S1041, Determine the state prediction reward based on the robot's predicted state at a preset time and the robot's actual state at the corresponding preset time.

[0075] S1042, Determine the task reward based on the actual robot state and environmental state obtained from the simulation engine;

[0076] S1043, based on the state prediction reward and task reward, update the parameters of the policy network and decoding network of the motion control model to be trained.

[0077] In the embodiments of this application, state prediction rewards and task rewards are used together for model reward processing, which improves the generalization ability, stability and reliability of the motion control model.

[0078] It should be further explained that, in order to improve the stability of the motion control model, the determination of the state prediction reward based on the robot's predicted state at a preset time and its corresponding actual state at the preset time may include:

[0079] The robot's predicted state at a preset time is compared with the robot's actual state at the preset time to determine the state prediction reward. The closer the robot's predicted state at the preset time is to the robot's actual state at the preset time, the closer the state prediction reward is to 1; otherwise, the state prediction reward is closer to 0.

[0080] The state prediction reward in this embodiment can be determined using the following formula: Where j represents the state prediction reward, and O represents the robot's actual state at a preset time. The robot's predicted state at a preset time.

[0081] It should be further explained that, in order to improve the accuracy of the reward, the above-mentioned updating of the parameters of the policy network and decoding network of the motion control model to be trained based on the state prediction reward and the task reward may include: summing the state prediction reward and the task reward as the training reward; and updating the parameters of the policy network and decoding network of the motion control model to be trained based on the training reward. This embodiment of the application improves the accuracy of rewarding the model by summing the state prediction reward and the task reward to obtain the total training reward, thereby updating the parameters of the motion control model to be trained based on the total training reward.

[0082] The robot motion control model training method provided in this application is essentially a predictive reinforcement learning model.

[0083] To facilitate understanding of this application, please refer to Figure 2. Figure 2 is a flowchart illustrating a robot motion control model training method provided in an embodiment of this application, which may specifically include:

[0084] S201: Based on the current actual state of the robot obtained from the simulation engine, the control commands and latent variables for the next preset number of frames are obtained using the policy network in the predictive reinforcement learning model to be trained.

[0085] The framework diagram corresponding to this embodiment is shown in Figure 3.

[0086] In this embodiment, the robot state includes the position and velocity feedback of each joint of the robot and the feedback from the robot's inertial sensors, and the control command is the desired position command for each joint of the robot.

[0087] In this embodiment, the policy network is a multilayer perceptron network (MLP) structure. The input layer dimension is the robot state dimension m, and the output layer dimension is s+l, where s is the dimension of the robot control command and l is the dimension of the latent variables (which can be selected by the user).

[0088] In this embodiment, the output of the policy network includes control commands for the next n frames and latent variables; where the latent variables are variables used to predict the robot's state at time t+Δt, t is the current time, and Δt is the model parameter update period during training; the control commands for the next n frames correspond to the future... There are a total of n control commands at different times.

[0089] S202, based on the latent variables, the robot's predicted state at a future preset time is obtained using the decoding network in the predictive reinforcement learning model to be trained.

[0090] In this embodiment, the latent variables are processed by a decoding network to generate a prediction of the robot's state at time t+Δt.

[0091] In this embodiment, the decoding network is also a multilayer perceptron network (MLP) structure, with the input layer dimension being the dimension l of the latent variables and the output layer dimension being the dimension m of the robot state.

[0092] S203 uses the simulation engine to execute each control instruction in the preset number of control instructions in sequence to obtain the actual state of the robot.

[0093] In this embodiment, the simulation engine is... The simulation cycle executes control commands at n time points sequentially to obtain the actual state of the robot at time t+Δt.

[0094] S204. Compare the robot's predicted state at a preset time with the robot's actual state at the preset time to determine the state prediction reward.

[0095] In this embodiment, the accuracy of the prediction is evaluated by comparing the robot's actual state with its predicted state, and a state prediction reward is obtained. Specifically, the reward is as follows: Where o represents the actual state of the robot at time t+Δt. Let be the robot's predicted state at time t+Δt. It can be seen that the closer the robot's predicted state is to its actual state, the closer the state prediction reward is to 1; the greater the deviation between the predicted and actual states, the closer the state prediction reward is to 0.

[0096] S205. Determine the total training reward based on the task reward and the state prediction reward, and update the policy network and decoding network in the predictive reinforcement learning model to be trained based on the total training reward, so as to obtain the trained motion control model.

[0097] In this embodiment, the task reward can be determined based on the robot's actual state, environmental state, and desired state. The task reward needs to be designed specifically for the training task by the user.

[0098] In this embodiment, the state prediction reward and the task reward are summed as the total training reward to simultaneously update the network parameters of the policy network and the decoding network. The network update algorithm can employ a proximal policy update algorithm.

[0099] The robot motion control model training method provided in this application provides that the motion control model outputs control commands and latent variables for future multiple frames during the training process, predicts the future robot state based on the latent variables, and updates the model parameters according to the predicted robot state and the actual robot state, so that the model's prediction of the future robot state is closer to the actual robot state. The control commands generated based on the model are more accurate, thereby improving the stability of subsequent robot control.

[0100] Please refer to Figure 4, which is a flowchart of a robot motion control method provided in an embodiment of this application. The method may include:

[0101] S301, Get the current robot status.

[0102] The execution subject of this embodiment is an electronic device, specifically a robot control device.

[0103] S302, based on the current robot state, makes decisions using the policy network of the trained motion control model to obtain control commands and latent variables for a preset number of future frames; wherein, the motion control model is the aforementioned trained motion control model.

[0104] S303, Select the preset frame control instruction from the preset frame control instructions to control the robot, without processing the other frame control instructions and the hidden variables.

[0105] In this embodiment, the preset frame can be the first frame.

[0106] In this embodiment, after controlling the robot according to the preset frame control command, a latest current robot state can be obtained, and control commands can be obtained based on the latest current robot state.

[0107] For example, when the application scenario is path planning, global path planning is performed based on known environmental information before the task begins to determine the basic route of the robot from the starting point to the end point. During execution, local path adjustments can be made based on the real-time robot status and preset frame control instructions to cope with changes and uncertainties in the actual environment.

[0108] The motion control model in this application outputs control instructions for future multiple frames. After the robot executes the first frame instruction each time, it can obtain the latest robot state corresponding to the instruction, and regenerate the latest future multiple frame control instructions based on the latest robot state corresponding to the instruction, thereby realizing the control of the robot.

[0109] To facilitate understanding of this application, please refer to Figure 5. Figure 5 is a flowchart illustrating a robot motion control method according to an embodiment of this application, which may specifically include:

[0110] S401, Get the current robot status.

[0111] The framework diagram corresponding to this embodiment is shown in Figure 6.

[0112] S402, based on the current robot state, use the policy network in the predictive reinforcement learning model to determine the control instructions and latent variables for the next preset number of frames; wherein, the predictive reinforcement learning model is the above-trained motion control model.

[0113] S403 sends the first frame control instruction from the preset frame control instructions to the robot, without processing the remaining frame control instructions and hidden variables.

[0114] In this embodiment, the computation cycle of the policy network during network deployment is... The policy network receives the current robot state and outputs control instructions and hidden variables for the next n frames. Only the first frame is considered. Control commands at specific times are sent to the robot for execution; control commands and latent variables at other times do not participate in the control process. A predictive reinforcement learning model is used, with a deployment control frequency of [missing information]. It is n times the training frequency, resulting in a higher control frequency than in typical reinforcement learning deployments, and thus achieving better control performance.

[0115] The robot motion control model training device provided in the embodiments of this application is described below. The robot motion control model training device described below and the robot motion control model training method described above can be referred to in correspondence.

[0116] Please refer to Figure 7 for details. Figure 7 is a schematic diagram of the structure of a robot motion control model training device provided in an embodiment of this application, which may include:

[0117] The determination module 100 is used to make decisions based on the current state of the training robot and the policy network of the motion control model to be trained, so as to obtain training control instructions and training latent variables for a preset number of future frames; wherein, the preset number of frames is greater than 1, and the training latent variables are unobservable random variables that play a role in the model's decision-making and prediction.

[0118] The prediction module 200 is used to predict the robot's predicted state at a future preset time based on the training latent variables and the decoding network of the motion control model to be trained; wherein the preset time is determined according to the parameter update cycle of the motion control model to be trained.

[0119] The execution module 300 is used to execute the training control instructions of the preset number of frames sequentially using the simulation engine to obtain the actual state of the robot;

[0120] The update module 400 is used to update the parameters of the policy network and the decoding network of the motion control model to be trained according to the robot's predicted state and the robot's actual state at the preset time, so as to obtain the trained motion control model.

[0121] Furthermore, based on the above embodiments, the relationship between the training control command for the preset number of frames and the preset time may include:

[0122] When the preset number of frames is n, the n time points corresponding to the n-frame training control command are respectively t+Δt is the preset time; where t is the current time and Δt is the parameter update period of the motion control model to be trained.

[0123] Furthermore, based on any of the above embodiments, the update module 400 may include:

[0124] The first reward determination unit is used to determine the state prediction reward based on the robot's predicted state at the preset time and the robot's actual state at the corresponding preset time.

[0125] The second reward determination unit is used to determine the task reward based on the actual state of the robot and the environmental state obtained by the simulation engine.

[0126] An update unit is used to update the parameters of the policy network and the decoding network of the motion control model to be trained based on the state prediction reward and the task reward.

[0127] Furthermore, based on the above embodiments, the first reward determination unit is specifically used to compare the robot's predicted state at the preset time with the robot's actual state at the preset time to determine the state prediction reward; wherein, the closer the robot's predicted state at the preset time is to the robot's actual state at the preset time, the closer the state prediction reward is to 1, otherwise the state prediction reward is closer to 0.

[0128] Furthermore, based on the above embodiments, the state prediction reward is determined using the following formula: Where j represents the predicted state reward, and O represents the actual state of the robot at the preset time. The robot's predicted state at the preset time.

[0129] Furthermore, based on any of the above embodiments, the above-mentioned updating unit may include:

[0130] The summation subunit is used to sum the state prediction reward and the task reward as the training reward.

[0131] An update subunit is used to update the parameters of the policy network and decoding network of the motion control model to be trained based on the training rewards.

[0132] Furthermore, based on the above embodiments, the policy network has a multilayer perceptron network structure, the input layer dimension of the policy network includes the robot's actual state, and the output layer dimension of the policy network includes robot control commands and latent variables; the decoding network has the same structure as the multilayer perceptron network, the input layer dimension of the decoding network includes latent variables, and the output layer dimension of the decoding network includes the robot's predicted state.

[0133] Furthermore, based on any of the above embodiments, the training latent variables include at least one of environmental features, robot historical behavior patterns, and robot load capacity.

[0134] Furthermore, based on any of the above embodiments, the training robot state includes at least one of the positions of the robot's joints, the speeds of the robot's joints, and data fed back from the robot's inertial sensors.

[0135] It should be noted that the order of the modules and units in the above-mentioned robot motion control model training device can be changed without affecting the logic.

[0136] Please refer to Figure 8 for details. Figure 8 is a structural schematic diagram of a robot motion control device provided in an embodiment of this application, which may include:

[0137] The status acquisition module 500 is used to acquire the current robot status;

[0138] The instruction generation module 600 is used to make decisions based on the current robot state using the policy network of the trained motion control model to obtain control instructions and latent variables for a preset number of future frames; wherein, the motion control model is the aforementioned trained motion control model.

[0139] The instruction selection module 700 is used to select a preset frame control instruction from the preset frame control instructions for controlling the robot, without processing the other frame control instructions and the hidden variables.

[0140] It should be noted that the order of the modules and units in the above-mentioned robot motion control device can be changed without affecting the logic.

[0141] The following describes an electronic device provided in an embodiment of this application. The electronic device described below can be referred to in correspondence with the robot motion control model training method and motion control method described above.

[0142] Please refer to Figure 9, which is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, which may include:

[0143] Memory 10 is used to store computer programs;

[0144] The processor 20 is used to execute computer programs to implement the robot motion control model training method and / or motion control method described above.

[0145] The memory 10, processor 20, and communication interface 30 all communicate with each other through the communication bus 40.

[0146] In this embodiment, the memory 10 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment, the memory 10 may store programs for implementing the following functions:

[0147] Based on the current state of the training robot, the policy network of the motion control model to be trained is used to make decisions, and training control instructions and training latent variables for a preset number of frames in the future are obtained; where the preset number of frames is greater than 1, the training latent variables are unobservable random variables that play a role in the model's decision-making and prediction.

[0148] Based on the training latent variables, the decoding network of the motion control model to be trained is used to predict the robot's predicted state at a future preset time. The preset time is determined according to the parameter update cycle of the motion control model to be trained.

[0149] The simulation engine is used to execute training control commands for a preset number of frames in sequence to obtain the actual state of the robot;

[0150] Based on the robot's predicted state and actual state at preset times, the parameters of the policy network and decoding network of the motion control model to be trained are updated to obtain the trained motion control model.

[0151] Or a program used to achieve the following functions:

[0152] Get the current robot status;

[0153] Based on the current state of the robot, the policy network of the trained motion control model is used to make decisions and obtain control instructions and latent variables for a preset number of frames in the future; wherein, the motion control model is the trained motion control model mentioned above.

[0154] The preset frame control instruction, selected from the preset frame control instructions, is used to control the robot. No other frame control instructions or the hidden variables are processed.

[0155] In one possible implementation, the memory 10 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; and the data storage area may store data created during use.

[0156] Furthermore, memory 10 may include read-only memory and random access memory, providing instructions and data to the processor. A portion of the memory may also include NVRAM. The memory stores operating systems and operating instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operating instructions may include various operating instructions for implementing various operations. The operating system may include various system programs for implementing various basic tasks and handling hardware-based tasks.

[0157] Processor 20 can be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic device. Processor 20 can be a microprocessor or any conventional processor. Processor 20 can call programs stored in memory 10.

[0158] The communication interface 30 can be an interface for the communication module, used to connect with other devices or systems.

[0159] [Correction 27.11.2024 according to Rule 91] Of course, it should be noted that the structure shown in FIG9 does not constitute a limitation on the electronic device in the embodiments of this application. In practical applications, the electronic device may include more or fewer components than those shown in FIG9, or combine certain components.

[0160] The following describes the computer-readable storage medium provided in the embodiments of this application. The computer-readable storage medium described below can be referred to in correspondence with the robot motion control model training method and / or motion control method described above.

[0161] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the steps of the above-described robot motion control model training method and / or motion control method. The computer-readable storage medium may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0162] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0163] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0164] Finally, it should be noted that in this document, relationships such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0165] The above provides a detailed description of a robot motion control model training method and a robot motion control method provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for training a robot motion control model, characterized in that, The method comprises: based on the current moment of the training robot state, using the policy network of the motion control model to be trained to make decisions, obtaining the training control instructions and the training hidden variables of a preset number of frames in the future; wherein, the preset number of frames is greater than 1, and the training hidden variable is a random variable that is unobservable but plays a role in the decision and prediction of the model; based on the training hidden variable, using the decoding network of the motion control model to be trained to make predictions, obtaining the robot prediction state at a preset time in the future; wherein, the preset time is determined according to the parameter update period of the motion control model to be trained; using the simulation engine to execute the training control instructions of the preset number of frames in turn, obtaining the actual state of the robot; according to the robot prediction state at the preset time and the actual state of the robot, updating the parameters of the policy network and the decoding network of the motion control model to be trained, obtaining the trained motion control model.

2. The robot motion control model training method according to claim 1, characterized by, The relationship between the training control instructions of the preset number of frames and the preset time comprises: When the preset frame number is n, n frames of training control instructions correspond to n time points respectively for the preset time; wherein, t is the current moment, and Δt is the parameter update period of the motion control model to be trained.

3. The robot motion control model training method according to claim 1, characterized by, According to the robot prediction state at the preset time and the actual state of the robot, updating the parameters of the policy network and the decoding network of the motion control model to be trained, comprises: determining the state prediction reward according to the robot prediction state at the preset time and the actual state of the robot at the corresponding preset time; determining the task reward according to the actual state of the robot and the environment state obtained by the simulation engine; updating the parameters of the policy network and the decoding network of the motion control model to be trained according to the state prediction reward and the task reward.

4. The robot motion control model training method according to claim 3, characterized by, According to the robot prediction state at the preset time and the actual state of the robot at the corresponding preset time, determining the state prediction reward, comprises: comparing the robot prediction state at the preset time and the actual state of the robot at the preset time to determine the state prediction reward; wherein, the closer the robot prediction state at the preset time and the actual state of the robot at the preset time, the closer the state prediction reward to 1, otherwise the state prediction reward is closer to 0.

5. The robot motion control model training method according to claim 3, wherein According to the state prediction reward and the task reward, updating the parameters of the policy network and the decoding network of the motion control model to be trained, comprises: summing the state prediction reward and the task reward as the training reward; updating the parameters of the policy network and the decoding network of the motion control model to be trained according to the training reward. 6.The robot motion control model training method according to any one of claims 1 to 5, characterized in that, The training hidden variable comprises at least one of the environment characteristics, the robot historical behavior mode and the load capacity of the robot.

7. A robot motion control method characterized by, The method comprises: obtaining the current robot state; based on the current robot state, using the policy network of the trained motion control model to make decisions, obtaining the control instructions and the hidden variables of a preset number of frames in the future; wherein, the motion control model is the trained motion control model of any one of claims 1 to 6. A preset frame control instruction in the control instructions of the preset frame number is selected for controlling the robot, and the remaining frame control instructions and the hidden variable are not processed.

8. A robot motion control model training device characterized by comprising: The method comprises the steps of: A determination module is configured to determine a future preset frame number of training control instructions and training hidden variables by using a policy network of a motion control model to be trained to make a decision based on a current time training robot state, wherein the preset frame number is greater than 1, and the training hidden variable is a random variable that is unobservable but plays a role in the decision and prediction of the model. A prediction module is configured to predict a future preset time robot predicted state by using a decoding network of the motion control model to be trained based on the training hidden variable, wherein the The preset time is determined according to a parameter update period of the motion control model to be trained. An execution module is configured to sequentially execute the preset frame number of training control instructions by using a simulation engine to obtain a robot actual state. An update module is configured to update parameters of the policy network and the decoding network of the motion control model to be trained based on the robot predicted state at the preset time and the robot actual state to obtain a trained motion control model.

9. A robot motion control apparatus characterized by comprising: The method comprises the steps of: A state acquisition module is configured to acquire a current robot state. An instruction generation module is configured to determine a future preset frame number of control instructions and hidden variables by using a policy network of a trained motion control model to make a decision based on the current robot state, wherein the motion control model is the trained motion control model of any one of claims 1 to 6. An instruction selection module is configured to select a preset frame control instruction in the control instructions of the preset frame number for controlling the robot, and not to process the remaining frame control instructions and the hidden variable.

10. An electronic device, comprising: The method comprises the steps of: A memory is configured to store a computer program. A processor is configured to execute the computer program to implement the steps of the method of any one of claims 1 to 7.

11. A computer readable storage medium characterized by, The computer program is stored on the computer readable storage medium, and the computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 7.

12. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instruction is executed by the processor to implement the steps of the method of any one of claims 1 to 7.

Images