Motion control model training method, motion control method, equipment and medium

Abstract

The invention provides a motion control model training method, a motion control method, equipment and a medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring first scanning point information, second scanning point information and first sensing information in an analog simulation environment; predicting first action information by adopting a teacher model according to the first scanning point information and the first perception information; according to the second scanning point information and the first perception information, predicting second action information based on a new strategy by adopting a student model; calculating a first training loss according to the first action information and the second action information; calculating second training loss according to the second action information and third action information, wherein the third action information is action information generated by the student model by adopting an old strategy in the previous training round; and training a strategy of the student model according to the first training loss and the second training loss to obtain a motion control model of the target robot. According to the invention, the performance of the motion control model in a real scene can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a motion control model training method, a motion control method, a device and a medium. Background Art

[0002] In recent years, humanoid robots have attracted much attention in academia and industry due to their high adaptability to the environment and human-like characteristics.

[0003] The development of reinforcement learning and deep learning has brought revolutionary changes to the walking control of humanoid robots, enabling them to go beyond the scope of preset actions and tasks and show more possibilities. Among the many tasks of humanoid robots, motion control is one of the most critical and challenging tasks.

[0004] Existing technologies usually first train a teacher model in a simulated environment, and then use distillation technology to transfer the learned privileged information to the student model in the form of latent features or actions. This method may limit the potential of the student model, so that it can only gradually approach the behavior of the teacher model but cannot surpass it. This is contrary to the core idea of ​​reinforcement learning to learn better strategies through continuous trial and error interaction with the environment, resulting in poor performance in real-world scenarios. Summary of the invention

[0005] The purpose of this application is to provide a motion control model training method, motion control method, device and medium to address the deficiencies in the above-mentioned prior art, so as to optimize the strategy of the student model by combining knowledge distillation and reinforcement learning, and improve the performance of the motion control model in real-world scenarios.

[0006] To achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows: In a first aspect, an embodiment of the present application provides a motion control model training method for a humanoid robot, the method comprising: Acquire first scanning point information, second scanning point information and first perception information of the target robot in the simulation environment, wherein the second scanning point information is information of adding a noise scanning point to the first scanning point information; Predicting the first action information of the target robot at the current moment using a pre-trained teacher model according to the first scanning point information and the first perception information; According to the second scanning point information and the first perception information, using the student model to be trained to predict the second action information of the target robot at the current moment based on a new strategy, wherein the new strategy is the strategy adopted by the student model in the current training round; Calculate a first training loss according to the first action information and the second action information; Calculate a second training loss according to the second action information and the third action information, wherein the third action information is action information generated by the student model using the old strategy in the previous training round; The strategy of the student model is trained according to the first training loss and the second training loss to obtain a motion control model of the target robot.

[0007] Optionally, the obtaining the first scanning point information in the simulation environment includes: Acquire three-dimensional point cloud data of the simulated environment through a sensor; Generating an elevation map of the simulated environment according to the three-dimensional point cloud data, wherein the elevation map is used to represent the terrain of the simulated environment; The elevation map is sampled at intervals to determine first scanning point information, where the first scanning point information includes terrain information of a plurality of scanning points sampled at intervals.

[0008] Optionally, predicting the first action information of the target robot at the current moment using a pre-trained teacher model according to the first scanning point information and the first perception information includes: Generate a first terrain feature, a first historical perception feature, and a first estimated perception feature using a feature extraction module in the teacher model according to the first scanning point information and the historical perception information; Encoding and splicing the first terrain feature, the first historical perception feature, and the first estimated perception feature to generate a first spliced ​​feature; According to the first splicing feature and the current perception information, the strategy module in the teacher model is used to predict the first joint position information, and the first action information includes: the first joint position information.

[0009] Optionally, predicting the second action information of the target robot at the current moment based on a new strategy using a student model to be trained according to the second scanning point information and the first perception information includes: According to the second scanning point information and the historical perception information, using the feature extraction module in the student model, generating a second terrain feature, a second historical perception feature, and a second estimated perception feature; encoding and concatenating the second terrain feature, the second historical perception feature, and the second estimated perception feature to generate a second concatenated feature; According to the second splicing feature and the current perception information, using the strategy module in the student model to predict the second joint position information; According to the current perception information, the evaluation module in the student model is used to determine the value information, and the second action information includes: the second joint position information and the value information.

[0010] Optionally, calculating the second training loss according to the second action information and the third action information includes: Calculating a strategy loss according to the second joint position information and the third joint position information in the third action information; Calculating a value function loss according to the second joint position information and the value information; Calculating an entropy reward according to the current perception information; The second training loss is calculated according to the policy loss, the value function loss and the entropy reward.

[0011] Optionally, calculating the value function loss according to the second joint position information and the value information includes: Calculating at least one reward value according to at least one predefined reward function and the second joint position information; Calculate the reward value of the second joint position information at a future time according to the at least one reward value; The value function loss is calculated according to the reward value and the value information.

[0012] Optionally, the reward function includes at least one of the following: a bipedal motion reward function of the target robot, a joint restriction reward function, a speed reward function, an acceleration reward function, an arm degree of freedom penalty function, a trunk offset reward function, and an instruction reward function.

[0013] In a second aspect, an embodiment of the present application further provides a motion control method for a humanoid robot, which is applied to a controller of the humanoid robot, and the method comprises: Acquire the third scanning point information and the second sensing information of the working environment; Generate action information using a pre-trained motion control model according to the third scanning point information and the second perception information, wherein the motion control model is trained using the motion control model training method for a humanoid robot according to any one of the first aspects; The humanoid robot is controlled to move according to the motion information.

[0014] In a third aspect, an embodiment of the present application provides a motion control model training device for a humanoid robot, the device comprising: A first information acquisition module is used to acquire first scanning point information, second scanning point information and first perception information of the target robot in the simulation environment, wherein the second scanning point information is information of adding noise scanning points to the first scanning point information; a first prediction module, configured to predict first action information of the target robot at a current moment using a pre-trained teacher model according to the first scanning point information and the first perception information; A second prediction module is used to predict the second action information of the target robot at the current moment based on a new strategy using a student model to be trained according to the second scanning point information and the first perception information, wherein the new strategy is a strategy adopted by the student model in a current training round; a loss calculation module, configured to calculate a first training loss according to the first action information and the second action information; The loss calculation module is further used to calculate the second training loss according to the second action information and the third action information, wherein the third action information is the action information generated by the student model using the old strategy in the previous training round; The model training module is used to train the strategy of the student model according to the first training loss and the second training loss to obtain a motion control model of the target robot.

[0015] Optionally, the first information acquisition module is used to acquire three-dimensional point cloud data of the simulated environment through a sensor; generate an elevation map of the simulated environment based on the three-dimensional point cloud data, and the elevation map is used to represent the terrain of the simulated environment; perform interval sampling on the elevation map to determine first scanning point information, and the first scanning point information includes terrain information of multiple scanning points sampled at intervals.

[0016] Optionally, the first prediction module is specifically used to generate a first terrain feature, a first historical perception feature and a first estimated perception feature based on the first scanning point information and historical perception information, using a feature extraction module in the teacher model; encode and splice the first terrain feature, the first historical perception feature and the first estimated perception feature to generate a first spliced ​​feature; and predict the first joint position information based on the first spliced ​​feature and current perception information, using a strategy module in the teacher model, wherein the first action information includes: first joint position information.

[0017] Optionally, the second prediction module is specifically used to generate a second terrain feature, a second historical perception feature and a second estimated perception feature according to the second scanning point information and the historical perception information, using the feature extraction module in the student model; encode and splice the second terrain feature, the second historical perception feature and the second estimated perception feature to generate a second spliced ​​feature; predict the second joint position information according to the second spliced ​​feature and the current perception information, using the strategy module in the student model; determine the value information according to the current perception information, using the evaluation module in the student model, the second action information including: the second joint position information and the value information.

[0018] Optionally, the loss calculation module is specifically used to calculate the strategy loss based on the second joint position information and the third joint position information in the third action information; calculate the value function loss based on the second joint position information and the value information; calculate the entropy reward based on the current perception information; calculate the second training loss based on the strategy loss, the value function loss and the entropy reward.

[0019] Optionally, the loss calculation module is also used to calculate at least one reward value based on at least one predefined reward function and the second joint position information; calculate the reward value of the second joint position information at a future time based on the at least one reward value; and calculate the value function loss based on the reward value and the value information.

[0020] Optionally, the reward function includes at least one of the following: a bipedal motion reward function of the target robot, a joint restriction reward function, a speed reward function, an acceleration reward function, an arm degree of freedom penalty function, a trunk offset reward function, and an instruction reward function.

[0021] In a fourth aspect, an embodiment of the present application further provides a motion control device for a humanoid robot, which is applied to a controller of the humanoid robot, and the device comprises: A second information acquisition module, used to acquire third scanning point information and second perception information of the working environment; a third prediction module, configured to generate motion information according to the third scanning point information and the second perception information by using a pre-trained motion control model, wherein the motion control model is trained by using the motion control model training method for a humanoid robot according to any one of the first aspects; A control module is used to control the humanoid robot to move according to the motion information.

[0022] In a fifth aspect, an embodiment of the present application further provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores program instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium through the bus, and the processor executes the program instructions to perform the steps of the motion control model training method for a humanoid robot as described in any one of the first aspect, or the steps of the motion control method for a humanoid robot as described in the second aspect.

[0023] In a sixth aspect, an embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the motion control model training method for a humanoid robot as described in any one of the first aspect, or the steps of the motion control method for a humanoid robot as described in the second aspect are executed.

[0024] The beneficial effects of this application are: The motion control model training method for a humanoid robot provided in the above embodiment optimizes the strategy of the student model by combining the distillation loss between the teacher model and the student model, and the strategy optimization loss of the student model for strategy adjustment in adjacent training rounds. This ensures that the student model is not only able to approximate the teacher model. By adding reinforcement learning, it can ensure that the student model generates better strategies, thereby improving the performance of the motion control model in real scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 Schematic diagram of the process of the motion control model training method for a humanoid robot provided in an embodiment of the present application Figure 1 ; Figure 2 Schematic diagram of the process of the motion control model training method for a humanoid robot provided in an embodiment of the present application Figure 2 ; Figure 3 A diagram of the student model training architecture provided in an embodiment of the present application; Figure 4 Schematic diagram of the process of the motion control model training method for a humanoid robot provided in an embodiment of the present application Figure 3 ; Figure 5Schematic diagram of the process of the motion control model training method for a humanoid robot provided in an embodiment of the present application Figure 4 ; Figure 6 A schematic diagram of a flow chart of a motion control method for a humanoid robot provided in an embodiment of the present application; Figure 7 A schematic diagram of the structure of a motion control model training device for a humanoid robot provided in an embodiment of the present application; Figure 8 A schematic diagram of the structure of a motion control device for a humanoid robot provided in an embodiment of the present application; Fig. 9 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0028] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0029] In addition, the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] It should be noted that, in the absence of conflict, the features in the embodiments of the present application may be combined with each other.

[0031] The motion control model training method for a humanoid robot provided in an embodiment of the present application may be implemented by a computer device as the electronic device used. The motion control method for a humanoid robot provided in an embodiment of the present application may be implemented by a controller in the humanoid robot as the electronic device used.

[0032] Figure 1 Schematic diagram of the process of the motion control model training method for a humanoid robot provided in an embodiment of the present application Figure 1 ,like Figure 1 As shown, the method may include: S101, obtaining first scanning point information, second scanning point information and first perception information of a target robot in a simulation environment, wherein the first scanning point information is information including noise scanning points.

[0033] In this embodiment, a simulation environment is generated by a simulator. The simulation environment may have uneven terrain, such as slopes, steps, etc. The uneven terrain of the model simulation environment is sampled at intervals to obtain scanning information of multiple first scanning points. The scanning information of the first scanning points may include: height information of the multiple first scanning points. Based on the height information of the multiple first scanning points, the geometric shape of the terrain of the simulation environment can be represented.

[0034] In order to ensure that the motion control model trained in a simulation environment can bridge the gap between simulation and display, it is necessary to add noise scanning points on the basis of the first scanning points to generate second scanning point information containing the first scanning points and the noise scanning points, wherein the noise scanning points can be generated randomly or by mixed weighted generation based on the first scanning points, and this embodiment does not impose any restrictions on this.

[0035] The first perception information of the target robot in the simulation environment includes perception state information generated by the target robot based on the perceived environment information. The perception state information may include: gait information, posture information, joint information and action information of the target robot.

[0036] In some embodiments, the first perception information may include perception information of the target robot in a simulated environment collected during the training of the teacher model, which is not limited in this embodiment.

[0037] S102: predicting the first action information of the target robot at the current moment using a pre-trained teacher model according to the first scanning point information and the first perception information.

[0038] In this embodiment, the pre-trained teacher model can be a model trained using scanning point information and perception information that does not contain noise scanning points. During the training of the teacher model, the action reward can be calculated according to the Markov decision process (MDP), and the action generation strategy of the teacher model can be optimized based on the action reward.

[0039] The first perception information at least includes the perception information at the current moment. The first scanning point information and the first perception information are input into the teacher model. The teacher model predicts the first action information of the target robot at the current moment based on the first scanning point information and the first perception information. The first action information is the action of the target robot at the current moment determined based on the perception state information of the target robot in the simulated environment at the current moment. After the target robot controls the limbs to move based on the first action information, the perception information will be updated to the perception information after the movement.

[0040] S103. According to the second scanning point and the first perception information, the student model to be trained is used to predict the second action information of the target robot at the current moment based on the new strategy, where the new strategy is the strategy adopted by the student model in the current training round.

[0041] In this embodiment, after the teacher model training is completed, the model weights of the teacher model are transferred to the student model to be trained through knowledge distillation, so that the student model to be trained inherits the model weights of the teacher model.

[0042] It should be noted that since the scale of the student model is much smaller than that of the teacher model, the model weights of the teacher model inherited by the student model are not exactly the same weights, but the weights after knowledge distillation. That is, even if the student model inherits the model weights of the teacher model, it is not equivalent to the teacher model.

[0043] The second scanning point information and the first perception information are simultaneously input into the student model to be trained, and the student model to be trained predicts the second action information of the target robot at the current moment based on the new strategy.

[0044] Among them, the new strategy is the action generation strategy of the student model to be trained in the current training round among multiple training rounds. The training process of the student model is actually the process of adjusting the action generation strategy of the student model, so that the student model can output action information with maximum cumulative reward based on scanning point information and perception information. The student model to be trained optimizes the old strategy adopted in the previous training round in the previous training round to obtain a new strategy.

[0045] S104. Calculate a first training loss according to the first action information and the second action information.

[0046] In this embodiment, the distillation loss between the teacher model and the student model is determined according to the first action information output by the teacher model and the second action information output by the student model. The distillation loss is the loss caused by distilling the model weights of the teacher model to the student model.

[0047] In some embodiments, the first training loss may be determined based on a mean square error between the first action information and the second action information.

[0048] S105. Calculate a second training loss according to the second action information and the third action information, where the third action information is the action information generated by the student model in the previous training round using the old strategy.

[0049] In this embodiment, reinforcement learning is used to optimize the model strategy so that the model outputs the action with the maximum cumulative reward. The third action information is the action information generated by the student model using the old strategy in the previous training round. In the previous training round, the distillation loss and the strategy optimization loss are calculated based on the third action information to optimize the action generation strategy of the student model and obtain the new strategy for the current training round.

[0050] According to the second action information of the current training round and the third action information of the previous training round, the proximal policy optimization (PPO) loss of the student model during the policy optimization process is determined.

[0051] The proximal policy optimization loss can limit the policy updates of the action generation model during the training of the student model, as well as the role of reinforcement learning rewards in the policy optimization of the student model.

[0052] S106. According to the first training loss and the second training loss, the strategy of the student model is trained to obtain a motion control model of the target robot.

[0053] In this embodiment, the first training loss and the second training loss are weighted, and the strategy of the student model is trained according to the weighted training loss. When the training rounds reach a preset number of rounds or the weighted training loss converges, the training of the student model is completed to obtain the motion control model of the target robot.

[0054] The motion control model training method for a humanoid robot provided in the above embodiment optimizes the strategy of the student model by combining the distillation loss between the teacher model and the student model, and the strategy optimization loss of the student model for strategy adjustment in adjacent training rounds. This ensures that the student model is not only able to approximate the teacher model. By adding reinforcement learning, it can ensure that the student model generates better strategies, thereby improving the performance of the motion control model in real scenarios.

[0055] In a possible implementation, the process of obtaining the first scanning point information in the simulation environment in S101 may include: Acquire three-dimensional point cloud data of a simulated environment through a sensor; generate an elevation map of the simulated environment based on the three-dimensional point cloud data, the elevation map being used to represent the terrain of the simulated environment; perform interval sampling on the elevation map to determine first scanning point information, the first scanning point information including terrain information of multiple scanning points sampled at intervals.

[0056] In this embodiment, the sensor may be, for example, a laser radar, which acquires three-dimensional point cloud data of a simulated environment through the sensor, renders the terrain of the simulated environment based on the three-dimensional point cloud data, and generates an elevation map of the simulated environment. The elevation map records the ground height of each point in the simulated environment, and the ground height is the ground height centered on the target robot.

[0057] By performing interval sampling on the elevation map, multiple first scanning points can be obtained. The first scanning point information is the terrain information of the multiple first scanning points, that is, the location height information. For example, the elevation map can be compressed into scanning points, and multiple first scanning points can be obtained with a preset distance as the sampling interval.

[0058] Furthermore, since the three-dimensional point cloud data collected by the sensor contains noise, a Kalman filter can be used to update the height information of the scanning point.

[0059] For example, the update formula can be expressed as:

[0060] in, is the estimated variance of the height of each cell in the elevation map, is the noise model variance of the sensor, the noise model , d is the distance between the scanning point and the sensor, is a hyperparameter, is the height measurement.

[0061] The motion control model training method for a humanoid robot provided in the above embodiment generates scanning point information based on an elevation map, so that the trained motion control model can control the motion of the humanoid robot in various complex environments, thereby improving the performance of the motion control model in a real environment.

[0062] In one possible implementation, Figure 2 Schematic diagram of the process of the motion control model training method for a humanoid robot provided in an embodiment of the present application Figure 2 ,like Figure 2 As shown, the process of predicting the first action information of the target robot at the current moment using the pre-trained teacher model according to the first scanning point information and the first perception information in S102 may include: S201. According to the first scanning point information and the historical perception information, a feature extraction module in the teacher model is used to generate a first terrain feature, a first historical perception feature and a first estimated perception feature.

[0063] S202: Encode and concatenate the first terrain feature, the first historical perception feature, and the first estimated perception feature to generate a first concatenated feature.

[0064] S203. According to the first splicing feature and the current perception information, the strategy module in the teacher model is used to predict the first joint position information. The first action information includes: the first joint position information.

[0065] In this embodiment, for example, Figure 3 The student model training architecture diagram provided in the embodiment of the present application is as follows: Figure 3 As shown, the teacher model may include: a feature extraction module, a strategy module and an evaluation module, wherein the evaluation model only calculates the state value of the robot when training the teacher model to optimize the strategy of the strategy module. After the model training is completed, the evaluation model is no longer used.

[0066] The feature extraction model of the teacher model extracts features from the first scanning point information to obtain a first terrain feature. For example, a one-bit convolution can be used to compress the first scanning point information to a latent space of a target dimension to obtain a first terrain feature.

[0067] The feature extraction module of the teacher model also extracts features from the historical perception information of a preset number of frames in the first perception information to obtain a first historical perception feature and a first estimated perception feature at the current moment estimated based on the historical perception information. For example, an external encoder can be used to compress the historical perception information into a latent space of a target dimension to obtain the first historical perception feature and the first estimated perception feature.

[0068] The first terrain feature, the first historical perception feature and the first estimated perception feature are encoded to obtain a first scanning point vector, a first historical perception vector and a first estimated perception vector. The first scanning point vector, the first historical perception vector and the first estimated perception vector are spliced ​​to generate a first splicing vector, which is a historical latent vector.

[0069] The first perception information also includes the current perception information at the current moment. According to the perception latent vector and the first splicing vector of the current perception information, the strategy module in the teacher model is used to perform prediction to generate the first joint position information of the target robot. The first joint position information is used to represent the action to be performed by the target robot based on the current perception information.

[0070] In one possible implementation, Figure 4Schematic diagram of the process of the motion control model training method for a humanoid robot provided in an embodiment of the present application Figure 3 ,like Figure 4 As shown, S103, according to the second scanning point and the first perception information, the process of using the student model to be trained to predict the second action information of the target robot at the current moment based on the new strategy may include: S301. According to the second scanning point information and the historical perception information, a feature extraction module in the student model is used to generate a second terrain feature, a second historical perception feature and a second estimated perception feature.

[0071] S302: Encode and concatenate the second terrain feature, the second historical perception feature, and the second estimated perception feature to generate a second concatenated feature.

[0072] S303. According to the second splicing feature and the current perception information, the strategy module in the student model is used to predict the second joint position information.

[0073] S304. According to the current perception information, the evaluation module in the student model is used to determine the value information. The second action information includes: second joint position information and value information.

[0074] In this embodiment, if Figure 3 As shown, the student model adopts a structure corresponding to the teacher model, and also includes a feature extraction module, a strategy module and an evaluation module. The feature extraction module of the student model performs feature extraction, encoding and splicing of the second scanning point information and the historical perception information, which is the same as the feature extraction module of the teacher model performs feature extraction, encoding and splicing of the first scanning point information and the historical perception information, and will not be elaborated here.

[0075] The current state information is used to refer to the current state s of the target robot. According to the second splicing feature and the current perception information, the strategy module in the student model is used to predict the action of the target robot at the current moment, that is, the second joint position information. According to the current perception information, the evaluation module in the student model is used to evaluate the current state s of the target robot to determine the value information of the current state.

[0076] In some embodiments, the process of calculating the first training loss according to the first action information and the second action information in S104 may include: A first training loss is calculated based on the first joint position information and the second joint position information.

[0077] In other embodiments, Figure 5 Schematic diagram of the process of the motion control model training method for a humanoid robot provided in an embodiment of the present application Figure 4 ,like Figure 5As shown, the process of calculating the second training loss according to the second action information and the third action information in S105 may include: S401. Calculate the strategy loss according to the second joint position information and the third joint position information in the third action information.

[0078] In this embodiment, the strategy module of the student model adopts a new strategy to generate multiple joint position information based on the second splicing feature and the current perception information. The multiple joint position information is the joint position information of the target robot generated by the student module for multiple actions that may be adopted. According to the mean and variance of the multiple joint position information, the Gaussian distribution probability of each joint position information is calculated, and the joint position information with the largest Gaussian distribution probability is determined to be the second joint position information.

[0079] The student model uses the same scheme in the previous round of training to determine the third joint position information generated by the old strategy.

[0080] The strategy loss is calculated based on the ratio of the Gaussian distribution probability of the second joint position information to the Gaussian distribution probability of the third joint position information. .

[0081] In some embodiments, the probability ratio of the new and old strategies is calculated based on the ratio of the Gaussian distribution probability of the second joint position information and the Gaussian distribution probability of the third joint position information, and the strategy loss is calculated based on the probability ratio combined with the advantage function of executing the action corresponding to the second joint position information in the current state s and the preset clipping parameters. .

[0082] For example, the first strategy loss is calculated according to the product of the probability ratio and the advantage function, the clipping probability ratio is calculated using the truncation function according to the probability ratio and the clipping parameter, the second strategy loss is calculated according to the product of the clipping probability ratio and the advantage function, and the minimum value of the first strategy loss and the second strategy loss is determined as the strategy loss .

[0083] Among them, the clipping parameters Used to limit the probability ratio to [1- , 1+ ] to avoid policy updates that are too large.

[0084] In some embodiments, based on the reward value and discount factor at the current moment determined based on the second joint position information, the reward value of using the second joint position information to perform an action from the current moment to a preset future moment, that is, the discounted cumulative value of the reward, is determined, and the advantage function is calculated based on the difference between the discounted cumulative value and the value information.

[0085] S402. Calculate the value function loss based on the second joint position information and value information.

[0086] In this embodiment, according to the reward value and discount factor at the current moment determined based on the second joint position information, the reward value of the action from the current moment to the preset future moment using the second joint position information is determined, that is, the discounted cumulative value of the reward, and the value function loss is calculated according to the mean square error of the value information minus the discounted cumulative value. .

[0087] S403: Calculate entropy reward based on current perception information.

[0088] In this embodiment, the entropy reward is calculated based on the Gaussian distribution probability and logarithmic probability of the student model outputting multiple joint position information under the current perception information. .

[0089] S404. Calculate the second training loss according to the strategy loss, the value function loss and the entropy reward.

[0090] In this example, the strategy loss Subtract the value function loss Plus the entropy bonus , and get the second training loss.

[0091] For example, the formula for calculating the second training loss can be expressed as:

[0092] Among them, c1 and c2 are the balance value function losses and Entropy Reward The coefficient of .

[0093] The above embodiment provides a motion control model training method for a humanoid robot. The strategy loss term ensures that the new strategy does not deviate too much from the old strategy through the clipping coefficient. The value function loss minimizes the difference between the estimated value and the actual return. The entropy reward explores more by maximizing the entropy estimation strategy of the strategy distribution. The student model is trained by the second training loss composed of these three data. The stability and efficiency of the student model training can be improved, the model training can be avoided from converging too early, and the model training effect can be ensured.

[0094] In a possible implementation, the process of calculating the value function loss according to the second joint position information and the value information in S402 may include: According to at least one predefined reward function and the second joint position information, at least one reward value is calculated; according to the at least one reward value, a reward value of the second joint position information at a future time is calculated; according to the reward value and the value information, a value function loss is calculated.

[0095] In this embodiment, at least one reward function is predefined, and the reward function is used to calculate the reward value of the target robot using the second joint position information to perform an action in one dimension. According to at least one reward value of at least one reward function, the total reward value at the current moment is determined. According to the total reward value and the discount factor at multiple future moments, the total reward value of using the second joint position information to perform the action from the current moment to multiple future moments is determined, that is, the discounted cumulative value of the reward. According to the mean square error of the value information minus the discounted cumulative value, the value function loss is calculated. .

[0096] In some embodiments, the reward function includes at least one of the following: a bipedal motion reward function of the target robot, a joint limitation reward function, a velocity reward function, an acceleration reward function, an arm degree of freedom penalty function, a torso offset reward function, and an instruction reward function.

[0097] In this embodiment, the bipedal motion reward function includes a swing phase reward and a support phase reward during bipedal motion. The swing phase is the stage when the foot moves in the air, and the support phase is the stage when the foot touches the ground. The reward function of each foot is a weighted function of the product of the phase indicator of the swing phase and the reward function of the swing phase, and the product of the phase indicator of the support phase and the reward function of the support phase. The reward function of the swing phase is calculated based on the square of the foot velocity, and the reward function of the support phase is calculated based on the square of the normal force of the foot.

[0098] The joint restriction reward function is used to limit the joint freedom of the target robot, ensure that the movement is within the physical capabilities of the target robot, and prevent actions that may damage the robot's mechanical structure or exceed the robot's operating range. The joint restriction reward function can be calculated based on the joint angle in the second joint position information, the preset minimum joint angle, and the preset maximum joint angle.

[0099] The speed reward function is used to ensure that the target robot maintains the optimal movement speed, and the acceleration reward function is used to ensure that the target robot accelerates and decelerates smoothly, thereby ensuring the stability of the target robot's movement.

[0100] The arm freedom penalty function is used to suppress the robot arm's transitional motion, and the torso excursion reward function is used to maintain the target robot's balance and correct motion direction.

[0101] The command reward function is used to encourage the target robot to move in the preset direction. The command reward function is calculated based on the robot's current movement speed and expected speed, as well as the reward weights in each direction.

[0102] The motion control model training method for a humanoid robot provided in the above embodiment calculates reward values ​​based on multiple reward functions, and calculates value function losses based on the reward values, so that when the student model is trained based on the value function loss, the student model can adjust the strategy to the optimal level to output the best motion information to control the motion of the target robot.

[0103] Based on the motion control model training method for a humanoid robot provided in the above-mentioned embodiment, an embodiment of the present application further provides a motion control method for a humanoid robot, which is applied to a controller of the humanoid robot.

[0104] Figure 6 A schematic diagram of a flow chart of a motion control method for a humanoid robot provided in an embodiment of the present application, such as Figure 6 As shown, the method may include: S501, obtaining the third scanning point information and the second perception information of the working environment.

[0105] S502: Generate action information using a pre-trained motion control model according to the third scanning point information and the second perception information.

[0106] S503: Control the humanoid robot to move according to the motion information.

[0107] In this embodiment, the third scanning point information of the working environment can be determined by scanning the working environment through a laser radar installed on the robot, and the second perception information can be obtained through various sensors installed on the robot. The type of sensor is determined by the type of information contained in the second perception information, and no limitation is given here.

[0108] The motion control model trained by the motion control model training method for a humanoid robot is deployed in the controller of the humanoid robot. The controller generates action information using the motion control model based on the third scanning point information and the second perception information, and controls the movement of the limbs of the humanoid robot according to the action information.

[0109] The motion control method of the humanoid robot provided in the above embodiment uses a trained motion control model to accurately control the motion of the humanoid robot and improve the performance of the humanoid robot in the working environment. The humanoid robot can operate in a working environment with complex terrain.

[0110] Based on the above method embodiments, an embodiment of the present application provides a motion control model training device for a humanoid robot. Figure 7 A schematic diagram of the structure of a motion control model training device for a humanoid robot provided in an embodiment of the present application, such as Figure 7 As shown, the device may include: The first information acquisition module 601 is used to acquire first scanning point information, second scanning point information and first perception information of the target robot in the simulation environment, wherein the second scanning point information is information of adding noise scanning points to the first scanning point information; A first prediction module 602 is used to predict the first action information of the target robot at the current moment using a pre-trained teacher model according to the first scanning point information and the first perception information; The second prediction module 603 is used to predict the second action information of the target robot at the current moment based on the new strategy using the student model to be trained according to the second scanning point information and the first perception information, where the new strategy is the strategy adopted by the student model in the current training round; A loss calculation module 604, configured to calculate a first training loss according to the first action information and the second action information; The loss calculation module 604 is further used to calculate the second training loss according to the second action information and the third action information, where the third action information is the action information generated by the student model using the old strategy in the previous training round; The model training module 605 is used to train the strategy of the student model according to the first training loss and the second training loss to obtain the motion control model of the target robot.

[0111] Optionally, the first information acquisition module 601 is used to acquire three-dimensional point cloud data of a simulated environment through a sensor; generate an elevation map of the simulated environment based on the three-dimensional point cloud data, the elevation map is used to represent the terrain of the simulated environment; perform interval sampling on the elevation map to determine first scanning point information, the first scanning point information includes terrain information of multiple scanning points sampled at intervals.

[0112] Optionally, the first prediction module 602 is specifically used to generate a first terrain feature, a first historical perception feature and a first estimated perception feature based on the first scanning point information and the historical perception information, using a feature extraction module in the teacher model; encode and splice the first terrain feature, the first historical perception feature and the first estimated perception feature to generate a first spliced ​​feature; and predict the first joint position information based on the first spliced ​​feature and the current perception information, using a strategy module in the teacher model. The first action information includes: the first joint position information.

[0113] Optionally, the second prediction module 603 is specifically used to generate a second terrain feature, a second historical perception feature and a second estimated perception feature according to the second scanning point information and the historical perception information, using a feature extraction module in the student model; encode and splice the second terrain feature, the second historical perception feature and the second estimated perception feature to generate a second spliced ​​feature; predict the second joint position information according to the second spliced ​​feature and the current perception information, using a strategy module in the student model; and determine the value information according to the current perception information, using an evaluation module in the student model, the second action information including: second joint position information and value information.

[0114] Optionally, the loss calculation module 604 is specifically used to calculate the strategy loss based on the second joint position information and the third joint position information in the third action information; calculate the value function loss based on the second joint position information and the value information; calculate the entropy reward based on the current perception information; calculate the second training loss based on the strategy loss, the value function loss and the entropy reward.

[0115] Optionally, the loss calculation module 604 is also used to calculate at least one reward value based on at least one predefined reward function and the second joint position information; calculate the reward value of the second joint position information at a future time based on the at least one reward value; and calculate the value function loss based on the reward value and the value information.

[0116] Optionally, the reward function includes at least one of the following: a bipedal motion reward function of the target robot, a joint restriction reward function, a speed reward function, an acceleration reward function, an arm degree of freedom penalty function, a torso offset reward function, and an instruction reward function.

[0117] On the basis of the above method embodiments, the embodiments of the present application further provide a motion control device for a humanoid robot, which is applied to a controller of the humanoid robot. Figure 8 A schematic diagram of the structure of the motion control device of the humanoid robot provided in the embodiment of the present application, as shown in FIG. Figure 8 As shown, the device may include: The second information acquisition module 701 is used to acquire the third scanning point information and the second perception information of the working environment; A third prediction module 702 is used to generate action information according to the third scanning point information and the second perception information by using a pre-trained motion control model, where the motion control model is trained by using the motion control model training method for a humanoid robot according to any one of the first aspects; The control module 703 is used to control the humanoid robot to move according to the motion information.

[0118] The above-mentioned device is used to execute the method provided by the aforementioned embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0119] The above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASICs), or one or more microprocessors, or one or more field programmable gate arrays (FPGAs). For another example, when a module is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0120] Fig. 9 A schematic diagram of an electronic device provided in an embodiment of the present application, the electronic device 800 may include: a processor 801, a storage medium 802 and a bus, the storage medium 802 stores program instructions executable by the processor 801, when the electronic device 800 is running, the processor 801 communicates with the storage medium 802 via the bus, and the processor 801 executes the program instructions to execute the above method embodiment. The specific implementation method and technical effect are similar and will not be repeated here.

[0121] Optionally, the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the above method embodiment is executed.

[0122] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0123] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0124] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0125] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (English: Read-Only Memory, referred to as: ROM), random access memory (English: Random Access Memory, referred to as: RAM), disk or optical disk and other media that can store program codes.

[0126] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A motion control model training method for a humanoid robot, characterized in that: The method comprises: Acquire first scanning point information, second scanning point information and first perception information of the target robot in the simulation environment, wherein the second scanning point information is information of adding a noise scanning point to the first scanning point information; Predicting the first action information of the target robot at the current moment using a pre-trained teacher model according to the first scanning point information and the first perception information; According to the second scanning point information and the first perception information, using the student model to be trained to predict the second action information of the target robot at the current moment based on a new strategy, wherein the new strategy is the strategy adopted by the student model in the current training round; Calculate a first training loss according to the first action information and the second action information; Calculate a second training loss according to the second action information and the third action information, wherein the third action information is action information generated by the student model using the old strategy in the previous training round; The strategy of the student model is trained according to the first training loss and the second training loss to obtain a motion control model of the target robot.

2. The method according to claim 1, characterized in that The obtaining of the first scanning point information in the simulation environment includes: Acquire three-dimensional point cloud data of the simulated environment through a sensor; Generating an elevation map of the simulated environment according to the three-dimensional point cloud data, wherein the elevation map is used to represent the terrain of the simulated environment; The elevation map is sampled at intervals to determine first scanning point information, where the first scanning point information includes terrain information of a plurality of scanning points sampled at intervals.

3. The method according to claim 1, characterized in that The step of using a pre-trained teacher model to predict the first action information of the target robot at the current moment according to the first scanning point information and the first perception information includes: Generate a first terrain feature, a first historical perception feature, and a first estimated perception feature using a feature extraction module in the teacher model according to the first scanning point information and the historical perception information; Encoding and splicing the first terrain feature, the first historical perception feature, and the first estimated perception feature to generate a first spliced ​​feature; According to the first splicing feature and the current perception information, the strategy module in the teacher model is used to predict the first joint position information, and the first action information includes: the first joint position information.

4. The method according to claim 1, characterized in that The method of predicting the second action information of the target robot at the current moment based on the new strategy using the student model to be trained according to the second scanning point information and the first perception information includes: According to the second scanning point information and the historical perception information, using the feature extraction module in the student model, generating a second terrain feature, a second historical perception feature, and a second estimated perception feature; encoding and concatenating the second terrain feature, the second historical perception feature, and the second estimated perception feature to generate a second concatenated feature; According to the second splicing feature and the current perception information, using the strategy module in the student model to predict the second joint position information; According to the current perception information, the evaluation module in the student model is used to determine the value information, and the second action information includes: the second joint position information and the value information.

5. The method according to claim 4, characterized in that The calculating the second training loss according to the second action information and the third action information includes: Calculating a strategy loss according to the second joint position information and the third joint position information in the third action information; Calculating a value function loss according to the second joint position information and the value information; Calculating an entropy reward according to the current perception information; The second training loss is calculated according to the policy loss, the value function loss and the entropy reward.

6. The method according to claim 5, characterized in that The calculating the value function loss according to the second joint position information and the value information includes: Calculating at least one reward value according to at least one predefined reward function and the second joint position information; Calculate the reward value of the second joint position information at a future time according to the at least one reward value; The value function loss is calculated according to the reward value and the value information.

7. The method according to claim 6, characterized in that The reward function includes at least one of the following: a bipedal motion reward function, a joint restriction reward function, a speed reward function, an acceleration reward function, an arm degree of freedom penalty function, a trunk offset reward function, and an instruction reward function of the target robot.

8. A motion control method for a humanoid robot, characterized in that: The controller applied to the humanoid robot comprises: Acquire the third scanning point information and the second sensing information of the working environment; Generate action information using a pre-trained motion control model according to the third scanning point information and the second sensing information, wherein the motion control model is trained using the motion control model training method for a humanoid robot according to any one of claims 1 to 7; The humanoid robot is controlled to move according to the motion information.

9. An electronic device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores program instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the program instructions to perform the steps of the motion control model training method for a humanoid robot as described in any one of claims 1 to 7, or the steps of the motion control method for a humanoid robot as described in claim 8.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, executes the steps of the motion control model training method for a humanoid robot as described in any one of claims 1 to 7, or the steps of the motion control method for a humanoid robot as described in claim 8.

Images