A Humanoid Robot Motion Training Method and Device Based on Real Scenarios

Abstract

The present application discloses a humanoid robot motion training method and device based on a real scenario, which are used to improve the performance of a humanoid robot in a real scenario. A world model for training the humanoid robot is constructed; an initial motion control strategy is generated for the humanoid robot; the initial motion control strategy of the humanoid robot is trained in a simulation environment; the humanoid robot is made to enter the real world to execute real-world scenario motions; real scenario motion data of the humanoid robot during the real-world scenario motion is collected in real time through sensors; the real scenario motion data and the real-time motion actions determined by the initial motion control strategy are input into the world model in real time for interactive training to generate the simulated state of the robot at the next moment; an error analysis is performed on the simulated state of the robot at the next moment and the real state of the robot at the corresponding moment in the real scenario motion data to generate a total prediction error; and the initial motion control strategy is updated according to the total prediction error.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of robot control, and in particular, to a method and device for training the motion of a humanoid robot based on a real scenario. Background Art

[0002] With the continuous development of humanoid robot technology, the application fields of humanoid robots have gradually expanded, and the complexity has been increasing. For example, rescue tasks in complex environments, industrial production tasks with fine operations, etc. all require humanoid robots to be able to stably and efficiently execute motion control instructions in real scenarios.

[0003] However, the main problem currently faced is that there is a large gap between the simulated training environment (simulation environment) and the real scenario, that is, the sim2real gap problem. For example, in a rescue scenario, a humanoid robot needs to move flexibly in a disaster scene full of uncertainties, avoid obstacles and accurately reach the target position. However, for existing robots based on simulated training, after entering the real scenario, due to the differences between the environment and the simulated training environment, the motion control strategies of humanoid robots are often less effective or even ineffective, and they cannot stably execute tasks. Secondly, in an industrial production scenario, a humanoid robot needs to accurately complete operations such as the assembly of parts. The strategies trained in the simulation environment are difficult to ensure stable motion accuracy in the real production line due to factors such as equipment vibration and light changes.

[0004] In summary, when the current humanoid robots are put into new real scenarios to execute tasks, the differences between the real scenario and the simulated training scenario lead to a reduction in the effect of the motion strategy, thereby reducing the performance of the humanoid robot in the real scenario. Summary of the Invention

[0005] The present application discloses a method and device for training the motion of a humanoid robot based on a real scenario, which is used to improve the performance of the humanoid robot in the real scenario.

[0006] The specific task is to design a technical solution for training the motion control of a humanoid robot that can solve the sim2real gap. Technically, it is required to use a world model and online learning to quickly fine-tune the motion control strategy on the real machine, so that the motion control of the robot in the real scenario is more stable and meets the requirements of the actual business scenario.

[0007] The first aspect of the present application discloses a control method for a humanoid robot to walk with straight knees, including:

[0008] Construct a world model for training the humanoid robot, where the world model is used to simulate the physical characteristics of the real world and the motion laws of the humanoid robot;

[0009] Generate an initial motion control strategy for the humanoid robot according to the target training task and the structural parameters of the humanoid robot;

[0010] Train the initial motion control strategy of the humanoid robot in a simulation environment;

[0011] Let the humanoid robot enter the real world to execute real-world scenario motions;

[0012] Collect real-world scenario motion data of the humanoid robot in real time during the real-world scenario motion through sensors;

[0013] Input the real-world scenario motion data and the real-time motion actions determined by the initial motion control strategy into the world model for interactive training in real time to generate the robot simulation state at the next moment;

[0014] Perform error analysis on the robot simulation state at the next moment and the real robot state at the corresponding moment in the real-world scenario motion data to generate the total prediction error;

[0015] Update the initial motion control strategy according to the total prediction error.

[0016] Optionally, the steps of training the initial motion control strategy of the humanoid robot in a simulation environment include:

[0017] Obtain the state information of the humanoid robot at the first moment and the operation environment information at the first moment;

[0018] Determine the control instruction at the next moment from the initial motion control strategy according to the state information at the first moment and the operation environment information at the first moment;

[0019] Control the humanoid robot to move through the control instruction;

[0020] Generate motion evaluation information for the humanoid robot after the motion is completed;

[0021] Adjust the initial motion control strategy according to the motion evaluation information and the preset reward function.

[0022] Optionally, the reward function includes a navigation task reward function and a balance posture control task reward function;

[0023] The steps of adjusting the initial motion control strategy according to the motion evaluation information and the preset reward function include:

[0024] Calculate the navigation task reward according to the distance data, obstacle handling data, and action stability data in the motion evaluation information and the preset navigation task reward function;

[0025] Calculate the balance posture control task reward according to the vertical pose data, angular velocity data, and action stability data in the motion evaluation information and the preset balance posture control task reward function;

[0026] Optimize the initial motion control strategy according to the navigation task reward and the balanced attitude control task reward.

[0027] Optionally, the steps of performing error analysis on the robot simulation state at the next moment and the real robot state at the corresponding moment in the real - world motion data to generate the total prediction error include:

[0028] Determine the speed data of each key joint and the pose data of each end - effector from the robot simulation state at the next moment and the real robot state at the corresponding moment in the real - world motion data;

[0029] Generate joint error weights for each key joint of the humanoid robot according to the target training task;

[0030] Determine the adjacent connection weights of each key joint according to the real - time motion actions;

[0031] Generate key joint data weights for the speed data of each key joint according to the joint error weights and the adjacent connection weights;

[0032] Generate pose weights for each end - effector of the humanoid robot according to the target training task, and bind the pose weights to the pose data of each corresponding end - effector;

[0033] Perform error analysis according to the key joint data weights, pose weights, the robot simulation state at the next moment and the real robot state at the corresponding moment in the real - world motion data to generate the total prediction error.

[0034] Optionally, the steps of updating the initial motion control strategy according to the total prediction error include:

[0035] Determine the key joint errors from the total prediction error;

[0036] Update the strategy weights corresponding to the key joints in the initial motion control strategy according to the key joint errors;

[0037] Update all the strategy weights in the initial motion control strategy according to the total prediction error.

[0038] Optionally, the steps of constructing a world model for training a humanoid robot include:

[0039] Collect the empirical motion data of the humanoid robot in the simulation world and the real world;

[0040] Construct a world model for training a humanoid robot from the empirical motion data.

[0041] Optionally, after the step of collecting real - world motion data of the humanoid robot in real - world scenarios in real - time through sensors, and before the step of inputting the real - world motion data into the world model for interactive training to generate the robot simulation state at the next moment, the humanoid robot motion training method further includes:

[0042] Denoise the collected real - world motion data;

[0043] Map the values of real - world motion data collected by different sensors into a unified interval.

[0044] The second aspect of this application discloses a humanoid robot motion training device based on a real - world scenario, including:

[0045] A construction unit for constructing a world model for training the humanoid robot, where the world model is used to simulate the physical characteristics of the real world and the motion laws of the humanoid robot;

[0046] A first generation unit for generating an initial motion control strategy for the humanoid robot according to the target training task and the structural parameters of the humanoid robot;

[0047] A training unit for training the initial motion control strategy of the humanoid robot in a simulated environment;

[0048] An execution unit for making the humanoid robot enter the real world to perform real - world scenario motions;

[0049] A collection unit for collecting real - world motion data of the humanoid robot in real - world scenarios in real - time through sensors;

[0050] A second generation unit for inputting the real - world motion data and the real - time motion actions determined by the initial motion control strategy into the world model for interactive training to generate the robot simulation state at the next moment;

[0051] A third generation unit for performing error analysis on the robot simulation state at the next moment and the real robot state at the corresponding moment in the real - world motion data to generate a total prediction error;

[0052] An update unit for updating the initial motion control strategy according to the total prediction error.

[0053] Optionally, the training unit includes:

[0054] An acquisition module for acquiring the state information of the humanoid robot at the first moment and the operating environment information at the first moment;

[0055] A determination module for determining the control instruction at the next moment from the initial motion control strategy according to the state information at the first moment and the operating environment information at the first moment;

[0056] Control: Control the humanoid robot to move through control instructions;

[0057] A generation module, configured to generate motion evaluation information for the humanoid robot after the motion is completed;

[0058] An adjustment module, configured to adjust the initial motion control strategy according to the motion evaluation information and a preset reward function.

[0059] Optionally, the reward function includes a navigation task reward function and a balance posture control task reward function;

[0060] The adjustment module includes:

[0061] Calculate the navigation task reward according to the distance data, obstacle handling data, and action stability data in the motion evaluation information and the preset navigation task reward function;

[0062] Calculate the balance posture control task reward according to the vertical posture data, angular velocity data, and action stability data in the motion evaluation information and the preset balance posture control task reward function;

[0063] Optimize the initial motion control strategy according to the navigation task reward and the balance posture control task reward.

[0064] Optionally, the third generation unit includes:

[0065] Determine the speed data of each key joint and the pose data of each end of the skeleton from the simulated state of the robot at the next moment and the real state of the robot at the corresponding moment in the real scene motion data;

[0066] Generate joint error weights for each key joint of the humanoid robot according to the target training task;

[0067] Determine the adjacent association weights of each key joint according to the real-time motion action;

[0068] Generate key joint data weights for the speed data of each key joint according to the joint error weights and the adjacent association weights;

[0069] According to the pose weights of each end of the skeleton of the humanoid robot for the target training task, bind the pose weights to the pose data of each end of the corresponding skeleton;

[0070] Perform error analysis according to the key joint data weights, pose weights, the simulated state of the robot at the next moment, and the real state of the robot at the corresponding moment in the real scene motion data, and generate a total prediction error.

[0071] Optionally, the update unit includes:

[0072] Determine the key joint error from the total prediction error;

[0073] Update the policy weights corresponding to the key joints in the initial motion control policy according to the key joint error;

[0074] Update all the policy weights in the initial motion control policy according to the total prediction error.

[0075] Optionally, the construction unit includes:

[0076] Collect the empirical motion data of the humanoid robot in the simulation world and the real world;

[0077] Construct the empirical motion data to train the world model of the humanoid robot.

[0078] Optionally, after the acquisition unit and before the second generation unit, the humanoid robot motion training device further includes:

[0079] A denoising unit for denoising the real-scene motion data collected;

[0080] A mapping unit for mapping the values of the real-scene motion data collected by different sensors into a unified interval.

[0081] The third aspect of this application provides a humanoid robot motion training device based on a real scene, including:

[0082] A processor, a memory, an input / output unit, and a bus;

[0083] The processor is connected to the memory, the input / output unit, and the bus;

[0084] The memory stores a program, and the processor calls the program to execute the humanoid robot motion training method as in the first aspect and any optional one of the first aspect.

[0085] The fourth aspect of this application provides a computer-readable storage medium, on which a program is stored, and when the program is executed on a computer, it executes the humanoid robot motion training method as in the first aspect and any optional one of the first aspect.

[0086] It can be seen from the above technical solutions that the embodiments of this application have the following advantages:

[0087] In this application, first, a world model for training a humanoid robot is constructed. The world model is used to simulate the physical characteristics of the real world and the motion laws of the humanoid robot. An initial motion control strategy is generated for the humanoid robot according to the target training task and the structural parameters of the humanoid robot. The initial motion control strategy of the humanoid robot is trained in a simulation environment. The humanoid robot is made to enter the real world to execute real-world scenario motions. Real scenario motion data of the humanoid robot during the real-world scenario motion is collected in real time through sensors. The real scenario motion data and the real-time motion actions determined by the initial motion control strategy are input into the world model in real time for interactive training to generate the simulated state of the robot at the next moment. Error analysis is performed on the simulated state of the robot at the next moment and the real state of the robot at the corresponding moment in the real scenario motion data to generate the total prediction error. The initial motion control strategy is updated according to the total prediction error.

[0088] The business requirements of the real scenario are realized by constructing a world model and using the method of online learning. First, a world model that can accurately simulate the physical characteristics of the real world and the motion laws of the robot is constructed, which is the basic support of the whole solution. In the simulation environment, the initial motion control strategy of the humanoid robot is trained based on the world model. Then, when the humanoid robot enters the real scenario, by using online learning means, the motion data of the robot in the real environment is collected in real time. These data are fed back to the world model to quickly fine-tune the initial strategy. The world model and online learning cooperate with each other to continuously optimize the strategy, enabling the humanoid robot to adapt to various changes in the real scenario, thereby realizing the stable operation of the motion control strategy in the real scenario and improving the performance of the humanoid robot in the real scenario operation. Brief Description of the Drawings

[0089] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0090] Figure 1 It is a schematic diagram of an embodiment of the method for training the motion of a humanoid robot based on a real scenario in the present application;

[0091] Figure 2 It is a schematic diagram of an embodiment of the method for training the initial motion control strategy of a humanoid robot in the present application;

[0092] Figure 3 It is a schematic diagram of an embodiment of the method for adjusting the initial motion control strategy in the present application;

[0093] Figure 4Schematic diagram of an embodiment of the method for generating the total prediction error of the present application;

[0094] Figure 5 Schematic diagram of an embodiment of the method for updating the initial motion control strategy of the present application;

[0095] Figure 6 Schematic diagram of an embodiment of the method for constructing a world model for training a humanoid robot of the present application;

[0096] Figure 7 Schematic diagram of an embodiment of the method for processing real - world motion data of the present application;

[0097] Figure 8 Schematic diagram of an embodiment of the real - world - based humanoid robot motion training device of the present application;

[0098] Figure 9 Schematic diagram of another embodiment of the real - world - based humanoid robot motion training device of the present application;

[0099] Figure 10 Schematic diagram of the overall framework process for training the motion of a humanoid robot based on a real scene of the present application;

[0100] Figure 11 Schematic diagram of the framework process for fine - tuning and learning of a humanoid robot of the present application in the real world. Detailed implementation mode

[0101] In the following description, for the purpose of illustration rather than limitation, specific details such as specific system architectures, technologies, etc. are presented to thoroughly understand the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well - known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present application.

[0102] It should be understood that when used in the specification of the present application and the appended claims, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0103] It should also be understood that the term "and / or" used in the specification of the present application and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0104] As used in the specification of this application and the appended claims, the term "if" may be construed as "when" or "once" or "in response to determining" or "in response to detecting" depending on the context. Similarly, the phrase "if determined" or "if [the described condition or event] is detected" may be construed as meaning "once determined" or "in response to determining" or "once [the described condition or event] is detected" or "in response to detecting [the described condition or event]" depending on the context.

[0105] In addition, in the description of the specification of this application and the appended claims, the terms "first", "second", "third", etc. are only used for differential description and cannot be construed as indicating or implying relative importance.

[0106] References in the specification of this application to "one embodiment" or "some embodiments" or the like mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in one or more embodiments of this application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification do not necessarily all refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized. The terms "comprising", "including", "having", and their variants all mean "including but not limited to", unless otherwise specifically emphasized.

[0107] In the prior art, the main problem currently faced is the large gap between the simulation training environment (simulation environment) and the real scenario, that is, the sim2real gap problem. For example, in a rescue scenario, a humanoid robot needs to move flexibly in a disaster scene full of uncertainties, avoid obstacles and accurately reach the target position. However, for existing robots based on simulation training, after entering the real scenario, due to the differences between the environment and the simulation training environment, the motion control strategies of humanoid robots are often less effective or even ineffective, and they cannot perform tasks stably. Secondly, in an industrial production scenario, a humanoid robot needs to accurately complete operations such as assembling parts. The strategies trained in the simulation environment are difficult to ensure stable motion accuracy in a real production line due to factors such as equipment vibration and light changes.

[0108] In summary, when the current humanoid robots are put into new real scenarios to perform tasks, the differences between the real scenario and the simulation training scenario lead to a reduction in the effect of the motion strategy, and thus reduce the performance of humanoid robots in real scenarios.

[0109] Based on this, this application discloses a motion training method and device for a humanoid robot based on a real scenario, which is used to improve the performance of a humanoid robot in a real scenario.

[0110] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0111] The method of the present application can be applied to a server, a device, a terminal, or other devices with logical processing capabilities. In this regard, the present application makes no limitation. For the convenience of description, the following takes the execution entity as a terminal as an example for description.

[0112] Please refer to Figure 1 , an embodiment of a method for training a humanoid robot's motion based on a real scenario provided by the present application includes:

[0113] 101. Construct a world model for training a humanoid robot, where the world model is used to simulate the physical characteristics of the real world and the motion laws of the humanoid robot;

[0114] In this embodiment, the terminal first constructs a world model according to the humanoid robot and the tasks that the humanoid robot needs to execute, and generates a world model for simulating the physical characteristics of the real world and the motion laws of the humanoid robot. Specifically, the terminal first collects simulation data, collects various data related to the motion of the humanoid robot in the simulation environment, including information such as the motion state of the robot and environmental parameters under different scenario settings, providing a data basis for constructing the world model. Then, the world model is constructed, and machine learning algorithms are used to analyze and process the collected simulation data to construct a world model that can simulate the physical characteristics of the real world and the motion laws of the robot.

[0115] 102. Generate an initial motion control strategy for the humanoid robot according to the target training task and the structural parameters of the humanoid robot;

[0116] 103. Train the initial motion control strategy of the humanoid robot in the simulation environment;

[0117] In this embodiment, the terminal generates an initial motion control strategy for the humanoid robot according to the target training task and the structural parameters of the humanoid robot. Then, the initial strategy is trained in the simulation environment based on the world model. The terminal uses the constructed world model to train the initial motion control strategy of the humanoid robot in the simulation environment with a suitable algorithm, enabling the humanoid robot to learn a preliminary effective motion method.

[0118] 104. Let the humanoid robot enter the real world to execute real-world scenario motions;

[0119] Humanoid robots enter the real world: Deploy the robots trained in a simulated environment to a real scenario to perform tasks.

[0120] 105. Real-time collect real-world motion data of a humanoid robot during its movement in a real-world scenario through sensors;

[0121] In this embodiment, the terminal real-time collects real-world motion data of a humanoid robot during its movement in a real-world scenario through sensors. Specifically, the sensors collect real-time motion data, and the humanoid robot in the real world collects motion data in real time through its own equipped sensors, such as information on joint angles, positions, speeds, etc.

[0122] The real-world motion data is the current robot state st: This is a multi-dimensional vector that contains various information describing the state of the robot at time t. The specific content includes joint angles, joint speeds, pose data of the robot, and humanoid robot sensor data or environmental sensor data.

[0123] Joint angle: The degree of bending of each joint, reflecting the posture of the robot's limbs.

[0124] Joint speed: The rate of change of the joint angle, reflecting the speed of the robot's limb movement.

[0125] Position and pose of the robot: Information such as coordinates and orientation in three-dimensional space.

[0126] Sensor data: Such as the force condition detected by a force sensor, environmental image features obtained by a vision sensor, etc. (if there are relevant sensors).

[0127] The current action at is the real-time motion action determined by the initial motion control strategy, representing the action instruction executed by the robot at time t, and is usually also a vector. The real-time motion action includes torque instructions, speed instructions of joints, etc., for example, the force to be applied to each joint or the motion speed to be achieved.

[0128] 106. Real-time input the real-world motion data and the real-time motion action determined by the initial motion control strategy into the world model for interactive training to generate the simulated state of the robot at the next moment;

[0129] 107. Analyze the error between the simulated state of the robot at the next moment and the real state of the robot at the corresponding moment in the real-world motion data to generate the total prediction error;

[0130] The terminal real-time inputs the real-world motion data and the real-time motion action determined by the initial motion control strategy into the world model for interactive training to generate the simulated state of the robot at the next moment.

[0131] Specifically, the humanoid robot state s^t+1 predicted by the terminal (the robot simulation state) is also a multi-dimensional vector, with the same dimension as the real-world motion data st, containing the state information of the robot predicted by the world model at time t+1, such as the predicted joint angles, velocities, positions, and postures. This prediction result will be compared with the real s^t+1 to calculate the corresponding error loss, thereby optimizing the parameters of the world model and the initial motion control strategy.

[0132] In this embodiment, the terminal performs error analysis on the robot simulation state at the next moment and the real robot state at the corresponding moment in the real-world motion data to generate the total prediction error. Specifically, it is necessary to compare the collected data with the prediction result of the world model to calculate the error, input the collected real motion data into the world model, and compare it with the prediction result of the model to calculate the error between the two.

[0133] Specifically, in this embodiment, in the world model, the main objective is to predict the robot state s^t+1 at the next moment based on the current robot state st and the current action at. Suppose there are N samples, and each sample contains the current state, action, and the corresponding real state at the next moment.

[0134] The MSE loss function L_MSE is used to measure the error between the predicted next moment state s^t+1 and the real next moment state st+1, and its formula is:

[0135]

[0136] Among them, N is the number of samples, that is, the number of state-action pairs at different moments used in the training process. d is the dimension of the humanoid robot's state vector. The humanoid robot state includes information such as joint angles, velocities, and positions, and d is the length of the vector composed of this information. st+1,ij represents the j-th element of the real next moment state vector in the i-th sample. s^t+1,ij represents the j-th element of the next moment state vector predicted by the model in the i-th sample.

[0137] When the humanoid robot enters the real world to perform tasks, its motion data, including joint states, positions, velocities, etc., is collected in real-time through the sensors on the humanoid robot and the authenticity of the environment.

[0138] When a humanoid robot enters the real world to perform a real-world task, its various sensors (such as joint position sensors, accelerometers, gyroscopes, and visual sensors) begin operating. These sensors collect real-time motion data from the humanoid robot, including joint angles, angular velocities, linear velocities, position coordinates, posture information, and information about the surrounding environment. For example, in a complex indoor environment, the visual sensor captures information such as the room layout and obstacle locations, while the joint position sensor records the real-time angles of each joint. This data is then input into a stored world model and compared with the world model's predictions to calculate the error. Based on this error, an online learning algorithm employs stochastic gradient descent to rapidly fine-tune the initial motion control strategy. This process needs to be repeated continuously during field tasks. As the humanoid robot continuously collects real-world data, each update continuously optimizes the robot's motion control strategy, gradually improving its stability and enhancing its effectiveness in real-world scenarios.

[0139] The error calculation here uses the loss function of the world model. The difference is that the true value is converted from the value obtained from the simulation to the true value in the real world.

[0140] The preprocessed data is input into the constructed world model. The world model predicts the state of the humanoid robot at the next moment based on the current input humanoid robot state and action. The predicted result is compared with the actual state at the next moment collected by the sensor, and the error between the two is calculated. The error calculation method used in this embodiment is the mean square error. For example, if the world model predicts a joint angle of 30 degrees, and the actual angle measured by the sensor is 32 degrees, the error in the joint angle prediction is calculated using the MSE formula, and the world model is trained using gradient descent.

[0141] In addition to the above-mentioned mean square error method, a task target weighted mean square error method is also created, which is described in detail in subsequent embodiments. The task target weighted mean square error method is used to analyze errors by combining important joints and specific end postures of the humanoid robot to accurately calculate the error between the real action and the predicted action of the humanoid robot in a specific task target.

[0142] 108. Update the initial motion control strategy based on the total prediction error.

[0143] Finally, the terminal fine-tunes the initial motion control strategy using an online learning algorithm and the total prediction error. Based on the calculated total prediction error, the initial motion control strategy is rapidly adjusted and optimized using the online learning algorithm. This optimized initial motion control strategy is fed back to the humanoid robot, making its motion control more stable in the real world. This process continues in a cycle, continuously improving the humanoid robot's real-world performance.

[0144] Please refer to Figure 10 and Figure 11 , Figure 10 is a schematic diagram of the overall framework process for training the motion of a humanoid robot based on a real scenario, Figure 11 is a schematic diagram of the framework process for fine-tuning and learning of a humanoid robot in the real world. The two figures respectively introduce the entire training scheme and the real-world fine-tuning and learning scheme. In this embodiment, the terminal uses online learning algorithms (stochastic gradient descent SGD and its variants Adagrad, Adadelta). First, the initial motion control strategy is adjusted according to the calculated error. Taking SGD as an example, the gradient of the error with respect to the parameters of the policy network is calculated, and then the parameters of the policy network are updated in the opposite direction of the gradient, so that the policy network can be closer to the real situation in subsequent predictions. If the gradient of a weight parameter w in the initial policy network is g and the learning rate is alpha, then the updated weight is (w' = w - alpha * g).

[0145] [[ID=^13]]Next, the terminal performs cyclic optimization in the real-world scenario, that is, continuously repeats the above steps of data collection, preprocessing, interaction with the world model, and policy update. As the running time of the humanoid robot in the real world increases, more and more data is accumulated, the policy network is continuously optimized, gradually adapts to the real-world environment, and the motion control ability is improved.

[0146] In this embodiment, first, a world model for training a humanoid robot is constructed. The world model is used to simulate the physical characteristics of the real world and the motion laws of the humanoid robot. An initial motion control strategy is generated for the humanoid robot according to the target training task and the structural parameters of the humanoid robot. The initial motion control strategy of the humanoid robot is trained in a simulation environment. The humanoid robot is made to enter the real world to execute real-world scenario motions. The real scenario motion data of the humanoid robot during the real-world scenario motion is collected in real time through sensors. The real scenario motion data and the real-time motion actions determined by the initial motion control strategy are input into the world model for interactive training in real time to generate the simulated state of the robot at the next moment. The error analysis is performed between the simulated state of the robot at the next moment and the real state of the robot at the corresponding moment in the real scenario motion data to generate the total prediction error. The initial motion control strategy is updated according to the total prediction error.

[0147] The real-world business requirements are achieved by constructing a world model and using online learning. First, a world model that can accurately simulate the physical characteristics of the real world and the motion laws of the robot is constructed, which is the basic support for the entire solution. In the simulation environment, the initial motion control strategy of the humanoid robot is trained based on the world model. Then, when the humanoid robot enters the real scenario, online learning means are used to collect the motion data of the robot in the real environment in real time. These data are fed back to the world model to quickly fine-tune the initial strategy. The world model and online learning cooperate with each other to continuously optimize the strategy, enabling the humanoid robot to adapt to various changes in the real scenario, thus realizing the stable operation of the motion control strategy in the real scenario and improving the performance of the humanoid robot in the real scenario operation.

[0148] Secondly, the motion control training of the real-world humanoid robot is realized with the help of the world model and online learning. The main function of this training method is to make the motion control of the humanoid robot more stable in the real environment and effectively solve the problem of the gap between simulation and reality. For example, in an industrial production scenario, after the robot is trained in the simulation environment and enters the real production line, it should be able to stably perform tasks such as grasping and assembling without making action mistakes due to environmental differences. It constructs a world model to simulate the physical characteristics of the real world and the motion laws of the robot. Online learning then uses the data collected in real time by the robot in the real environment to quickly fine-tune the strategy trained based on the world model. In this way, the robot can quickly adapt to the real environment, making the motion control strategy more stable and reliable and accurately completing various tasks.

[0149] Secondly, this embodiment also has the beneficial effect of quick fine-tuning, efficiently solving the sim2real gap: through online learning, the initial strategy trained based on the world model is quickly fine-tuned using the data collected in the real world, greatly reducing the gap between simulation and reality and enabling the robot to quickly adapt to the real environment.

[0150] Secondly, this embodiment also has the beneficial effect of enhancing the stability of motion control. By adjusting the strategy based on the world model prediction and real-time data feedback, the stability of the motion control of the humanoid robot in the real scenario is effectively improved, the action error rate is reduced, and the accuracy of task execution is guaranteed.

[0151] Secondly, in this embodiment, the world model is constructed relying on simulation, with a relatively low cost. Only the simulation data is needed to construct the world model, avoiding the high cost and potential risks of large-scale data collection in the real scenario. At the same time, the repeatability and controllability of the simulation environment are used to improve the model training efficiency.

[0152] Secondly, in this embodiment, real - machine iteration optimization is also used. During the real - machine operation, data is continuously collected, and the motion control strategy is continuously iteratively optimized. As time goes by, the robot's motion control performance becomes better and better, and it can better adapt to complex and changeable real scenarios.

[0153] Please refer to Figure 2 , an embodiment of a method for training an initial motion control strategy of a humanoid robot is provided in this application, including:

[0154] 201. Obtain the state information of the humanoid robot at the first moment and the operation environment information at the first moment;

[0155] 202. Determine the control instruction for the next moment from the initial motion control strategy according to the state information at the first moment and the operation environment information at the first moment;

[0156] 203. Control the humanoid robot to move through the control instruction;

[0157] 204. Generate motion evaluation information for the humanoid robot after the motion is completed;

[0158] 205. Adjust the initial motion control strategy according to the motion evaluation information and the preset reward function.

[0159] In this embodiment, in the simulated operation environment, with the help of the constructed world model, the initial motion control strategy of the humanoid robot is trained using the Proximal Policy Optimization (PPO) reinforcement learning algorithm. Taking the humanoid robot to complete a specific task (such as walking to a specified location in a simulated obstacle environment) as the goal, a corresponding reward mechanism (reward function) is set. Then, according to the results predicted by the world model, the humanoid robot continuously adjusts its own actions and gradually learns an effective motion strategy.

[0160] In this embodiment, the terminal obtains the state information of the humanoid robot at the first moment and the operation environment information at the first moment. The operation environment information at the first moment is the input (obs - observation value), specifically mainly including the current state information of the humanoid robot. Specifically, it covers the angles, angular velocities, and angular accelerations of each joint of the humanoid robot, which are used to accurately describe the posture and motion trend of the humanoid robot's limbs. It also includes the position coordinates of the humanoid robot in space (such as the x, y, z coordinates in the Cartesian coordinate system) and the posture (such as the orientation represented by Euler angles), to determine the three - dimensional position and direction of the humanoid robot in the virtual environment. In addition, it also includes the data collected by various sensors, such as the ground reaction force sensed by the pressure sensor, the contact situation with surrounding objects detected by the collision sensor, and the environmental feature information extracted by the vision sensor (such as the position, shape, distance, etc. of obstacles). These information combined provide a comprehensive description of the robot's current environment and its own state for the reinforcement learning algorithm, enabling the algorithm to make appropriate decisions based on this.

[0161] Based on the state information at the first moment and the operating environment information at the first moment, the terminal determines the control instruction for the next moment from the initial motion control strategy. The output of the initial motion control strategy is a motion instruction (action), which is used to drive the motion of the humanoid robot. This includes control signals for each joint, specifying the target angle, target angular velocity, or target torque for each joint, so as to precisely adjust the actions of the robot's limbs. At the same time, it also involves the overall motion instruction of the humanoid robot, such as setting the speed and direction of forward, backward, and turning, enabling the humanoid robot to move in the environment as expected and complete specific tasks.

[0162] After obtaining the motion instruction, the terminal controls the humanoid robot to move through the control instruction, and then generates motion evaluation information for the humanoid robot after the motion is completed. The motion evaluation information refers to the respective motion data of the humanoid robot collected from the start to the end of this motion instruction. Finally, the terminal adjusts the initial motion control strategy according to the motion evaluation information and the preset reward function to achieve the effect of adjusting the initial motion control strategy.

[0163] Please refer to Figure 3 , an embodiment of a method for adjusting an initial motion control strategy provided by this application includes:

[0164] 301. Calculate the navigation task reward according to the distance data, obstacle handling data, and action stability data in the motion evaluation information and the preset navigation task reward function;

[0165] 302. Calculate the balance attitude control task reward according to the vertical pose data, angular velocity data, and action stability data in the motion evaluation information and the preset balance attitude control task reward function;

[0166] 303. Optimize the initial motion control strategy according to the navigation task reward and the balance attitude control task reward.

[0167] The reward function is key to guiding humanoid robots to learn effective strategies. In the early stages of training, positive rewards are given for positive behaviors such as the humanoid robot moving toward the target position, avoiding collisions with obstacles, and maintaining body balance. In this embodiment, the reward value increases every time the humanoid robot approaches a certain distance from the target. If no collision occurs within a certain period of time, an additional reward is given. Successfully maintaining balance in complex terrain will also receive a corresponding reward. Conversely, negative rewards are given for negative behaviors such as deviation from the target, collision, and loss of balance. For example, the reward value decreases every time the robot moves a certain distance away from the target; a large amount of reward is deducted if a collision occurs; and a severe negative reward is given if the robot loses balance and falls. By rationally designing the reward function, the robot can gradually learn the optimized movement strategy through continuous attempts, and develop towards completing the task and obtaining the maximum reward. The reward function in this embodiment is mainly based on maintaining balance while navigating the task, so it is set as a navigation task reward and a balance posture control task reward.

[0168] In this embodiment, the terminal first calculates the navigation task reward based on the distance data, obstacle handling data, and action stability data in the motion evaluation information and a preset navigation task reward function. Specifically, in a navigation task, the goal is to move the robot from a starting point to a designated target point. The reward function encourages the robot to approach the target while avoiding collisions with obstacles.

[0169]

[0170] Where R is the reward value of the navigation task reward function. dt is the distance between the robot and the target point at time t. This is a very small positive number to prevent the denominator from being zero, usually set to 10^-6. collision is the collision flag, which is 1 if a collision occurs and 0 otherwise. fall is the fall flag, which is 1 if the humanoid robot falls and 0 otherwise. 、 、 is the weight coefficient, which is used to adjust the importance of each item in the reward function. For example, =10, =20, =50.

[0171] Balance and Posture Control Task Reward,In the balance and posture control task, the focus is on making the robot maintain a good,posture and balance.

[0172]

[0173] Where: R here is the reward value of the balance posture control task reward function, It is the angle between the humanoid robot's body and the vertical direction. The smaller the angle, the more balanced the posture. is the angular velocity of the robot body. The smaller the angular velocity, the more stable it is. , , are weight coefficients. For example, in this embodiment, = 15, = 5, = 30.

[0174] Please refer to Figure 4 , an embodiment of a method for generating a total prediction error provided by this application includes:

[0175] 401. Determine the speed data of each key joint and the pose data of each end of the skeleton from the robot simulation state at the next moment and the real state of the robot at the corresponding moment in the real - scene motion data;

[0176] In this embodiment, determine the speed data of each key joint from the robot simulation state at the next moment, and the speed data of each key joint in the real - scene motion data at the next moment. The two sets of speed data are in one - to - one correspondence through the key joints. Next, determine the pose data of each end of the skeleton from the robot simulation state at the next moment, and the pose data of each end of the skeleton in the real - scene motion data at the next moment.

[0177] 402. Generate joint error weights for each key joint of the humanoid robot according to the target training task;

[0178] The terminal determines the running accuracy importance for each key joint at the next moment according to different training tasks of the humanoid robot. Because there are multiple core actions in different target training tasks, and each action has different functions, this makes the importance of key joints different in each action, and the importance of the key joints used in each key action is also different. For example, several key joints in the core of the squatting action and several key joints in the core of the lifting action. In different actions, the same key joint plays different roles, so the possible deformation amount brought by this key joint is also different, resulting in different importance of key joints in different actions. The joint error weight is a value greater than or equal to 1.

[0179] 403. Determine the adjacent association weight of each key joint according to the real - time motion action;

[0180] In this embodiment, since different key joints have different motions under one action, and the motions of some key joints can affect several adjacent key joints, the deformation of the motion on this key joint will drive the deformation of other adjacent key joints. At this time, we give a contiguous association weight to this part of the key joints, so that when this key joint deforms, it can increase its corresponding error through this association weight, thereby increasing the total error. The contiguous association weight is a value greater than 1 or equal to 1.

[0181] 404. Generate key joint data weights for the velocity data of each key joint according to the joint error weight and the contiguous association weight;

[0182] In this embodiment, the terminal generates key joint data weights for the velocity data of each key joint according to the joint error weight and the contiguous association weight. In this embodiment, the joint error weight and the contiguous association weight are multiplied to obtain the key joint data weights. The key joint data weights are inserted into the MSE loss function in step 107. After taking the difference and then squaring each state in the true state of the robot at the corresponding moment between the simulated state of the robot and the motion data of the real scene, if it belongs to the key joint with key joint data weights, it needs to be multiplied by the key joint data weights and then divided by N after squaring. After adding up all the errors, the error of a specific key joint is amplified.

[0183] 405. According to the target training task, determine the pose weights for the end of each skeleton of the humanoid robot, and bind the pose weights to the skeleton end pose data corresponding to each skeleton end;

[0184] In this embodiment, the terminal determines the pose weights for the end of each skeleton of the humanoid robot according to the target training task, and binds the pose weights to the skeleton end pose data corresponding to each skeleton end. Since the pose of the skeleton end represents the overall completion degree of the action, and the humanoid robot's end has different pose accuracy requirements according to different tasks, in a specific target training task, when different skeleton ends complete the action, the required accuracies are different, such as grasping actions and gesture actions. The pose weight of each skeleton end is a value greater than 1. Subsequently, it is inserted into the MSE loss function in step 107, and the method is the same as that of the key joint data weights in step 404, which will not be elaborated here.

[0185] 406. Perform error analysis according to the key joint data weights, pose weights, the simulated state of the robot at the next moment, and the true state of the robot at the corresponding moment in the real scene motion data to generate the total prediction error.

[0186] Finally, the terminal performs error analysis based on the key joint data weights, pose weights, the simulated state of the robot at the next moment, and the real state of the robot at the corresponding moment in the real - world motion data to generate the total prediction error. In this embodiment, by setting corresponding deformation influence weights for different key joints and the end of the skeleton, if deformation errors that can be detected occur in the parameters of these positions during the motion process, these errors will be amplified according to the magnitude of their influence, increasing their importance and making them more easily noticed. The generated total prediction error can increase the accuracy of subsequent adjustment of the initial motion control strategy.

[0187] Please refer to Figure 5 , an embodiment of a method for updating an initial motion control strategy provided by this application includes:

[0188] 501. Determine the key joint error from the total prediction error;

[0189] 502. Update the strategy weights corresponding to the key joints in the initial motion control strategy according to the key joint error;

[0190] 503. Update all the strategy weights in the initial motion control strategy according to the total prediction error.

[0191] In this embodiment, the terminal not only needs to adjust the entire initial motion control strategy through the total prediction error, but also needs to separately adjust the control strategies of different key joints to make the adjustment more targeted. Specifically, in this embodiment, after determining the key joint error from the total prediction error, determine each strategy weight related to the key joint from the initial motion control strategy, and use the key joint error to separately update this part of the strategy weights. Such an update strategy can specifically adjust the action strategies of specific key joints for the error of each action, increasing the accuracy of the initial motion control strategy.

[0192] Please refer to Figure 6 , an embodiment of a method for constructing a world model for training a humanoid robot provided by this application includes:

[0193] 601. Collect the empirical motion data of the humanoid robot in the simulation world and the real world;

[0194] 602. Construct a world model for training the humanoid robot from the empirical motion data.

[0195] In this embodiment, the terminal first collects a large amount of data related to the movement of humanoid robots in a simulated world or the real world. Specifically, it is necessary to cover the joint angles, speeds, accelerations, and force conditions of the robot during movement under various environmental parameters such as task terrains, lighting conditions, and temperature environments, as well as information such as the positions, shapes, and materials of surrounding obstacles and targets. Then, machine learning algorithms and deep neural networks are used to analyze and process the collected data to construct a world model that can accurately simulate the physical characteristics of the real world and the movement laws of the robot. The function of this world model is to predict the operating state at the next moment based on the current state and actions of the humanoid robot.

[0196] Please refer to Figure 7 , an embodiment of a method for processing real-scene motion data provided by this application includes:

[0197] 701. Denoise the collected real-scene motion data;

[0198] 702. Map the values of real-scene motion data collected by different sensors to a unified interval.

[0199] The terminal processes the collected real-scene motion data. Remove the noise data. Specifically, the terminal removes the random noise in the sensor measurement process through a filtering algorithm (such as Kalman filtering), and then normalizes the data to map the values collected by different sensors to a unified interval, such as [0, 1] or [-1, 1], so that the neural network can better process the data and improve the training efficiency and stability.

[0200] Please refer to Figure 8 , an embodiment of a humanoid robot motion training device based on a real scene provided by this application includes:

[0201] A construction unit 801 for constructing a world model for training a humanoid robot, where the world model is used to simulate the physical characteristics of the real world and the movement laws of the humanoid robot;

[0202] Optionally, the construction unit 801 includes: collecting the empirical motion data of the humanoid robot in the simulated world and the real world; constructing a world model for training the humanoid robot from the empirical motion data.

[0203] A first generation unit 802 for generating an initial motion control strategy for the humanoid robot according to the target training task and the structural parameters of the humanoid robot;

[0204] A training unit 803 for training the initial motion control strategy of the humanoid robot in a simulated environment;

[0205] Optionally, the training unit 803 includes:

[0206] An acquisition module for acquiring the state information of the humanoid robot at the first moment and the operation environment information at the first moment;

[0207] A determination module for determining the control instruction for the next moment from the initial motion control strategy according to the state information at the first moment and the operation environment information at the first moment;

[0208] A control module for controlling the humanoid robot to move through the control instruction;

[0209] A generation module for generating motion evaluation information for the humanoid robot after the motion is completed;

[0210] An adjustment module for adjusting the initial motion control strategy according to the motion evaluation information and a preset reward function.

[0211] Optionally, the reward function includes a navigation task reward function and a balance posture control task reward function;

[0212] The adjustment module includes: calculating the navigation task reward according to the distance data, obstacle handling data, and action stability data in the motion evaluation information and the preset navigation task reward function; calculating the balance posture control task reward according to the vertical pose data, angular velocity data, and action stability data in the motion evaluation information and the preset balance posture control task reward function; optimizing the initial motion control strategy according to the navigation task reward and the balance posture control task reward.

[0213] An execution unit 804 for making the humanoid robot enter the real world to execute real-world scenario motions;

[0214] An acquisition unit 805 for collecting real scenario motion data of the humanoid robot in real time during the real-world scenario motion through sensors;

[0215] A denoising unit 806 for denoising the collected real scenario motion data;

[0216] A mapping unit 807 for mapping the values of the real scenario motion data collected by different sensors into a unified interval;

[0217] A second generation unit 808 for interactively training the real scenario motion data and the real-time motion actions determined by the initial motion control strategy by inputting them into the world model in real time to generate the simulated state of the robot for the next moment;

[0218] A third generation unit 809 for performing error analysis on the simulated state of the robot for the next moment and the real state of the robot at the corresponding moment in the real scenario motion data to generate the total prediction error;

[0219] Optionally, the third generation unit 809 includes: determining the speed data of each key joint and the pose data of each end of the skeleton from the robot simulation state at the next moment and the real state of the robot at the corresponding moment in the real scene motion data; generating joint error weights for each key joint of the humanoid robot according to the target training task; determining the adjacent association weights of each key joint according to the real-time motion action; generating key joint data weights for the speed data of each key joint according to the joint error weights and the adjacent association weights; generating pose weights for each end of the skeleton of the humanoid robot according to the target training task, and binding the pose weights to the pose data of each end of the skeleton corresponding thereto; performing error analysis according to the key joint data weights, the pose weights, the robot simulation state at the next moment and the real state of the robot at the corresponding moment in the real scene motion data, and generating a total prediction error.

[0220] The updating unit 810 is configured to update the initial motion control strategy according to the total prediction error.

[0221] Optionally, the updating unit 810 includes: determining the key joint error from the total prediction error; updating the strategy weights corresponding to the key joints in the initial motion control strategy according to the key joint error; and updating all the strategy weights in the initial motion control strategy according to the total prediction error.

[0222] Please refer to Figure 9 , this application provides a humanoid robot motion training device based on a real scene, including:

[0223] A processor 901, a memory 902, an input / output unit 903, and a bus 904.

[0224] The processor 901 is connected to the memory 902, the input / output unit 903, and the bus 904.

[0225] The memory 902 stores a program, and the processor 901 calls the program to execute the humanoid robot motion training method as described in Figure 1 , Figure 2 and Figure 3 , Figure 4 , Figure 5 , Figure 6 and Figure 7 .

[0226] This application provides a computer-readable storage medium, on which a program is stored, and when the program is executed on a computer, it executes the humanoid robot motion training method as described in Figure 1 , Figure 2 and Figure 3 , Figure 4 , Figure 5 , Figure 6 and Figure 7The humanoid robot motion training method in

[0227] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the above-described systems, devices, and units can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0228] In several embodiments provided in the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there can be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, indirect couplings or communication connections of devices or units, and can be in electrical, mechanical, or other forms.

[0229] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0230] In addition, the functional units in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0231] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage media include: USB flash drives, mobile hard disks, read-only memories (ROM, read-only memory), random access memories (RAM, random access memory), magnetic disks, or optical disks and other various media that can store program codes.

Claims

1. A method for training the motion of a humanoid robot based on a real scenario, characterized in that, Including: Construct a world model for training a humanoid robot, where the world model is used to simulate the physical characteristics of the real world and the motion laws of the humanoid robot; Generate an initial motion control strategy for the humanoid robot according to the target training task and the structural parameters of the humanoid robot; Train the initial motion control strategy of the humanoid robot in a simulation environment; Let the humanoid robot enter the real world to perform real-world scenario motions; Real-time collect the real scenario motion data of the humanoid robot during the real-world scenario motion through sensors; Input the real scenario motion data and the real-time motion actions determined by the initial motion control strategy into the world model for interactive training in real time to generate the robot simulation state at the next moment; Perform error analysis on the robot simulation state at the next moment and the real robot state at the corresponding moment in the real scenario motion data to generate the total prediction error; Update the initial motion control strategy according to the total prediction error.

2. The humanoid robot motion training method according to claim 1, wherein, The step of training the initial motion control strategy of the humanoid robot in the simulation environment includes: Obtain the state information of the humanoid robot at the first moment and the operation environment information at the first moment; Determine the control instruction at the next moment from the initial motion control strategy according to the state information at the first moment and the operation environment information at the first moment; Control the humanoid robot to move through the control instruction; Generate motion evaluation information for the humanoid robot after the motion is completed; Adjust the initial motion control strategy according to the motion evaluation information and a preset reward function.

3. The humanoid robot motion training method according to claim 2, wherein The reward function includes a navigation task reward function and a balance posture control task reward function; The step of adjusting the initial motion control strategy according to the motion evaluation information and the preset reward function includes: Calculate the navigation task reward according to the distance data, obstacle handling data, and action stability data in the motion evaluation information and the preset navigation task reward function; Calculate the balance posture control task reward according to the vertical pose data, angular velocity data, and action stability data in the motion evaluation information and the preset balance posture control task reward function; Optimize the initial motion control strategy according to the navigation task reward and the balance posture control task reward.

4. The humanoid robot motion training method according to claim 1, characterized in that, The step of performing error analysis on the robot simulation state at the next moment and the real robot state at the corresponding moment in the real scenario motion data to generate the total prediction error includes: Determine the speed data of each key joint and the pose data of each end of the skeleton from the robot simulation state at the next moment and the real robot state at the corresponding moment in the real scenario motion data; Generate joint error weights for each key joint of the humanoid robot according to the target training task; Determine the adjacent connection weight of each key joint according to the real-time motion action, where the meaning of the adjacent connection weight is that when a key joint deforms, its corresponding error needs to be increased through the corresponding adjacent connection weight, thereby increasing the total error; Generate key joint data weights for the speed data of each key joint according to the joint error weights and the adjacent connection weights; Generate pose weights for each end of the skeleton of the humanoid robot according to the target training task, and bind the pose weights to the skeleton end pose data corresponding to each end of the skeleton; Generate a total prediction error through error analysis based on the key joint data weights, the pose weights, the robot simulation state at the next moment, and the real robot state at the corresponding moment in the real scene motion data.

5. The method for humanoid robot motion training according to claim 4, characterized in that, The step of updating the initial motion control strategy according to the total prediction error includes: Determine the key joint error from the total prediction error; Update the strategy weights corresponding to the key joints in the initial motion control strategy according to the key joint error; Update all the strategy weights in the initial motion control strategy according to the total prediction error.

6. The humanoid robot motion training method according to any one of claims 1 to 5, characterized in that, The step of constructing a world model for training a humanoid robot includes: Collect the empirical motion data of the humanoid robot in the simulation world and the real world; Construct a world model for training a humanoid robot with the empirical motion data.

7. The humanoid robot motion training method according to any one of claims 1 to 5, characterized in that, After the step of collecting the real scene motion data of the humanoid robot in the real world scene motion process in real time through the sensor, and before the step of inputting the real scene motion data and the real-time motion actions determined by the initial motion control strategy into the world model for interactive training to generate the robot simulation state at the next moment, the humanoid robot motion training method further includes: Denoise the collected real scene motion data; Map the numerical values of the real scene motion data collected by different sensors into a unified interval.

8. A humanoid robot motion training device based on a real scenario, characterized in that, Include: A construction unit for constructing a world model for training a humanoid robot, where the world model is used to simulate the physical characteristics of the real world and the motion law of the humanoid robot; A first generation unit for generating an initial motion control strategy for the humanoid robot according to the target training task and the structural parameters of the humanoid robot; A training unit for training the initial motion control strategy of the humanoid robot in a simulation environment; An execution unit for making the humanoid robot enter the real world to perform real world scene motions; A collection unit for collecting the real scene motion data of the humanoid robot in the real world scene motion process in real time through a sensor; A second generation unit for inputting the real scene motion data and the real-time motion actions determined by the initial motion control strategy into the world model for interactive training to generate the robot simulation state at the next moment; A third generation unit for performing error analysis on the robot simulation state at the next moment and the real robot state at the corresponding moment in the real scene motion data to generate a total prediction error; An update unit for updating the initial motion control strategy according to the total prediction error.

9. The humanoid robot motion training device according to claim 8, characterized in that, The training unit includes: An acquisition module for acquiring the state information of the humanoid robot at the first moment and the operation environment information at the first moment; A determination module, configured to determine a control instruction for the next moment from the initial motion control strategy according to the state information at the first moment and the operating environment information at the first moment; A control module, configured to control the humanoid robot to perform motion through the control instruction; A generation module, configured to generate motion evaluation information for the humanoid robot after the motion is completed; An adjustment module, configured to adjust the initial motion control strategy according to the motion evaluation information and a preset reward function.

10. A humanoid robot motion training device based on a real scenario, characterized in that, Comprising: A processor, a memory, an input / output unit, and a bus, where the processor is connected to the memory, the input / output unit, and the bus, the memory stores a program, and the processor calls the program to execute the humanoid robot motion training method according to any one of claims 1 to 7.

Images