Humanoid robot motion control model training method and device based on imitation learning

Abstract

The present disclosure provides a kind of humanoid robot motion control model training method and device based on imitation learning, it is related to sensor and robot technical field.The method comprises: obtaining the action data set comprising multiple reference human action sequences, and pre-training strategy network based on action data set;On the basis of pre-training strategy network, introduce low-rank adaptive parameter matrix for each linear transformation module in strategy network, and the low-rank adaptive parameter matrix and original weight matrix are combined for subsequent action output generation;The target action is input into the strategy network after adding low-rank adaptive parameter matrix, and the pose tracking reward is calculated by the deviation of reference posture;Based on pose tracking reward, update low-rank adaptive parameter matrix to complete the strategy fine tuning of target action.The present disclosure can reduce the cost of model training, and on this basis, improve the efficiency of model training, while the model can still maintain stable action output under unseen action task.

Description

Technical Field

[0001] This disclosure relates to the fields of sensor and robotics technology, and to a method and apparatus for training a humanoid robot motion control model based on imitation learning. Background Technology

[0002] While mimic learning methods in related technologies can improve the sample efficiency of reinforcement learning training by using human action databases for policy pre-training, they still have significant shortcomings in adaptability to specific target actions, especially in unseen action tasks outside the training distribution, where their action generalization ability is limited.

[0003] Specifically, basic imitation models are typically trained only on a limited set of actions. When faced with unseen action tasks, the model parameters cannot effectively generalize to new motion targets, leading to low efficiency in policy learning, long training times, and even convergence problems. To improve the model's action adaptability, one approach is to retrain all parameters in the model so that it can learn new action tasks. However, this method requires a long training time and more interaction data, increasing both the training cost and efficiency. Another approach is to fine-tune the basic imitation model with a small number of supervised samples. However, this method is prone to overfitting the target trajectory in scenarios with few samples, sacrificing overall controllability and making it difficult to maintain stable generalization in unseen action tasks.

[0004] For example, in Chinese patent application CN120116218A, human motion data is collected and motion recognition is performed. The identified motion features are then input into a mapping model to generate a target state that the robot can execute. The control module then outputs drive signals to control the robot to perform the corresponding actions, thereby achieving human motion imitation control. This method employs a supervised imitation learning framework, primarily relying on posture errors for parameter optimization, and has limited generalization ability for unseen motion tasks outside the training distribution.

[0005] Therefore, how to improve the model's generalization ability while reducing the model's training cost and increasing its training efficiency has become an urgent problem to be solved in the training process of imitation learning control strategies.

[0006] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0007] This disclosure provides a method and apparatus for training a humanoid robot motion control model based on imitation learning, in order to overcome the problems of insufficient model action generalization ability, high model training cost and low model training efficiency in related technologies.

[0008] Additional aspects and advantages of this disclosure will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this disclosure.

[0009] According to a first aspect of this disclosure, a method for training a humanoid robot motion control model based on imitation learning is provided, comprising:

[0010] Obtain an action dataset containing multiple reference human action sequences, and pre-train a policy network based on imitation learning based on the action dataset. The input of the policy network includes ontology perception information and relative target position information.

[0011] Based on the pre-trained policy network, a low-rank adaptation parameter matrix is ​​introduced into each linear transformation module in the policy network, and the low-rank adaptation parameter matrix is ​​combined with the original weight matrix for subsequent action output generation.

[0012] The target action is input into the policy network after adding a low-rank adaptation parameter matrix, and the attitude tracking reward is calculated by the deviation from the reference attitude.

[0013] Based on posture tracking rewards, the strategy is fine-tuned by updating the low-rank adaptation parameter matrix to complete the target action.

[0014] In one exemplary embodiment of this disclosure, pre-training a policy network based on imitation learning using an action dataset includes:

[0015] Randomly select the starting frame of the slice, and extract continuous frame motion data from the motion dataset according to a preset time length to generate training segments;

[0016] Based on the training segments, the policy network is pre-trained using imitation learning.

[0017] In one exemplary embodiment of this disclosure, imitation learning pre-training is performed on the policy network based on training segments, including:

[0018] Extract the corresponding reference human actions frame by frame from the training clips;

[0019] Acquire the body perception information of the humanoid robot and determine the current state of the humanoid robot based on the body perception information;

[0020] Based on the current state and the reference human action in the corresponding frame, the relative target position information of the humanoid robot is calculated;

[0021] The humanoid robot's proprioceptive information and relative target position information are input into the policy network to perform imitation learning pre-training on the policy network.

[0022] In one exemplary embodiment of this disclosure, the attitude tracking reward is calculated by the deviation from the reference attitude, including:

[0023] Obtain the predicted pose vector corresponding to the control action output by the policy network, and the reference pose vector corresponding to the reference human action in the corresponding frame.

[0024] Calculate the attitude deviation between the predicted attitude vector and the reference attitude vector;

[0025] The attitude deviation is input into the reward function to obtain the attitude tracking reward.

[0026] In one exemplary embodiment of this disclosure, the attitude deviation is input into a reward function to obtain an attitude tracking reward, including:

[0027]

[0028] in, For attitude tracking rewards, For attitude deviation, For reference to human movements t The reference pose vector at time step, For policy networks in t The predicted attitude vector corresponding to the control action output at any given time. This is the tolerance parameter for attitude tracking rewards.

[0029] In one exemplary embodiment of this disclosure, after inputting the target action into the policy network with the low-rank adaptation parameter matrix added, the method further includes:

[0030] The contact matching reward is calculated based on the deviation from the reference contact state.

[0031] Based on attitude tracking rewards and contact matching rewards, the strategy fine-tuning is used to update the low-rank adaptation parameter matrix to complete the target action.

[0032] In one exemplary embodiment of this disclosure, calculating a contact matching reward by means of deviation from a reference contact state includes:

[0033] Obtain the predicted contact state vector corresponding to the control action output by the policy network, and the reference contact state vector corresponding to the reference human action of the corresponding frame.

[0034] Calculate the contact state deviation between the predicted contact state vector and the reference contact state vector;

[0035] The contact state deviation is input into the reward function to obtain the contact matching reward.

[0036] In one exemplary embodiment of this disclosure, the contact state deviation is input into a reward function to obtain a contact matching reward, including:

[0037]

[0038] in, To access matchmaking rewards, For contact state deviation, To reference human movements t The reference contact state vector at time 1. For policy networks in t The predicted contact state vector corresponding to the control action output at any given time. Tolerance parameters for accessing matching rewards.

[0039] In one exemplary embodiment of this disclosure, the method further includes:

[0040] Speed ​​matching rewards are calculated based on the deviation from a reference motion speed;

[0041] Based on attitude tracking rewards, contact matching rewards, and velocity matching rewards, the low-rank adaptation parameter matrix is ​​updated to fine-tune the strategy for completing the target action.

[0042] In one exemplary embodiment of this disclosure, calculating a speed matching reward by means of deviation from a reference motion speed includes:

[0043] Obtain the predicted velocity vector corresponding to the control action output by the policy network, and the reference velocity vector corresponding to the reference human action in the corresponding frame;

[0044] Calculate the velocity deviation between the predicted velocity vector and the reference velocity vector;

[0045] The speed deviation is input into the reward function to obtain the speed matching reward.

[0046] In one exemplary embodiment of this disclosure, the motion speed deviation is input into a reward function to obtain a speed matching reward, including:

[0047]

[0048] in, For speed-matching rewards, For the deviation in motion speed, To reference human movements t The reference velocity vector at time t, For policy networks in tThe predicted velocity vector corresponding to the control action output at any given time. Tolerance parameters for speed matching rewards.

[0049] In one exemplary embodiment of this disclosure, the method further includes:

[0050] The reward function is dynamically adjusted based on the course learning strategy.

[0051] In one exemplary embodiment of this disclosure, the reward function is dynamically adjusted based on the course learning strategy, including:

[0052] During training, the tolerance parameters of each reward item in the reward function are gradually reduced as the training progresses.

[0053] In one exemplary embodiment of this disclosure, the method further includes:

[0054] During training, the weight parameters in the original weight matrix are frozen, and only the adaptation parameters in the low-rank adaptation parameter matrix are updated.

[0055] In one exemplary embodiment of this disclosure, the low-rank adaptation parameter matrix is ​​constructed from a first trainable matrix and a second trainable matrix, wherein the rank of the first trainable matrix and the rank of the second trainable matrix are both less than the rank of the original weight matrix.

[0056] In one exemplary embodiment of this disclosure, constructing a low-rank adaptation parameter matrix includes:

[0057] Construct a low-rank fitting parameter matrix based on the first trainable matrix and the transpose of the second trainable matrix.

[0058] According to a second aspect of this disclosure, a motion control method for a humanoid robot is provided, comprising:

[0059] Acquire the body perception information of the humanoid robot;

[0060] The proprioceptive information is input into the trained policy network, and the action policy for controlling the humanoid robot's movement is output based on the policy network.

[0061] The policy network is obtained according to the training method for humanoid robot motion control model based on imitation learning in the first aspect of this disclosure.

[0062] According to a third aspect of this disclosure, a training device for a humanoid robot motion control model based on imitation learning is provided, comprising:

[0063] The network pre-training module is used to acquire an action dataset containing multiple reference human action sequences and pre-train a policy network based on imitation learning based on the action dataset. The input of the policy network includes ontology perception information and relative target position information.

[0064] The network adjustment module is used to introduce low-rank adaptation parameter matrices into each linear transformation module in the policy network based on the pre-trained policy network, and combine the low-rank adaptation parameter matrices with the original weight matrix for subsequent action output generation.

[0065] The reward calculation module is used to input the target action into the policy network after adding a low-rank adaptation parameter matrix, and calculate the attitude tracking reward by the deviation from the reference attitude.

[0066] The post-training module is used to fine-tune the policy by updating the low-rank adaptation parameter matrix based on pose tracking rewards to complete the target action.

[0067] According to a fourth aspect of this disclosure, a motion control device for a humanoid robot is provided, comprising:

[0068] The input information acquisition module is used to acquire the body perception information of the humanoid robot;

[0069] The motion strategy output module is used to input the body perception information into the trained policy network and output the motion strategy to control the humanoid robot's movement based on the policy network.

[0070] The policy network is obtained according to the training method for humanoid robot motion control model based on imitation learning in the first aspect of this disclosure.

[0071] According to a fifth aspect of this disclosure, an electronic device is provided, comprising:

[0072] Processor; and

[0073] A memory that stores computer-readable instructions, which, when executed by a processor, implement the method as described in the above embodiments.

[0074] According to a sixth aspect of this disclosure, a humanoid robot is provided, comprising:

[0075] Processor; and

[0076] A memory that stores computer-readable instructions, which, when executed by a processor, implement the method as described in the above embodiments.

[0077] According to a seventh aspect of this disclosure, a computer-readable storage medium is provided that stores computer program code instructions that, when invoked by a processor, cause the processor to perform the method as described in the above embodiments.

[0078] As can be seen from the above technical solution, this disclosure possesses at least one of the following advantages and positive effects:

[0079] This disclosure pre-trains a policy network based on a reference human action dataset, enabling the network to directly reuse existing action representations in new action tasks. This provides strong action prior knowledge from the initial model stage, significantly reducing the training cycle required for policy transfer. Furthermore, a low-rank adaptation parameter matrix is ​​introduced into each linear transformation module and combined with the original weight matrix for action output generation, allowing the model to maintain its pre-trained structure while possessing parameter adjustability. During fine-tuning of the target action, a pose tracking reward is calculated based on the deviation from the reference pose, and the low-rank adaptation parameter matrix is ​​updated based on this reward. This allows for gradual optimization of the action output without retraining all network parameters, achieving efficient tracking and rapid adaptation of the target action. Therefore, this disclosure enables efficient parameter tuning for new action tasks based on pre-trained models that mimic existing action patterns, thereby reducing model training costs and improving training efficiency. Simultaneously, it allows the model to maintain stable action output even without encountering new action tasks. Attached Figure Description

[0080] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0081] Figure 1 A system architecture diagram is shown that can be applied to the humanoid robot motion control model training method and the humanoid robot motion control method based on imitation learning in the embodiments of this disclosure.

[0082] Figure 2 A flowchart illustrating a method for training a humanoid robot motion control model based on imitation learning, as described in an embodiment of this disclosure, is shown.

[0083] Figure 3 A schematic diagram of an action sequence length distribution histogram is shown in an embodiment of this disclosure.

[0084] Figure 4 The illustration shows a flowchart of a process for pre-training a policy network based on imitation learning using an action dataset, according to an embodiment of this disclosure.

[0085] Figure 5 This illustration shows a flowchart of a policy network pre-training based on training segments according to an embodiment of the present disclosure.

[0086] Figure 6 A schematic diagram of a process for calculating attitude tracking reward by deviation from a reference attitude is shown in an embodiment of this disclosure.

[0087] Figure 7 A schematic diagram of a process for calculating a contact matching reward based on a deviation from a reference contact state is shown in an embodiment of this disclosure.

[0088] Figure 8 A schematic diagram of a process for calculating speed matching rewards by means of deviation from a reference motion speed is shown in an embodiment of this disclosure.

[0089] Figure 9 A flowchart illustrating a motion control method for a humanoid robot according to an embodiment of this disclosure is shown.

[0090] Figure 10 A schematic diagram comparing the execution effects of different training strategies in an embodiment of this disclosure is shown.

[0091] Figure 11 A block diagram of a humanoid robot motion control model training device based on imitation learning is shown in an embodiment of this disclosure.

[0092] Figure 12 A block diagram of a humanoid robot motion control device according to an embodiment of the present disclosure is shown.

[0093] Figure 13 A schematic diagram of a humanoid robot according to an embodiment of this disclosure is shown.

[0094] Figure 14 A schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present disclosure is shown.

[0095] Figure 15 A schematic diagram of a computer-readable storage medium according to an embodiment of the present disclosure is shown. Detailed Implementation

[0096] In this disclosure, the terms "first" and "second" are used for description only and do not indicate relative importance or imply the number of technical features. Therefore, the features referred to as "first" or "second" may explicitly or implicitly include at least one of those features. "A plurality of" means at least two, unless otherwise expressly defined.

[0097] First, the relevant terms used in the exemplary embodiments of this disclosure will be explained:

[0098] Imitation learning refers to a learning method that uses external demonstration data as a learning basis, and obtains a motion control strategy by making the model's output actions or postures consistent with the demonstration actions in terms of time and space. In the embodiments of this disclosure, imitation learning is used to pre-train a policy network using human action sequences, enabling the policy network to have basic human motion pattern representation capabilities.

[0099] Reference human motion sequences refer to model data used for imitation learning and training, which can represent continuous motion processes with semantic features of human actions. Reference human motion sequences can be derived from real robot execution logs, human motion data collected by motion capture systems (after being mapped by robot kinematics), trajectories generated by high-fidelity simulation platforms, or publicly available motion datasets.

[0100] A policy network based on imitation learning refers to a motion control model constructed using a neural network structure. Its parameters are trained through imitation learning to output corresponding action policies based on input state information. In this embodiment, the policy network is trained by referencing human action sequences during the pre-training phase and fine-tuned in subsequent phases based on a low-rank adaptation parameter matrix.

[0101] Propriocity perception information refers to data collected by the robot's own sensors that describes the robot's current motion state, including but not limited to joint angles, joint angular velocities, attitude information, and inertial measurement information. Propriocity perception information serves as one of the inputs to the policy network, characterizing the robot's own motion state.

[0102] Relative target position information refers to information describing the relative relationship between the robot's current state and a reference human action or target action, such as the displacement or directional difference of the current posture relative to the reference posture. In this embodiment, relative target position information can be used together with ontology perception information as input to the policy network.

[0103] A linear transformation module is a network structural unit in the policy network that performs linear mapping operations, typically combining input features linearly in the form of a weight matrix. In this embodiment, the linear transformation module includes a network layer that can incorporate a low-rank adaptation parameter matrix.

[0104] The original weight matrix refers to the parameter matrix obtained by each linear transformation module in the policy network during the imitation learning pre-training stage. It is used to perform the linear mapping operation from input features to output features. After introducing the low-rank adaptation parameter matrix, the original weight matrix remains unchanged and participates in the calculation of subsequent action outputs together with the low-rank adaptation parameter matrix.

[0105] Low-rank adaptation parameter matrix: refers to a set of trainable parameter matrices with limited dimensions and lower rank compared to the original weight matrix. It is used to adapt and adjust the network output while keeping the original weight matrix unchanged. In this embodiment, the low-rank adaptation parameter matrix is ​​used to fine-tune the target action of the policy network after pre-training.

[0106] Pose tracking reward: This refers to the reward amount calculated based on the deviation between the action result output by the policy network and the corresponding pose in the reference human action sequence. It is used to measure the degree to which the robot's current pose tracks the reference pose. In this embodiment, the pose tracking reward is used to guide the update of the low-rank adaptation parameter matrix.

[0107] Policy fine-tuning refers to the process of further training only some parameters in the policy network after the policy network has completed imitation learning pre-training, for a specific target action or action style. In the embodiments of this disclosure, policy fine-tuning is achieved by updating only the low-rank adaptation parameter matrix.

[0108] Slicing refers to the process of dividing a continuous sequence of actions into multiple sub-sequences according to a predetermined time length or number of steps, used to construct training samples or improve the stability of the training process. In the embodiments of this disclosure, slicing is used to process a reference human action sequence to generate action fragments suitable for training.

[0109] Contact matching reward: refers to the reward amount calculated based on the consistency between the contact state generated by the robot during the execution of an action and the corresponding contact state in a reference human action. It is used to constrain the robot to maintain reasonable contact behavior during the execution of an action.

[0110] Speed ​​matching reward: refers to the reward amount calculated based on the difference between the current movement speed of the robot's joints or body parts and the corresponding speed in the reference human movement, used to constrain the speed change characteristics of the robot during movement.

[0111] Course learning strategy: This refers to a training strategy that gradually adjusts the training difficulty or reward constraint strength during the training process, allowing the model to gradually transition from more lenient constraints to more stringent constraints. In the embodiments of this disclosure, the course learning strategy can be used to gradually increase the constraint level of any one of the posture tracking reward, contact matching reward, and velocity matching reward.

[0112] The tolerance parameter for the reward item refers to a parameter used to control the allowable error range during reward calculation, adjusting the sensitivity of the reward function to attitude, velocity, or contact deviations. In this disclosure, the tolerance parameter can be adjusted during the training phase to align with the implementation of the course learning strategy.

[0113] Figure 1 A system architecture diagram is shown that can be applied to the humanoid robot motion control model training method based on imitation learning and the humanoid robot motion control method in the embodiments of this disclosure. Figure 1 As shown, the system architecture 100 may include a terminal device 101, a robot 102, a network 103, and a server 104.

[0114] The terminal device 101 includes, but is not limited to, desktop computers, laptops, smartphones, and tablets. The terminal device 101 is equipped with a graphical user interface, providing visualization capabilities to display the robot 102's operating status, proprioceptive perception information, and action results output by the policy network. Furthermore, the terminal device 101 also provides parameter configuration and management functions, allowing users to set parameters related to imitation learning training, target action selection, posture tracking reward calculation, etc., and to collaborate with the server 104 to complete the visualization analysis and management of the training process.

[0115] Robot 102 can be a humanoid robot, equipped with inertial measurement units, joint angle encoders, and other proprioceptive sensors to collect its own joint states, posture information, and motion state data related to target actions in real time. It is important to note that robot 102 includes at least a main controller and joint actuators. The main controller deploys and runs a trained policy network, receives proprioceptive information, and inputs this information into the policy network to output motion policies for controlling the humanoid robot's movement. The joint actuators receive the motion policies from the main controller and drive the corresponding joints to perform movements accordingly, thereby achieving motion control of the humanoid robot.

[0116] Server 104 is equipped with a model training module for humanoid robot motion control. This module acquires a motion dataset containing multiple reference human motion sequences and pre-trains a policy network based on imitation learning using this dataset. Following pre-training, a low-rank adaptation parameter matrix is ​​introduced into the linear transformation module of the policy network. Based on the pose tracking reward corresponding to the target action, only the low-rank adaptation parameter matrix is ​​updated to fine-tune the policy. After training, server 104 can deploy the trained policy network to robot 102 for real-time motion policy generation during its operation.

[0117] Network 103 serves as a medium to provide a communication link between terminal device 101, robot 102, and server 104, enabling the exchange of sensor data transmission, model parameter distribution, and training process control information. Network 103 can include various connection types, such as wired, wireless communication links, or fiber optic cables, etc. It should be understood that... Figure 1 The number and type of terminal devices, robots, networks, and servers shown are merely illustrative. Depending on implementation needs, there can be any number and type of terminal devices, robots, networks, and servers; for example, the servers could be cloud servers.

[0118] Through the coordinated operation of various parts in the system architecture 100, a complete process can be realized, from human motion data-driven imitation learning pre-training, target motion fine-tuning based on low-rank adaptation parameter matrix, to real-time motion strategy output and execution of humanoid robots, thereby improving the humanoid robot's adaptability to target motions while ensuring model stability.

[0119] This disclosure provides an example implementation of a method for training a humanoid robot motion control model based on imitation learning. (See references) Figure 2 As shown, the method may include the following steps S201 to S204:

[0120] Step S201: Obtain an action dataset containing multiple reference human action sequences, and pre-train a policy network based on imitation learning based on the action dataset. The input of the policy network includes ontology perception information and relative target position information.

[0121] Step S202: Based on the pre-trained policy network, a low-rank adaptation parameter matrix is ​​introduced for each linear transformation module in the policy network, and the low-rank adaptation parameter matrix is ​​combined with the original weight matrix for subsequent action output generation.

[0122] Step S203: Input the target action into the policy network after adding the low-rank adaptation parameter matrix, and calculate the attitude tracking reward by the deviation from the reference attitude.

[0123] Step S204: Based on the posture tracking reward, update the low-rank adaptation parameter matrix to fine-tune the strategy for completing the target action.

[0124] This disclosure pre-trains the policy network based on a reference human action dataset, enabling the model to directly reuse existing action representations in new action tasks. This provides the model with action prior knowledge in the early stages of training, shortening the training cycle required for policy transfer. Furthermore, by introducing low-rank adaptation parameter matrices into each linear transformation module and updating these matrices based on pose tracking rewards during the target action fine-tuning phase, the model achieves rapid adaptation and optimization to the target action while maintaining its pre-trained structure. This reduces training costs while improving training efficiency and allows the model to maintain stable action output in unseen action tasks.

[0125] The following will provide a detailed description of the training method for the humanoid robot motion control model based on imitation learning in this example embodiment.

[0126] In step S201, an action dataset containing multiple reference human action sequences is obtained, and a policy network based on imitation learning is pre-trained based on the action dataset. The input of the policy network includes ontology perception information and relative target position information.

[0127] In the exemplary embodiments of this disclosure, the action dataset may include multiple reference human action sequences. During model learning and training, each reference human action sequence may be represented in the form of state data to describe the changes in the motion state of the reference human action over time, serving as input or reference for the policy network to learn the semantic features of human actions. State data includes, but is not limited to: angles, angular velocities, angular accelerations of each joint, contact states of the foot or contact point (e.g., whether it is in contact with the ground, the magnitude of the contact force), proprioceptive sensor data (e.g., attitude, angular velocity, linear acceleration output by the inertial measurement unit), and physical quantities or control parameters describing the overall motion state, such as body pose (e.g., torso orientation, position in the global coordinate system).

[0128] For example, a reference human movement sequence in the motion dataset can be used as a demonstration of a standing jump. The state data corresponding to this reference human movement sequence can include continuous changes in joint and center-of-mass states from knee flexion and charging, to push-off and takeoff. As another example, another reference human movement sequence in the motion dataset can serve as a demonstration of walking, standing, or posture adjustment, and corresponding state data reflecting the movement process can be constructed. It should be understood that the motion dataset may also include other types of reference human movement sequences, and this disclosure does not limit this.

[0129] During pre-training, state data from reference human action sequences can be used as a reference target for the policy network during imitation learning training. By comparing the reference target with the current motion state of the humanoid robot, the policy network learns the correspondence between the humanoid robot's motion state and the reference human actions, thereby gaining the ability to represent human action features.

[0130] For example, the acquired motion dataset may include several AMASS (Archive of MotionCapture as Surface Shapes) motion sequences. (See reference) Figure 3 As shown, a histogram of action sequence length distribution is presented through statistical analysis of several AMASS action sequences. The horizontal axis of the histogram represents the duration of the action sequence in seconds, and the vertical axis represents the number of action sequences within the corresponding duration interval. The histogram shows that AMASS action sequences are concentrated in shorter duration intervals, and their number gradually decreases as the duration of the action sequence increases, exhibiting a clear long-tail distribution characteristic.

[0131] Through such Figure 3The action dataset size and action sequence duration distribution shown enable the policy network to access demonstration data covering different action durations and different motion patterns during pre-training. This facilitates the policy network's learning of diverse human action features and provides a stable parameter initialization foundation for subsequent policy fine-tuning based on target actions.

[0132] After obtaining the action dataset, training samples can be constructed based on reference human action sequences. These training samples are then input into a policy network for pre-training, enabling the policy network to learn the motion patterns corresponding to the reference human action sequences. Through this pre-training process, the policy network obtains initial parameters to represent the features of human actions, providing a foundation for subsequent policy fine-tuning based on a low-rank fitting parameter matrix.

[0133] In some example implementations, the pre-training process can be carried out by randomly slicing the action dataset and constructing training samples, and then updating the parameters of the policy network in conjunction with an imitation learning reward mechanism.

[0134] refer to Figure 4 As shown, the process of pre-training a policy network based on imitation learning using an action dataset may include the following steps S401 and S402:

[0135] Step S401: Randomly select the starting frame of the slice, and extract continuous frame motion data from the motion dataset according to the preset time length to generate training segments.

[0136] Specifically, a starting frame of a slice can be randomly selected from the action dataset, and continuous frames of action data can be extracted from the corresponding reference human action sequence according to a preset time length to obtain a training segment. For example, the preset time length can be set to 10 seconds, that is, starting from the starting frame of the slice, continuous frames of data corresponding to a duration of 10 seconds are extracted as a training segment. The preset time length is used to limit the duration of each training segment to ensure that the training segment has complete local action continuity in the time dimension. This disclosure does not limit the specific value of the preset time length.

[0137] By randomly selecting the starting frame of a slice, different training segments can cover different time intervals in the same reference human action sequence, thereby enabling the generated training segments to encompass the state change information corresponding to the start phase, intermediate transition phase, and end phase of the action.

[0138] For example, for a reference human action sequence that includes a standing take-off, training segments can be generated by random slicing that only cover the knee flexion and power-gathering phase, or training segments that cover the push-off and take-off phases, so that the policy network is exposed to different temporal segments of the same action during pre-training.

[0139] Step S402: Based on the training segments, perform imitation learning pre-training on the policy network.

[0140] Specifically, training samples can be constructed based on continuous frame data in the training segment, and an imitation learning reward mechanism can be introduced during training to update the parameters of the policy network. In the example implementation of this disclosure, the imitation learning reward is used to measure the consistency between the action output by the policy network and the corresponding reference human action in the training segment. It may include pose tracking reward, contact matching reward, etc., and is used to constrain the action output by the policy network to maintain consistency with the corresponding reference human action in the training segment in terms of pose changes and contact behavior.

[0141] For example, for any given moment in a training segment, the corresponding training sample can be input into the policy network to obtain the policy network output, and the deviation between the policy network output and the corresponding reference state in the training segment can be calculated. Subsequently, the policy network parameters can be updated based on the imitation learning reward corresponding to the deviation, thereby guiding the policy network to gradually approximate the motion state change pattern corresponding to the reference human action.

[0142] By repeatedly performing the pre-training process within the time frame of the training segments and iteratively training multiple training segments, the policy network can gradually learn the motion patterns and semantic features corresponding to human actions during the pre-training phase, thereby forming general initial model parameters and providing a foundation for subsequent policy fine-tuning based on target actions.

[0143] In the exemplary embodiments of this disclosure, the input to the policy network may include proprioceptive perception information and relative target position information. The proprioceptive perception information characterizes the humanoid robot's current motion state, while the relative target position information characterizes the relative relationship between the humanoid robot's current state and the corresponding target state in a reference human action sequence. By simultaneously introducing proprioceptive perception information and relative target position information, the policy network can establish a correspondence between the humanoid robot's state and the reference human actions during the pre-training phase.

[0144] Accordingly, refer to Figure 5 As shown, step S402 may further include steps S501 to S504:

[0145] Step S501: Extract the corresponding reference human actions frame by frame from the training clip.

[0146] The training segment consists of consecutive frames extracted from the action dataset, with each frame having a clear temporal order. Therefore, each frame in the training segment can be considered a discrete-time reference human action sample, and the corresponding reference human action at that moment can be extracted for the construction of subsequent training inputs and the calculation of relative target position information.

[0147] For example, reference human motion information can be read sequentially according to the frame number order of the training segment. For instance, if the training segment contains frames k to k+N, the reference human motion corresponding to frame k, frame k+1, up to frame k+N can be extracted sequentially.

[0148] By reading frame by frame, it is possible to ensure that the reference human actions and the current state information are strictly aligned in the time dimension during subsequent training, thus avoiding training bias caused by frame misalignment.

[0149] For example, when a training segment corresponds to a local phase of a standing take-off, the reference human movements extracted frame by frame can sequentially reflect the posture changes in the phases of knee flexion, push-off, and take-off, thus providing a clear temporal demonstration for the policy network and guiding it to learn the continuous motion patterns within that phase.

[0150] Step S502: Obtain the body perception information of the humanoid robot and determine the current state of the humanoid robot based on the body perception information.

[0151] The ontological perception information of a humanoid robot can be provided by robot sensors or a simulation environment, and is used to characterize the robot's own motion at the current moment. Based on the ontological perception information, a current state representation can be constructed for input to the policy network.

[0152] It should be noted that during pre-training, the current state of the humanoid robot corresponds to the state input during training. It can be directly constructed from proprioceptive perception information, or obtained by combining, selecting, or formatting proprioceptive perception information as necessary, thereby satisfying the policy network's requirements for input format. It should be understood that the specific method of constructing the current state can be set according to the structural design of the policy network and training requirements; this disclosure does not limit this approach.

[0153] Specifically, proprioceptive information can be directly input into the policy network as the current state. For example, collected joint state information, posture information, or other proprioceptive data can be concatenated in a preset order to form a state vector, which is then used as input to the policy network. Alternatively, some state variables relevant to the current training phase can be selected from the proprioceptive information as state input, or different types of proprioceptive information can be normalized, scaled, or dimensionally aligned to form a state representation that meets the input dimension requirements of the policy network. Furthermore, a state vector containing multi-dimensional state variables can be constructed based on proprioceptive information to reflect the comprehensive motion state of the humanoid robot at the joint space and overall posture levels.

[0154] The state construction method in this step enables the current state to be used as training input in a form suitable for policy network processing without changing the semantics of the ontology perception information, thus providing a stable data foundation for imitation learning training.

[0155] Step S503: Calculate the relative target position information of the humanoid robot based on the current state and the reference human action of the corresponding frame.

[0156] Relative target position information is used to provide the policy network with motion instructions relative to the reference human action during training, enabling the policy network to gradually approach the target state corresponding to the reference human action based on the current state.

[0157] For example, when the reference human action in the corresponding frame is in the knee-bending, power-building phase of a jump, the relative target position information can reflect the difference between the current trunk height or joint posture and the reference posture. When the reference human action is in the airborne phase, the relative target position information can reflect the relative relationship between the current center of mass position and the reference center of mass position, thus providing the policy network with phased and clear motion guidance.

[0158] By introducing relative target position information, the policy network can not only perceive its current state, but also obtain the relative relationship with the reference human actions, thereby improving the action alignment ability in the imitation learning process.

[0159] Specifically, the relative target position information can be determined by the difference or relative transformation relationship between the current state of the humanoid robot and the target position information extracted from the corresponding frame's reference human action. This reflects the degree of relative deviation between the current state and the reference target state in terms of spatial position, orientation, or posture. Alternatively, it can be obtained by calculating the differences in relative displacement, relative direction, or relative posture between the current state and the reference target state. Furthermore, the reference target state can be represented in a relative coordinate system based on the current state to reduce the impact of absolute coordinate changes on the training process.

[0160] It should be understood that the specific construction method and representation form of relative target position information can be set according to training needs and policy network structure, and this disclosure does not limit it in this regard. Based on this, relative target position information can provide clear and effective motion guidance for the policy network, improving the stability and effectiveness of imitation learning training.

[0161] Step S504: Input the humanoid robot's body perception information and relative target position information into the policy network to perform imitation learning pre-training on the policy network.

[0162] After inputting ontological perception information and relative target position information into the policy network, the policy network outputs a motion policy for controlling the humanoid robot's movement. Subsequently, an imitation learning training signal can be constructed based on the consistency between the motion policy output by the policy network and the corresponding frame reference human action, and the policy network parameters can be updated accordingly, so that the policy network output gradually approximates the motion law corresponding to the reference human action.

[0163] By repeatedly executing steps S501 to S504 on each frame of the training segment and iteratively training multiple training segments, the policy network can gradually learn the motion patterns contained in different action sequences and different time stages during the pre-training phase, thereby forming universal initial model parameters and providing a foundation for subsequent policy fine-tuning based on target actions.

[0164] In step S202, based on the pre-trained policy network, a low-rank adaptation parameter matrix is ​​introduced for each linear transformation module in the policy network, and the low-rank adaptation parameter matrix is ​​combined with the original weight matrix for subsequent action output generation.

[0165] After pre-training, each linear transformation module in the policy network has a parameter matrix obtained during the pre-training phase, which is retained as the original weight matrix. To achieve efficient parameter updates in the subsequent fine-tuning phase, the exemplary implementation of this disclosure can introduce a low-rank adaptation parameter matrix corresponding to at least one linear transformation module. This low-rank adaptation parameter matrix is ​​used to incrementally adjust the linear mapping capability of the original weight matrix in a low-rank form.

[0166] For example, for any linear transformation module in a policy network, the original weight matrix can be denoted as W1. After introducing the low-rank adaptation parameter matrix W2, a combined weight matrix W3 can be constructed for this linear transformation module. W3 satisfies a preset combination relationship with W1 and W2, such as W3 = W1 + W2, or satisfies other equivalent linear combination relationships.

[0167] By combining the weight matrix W3, the linear transformation module can simultaneously reflect the original mapping capability obtained from pre-training and the incremental adjustment introduced by the low-rank adaptation parameter matrix during forward computation, thus enabling its use in subsequent action output generation.

[0168] It should be noted that the low-rank adaptation parameter matrix can be introduced to cover multiple linear transformation modules in the policy network, or it can cover only some of them. Accordingly, the rank, dimension, and combination method of the low-rank adaptation parameter matrix with the original weight matrix can be configured according to implementation requirements, and this disclosure does not impose any restrictions on this.

[0169] To ensure the inheritance of pre-training capabilities, optionally, the original weight matrix remains unchanged after the introduction of the low-rank adaptation parameter matrix, and remains frozen in the subsequent policy fine-tuning stage, with only the low-rank adaptation parameter matrix being updated.

[0170] This step enables an adjustable extension of the linear mapping capability while maintaining the original structure and function of the policy network. It provides a parameter basis for subsequent input of target actions and policy fine-tuning based on posture tracking rewards, allowing the policy network to adapt to the target action requirements while retaining the semantic features of pre-trained actions.

[0171] In some example implementations, the low-rank fitting parameter matrix can be constructed from a first trainable matrix and a second trainable matrix. Both the first and second trainable matrices are parameter matrices that participate in updates during subsequent training phases, and their ranks are both less than the rank of the original weight matrix. By restricting the low-rank fitting parameter matrix to a lower-rank parameter space, its parameter size is significantly reduced relative to the original weight matrix, thereby reducing the number of parameters that need to be updated during subsequent training, lowering training computational overhead, and improving training efficiency.

[0172] Optionally, a low-rank adaptation parameter matrix can be constructed based on the first trainable matrix and the transpose of the second trainable matrix. For example, by performing matrix multiplication or an equivalent linear combination operation on the transpose of the first and second trainable matrices, a low-rank adaptation parameter matrix for adjusting the original weight matrix can be obtained. The resulting low-rank adaptation parameter matrix can characterize the adjustment components of the linear mapping result of the original weight matrix in low-dimensional parameter form.

[0173] After introducing the low-rank adaptation parameter matrix, the linear transformation module can perform linear mapping calculations based on the combination of the original weight matrix and the low-rank adaptation parameter matrix during forward computation. Specifically, the linear mapping introduces additional mapping components provided by the low-rank adaptation parameter matrix while keeping the original weight matrix unchanged, thereby allowing the output of the linear transformation module to be appropriately adjusted based on the pre-trained mapping results.

[0174] Since the low-rank adaptation parameter matrix is ​​constructed from the rank-constrained first and second trainable matrices, its influence on the linear mapping result is mainly concentrated within the low-dimensional subspace spanned by the low-rank parameters. Adjusting within this low-dimensional subspace facilitates fine-tuning of the pose or motion features related to the target action in the subsequent policy fine-tuning stage, while avoiding significant interference with the general action pattern represented by the original weight matrix.

[0175] It should be noted that the specific dimension settings, rank selection, and combination methods of the first and second trainable matrices can be configured according to the parameter scale and training requirements of different linear transformation modules, and this disclosure does not limit them.

[0176] In step S203, the target action is input into the policy network after adding the low-rank adaptation parameter matrix, and the attitude tracking reward is calculated by the deviation from the reference attitude.

[0177] After pre-training and introducing a low-rank adaptation parameter matrix, the policy network can be further trained for the target action. In this stage, pose tracking rewards can be used as a key reward in the policy fine-tuning phase, participating in the updating of the policy network parameters. By introducing pose tracking rewards, the policy network can be guided to gradually adapt to the pose change features corresponding to the target action while maintaining the general action representation obtained in the pre-training phase.

[0178] The target action represents the type of action that the policy network needs to focus on learning and adapting. It can be trained using a reference pose sequence or an equivalent form. After introducing a low-rank adaptation parameter matrix, the policy network can generate corresponding action outputs based on the target action, which describe the humanoid robot's motion posture predicted by the policy network at the current moment.

[0179] For example, refer to Figure 6 As shown, the process of calculating the attitude tracking reward by the deviation from the reference attitude may include the following steps S601 to S603:

[0180] Step S601: Obtain the predicted pose vector corresponding to the control action output by the policy network, and the reference pose vector corresponding to the reference human action of the corresponding frame.

[0181] The policy network outputs control actions based on the input ontology perception information and relative target position information. The predicted pose vector corresponding to this control action is used to characterize the humanoid robot's motion posture at that moment. Simultaneously, reference pose vectors for corresponding frames are extracted from the reference human action sequence. These reference pose vectors characterize the expected pose state of the reference human action at the same time point, serving as a reference benchmark for posture tracking.

[0182] Step S602: Calculate the attitude deviation between the predicted attitude vector and the reference attitude vector.

[0183] Specifically, attitude bias is used to measure the degree of difference between the predicted attitude vector and the reference attitude vector, that is, to reflect the attitude consistency between the policy network's output control action and the reference human action. For example, attitude bias can be obtained by calculating the vector difference between the predicted attitude vector and the reference attitude vector, or by using methods such as Euclidean distance to calculate the degree of deviation between the two, thereby guiding the policy network to optimize its output action policy.

[0184] Step S603: Input the attitude deviation into the reward function to obtain the attitude tracking reward.

[0185] In this step, the reward value is typically inversely proportional to the magnitude of the attitude deviation; that is, a smaller attitude deviation results in a larger attitude tracking reward, and vice versa. By mapping the attitude deviation to a reward function, the attitude tracking reward can reflect the policy network's tracking performance of the target action at the attitude level, thereby effectively guiding the policy network to adjust its control actions and improving the accuracy of the target action.

[0186] For example, by inputting the attitude deviation into the reward function to obtain the attitude tracking reward, we have:

[0187]

[0188] in, For attitude tracking rewards, For attitude deviation, For reference to human movements t The reference pose vector at time step, For policy networks in t The predicted attitude vector corresponding to the control action output at any given time. The tolerance parameter for attitude tracking reward. exp () represents exponentiation, |·| 2 This represents the square Euclidean norm.

[0189] By continuously calculating posture deviations during training and updating the policy network based on posture tracking rewards, the difference between the predicted posture output by the policy network and the reference human action can be gradually reduced, thereby improving the posture tracking accuracy of the policy network for the target action and providing a more stable and natural motion control strategy for humanoid robots to perform target actions.

[0190] In step S204, based on the attitude tracking reward, the low-rank adaptation parameter matrix is ​​updated to complete the policy fine-tuning of the target action.

[0191] After completing the pre-training based on imitation learning, the policy network can be post-trained to further optimize the policy network for specific target actions.

[0192] In the exemplary implementation of this disclosure, the post-training phase is carried out on the basis of a policy network with a low-rank adaptation parameter matrix, and the posture tracking reward is used as the main optimization signal to guide the policy network to gradually adapt to the posture change features corresponding to the target action.

[0193] Specifically, during policy fine-tuning, the weight parameters in the original weight matrix obtained during the pre-training phase can be kept out of the update process, and only the low-rank fitting parameter matrix can be trained and updated. Since the parameter size of the low-rank fitting parameter matrix is ​​significantly smaller than the total number of parameters in the policy network, the stability and efficiency of the training process can be improved while reducing the number of trainable parameters.

[0194] This approach allows the policy network to adjust the linear mapping results using a low-rank fitting parameter matrix without altering the general motion representation formed during the pre-training phase. This gradually reduces the deviation between the predicted and reference postures, enabling the policy network to converge to a parameter configuration suitable for the target motion, thus achieving fine-tuning of specific target actions. Based on this, the resulting policy network can be used for subsequent humanoid robot motion control, thereby improving control performance.

[0195] In some example implementations, to further constrain the actions output by the policy network to maintain consistency with reference human actions at the contact behavior level, a contact matching reward can be introduced during training. The contact matching reward measures the consistency between the contact state corresponding to the policy network's output action and the contact state in the reference human action, thereby guiding the policy network to learn motion behaviors that conform to the contact timing and contact pattern of the reference action.

[0196] For example, the contact matching reward is calculated based on the deviation from the reference contact state, and the low-rank adaptation parameter matrix is ​​updated based on the attitude tracking reward and the contact matching reward to complete the policy fine-tuning of the target action. Specifically, in the policy fine-tuning stage, the contact matching reward and the attitude tracking reward can be used together as optimization signals to participate in the updating of the policy network parameters.

[0197] refer to Figure 7 As shown, the process of calculating the contact matching reward based on the deviation from the reference contact state may include the following steps S701 to S703:

[0198] Step S701: Obtain the predicted contact state vector corresponding to the control action output by the policy network, and the reference contact state vector corresponding to the reference human action of the corresponding frame.

[0199] During frame-by-frame training of the training segment, the policy network outputs control actions based on the input information at the current moment. Based on the control actions, the predicted contact state vector of the humanoid robot at that moment can be determined, which represents the contact status of each contact part of the robot (such as the feet or other predefined contact points) at the current moment. Simultaneously, the reference contact state vector of the corresponding frame can be extracted from the reference human action in the training segment corresponding to the current moment, which represents the contact state of the reference human action at that moment. By obtaining the predicted contact state vector and the reference contact state vector in the same time frame, the consistency of subsequent deviation calculations in the time dimension can be guaranteed.

[0200] Furthermore, this disclosure does not limit the specific representation, dimension, or value method of the predicted contact state vector and the reference contact state vector, as long as they can reflect the contact behavior during the action execution process.

[0201] Step S702: Calculate the contact state deviation between the predicted contact state vector and the reference contact state vector.

[0202] For example, the predicted contact state vector and the reference contact state vector can be compared element-by-element to obtain the contact state deviation, which characterizes the degree of difference between the two. The contact state deviation is used to quantify the degree of deviation of the contact behavior output by the policy network from the reference human action; the smaller the deviation, the more consistent the predicted contact state is with the reference contact state.

[0203] Step S703: Input the contact state deviation into the reward function to obtain the contact matching reward.

[0204] The reward function maps the contact state deviation to a reward value, so that a higher contact matching reward is obtained when the predicted contact state is more consistent with the reference contact state, while the contact matching reward decreases accordingly when the difference between the two is large.

[0205] By introducing contact matching rewards during training, the policy network can be guided to gradually learn and refer to the contact timing and contact patterns corresponding to human actions when outputting control actions, thereby improving the stability and rationality of humanoid robots in the process of action execution.

[0206] For example, by inputting the contact state deviation into the reward function, we obtain the contact matching reward, which is:

[0207]

[0208] in, To access matchmaking rewards, For contact state deviation, To reference human movements t The reference contact state vector at time 1. For policy networks int The predicted contact state vector corresponding to the control action output at any given time. Tolerance parameters for accessing matching rewards.

[0209] Within the time range corresponding to the training segment, steps S701 to S703 can be repeated at each moment, and the obtained contact matching reward can be used as one of the reward items in the imitation learning training process to participate in the update of the policy network parameters, so that the policy network gradually reduces the deviation between the predicted contact state and the reference contact state, thereby more accurately reproducing the contact behavior characteristics in the reference human action.

[0210] Furthermore, based on the joint constraint of pose tracking reward and contact matching reward, the policy network with introduced low-rank adaptation parameter matrix is ​​updated. During the update process, the original weight matrix obtained in the pre-training stage is not included in the update; only the low-rank adaptation parameter matrix is ​​trained and updated. In this way, the policy network gradually approximates the reference human action at the pose level while maintaining consistency with the reference action at the contact behavior level, thereby completing the policy fine-tuning for the target action.

[0211] By introducing contact matching rewards in conjunction with posture tracking rewards, the policy network can simultaneously take into account posture change features and contact behavior features without compromising the general action representation capabilities of the pre-training phase. This is beneficial for improving the rationality of the target action execution process and the overall control effect.

[0212] In some example implementations, to further constrain the motion velocity output by the policy network to maintain consistency with the reference human action during subsequent training, a velocity matching reward can be calculated based on the deviation from the reference motion velocity. During the policy fine-tuning phase, the policy network, after incorporating the low-rank adaptation parameter matrix, can be jointly optimized based on posture tracking rewards, contact matching rewards, and velocity matching rewards, thereby updating the low-rank adaptation parameter matrix to complete the policy fine-tuning for the target action.

[0213] Reference motion velocity refers to the velocity information obtained from reference human motion, used to characterize the speed and trend of the reference human motion at a corresponding moment. Reference motion velocity can be calculated from the state data in the reference human motion sequence, for example, determined by the changes between reference posture, position, or joint state in adjacent time frames, and is used to reflect the motion velocity characteristics of the reference human motion at that moment.

[0214] For example, refer to Figure 8 As shown, the process of calculating the speed matching reward by deviating from the reference motion speed may include the following steps S801 to S803:

[0215] Step S801: Obtain the predicted velocity vector corresponding to the control action output by the policy network, and the reference velocity vector corresponding to the reference human action of the corresponding frame.

[0216] During frame-by-frame training of the training segment, the policy network outputs control actions based on the input information at the current moment. Based on the control actions, a predicted velocity vector corresponding to the control action can be determined, which represents the humanoid robot's motion velocity at the current moment under the action of the control action. Simultaneously, a reference velocity vector can be obtained from the reference human action corresponding to the current moment in the training segment, which represents the motion velocity of the reference human action at that corresponding moment. By obtaining the predicted velocity vector and the reference velocity vector at the same time, it is ensured that subsequent deviation calculations are aligned in the time dimension.

[0217] The predicted velocity vector and the reference velocity vector are used to characterize the motion velocity information of the humanoid robot and the reference human action in the same representation space, respectively, so as to facilitate the comparison between the two. This disclosure does not limit the specific representation dimension and form of the two.

[0218] Step S802: Calculate the velocity deviation between the predicted velocity vector and the reference velocity vector.

[0219] Specifically, the predicted velocity vector can be compared with the reference velocity vector based on vector differences to obtain the motion velocity deviation, which reflects the degree of inconsistency between the two. The motion velocity deviation is used to quantify the degree of deviation of the policy network's output control action from the reference human action in terms of velocity.

[0220] For example, the velocity deviation can be determined by the difference vector between the predicted velocity vector and the reference velocity vector, or by the magnitude, norm, etc. of the difference vector; this disclosure does not limit this.

[0221] Step S803: Input the motion speed deviation into the reward function to obtain the speed matching reward.

[0222] The reward function maps the velocity deviation to a reward value, so that a higher velocity matching reward is obtained when the predicted velocity vector is closer to the reference velocity vector, while the velocity matching reward decreases accordingly when the deviation between the two increases.

[0223] By introducing speed-matching rewards during training, the control actions output by the policy network can be guided to maintain consistency with the speed change trend of the reference human action, thereby improving the continuity and stability of motion imitation.

[0224] For example, by inputting the speed deviation into the reward function, we obtain the speed matching reward, which is:

[0225]

[0226] in, For speed-matching rewards, For the deviation in motion speed, To reference human movements t The reference velocity vector at time t, For policy networks in t The predicted velocity vector corresponding to the control action output at any given time. Tolerance parameters for speed matching rewards.

[0227] Within the time frame of the training segment, steps S801 to S803 can be repeated at each moment to obtain the speed matching reward for the corresponding moment. The speed matching reward is then used as one of the reward items in the imitation learning training process to participate in the parameter update of the policy network, thereby enabling the policy network to gradually learn the motion speed pattern corresponding to human actions.

[0228] It should be noted that, in order to improve the stability of the policy network training process and gradually improve the tracking accuracy of the policy network for reference human actions, a course learning strategy can be introduced during the training process to adjust the constraint strength of each reward item in stages.

[0229] Specifically, during training, tolerance parameters can be set for each reward item in the reward function. These tolerance parameters adjust the range to which the reward function tolerates deviations between the predicted and reference results. In the early stages of training, the tolerance parameters can be set to larger values, allowing the reward function to have a higher tolerance for attitude deviations, velocity deviations, or contact state deviations. This ensures relatively smooth reward feedback even before the policy network has developed a stable action output capability.

[0230] As training progresses, the tolerance parameters corresponding to each reward item in the reward function can be gradually reduced based on the training progress. Training progress can be correlated with the number of training epochs, training steps, or the performance status of the policy network during training.

[0231] Taking posture tracking rewards as an example, in the early stages of training, the tolerance parameter can be set to 2, allowing the reward function to have a higher tolerance for deviations between the predicted and reference postures. At this stage, the policy network can still obtain effective reward signals even with large posture deviations. This helps the policy network prioritize learning the basic characteristics of the reference human actions in the overall posture change trend, avoiding instability or convergence difficulties caused by overly strict constraints.

[0232] As training progresses, the tolerance parameter in the pose tracking reward is gradually reduced, for example, from 2 to 1, so that the penalty for pose deviation in the reward function gradually increases. In the later stages of training, the smaller tolerance parameter makes the pose tracking reward more sensitive to subtle differences between the predicted pose and the reference pose, thereby guiding the policy network to further improve the tracking accuracy of the reference pose based on its understanding of the overall movement trend.

[0233] By gradually reducing the tolerance parameter, the reward function's constraint on the bias is gradually strengthened, thereby guiding the policy network to more accurately match the posture changes, movement speed, and contact behavior corresponding to the reference human actions in subsequent training stages.

[0234] Of course, besides adjusting the tolerance parameters based on the training progress, the tolerance parameters for each reward item can also be dynamically set according to the performance of the policy network during training, the type of the target action, or the stage of action execution. For example, the tolerance parameters can be adaptively adjusted based on the statistical characteristics of posture deviation or the trend of reward changes, or a larger tolerance parameter can be used in stages with large action changes, while a smaller tolerance parameter can be used in stages requiring fine control. In this way, the constraint strength of the reward function can be flexibly adapted to different training states and action characteristics, thereby further improving the stability and convergence effect of the training process.

[0235] By introducing a dynamic adjustment method for the reward function based on the course learning strategy during the training process, the policy network can complete the initial action learning under relatively relaxed reward constraints, and gradually transition to high-precision action tracking in the later stage of training. This is beneficial to improve the stability of the imitation learning training process and the final convergence effect, while avoiding the training instability problem caused by premature tightening of reward constraints.

[0236] This disclosure also provides a humanoid robot motion control method, which, after model training is completed, applies the trained policy network to the humanoid robot's operation phase, enabling the humanoid robot to utilize the pre-trained general motion representation capabilities and, in conjunction with fine-tuned parameter configurations, generate a motion control strategy that meets the target motion requirements, thereby achieving real-time control of the humanoid robot's motion.

[0237] refer to Figure 9 As shown, the method may include the following steps S901 and S902:

[0238] Step S901: Obtain the body perception information of the humanoid robot.

[0239] During the operation of a humanoid robot, its own sensors can collect real-time proprioceptive information to characterize its current motion state. This proprioceptive information reflects the robot's joint states, overall posture, and motion changes at any given moment, describing its current motion state and serving as the basis for subsequent action decisions by the policy network.

[0240] After acquiring the ontology perception information, it can be organized, filtered, or formatted according to the input structure of the policy network to construct a state input vector that meets the input requirements of the policy network, thereby ensuring that the acquired ontology perception information can be effectively received by the policy network and used for subsequent action decisions.

[0241] Step S902: Input the ontology perception information into the trained policy network, and output the action policy for controlling the humanoid robot's movement based on the policy network.

[0242] In this step, the policy network outputs a motion policy based on the currently input ontology perception information and combined with the human action semantic features and motion patterns learned during the training phase. The motion policy can be represented in the form of joint control variables, joint velocity commands, torque commands, or equivalent control commands. It instructs the humanoid robot on the motion control behavior to be performed at the current moment and is executed by the humanoid robot's joint actuators to drive each joint to complete the corresponding movement.

[0243] It should be noted that the policy network in this example implementation is based on another example implementation, such as... Figure 2 The training method shown is for a humanoid robot motion control model based on imitation learning.

[0244] By repeatedly executing steps S901 and S902 in consecutive time steps, the humanoid robot can continuously update its motion strategy based on real-time acquired proprioceptive information, thereby achieving continuous, smooth, and stable motion control.

[0245] refer to Figure 10 As shown, a schematic diagram comparing the execution effects of different training strategies is presented. During the verification process, a unified experimental setup was used to compare and verify different training strategies, and the number of training steps required for the imitation learning-based policy network to reach a preset control performance threshold was used as the evaluation metric. Lateral walking was set as the new objective task, and all training strategies were trained under the same environmental configuration, reward function, and convergence criterion. Figure 10 In the diagram, the horizontal axis represents the number of training steps, and the vertical axis represents the performance metric, with a performance threshold set to 0.935. Discrete data points represent the raw performance data for each training step, while the curves represent the performance trend after smoothing the raw performance data.

[0246] like Figure 10 As shown, the training strategies include training from scratch and training with a low-rank adaptation parameter matrix (hereinafter referred to as the low-rank matrix) introduced on top of pre-training. For training from scratch, the policy network is trained directly on the lateral walking task, requiring 2333 training steps to reach the control performance threshold. For training with a low-rank adaptation parameter matrix introduced on top of pre-training, the policy network is first pre-trained using an action dataset containing 40 walking actions. After pre-training, the low-rank adaptation parameter matrix is ​​used for training adjustments, allowing the policy network to reach the control performance threshold in the lateral walking task with only 444 training steps. Compared to training from scratch, the required training steps are reduced by approximately 5.25 times, significantly improving the efficiency of training data utilization. Furthermore, the model convergence speed is significantly faster during training before reaching the control performance threshold, especially before the control performance index reaches 0.8.

[0247] Therefore, under the same training conditions and evaluation criteria, the number of training steps required for different training strategies to reach the preset control performance threshold varies significantly. The technical solution disclosed herein, by adjusting only the low-rank parameters relevant to the new task while maintaining the stability of the pre-training backbone parameters, makes parameter adjustments more concentrated during the training process for the new task. This improves the training efficiency and convergence speed of the model in the new task without increasing the scale of the training data.

[0248] In an exemplary embodiment of this disclosure, a training device for a humanoid robot motion control model based on imitation learning is also provided. (Reference) Figure 11 As shown, the humanoid robot motion control model training device 1100 based on imitation learning includes a network pre-training module 1101, a network adjustment module 1102, a reward calculation module 1103, and a network post-training module 1104, wherein:

[0249] The network pre-training module 1101 is used to acquire an action dataset containing multiple reference human action sequences and pre-train a policy network based on imitation learning based on the action dataset. The input of the policy network includes ontology perception information and relative target position information.

[0250] The network adjustment module 1102 is used to introduce low-rank adaptation parameter matrices to each linear transformation module in the policy network based on the pre-trained policy network, and to combine the low-rank adaptation parameter matrices with the original weight matrix for subsequent action output generation.

[0251] The reward calculation module 1103 is used to input the target action into the policy network after adding the low-rank adaptation parameter matrix, and calculate the attitude tracking reward by the deviation from the reference attitude.

[0252] The post-training module 1104 is used for policy fine-tuning by updating the low-rank adaptation parameter matrix based on pose tracking rewards to complete the target action.

[0253] The specific details of each module in the above-mentioned humanoid robot motion control model training device based on imitation learning have been described in detail in the corresponding humanoid robot motion control model training method based on imitation learning, so they will not be repeated here.

[0254] In an exemplary embodiment of this disclosure, a motion control device for a humanoid robot is also provided. (See reference...) Figure 12 As shown, the humanoid robot motion control device 1200 includes an input information acquisition module 1201 and a motion strategy output module 1202, wherein:

[0255] The input information acquisition module 1201 is used to acquire the body perception information of the humanoid robot;

[0256] The motion strategy output module 1202 is used to input the body perception information into the trained policy network and output the motion strategy to control the humanoid robot's movement based on the policy network.

[0257] Among them, the policy network is based on, for example, Figure 2 The training method shown is for a humanoid robot motion control model based on imitation learning.

[0258] The specific details of each module in the aforementioned humanoid robot motion control device have been described in detail in the corresponding humanoid robot motion control method, so they will not be repeated here.

[0259] In an exemplary embodiment of this disclosure, a humanoid robot is also provided. The humanoid robot includes a processor and a memory, the memory storing computer-readable instructions. When the computer-readable instructions are executed by the processor, they implement the described method. (Reference) Figure 13 The diagram shown is a schematic of a humanoid robot.

[0260] refer to Figure 14 As shown, an electronic device capable of implementing the above method is also provided. The electronic device 1400 includes a processor 1401 and a memory 1402. The memory 1402 stores computer-readable instructions, which, when executed by the processor 1401, implement the method of this disclosure.

[0261] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, having stored thereon computer program code instructions that, when invoked by a robot's processor, cause the robot to perform the method as described in the embodiments.

[0262] refer to Figure 15 As shown, a program product 1500 for implementing the above-described method according to an embodiment of the present disclosure is described. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0263] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0264] Finally, the above preferred embodiments are only used to illustrate the technical solutions of this application and are not restrictive. Although this application has been described in detail, those skilled in the art should understand that changes in form and detail can be made without departing from the scope defined by the claims of this application. The dimensions in the drawings are not related to the specific physical object, and the physical object dimensions can be arbitrarily changed.

Claims

1. A method for training a humanoid robot motion control model based on imitation learning, characterized in that, include: A motion dataset containing multiple reference human motion sequences is obtained, and a policy network based on imitation learning is pre-trained based on the motion dataset. The input of the policy network includes ontology perception information and relative target position information. Based on the pre-trained policy network, a low-rank adaptation parameter matrix is ​​introduced for each linear transformation module in the policy network, and the low-rank adaptation parameter matrix is ​​combined with the original weight matrix for subsequent action output generation; The target action is input into the policy network after adding the low-rank adaptation parameter matrix, and the attitude tracking reward is calculated by the deviation from the reference attitude. Based on the posture tracking reward, the low-rank adaptation parameter matrix is ​​updated to fine-tune the strategy for completing the target action.

2. The training method for humanoid robot motion control model based on imitation learning according to claim 1, characterized in that, The pre-training of the imitation-based policy network based on the action dataset includes: Randomly select the starting frame of the slice, and extract continuous frame motion data from the motion dataset according to a preset time length to generate a training segment; Based on the training segments, the policy network is pre-trained using imitation learning.

3. The training method for humanoid robot motion control model based on imitation learning according to claim 2, characterized in that, The step of performing imitation learning pre-training on the policy network based on the training segments includes: Extract the corresponding reference human actions frame by frame from the training clips; Acquire the body perception information of the humanoid robot, and determine the current state of the humanoid robot based on the body perception information; Based on the current state and the reference human action in the corresponding frame, the relative target position information of the humanoid robot is calculated. The humanoid robot's body perception information and its relative target position information are input into the policy network to perform imitation learning pre-training on the policy network.

4. The method for training a humanoid robot motion control model based on imitation learning according to claim 1, characterized in that, The calculation of attitude tracking reward based on the deviation from the reference attitude includes: Obtain the predicted pose vector corresponding to the control action output by the policy network, and the reference pose vector corresponding to the reference human action of the corresponding frame; Calculate the attitude deviation between the predicted attitude vector and the reference attitude vector; The attitude deviation is input into the reward function to obtain the attitude tracking reward.

5. The training method for a humanoid robot motion control model based on imitation learning according to claim 4, characterized in that, The step of inputting the attitude deviation into the reward function to obtain the attitude tracking reward includes: in, For attitude tracking rewards, For attitude deviation, For reference to human movements t The reference pose vector at time step, For the policy network in t The predicted attitude vector corresponding to the control action output at any given time. This is the tolerance parameter for attitude tracking rewards.

6. The training method for a humanoid robot motion control model based on imitation learning according to claim 1, characterized in that, After inputting the target action into the policy network after adding the low-rank adaptation parameter matrix, the method further includes: The contact matching reward is calculated based on the deviation from the reference contact state. Based on the posture tracking reward and the contact matching reward, the low-rank adaptation parameter matrix is ​​updated to complete the policy fine-tuning of the target action.

7. The method for training a humanoid robot motion control model based on imitation learning according to claim 6, characterized in that, The calculation of the contact matching reward based on the deviation from the reference contact state includes: Obtain the predicted contact state vector corresponding to the control action output by the policy network, and the reference contact state vector corresponding to the reference human action of the corresponding frame; Calculate the contact state deviation between the predicted contact state vector and the reference contact state vector; The contact state deviation is input into the reward function to obtain the contact matching reward.

8. The method for training a humanoid robot motion control model based on imitation learning according to claim 7, characterized in that, The step of inputting the contact state deviation into the reward function to obtain the contact matching reward includes: in, To access matchmaking rewards, For contact state deviation, To reference human movements t The reference contact state vector at time 1. For the policy network in t The predicted contact state vector corresponding to the control action output at any given time. Tolerance parameters for accessing matching rewards.

9. The method for training a humanoid robot motion control model based on imitation learning according to claim 6, characterized in that, The method further includes: Speed ​​matching rewards are calculated based on the deviation from a reference motion speed; Based on the attitude tracking reward, the contact matching reward, and the velocity matching reward, the low-rank adaptation parameter matrix is ​​updated to complete the strategy fine-tuning of the target action.

10. The method for training a humanoid robot motion control model based on imitation learning according to claim 9, characterized in that, The calculation of the speed matching reward based on the deviation from the reference motion speed includes: Obtain the predicted velocity vector corresponding to the control action output by the policy network, and the reference velocity vector corresponding to the reference human action of the corresponding frame; Calculate the velocity deviation between the predicted velocity vector and the reference velocity vector; The speed deviation is input into the reward function to obtain the speed matching reward.

11. The method for training a humanoid robot motion control model based on imitation learning according to claim 10, characterized in that, The step of inputting the motion speed deviation into the reward function to obtain the speed matching reward includes: in, For speed-matching rewards, For the deviation in motion speed, To reference human movements t The reference velocity vector at time t, For the policy network in t The predicted velocity vector corresponding to the control action output at any given time. Tolerance parameters for speed-matching rewards.

12. The training method for a humanoid robot motion control model based on imitation learning according to any one of claims 4 to 11, characterized in that, The method further includes: The reward function is dynamically adjusted based on the course learning strategy.

13. The training method for a humanoid robot motion control model based on imitation learning according to claim 12, characterized in that, The dynamic adjustment of the reward function based on the course learning strategy includes: During training, the tolerance parameters of each reward item in the reward function are gradually reduced as the training progresses.

14. The method for training a humanoid robot motion control model based on imitation learning according to claim 1, characterized in that, The method further includes: During training, the weight parameters in the original weight matrix are frozen, and only the adaptation parameters in the low-rank adaptation parameter matrix are updated.

15. The method for training a humanoid robot motion control model based on imitation learning according to claim 1, characterized in that, The low-rank adaptation parameter matrix is ​​constructed from a first trainable matrix and a second trainable matrix, wherein the rank of the first trainable matrix and the rank of the second trainable matrix are both less than the rank of the original weight matrix.

16. The method for training a humanoid robot motion control model based on imitation learning according to claim 15, characterized in that, Constructing the low-rank adaptation parameter matrix includes: The low-rank adaptation parameter matrix is ​​constructed based on the first trainable matrix and the transpose of the second trainable matrix.

17. A method for motion control of a humanoid robot, characterized in that, include: Acquire the body perception information of the humanoid robot; The ontological perception information is input into a trained policy network, and an action strategy for controlling the humanoid robot's movement is output based on the policy network. The policy network is obtained using the humanoid robot motion control model training method based on imitation learning according to any one of claims 1 to 16.

18. A training device for a humanoid robot motion control model based on imitation learning, characterized in that, include: The network pre-training module is used to acquire an action dataset containing multiple reference human action sequences, and to pre-train a policy network based on imitation learning based on the action dataset. The input of the policy network includes ontology perception information and relative target position information. The network adjustment module is used to introduce low-rank adaptation parameter matrices into each linear transformation module in the policy network based on the pre-trained policy network, and combine the low-rank adaptation parameter matrices with the original weight matrix for subsequent action output generation; The reward calculation module is used to input the target action into the policy network after adding a low-rank adaptation parameter matrix, and calculate the attitude tracking reward by the deviation from the reference attitude. The network post-training module is used to update the low-rank adaptation parameter matrix based on the pose tracking reward to complete the policy fine-tuning of the target action.

19. A motion control device for a humanoid robot, characterized in that, include: The input information acquisition module is used to acquire the body perception information of the humanoid robot; The motion strategy output module is used to input the ontology perception information into the trained policy network and output the motion strategy for controlling the humanoid robot's movement based on the policy network. The policy network is obtained using the humanoid robot motion control model training method based on imitation learning according to any one of claims 1 to 16.

20. An electronic device, characterized in that, include: processor; as well as A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 17.

21. A humanoid robot, characterized in that, include: processor; as well as A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in claim 17.

22. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program code instructions that, when invoked by a processor, cause the processor to perform the method as described in any one of claims 1 to 17.

Images