Progressive motion interpolation training method for whole body motion learning of humanoid robot

Abstract

The application relates to a progressive motion interpolation training method for whole-body motion learning of a humanoid robot, which comprises the following steps: S1, obtaining a target human motion reference trajectory, converting the target human motion reference trajectory into a robot reference trajectory meeting joint constraints of the humanoid robot, and taking the robot reference trajectory as a target motion of the humanoid robot; S2, generating reference trajectories of different difficulty levels by interpolation and alignment between a standard standing posture and the target motion; S3, constructing a reinforcement learning environment based on the reference trajectories of different difficulty levels, constructing a reinforcement learning network, and training a tracking strategy in the reinforcement learning environment by using the reinforcement learning network, wherein in the training process, the difficulty is gradually increased from easy to difficult according to a tracking success rate of the trajectories under different difficulty levels and a rule is used to update the difficulty; and S4, judging whether an end condition is reached, if yes, ending the training, and if not, returning to step S2. Compared with the prior art, the application has the advantages of realizing progressive training from easy to difficult and the like.

Description

Technical Field

[0001] This invention relates to the field of humanoid robot control and intelligent motion learning, and in particular to a progressive motion interpolation training method for learning full-body movements in humanoid robots. Background Technology

[0002] Humanoid robots, with their near-human morphological structure, are widely envisioned for applications in diverse human-like behavioral scenarios, including daily assistance, specialized operations, performances, and complex tasks such as dance. However, directly training strategies for humanoid robots is extremely challenging. This is primarily because humanoid robots typically possess numerous joints and high degrees of freedom, resulting in highly coupled dynamics. Their high center of gravity leads to inherent instability, and maintaining motion continuity and dynamic balance is crucial when executing complex trajectories. In recent years, researchers have attempted to utilize large-scale human motion capture datasets (such as AMASS and LAFAN) combined with reinforcement learning methods to generate feasible trajectories adapted to robot mechanisms, enabling humanoid robots to mimic human movement and gradually master diverse skills. These methods lay the foundation for motion learning in humanoid robots, but issues such as slow convergence speed, training instability, and imprecise control remain problematic in learning highly complex motions, making it difficult to meet practical deployment requirements.

[0003] There are two main types of current motion learning methods. The first type is based on direct imitation and reinforcement learning, which converts human motion capture data into target trajectories and trains strategies to approximate reference actions in a simulation environment. However, this method suffers from extremely sparse reward signals in the early stages of high-difficulty actions, making it difficult for the robot to maintain balance and complete the target posture, resulting in slow training progress or even failure to converge. In complex postures, the robot is in a critical stable state, where even slight deviations can accumulate rapidly, easily leading to training failure. Furthermore, globally uniform rewards cannot reflect the independent load and coordination requirements of each joint, potentially causing some joints to overexert themselves and resulting in overall control instability. The second type is a phased training method involving teachers and students. A teacher model with global information is trained first, and then students, relying only on deployable observations, imitate the teacher's output. This method alleviates the training difficulty to some extent, but it requires fully training the teacher model first, followed by separate training of the student model, which is time-consuming and computationally expensive. The distribution of observation information between teachers and students differs, leading to performance degradation of students in real-world deployments. Furthermore, the teacher's failures or instability in high-difficulty actions are inherited and amplified by the students, affecting final convergence.

[0004] Furthermore, existing reinforcement learning algorithms typically employ a single global tolerance in reward design, making them susceptible to being dominated by simple actions and unable to obtain sufficient optimization signals for complex trajectories. Joint control is mostly based on uniform torque constraints rather than joint-specific optimization, failing to prevent increased energy consumption and hardware damage caused by excessive local actuation. Therefore, existing technologies still have significant shortcomings in terms of rapid convergence, reward balancing, safe control, and realistic deployability for complex, high-degree-of-freedom whole-body motions. There is an urgent need for a novel training framework that can progressively reduce learning difficulty, refine control constraints, and adapt to actual robot deployment. Summary of the Invention

[0005] The purpose of this invention is to provide a progressive motion interpolation training method for learning full-body movements of humanoid robots, which enables progressive training from easy to difficult at different difficulty levels.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] A progressive motion interpolation training method for full-body motion learning in humanoid robots includes the following steps:

[0008] S1. Obtain the target human motion reference trajectory and convert it into a robot reference trajectory that satisfies the joint constraints of the humanoid robot through the posture repositioning method, which serves as the target motion of the humanoid robot.

[0009] S2. Based on the robot reference trajectory, reference trajectories of different difficulty levels are generated between the standard standing posture and the target action through interpolation and alignment.

[0010] S3. Construct a reinforcement learning environment based on the reference trajectories of different difficulty levels, and construct a reinforcement learning network. In the reinforcement learning environment, use the reinforcement learning network to train a tracking strategy. During the training process, update the difficulty according to the trajectory tracking success rate at different difficulty levels to achieve progressive training from easy to difficult.

[0011] S4. Determine if the termination condition has been met. If yes, end the training. If no, return to step S2 until the termination condition is met.

[0012] Furthermore, the attitude reorientation method is either the PHC method or the GMR method.

[0013] Furthermore, the steps for generating the reference trajectories of different difficulty levels include:

[0014] Calculate the angular difference between the target action and the standard standing posture in the root orientation, wherein the expression for calculating the angular difference is:

[0015] Δψ = unwrap(ψ ref- ψ stand ),

[0016] In the formula, Δψ is the angle difference, and unwrap expands the angle difference to the continuous real number domain. ref ψ is the root orientation of the target motion. stand The orientation of the root in the standard standing posture;

[0017] Based on the aforementioned angle difference Δψ, the interpolation coefficient α (0 ≤ α ≤ 1) is used as the difficulty level to obtain the progressive root orientation ψ(α), expressed as:

[0018] ψ(α) = ψ stand + α·Δψ,

[0019] In the formula, ψ(α) represents the orientation of the root at difficulty level α;

[0020] First, the key points of the standard standing posture and the target action are aligned to the progressive root orientation through rotation transformation. Then, they are fused using a linear interpolation method to finally generate a reference trajectory with a difficulty level of α, represented as:

[0021] ,

[0022] ,

[0023] ,

[0024] In the formula, Let t be the new position of the k-th keypoint in the standard standing posture at time t with difficulty level α, after coordinate alignment. For rotational transformation, For the root orientation at time t with difficulty level α, The root orientation at time t is the standard standing posture. Let K be the coordinates of the k-th key point in the standard standing posture. These are the coordinates of the root reference point in the standard standing posture. The target root location, Let be the new position of the k-th key point on the tracking trajectory after coordinate alignment when the difficulty level is α at time t. To track the root orientation of the trajectory at time t, To track the coordinates of the k-th key point on the trajectory, To track the coordinates of the root reference point on the trajectory, This is a reference trajectory with a difficulty level of α. This is the attitude vector of the entire tracking trajectory after alignment. This is the pose vector for the entire standing posture after alignment.

[0025] Furthermore, the reinforcement learning environment includes:

[0026] The state space S includes joint position q and joint velocity. Root linear velocity v root angular velocity ω root and dynamic randomization parameters;

[0027] Action space A represents the torque or angle increments of each joint of the robot;

[0028] Environmental dynamics p(s) t+1 |s t , a t The state transitions and rewards are calculated by the physics simulation engine.

[0029] Reward function R.

[0030] Furthermore, the step of training the tracking strategy using a reinforcement learning network includes:

[0031] (1) Interactive sampling:

[0032] Get the initial state s0;

[0033] For each time step t:

[0034] The actor network receives its deployable observation information. t Output executable action a t ~ π θ (a t | o t ), π θ This represents the trajectory tracking strategy for a humanoid robot determined by parameter θ, wherein the observation information o t The next state transition s is calculated by taking into account the current joint position, velocity, root angular velocity, gravity projection, and short-time reference trajectory fragments of the humanoid robot. t+1 and reward r t ;

[0035] Transform tuple (s) t , o t , a t , r t , s t+1 , d, τ(α)) are stored in the experience replay buffer D, where d is the dynamic randomization parameter and τ(α) is the trajectory with difficulty level α;

[0036] (2) Network optimization:

[0037] Sampling: Sample batch data from the experience playback buffer D;

[0038] Critics Network Update: Based on the sampled batch data, the critics network estimates the value Q of the current state-action pair. φ (s t , a t , d, τ), and simultaneously calculate the target value y t and using the target value y t Minimize the difference error L(φ) to update the commentator parameter φ;

[0039] Actor Network Update: Based on sampled batch data, the actor network proposes action a. t Critics' online assessment of action a t Value Q φ (s t , a t , d, τ), and update the actor parameters θ using the policy gradient method to maximize the value of the critic network estimate;

[0040] Soft update: The target network parameters corresponding to the critic network and actor network are synchronized using a soft update method;

[0041] (3) Update:

[0042] Obtain newly generated trajectories at different difficulty levels within the current iteration training cycle, and calculate the trajectory tracking success rate at different difficulty levels based on them;

[0043] Based on the trajectory tracking success rate at different difficulty levels, the reward function is updated, and the difficulty level is adjusted using the difficulty scheduler to obtain the updated reward function and difficulty level, which are then used for the next iteration of training until the training ends.

[0044] Furthermore, in the network optimization process, the actor network update process serves as a policy update process. A joint-level torque loss function is introduced during this process to optimize the policy parameters based on the joint-level torque loss. The expression for the joint-level torque loss function is as follows:

[0045] L JTC = - E[min( r t,j × A t,j torque clip(r) t,j , 1-ε, 1+ε) × A t,j torque ) ],

[0046] r t,j = exp( log π θ,j - log π old,j ),

[0047] In the formula, L JTC Let E be the joint-level torque loss function, and r be the expected value. t,j Let A be the ratio of the old and new strategies for joint j. t,j torque For joints j At time t, the advantage estimate of torque action is clip for r. t,j Truncate the data to the interval [1-ϵ, 1+ϵ], where ε is the truncation threshold and π is the truncation value. θ,j For the new strategy of joint j, π old,j This is the old strategy for joint j.

[0048] Furthermore, within each iteration of the training cycle, the reward function is updated using a multi-trajectory adaptive reward shaping mechanism. The specific steps include:

[0049] Initialize each reference trajectory τ m The three reward tolerances: σ m,0 pose = σ init pose , σ m,0 pos = σ init pos , σ m,0 vel = σ init vel , where σ m,0 pose σ init pose The reward tolerance for the initialized attitude term, σ m,0 pos σ init pos The reward tolerance for the initial position item, σ m,0 vel σ init vel Reward tolerance for the initial speed term;

[0050] During the execution of the reference trajectory τ m For each time step t, calculate the three instantaneous tracking errors:

[0051] ,

[0052] ,

[0053] ,

[0054] In the formula, , , For the real-time tracking error in attitude, position, and velocity, , Let t be the true global position and expected / reference global position of each joint of the humanoid robot. , Let be the actual joint angles and the desired joint angles for each joint at time t. , Let be the actual joint angular velocity and the expected joint angular velocity of each joint at time t;

[0055] Calculate the exponential moving average for each real-time tracking error. And update the reward tolerance:

[0056] σ m,t = min(σ m,t-1 , ),

[0057] In the formula, σ m,t σ m,t-1 These are the reward tolerances at times t and t-1, respectively.

[0058] Based on the updated reward tolerance, update the reward function:

[0059] r t = exp( - x m,t / σ m,t ),

[0060] In the formula, r t The reward at time t.

[0061] Furthermore, the step of adjusting the difficulty level using the difficulty scheduler includes:

[0062] Define a discrete difficulty set A = { α0, α1, …, α D-1}, where α0 = 0 represents standing, α D-1 = 1 represents the complete reference trajectory, D is the total number of difficulty levels, and the difficulty levels are indexed by integers d∈{0,1,…,D-1}, with the corresponding actual difficulty value being the set element α. d ;

[0063] Count the number of samples N for each newly generated trajectory at the current difficulty d. m,d With trajectory tracking success rate s m,d ,

[0064] When N m,d ≥ N min And s m,d ≥ s upWhen the difficulty level is increased, d ← min(d+1, D-1), N min s is the minimum number of samples. up To increase the success rate threshold for higher difficulty;

[0065] When s m,d ≤ s down When the difficulty level is reduced, d ← max(d-1, 0);

[0066] Then, an exponential moving average is used to smooth the difficulty distribution, completing the difficulty level adjustment process.

[0067] Furthermore, the output of the actor network includes a root height prediction value. Based on the predicted root height value Construct a supervised loss function to achieve the predicted root height value. Aligned with the true root height, the expression for the supervised loss function is:

[0068] L height = || - h t || 2 ,

[0069] In the formula, L height For the supervised loss function, h t This represents the actual root height.

[0070] Furthermore, the reinforcement learning network employs an asymmetric actor-critic architecture.

[0071] Compared with the prior art, the present invention has the following beneficial effects:

[0072] (1) This invention generates reference trajectories of different difficulty levels by first generating them through values ​​and alignment, and uses them as training sets. It then uses a reinforcement learning network to train from the initial pose to the target pose at multiple difficulty levels, and combines the trajectory tracking success rate under different difficulty levels to achieve progressive training from easy to difficult.

[0073] (2) This invention generates multi-level difficulty trajectories through progressive motion interpolation and dynamically adjusts the training difficulty in combination with a difficulty scheduler, which significantly improves the problem of sparse early rewards and unstable training for high-difficulty movements and improves the policy convergence speed.

[0074] (3) The present invention introduces a multi-track adaptive reward shaping mechanism, which enables movements of different difficulty to maintain independent reward sensitivity, prevents simple movements from dominating training, and thus effectively enhances the learning ability of complex skills.

[0075] (4) The present invention introduces a joint-level torque loss function to refine the control of each joint and prevent local overload, thereby ensuring the stability of the overall control and the safety of hardware execution.

[0076] (5) The present invention adds a root height prediction value to the output end of the actor network, and realizes the alignment of the root height prediction value with the real root height by constructing a supervised loss function, providing a global pose reference, which significantly improves the training strategy in terms of spatial consistency and long-term stability, and supports the direct transfer of simulation strategy to real robot.

[0077] (6) This invention reduces the reliance on teacher-student two-stage training by using an asymmetric actor-critic structure, simplifies the training process, and reduces computational costs and development cycle. Attached Figure Description

[0078] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0079] Figure 2 This is a schematic diagram of the framework of the present invention. Detailed Implementation

[0080] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0081] This embodiment provides a progressive motion interpolation training method for learning full-body movements in a humanoid robot, such as... Figure 1 and Figure 2 As shown, the method includes the following steps:

[0082] Step 1: Initialize the reference trajectory

[0083] The target human motion reference trajectory is selected from the human motion capture dataset and used as the baseline sequence for subsequent interpolation and tracking.

[0084] Step 2: Redirect the reference trajectory to the humanoid robot

[0085] The target human motion reference trajectory (human motion posture) is mapped to a robot reference trajectory that satisfies the joint constraints of the humanoid robot (as the target motion of the humanoid robot) by using PHC (Perceptual Homing Control) or GMR (General Motion Redirection). This trajectory is then used as the target motion for the policy network to learn and track in subsequent training.

[0086] Step 3: Generate reference trajectories of different difficulty levels based on the target trajectory difficulty.

[0087] Between the standard standing posture (robot zero position, as the starting posture) and the target motion (robot joint trajectory, as the endpoint of the target posture sequence), reference trajectories of different difficulty levels are generated through interpolation and alignment, specifically including:

[0088] (1) Calculate the difference between the target movement and the standard standing posture in terms of root orientation:

[0089] Δψ = unwrap(ψ ref - ψ stand ),

[0090] Here, `unwrap` represents expanding the angle difference into a continuous real number field to avoid discontinuities caused by the periodicity of the angle; ψ ref For reference, the orientation of the root of the action, ψ stand The root orientation is for the standard standing posture. The interpolation coefficient α (0 ≤ α ≤ 1) is used as the difficulty level to obtain the progressive root orientation ψ(α), expressed as:

[0091] ψ(α) = ψ stand + α·Δψ,

[0092] In the formula, ψ(α) represents the root orientation of the difficulty level α.

[0093] (2) First, the key points of the standard standing posture and the target action are aligned to a progressive root orientation through rotation transformation. Then, they are fused using linear interpolation to finally generate a reference trajectory with a difficulty level of α, represented as:

[0094] ,

[0095] ,

[0096] ,

[0097] In the formula, Let t be the new position of the k-th keypoint in the standard standing posture at time t with difficulty level α, after coordinate alignment. For rotational transformation, For the root orientation at time t with difficulty level α, The root orientation at time t is the standard standing posture. Let K be the coordinates of the k-th key point in the standard standing posture. These are the coordinates of the root reference point in the standard standing posture. The target root location, Let be the new position of the k-th key point on the tracking trajectory after coordinate alignment when the difficulty level is α at time t. To track the root orientation of the trajectory at time t, To track the coordinates of the k-th key point on the trajectory, To track the coordinates of the root reference point on the trajectory, This is a reference trajectory with a difficulty level of α. This is the attitude vector of the entire tracking trajectory after alignment. This is the pose vector for the entire standing posture after alignment.

[0098] (3) Perform a smooth transition of attitude and velocity in a unified coordinate system to maintain the continuity and executability of the trajectory in attitude and dynamics.

[0099] Step 4: Reinforcement Learning Training Tracking Strategy

[0100] This embodiment employs a reinforcement learning network for training, which utilizes an asymmetric actor-critic structure and optimizes the policy network using the interpolation trajectory of the current difficulty level as a reference. Specifically, this step includes:

[0101] (1) Construct reinforcement learning environments based on reference trajectories of different difficulty levels, including:

[0102] The state space S includes joint position q and joint velocity. Root linear velocity v root angular velocity ω root and dynamic randomization parameters;

[0103] Action space A represents the torque or angle increments of each joint of the robot;

[0104] Environmental dynamics p(s) t+1 |s t , a t The state transitions and rewards are calculated by the physics simulation engine.

[0105] Reward function R.

[0106] (2) Interactive sampling:

[0107] Get the initial state s0;

[0108] For each time step t:

[0109] The actor network receives its deployable observation information. t Output executable action a t ~ π θ (a t | o t ), π θ This represents the trajectory tracking strategy for a humanoid robot determined by parameter θ, where the observation information o tThe next state transition s is calculated by taking into account the current joint position, velocity, root angular velocity, gravity projection, and short-time reference trajectory fragments of the humanoid robot. t+1 and reward r t ;

[0110] Transform tuple (s) t , o t , a t , r t , s t+1 , d, τ(α)) are stored in the experience replay buffer D, where d is the dynamic randomization parameter and τ(α) is the trajectory with difficulty level α;

[0111] (3) Network optimization:

[0112] Sampling: Sample batch data from the experience playback buffer D;

[0113] Critics Network Update: Based on the sampled batch data, the critics network estimates the value Q of the current state-action pair. φ (s t , a t , d, τ), and simultaneously calculate the target value y t and utilize the target value y t Minimize the difference error L(φ) to update the commentator parameter φ;

[0114] Actor Network Update: Based on sampled batch data, the actor network proposes action a. t Critics' online assessment of action a t Value Q φ (s t , a t , d, τ), and update the actor parameters θ using the policy gradient method to maximize the value of the critic network estimate;

[0115] Soft update: The target network parameters corresponding to the critic network and actor network are synchronized using a soft update method;

[0116] (4) Update:

[0117] Obtain newly generated trajectories at different difficulty levels within the current iteration training cycle, and calculate the trajectory tracking success rate at different difficulty levels based on them;

[0118] Based on the trajectory tracking success rate at different difficulty levels, the reward function is updated, and the difficulty level is adjusted using a difficulty scheduler to obtain the updated reward function and difficulty level, which are then used for the next iteration of training until the training ends.

[0119] Preferably, in the above network optimization process, the actor network update process is used as a strategy update process. To ensure overall coordination and hardware safety, a joint-level torque loss function is introduced during the strategy update process. The expression of the joint-level torque loss function is as follows:

[0120] L JTC = - E[min( r t,j × A t,j torque clip(r) t,j , 1-ε, 1+ε) × A t,j torque ) ],

[0121] r t,j = exp( log π θ,j - log π old,j ),

[0122] In the formula, L JTC Let E be the joint-level torque loss function, and r be the expected value. t,j Let A be the ratio of the old and new strategies for joint j. t,j torque For joints j At time t, the advantage estimate of torque action is clip for r. t,j Truncate the data to the interval [1-ϵ, 1+ϵ], where ε is the truncation threshold and π is the truncation value. θ,j For the new strategy of joint j, π old,j This is the old strategy for joint j.

[0123] This embodiment is based on L JTC By optimizing the strategy parameters, excessive amplification of torque at individual joints is limited, achieving both balance control and hardware safety.

[0124] Preferably, this embodiment designs a multi-trajectory adaptive motion tracking mechanism. In this mechanism, each reference trajectory τ m Each parameter independently maintains the reward tolerance σ for the three tracking errors: attitude, position, and velocity. m pose σ m pos σ m vel This achieves "one trajectory, three tolerances," completely avoiding signal mismatch issues caused by using the same reward scale for trajectories of different difficulties. The update steps of the reward function include:

[0125] Initialize each reference trajectory τ mThe three reward tolerances: σ m,0 pose = σ init pose , σ m,0 pos = σ init pos , σ m,0 vel = σ init vel ;

[0126] During the execution of the reference trajectory τ m For each time step t, calculate the three instantaneous tracking errors:

[0127] ,

[0128] ,

[0129] ,

[0130] In the formula, , , For the real-time tracking error in attitude, position, and velocity, , Let t be the true global position and expected / reference global position of each joint of the humanoid robot. , Let be the actual joint angles and the desired joint angles for each joint at time t. , Let be the actual joint angular velocity and the expected joint angular velocity of each joint at time t;

[0131] Calculate the exponential moving average for each real-time tracking error. And update the reward tolerance:

[0132] σ m,t = min(σ m,t-1 , ),

[0133] In the formula, σ m,t σ m,t-1 These are the reward tolerances at times t and t-1, respectively.

[0134] Based on the updated reward tolerance, update the reward function:

[0135] r t = exp( - x m,t / σ m,t ),

[0136] In the formula, rt The reward at time t.

[0137] This embodiment statistically analyzes and independently adjusts the attitude, position, and velocity errors of each trajectory, ensuring appropriate reward sensitivity for different trajectories and error dimensions. This significantly improves the training stability and final tracking accuracy of complex full-body movements in progressive learning courses.

[0138] Preferably, the step of adjusting the difficulty level using the difficulty scheduler includes:

[0139] Define a discrete difficulty set A = { α0, α1, …, α D-1}, where α0 = 0 represents standing, α D-1 = 1 represents the complete reference trajectory, D is the total number of difficulty levels, and the difficulty levels are indexed by integers d∈{0,1,…,D-1}, with the corresponding actual difficulty value being the set element α. d .

[0140] Count the number of samples N for each newly generated trajectory at the current difficulty d. m,d With trajectory tracking success rate s m,d ,

[0141] When N m,d ≥ N min And s m,d ≥ s up When the difficulty level is increased, d ← min(d+1, D-1), N min s is the minimum number of samples. up To increase the success rate threshold for higher difficulty;

[0142] When s m,d ≤ s down When the difficulty level is reduced, d ← max(d-1, 0);

[0143] Then, an exponential moving average is used to smooth the difficulty distribution, completing the difficulty level adjustment process.

[0144] This embodiment collects success rates under different trajectory difficulties and updates the difficulty according to rules, thus achieving progressive training from easy to difficult.

[0145] Preferably, the output of the actor network includes a root height prediction value. Based on the predicted value of root height Construct a supervised loss function to achieve the root height prediction value. Aligned with the true root height, the expression for the supervised loss function is:

[0146] Lheight = || - h t || 2 ,

[0147] In the formula, L height For the supervised loss function, h t This represents the actual root height.

[0148] 5) Determine if the training termination condition has been met. If yes, return to step 3) to perform iterative training. If no, training ends.

[0149] This step determines the convergence of the strategy's performance metrics, including global position error (e.g., E). g-mpbpe ), joint position error (e.g., E) mpjpe And the success rate of tasks at different difficulty levels. m,d When the above indicators meet the preset threshold requirements (the change is less than the threshold or reaches the target value) in several consecutive training rounds, the strategy is considered to have converged and the training ends; otherwise, return to step 3) to continue optimization.

[0150] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0151] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present invention can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0152] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0153] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0154] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0155] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0156] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A progressive motion interpolation training method for whole body motion learning of a humanoid robot, characterized by, The method comprises the following steps: S1, obtaining a target human motion reference trajectory, converting it into a robot reference trajectory satisfying the joint constraints of a humanoid robot through a pose reorientation method, and taking it as a target action of the humanoid robot; S2, based on the robot reference trajectory, generating reference trajectories of different difficulty levels between the standard standing pose and the target action through interpolation and alignment, the generation steps of the reference trajectories of different difficulty levels comprising: calculating the angle difference of the target action and the standard standing pose in the root orientation, wherein the calculation expression of the angle difference is: Δψ = unwrap(ψ ref - ψ stand ), where Δψ is the angle difference, unwrap is unwrapping the angle difference to the continuous real number domain, ψ ref is the root orientation of the target motion, and ψ stand is the root orientation of the standard standing posture. based on the angle difference Δψ, taking the interpolation coefficient α (0 ≤ α ≤ 1) as the difficulty level to obtain the progressive root orientation ψ(α), which is expressed as: ψ(α) = ψ stand + α·Δψ, In the formula, ψ(α) is the root orientation with difficulty level α; align the key points of the standard standing pose and the target action to the progressive root orientation through a rotation transformation, and then fuse them through a linear interpolation method to finally generate a reference trajectory with difficulty level α, which is expressed as: ， ， ， wherein, is the new position of the kth key point on the tracking trajectory at time t with difficulty level a after coordinate alignment, is the rotation transformation, is the root orientation of the tracking trajectory at time t with difficulty level a, is the root orientation of the standard standing pose at time t, is the coordinate of the kth key point in the standard standing pose, is the coordinate of the root reference point in the standard standing pose, is the target root position, is the new position of the kth key point on the tracking trajectory at time t with difficulty level a after coordinate alignment, is the root orientation of the tracking trajectory at time t with difficulty level a, is the coordinate of the kth key point on the tracking trajectory, is the coordinate of the root reference point on the tracking trajectory, is the reference trajectory with difficulty level a, is the pose vector of the aligned entire tracking trajectory, is the pose vector of the aligned entire standing pose; S3, constructing a reinforcement learning environment based on the reference trajectories of different difficulty levels, and constructing a reinforcement learning network, training a tracking strategy in the reinforcement learning environment using the reinforcement learning network, wherein in the training process, according to the trajectory tracking success rate under different difficulty levels, the difficulty level is updated according to the rule to realize the progressive training from easy to difficult; S4, judging whether the end condition is reached, if yes, ending the training, if not, returning to step S2 until the end condition is reached. 2.The progressive motion interpolation training method for whole body motion learning of a humanoid robot according to claim 1, wherein, The pose reorientation method is a PHC method or a GMR method. 3.The progressive motion interpolation training method for whole body motion learning of a humanoid robot according to claim 1, wherein, The reinforcement learning environment comprises: State space S, including joint positions q, joint velocities , root line velocities v root , angular velocities ω root and dynamics randomization parameters; an action space A, which is the torque or angle increment of each joint of the robot; environment dynamics p(s t+1 |s t , a t ) computed by the physics simulation engine a reward function R. 4.The progressive motion interpolation training method for whole body motion learning of a humanoid robot according to claim 3, wherein, The step of training the tracking strategy using the reinforcement learning network comprises: (1) interactive sampling: obtaining an initial state s0; for each time step t: The actor network receives its deployable observation information o t , including the current joint position, velocity, root angular velocity, gravity projection and short-time reference trajectory segment of the robot, for output executable action a t ~ π θ (a t | o t ), π θ represents the humanoid robot trajectory tracking strategy determined by the parameter θ, calculates the next state transition s t+1 and reward r t ; The transition tuple (s t , o t , a t , r t , s t+1 , d, τ(α)) is stored in the experience replay buffer D, where d is a dynamics randomization parameter and τ(α) is a trajectory of difficulty level α; (2) network optimization: sampling: sampling batch data from the experience replay buffer D; Critic network update: According to the sampled batch data, the critic network additionally introduces robot dynamics parameters and environment reference signals in the input, including mass center position, link mass, friction coefficient, control delay, dynamic randomization parameters of PD gain, and reference trajectory sequence of future several frames, to estimate the value Q of the current state-action pair in the training stage φ (s t , a t , d, τ), while calculating the target value y t , and minimizing the difference error L(φ) using the target value y t to update the critic parameters φ; Actor network update: the actor network proposes an action a based on the sampled batch data t , the critic network evaluates the value Q t (s φ , a t , d, τ) of the action a t , and updates the actor parameters θ by a policy gradient method to maximize the value estimated by the critic network; soft update: synchronously updating the target network parameters of the critic network and the actor network in a soft update manner; (3) update: obtaining the trajectories of different difficulty levels generated in the current iteration training period, and calculating the trajectory tracking success rate under different difficulty levels according to the trajectories; updating the reward function according to the trajectory tracking success rate under different difficulty levels, and adjusting the difficulty level using a difficulty scheduler to obtain the updated reward function and difficulty level, which are used for the next iteration training until the training is completed.

5. The progressive motion interpolation training method for whole body motion learning of a humanoid robot according to claim 4, wherein In the network optimization process, the actor network update process is the policy update process, and a joint-level torque loss function is introduced in the policy update process to optimize the policy parameters based on the joint-level torque loss, wherein the expression of the joint-level torque loss function is: L JTC = - E[ min( r t,j × A t,j torque , clip(r t,j , 1-ε, 1+ε) × A t,j torque ) ] r t,j = exp( log π θ,j - log π old,j ), where L JTC is the joint-level torque loss function, E is the expectation, r t,j is the new-to-old policy probability ratio for joint j, A t,j torque is the new policy for joint j At time t, the advantage estimate for the torque action, clip is the clipping of r t,j to limit it to the interval [1−ε, 1+ε], ε is the clipping threshold, π θ,j is the new policy for joint j, π old,j is the old policy for joint j. 6.The progressive motion interpolation training method for whole body motion learning of humanoid robots according to claim 4, wherein, In each iteration training period, the reward function is updated using a multi-trajectory adaptive reward shaping mechanism, and the specific steps comprise: Initialize the three reward tolerances for each reference trajectory τ m : σ m,0 pose = σ init pose , σ m,0 pos = σ init pos , σ m,0 vel = σ init vel , where σ m,0 pose , σ init pose are the reward tolerances for the initialized pose terms, σ m,0 pos , σ init pos are the reward tolerances for the initialized position terms, and σ m,0 vel , σ init vel are the reward tolerances for the initialized velocity terms. At each time step t of executing the reference trajectory τ m Three instantaneous tracking errors are computed, respectively: ， ， ， In the formula, , , is the instantaneous tracking error on the three items of attitude, position and velocity, , is the real global position of each joint of the humanoid robot at time t and the expected / reference global position, , is the real joint angle of each joint at time t and the expected joint angle, , is the real joint angular velocity of each joint at time t and the expected joint angular velocity. An exponential moving average is calculated for each instantaneous tracking error separately and the reward tolerance is updated: σ m,t = min(σ m,t-1 , ), where σ m,t , σ m,t-1 are the reward tolerances at times t and t-1, respectively. updating the reward function based on the updated reward tolerance: r t = exp( - x m,t / σ m,t ), where r t is the reward at time t.

7. The progressive motion interpolation training method for whole body motion learning of a humanoid robot according to claim 4, wherein The step of adjusting the difficulty level using the difficulty scheduler comprises: A = { α0, α1, …, α D-1 } is defined, where α0= 0 represents standing, α D-1 = 1 represents a complete reference trajectory, D is the total number of difficulty levels, the difficulty level is indexed by an integer d ∈ {0, 1, …, D-1}, and the actual difficulty value corresponding to the difficulty level is the set element α d ​ counting the number of samples N of each newly generated trajectory at the current difficulty d m,d with the trajectory tracking success rate s m,d , When N m,d ≥ N min and s m,d ≥ s up , the difficulty level d is raised by d ← min(d+1, D-1), N min is the minimum sampling number, and s up is the success rate threshold for raising the difficulty. When s m,d ≤ s down , decrease the difficulty level d ← max(d-1, 0). using an exponential moving average to smooth the difficulty distribution to complete the difficulty level adjustment process. 8.The progressive motion interpolation training method for whole body motion learning of a humanoid robot according to claim 4, wherein, The actor network outputs a root height prediction value. Based on the predicted root height value Construct a supervised loss function to achieve the predicted root height value. Aligned with the true root height, the expression for the supervised loss function is: L height = || - h t || 2 , In the formula, L height is a supervision loss function, h t is a true root height. 9.The progressive motion interpolation training method for whole body motion learning of humanoid robots according to claim 1, wherein, The reinforcement learning network employs an asymmetric actor critic architecture. The reinforcement learning network employs an asymmetric actor critic architecture. The reinforcement learning network employs an asymmetric actor critic architecture. The reinforcement learning network employs

Images