An Algorithm for Progressive Humanoid Robot Imitation and Control under Illegible Data Constraints

Abstract

This application relates to the field of robot control technology, and particularly to a progressive humanoid robot imitation and control algorithm under ill-conditioned data constraints. The algorithm includes the following steps: First, based on a constrained progressive imitation learning algorithm and an unsupervised reinforcement learning algorithm, a low-level policy model is pre-trained to obtain multiple general skills; then, all parameters of the low-level policy model are frozen, and a task policy is trained to combine multiple general skills to complete a specific complex task; finally, the trained task policy is verified in a simulation environment. This application first utilizes a large dataset of unstructured motion clippings to obtain a series of reusable general skills by pre-training a low-level policy, then freezes the parameters of the low-level policy before training the task policy, enabling the humanoid robot to perform coherent human-like movements while executing complex motion skills.

Description

Technical Field

[0001] This application relates to the field of robot control technology, and in particular to a progressive humanoid robot imitation and control algorithm under ill-conditioned data constraints. Background Technology

[0002] Humanoid robots possess human-like appearance and behavioral characteristics, making them more capable of performing complex tasks across various sectors of human society. For example, their physical structure allows them to communicate and interact with humans more naturally, effectively improving the efficiency of human-robot collaboration. Secondly, in hazardous environments, humanoid robots can replace humans in performing high-risk tasks, such as fire rescue and nuclear radiation monitoring, ensuring personnel safety while also improving task execution efficiency. Furthermore, humanoid robots can adapt to various application scenarios and work environments, providing intelligent and efficient solutions from industrial manufacturing to home services.

[0003] In terms of motion control, humanoid robots face multiple challenges. Their unique bipedal structure makes it difficult to maintain balance and stability during movement, and the large number of joints and greater degrees of freedom further increase the difficulty of system control. In recent years, learning-based methods, such as deep reinforcement learning and imitation learning, have provided new directions for controlling humanoid robots. By referencing expert data, imitation learning (IL) enables robots to reproduce expert behavior, achieving more coherent movements. For example, generative adversarial imitation learning (GAIL) employs the idea of ​​generative adversarial networks, using a discriminator to distinguish between generated and expert data, and a reinforcement learning strategy to encourage the agent to generate motion data similar to the expert data. However, for the ubiquitous unlabeled motion datasets in reality, supervised learning cannot find a correspondence between a policy and the individual skills it imitates. In this case, unsupervised learning methods can enable robots to learn more diverse skills by maximizing the discriminability of skills without explicit task rewards. Unsupervised reinforcement learning strategies are typically trained using a variational construct approximation, which uses a skill discriminator to identify potential skills, ensuring that the generated behaviors are diverse and distinguishable in the skill space.

[0004] In traditional imitation learning algorithms, the entire control strategy needs to be retrained for each new task, making it difficult for bipedal robots to achieve multi-motor skills. In contrast, humans, through natural selection, develop reusable and general skills that can be used for any new task. Inspired by this, researchers have proposed an adversarial skill embedding (ASE) method, which mines motion information from unstructured motion datasets, enabling virtual animated characters to learn general skills. However, controlling robots to achieve a series of complex movements still presents many challenges. First, existing imitation learning emphasizes reproducing behaviors similar to expert data, but this approach suffers from a large exploration space leading to slow convergence and unnatural learned movements. Furthermore, existing learning methods typically generate motion information from all joints when generating human motion, focusing more on overall similarity. However, the human joint chain consists of multiple joints, and even a small difference in one joint can trigger a chain reaction throughout the chain, potentially causing significant changes in movement, which affects the effectiveness of the imitation. Summary of the Invention

[0005] This application provides a progressive humanoid robot imitation and control algorithm under pathological data constraints. First, it utilizes a large dataset of unstructured motion clips to obtain a series of reusable general skills by pre-training a low-level policy. Then, it freezes the parameters of the low-level policy and trains the task policy, enabling the humanoid robot to perform coherent humanoid actions while executing complex motion skills.

[0006] To address the aforementioned technical problems, this application provides a progressive humanoid robot imitation and control algorithm under ill-conditioned data constraints. The algorithm includes the following steps: First, based on a constrained progressive imitation learning algorithm and an unsupervised reinforcement learning algorithm, a low-level policy model is pre-trained to obtain various general skills; then, all parameters of the low-level policy model are frozen, and a task policy is trained to combine various general skills to complete a specific complex task; finally, the trained task policy is verified in a simulation environment.

[0007] In some exemplary embodiments, before pre-training the underlying policy model, the aforementioned progressive humanoid robot imitation and control algorithm under ill-conditioned data constraints further includes: acquiring a dataset; the dataset includes a normal gait dataset and an ill-conditioned gait dataset; the normal gait dataset is a dataset M = {m} of human motion segments used for imitation. i};Pathological gait dataset patient motion segment dataset M'={m i}; where each motion segment m i A sequence of states representing a specific behavior

[0008] In some exemplary embodiments, the underlying policy model is pre-trained based on a constrained progressive imitation learning algorithm and an unsupervised reinforcement learning algorithm to obtain a variety of general skills. This includes: sequentially using an unsupervised reinforcement learning algorithm, an adversarial imitation learning algorithm under ill-conditioned data constraints, and a progressive imitation learning algorithm to pre-train the underlying policy model to obtain a variety of general skills.

[0009] In some exemplary embodiments, an unsupervised reinforcement learning algorithm is used to pre-train the underlying policy model, including: obtaining a set of skills indexed by latent variables z∈Z by establishing intrinsic rewards in the absence of external rewards; the establishment of intrinsic rewards includes: sampling a specific latent variable z~p at the beginning of each trajectory. z And by maximizing the latent variable z and extracting the feature trajectory The mutual information between them makes the acquired skills distinguishable, enabling the completion of different behavioral actions.

[0010] In some exemplary embodiments, by maximizing the latent variable z with the extracted feature trajectory The mutual information between them can uncover a set of potential and useful skills. This goal can be expressed by variational approximation as:

[0011]

[0012] Learning a parametric model using variational approximation We use variational approximation to obtain a lower bound for F(θ); by optimizing the lower bound of the policy parameter θ, we make the behavior generated by the policy diverse and distinguishable in the skill space.

[0013]

[0014] The skill space z is modeled as a hypersphere Z = {z: ||z|| = 1}. The final sample is obtained by normalizing samples drawn from a standard Gaussian distribution, with each skill having an equal probability of being selected.

[0015]

[0016] In some exemplary embodiments, the adversarial imitation learning algorithm under ill-conditioned data constraints pre-trains the underlying policy model, including: adding motion data and ill-conditioned data for imitation to the constrained adversarial imitation learning algorithm so that the generated data is close to the imitation data while staying away from the ill-conditioned data; using a discriminator to distinguish between the generated data and expert data, and using a least-squares generative adversarial network to optimize the discriminator.

[0017] In some exemplary embodiments, a discriminator is used to distinguish between generated data and expert data, and a least-squares generative adversarial network is employed to optimize the discriminator. The optimization process is represented as follows:

[0018]

[0019] Where, d ψ Indicates the discriminator, This represents a trajectory segment that mimics the data. f represents a trajectory segment of ill-conditioned data I (s) represents the trajectory segment generated by the strategy, d M Indicates the distribution of reference health data, d M ' represents the distribution of ill-conditioned data, d π The distribution of generated data is represented; the reward function used to train the policy is as follows:

[0020] r I =max[0,1-0.25(d) ψ (f I (s))-1) 2 (5)

[0021] Where, r I For imitation rewards, imitation reward r I A bounded output between 0 and 1 is provided to simplify downstream policy learning.

[0022] In some exemplary embodiments, a progressive imitation learning algorithm is used to pre-train the underlying policy model, including: first, the robot imitating the robot's knee joint, elbow joint and more distal joint information; then, the robot imitating the robot's hip joint, shoulder joint and more distal joint information; and finally, the robot imitating all joints in the body to generate full-body movements.

[0023] In some exemplary embodiments, the formula for pre-training the underlying policy model using a progressive imitation learning algorithm is as follows:

[0024]

[0025] Where i ranges from [1,3], representing the imitation step performed at step i; and the reward function r is used to train the progressive imitation policy. I As shown below:

[0026]

[0027]

[0028] in, δ1 and δ2 represent the rewards for the first, second, and third stages of imitation learning, respectively, and the thresholds for activating the rewards for the next stage are the thresholds for the next stage.

[0029] In some exemplary embodiments, training a task policy includes: the task policy ω(z|s,g) takes the current state s and the target g as inputs, takes the latent variable z as outputs, and inputs the latent variable z and the current state s into the underlying policy π(a|s,z) to obtain the action a that the robot is to perform.

[0030] The technical solution provided in this application has at least the following advantages:

[0031] This application provides a progressive humanoid robot imitation and control algorithm under ill-conditioned data constraints. The algorithm includes the following steps: First, based on a constrained progressive imitation learning algorithm and an unsupervised reinforcement learning algorithm, a low-level policy model is pre-trained to obtain a variety of general skills; then, all parameters of the low-level policy model are frozen, and a task policy is trained to combine the various general skills to complete a specific complex task; finally, the trained task policy is verified in a simulation environment.

[0032] This application provides a progressive humanoid robot imitation and control algorithm under pathological data constraints. First, it utilizes a large dataset of unstructured motion clippings to pre-train a low-level policy, acquiring a series of reusable general skills. Then, it freezes the parameters of the low-level policy before training the task policy. Specifically, this application first proposes a constrained adversarial imitation learning approach. By introducing pathological data from patients, the generated data approximates a healthy data distribution while distancing itself from the pathological data distribution of patients. This not only accelerates training but also makes the robot learn movements more natural. Second, this application proposes a progressive imitation strategy. This approach first ensures the imitation effect of the distal joints of the human body, and then progressively imitates the root joints. Specifically, this application sets a novel reward function so that the reward for the distal joints directly affects the imitation reward for the root joints, thus balancing overall and local imitation. Attached Figure Description

[0033] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations do not constitute a limitation on the embodiments, and unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0034] Figure 1 This is a flowchart illustrating a progressive humanoid robot imitation and control algorithm under ill-conditioned data constraints, provided as an embodiment of this application.

[0035] Figure 2This is a schematic diagram of the framework of a progressive humanoid robot imitation and control algorithm under ill-conditioned data constraints provided in an embodiment of this application.

[0036] Figure 3 A visual representation of a humanoid robot in an Isaac gym simulation environment, as provided in an embodiment of this application.

[0037] Figure 4 This is a schematic diagram illustrating the specific imitation steps of progressive imitation learning provided in an embodiment of this application. Detailed Implementation

[0038] As the background technology indicates, existing imitation learning methods suffer from problems such as slow convergence speed due to a large exploration space and unnatural learned movements. Furthermore, existing methods also have the technical problem of neglecting subtle differences in joints, leading to significant variations in movement and affecting the imitation effect.

[0039] To address the aforementioned technical problems, this application provides a progressive humanoid robot imitation and control algorithm under ill-conditioned data constraints. The algorithm includes the following steps: First, a low-level policy model is pre-trained based on a constrained progressive imitation learning algorithm and an unsupervised reinforcement learning algorithm to acquire various general skills; then, all parameters of the low-level policy model are frozen, and a task policy is trained, combining various general skills to complete a specific complex task; finally, the trained task policy is verified in a simulation environment. This application, by providing a progressive humanoid robot imitation and control algorithm under ill-conditioned data constraints, enables humanoid robots to perform complex motor skills while making coherent human-like movements.

[0040] The embodiments of this application will now be described in detail with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been provided in the embodiments of this application to facilitate a better understanding of the application. However, the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments.

[0041] refer to Figure 1 This application provides a progressive humanoid robot imitation and control algorithm under ill-conditioned data constraints, including the following steps:

[0042] Step S1: Based on the constrained progressive imitation learning algorithm and the unsupervised reinforcement learning algorithm, the underlying policy model is pre-trained to obtain a variety of general skills.

[0043] Step S2: Freeze all parameters of the underlying strategy model and train the task strategy to combine multiple general skills to complete a specific complex task.

[0044] Step S3: Validate the trained task strategy in a simulation environment.

[0045] Controlling humanoid robots to perform complex motor skills is a highly challenging task. In recent years, the use of motion capture systems to collect human motion data and employing imitation learning to enable robots to perform human-like motion trajectories has garnered significant attention. This application introduces a progressively constrained imitation learning algorithm, enabling humanoid robots to perform coherent human-like movements while executing complex motor skills. First, this application proposes a constrained imitation learning method that incorporates pathological motion data from patients along with motion data used for imitation, thereby constraining the robot's exploration space to generate more natural movements and accelerate training. Furthermore, this application employs a progressive imitation learning mode, allowing the robot to first imitate the movements of distal joints such as the limbs, and then further imitate the movements of root joints such as the torso. This addresses the problem of previous imitation learning methods that only focused on overall similarity, thus ensuring the reliability of the imitation. Simultaneously, existing mainstream humanoid robots can provide a reliable experimental platform for validating this model.

[0046] In some embodiments, before pre-training the underlying policy model in step S1, the aforementioned progressive humanoid robot imitation and control algorithm under ill-conditioned data constraints further includes: acquiring a dataset; the dataset includes a normal gait dataset and an ill-conditioned gait dataset; the normal gait dataset is a dataset of human motion segments M = {m} used for imitation. i};Pathological gait dataset patient motion segment dataset M'={m i}; where each motion segment m i A sequence of states representing a specific behavior

[0047] Figure 2 A flowchart illustrating a progressive humanoid robot imitation and control algorithm under ill-conditioned data constraints is shown. In the pre-training phase, a dataset of human motion segments for imitation and a dataset of patient motion segments are first collected. This application trains a low-level policy π(a|s,z) based on a progressively constrained imitation learning method, which uses unsupervised reinforcement learning to learn skills from unlabeled demonstration data in a generative adversarial network.

[0048] In some embodiments, step S1 pre-trains the underlying policy model based on a constrained progressive imitation learning algorithm and an unsupervised reinforcement learning algorithm to obtain a variety of general skills, including: sequentially using an unsupervised reinforcement learning algorithm, an adversarial imitation learning algorithm under ill-conditioned data constraints, and a progressive imitation learning algorithm to pre-train the underlying policy model to obtain a variety of general skills.

[0049] In some embodiments, an unsupervised reinforcement learning algorithm is used to pre-train the underlying policy model, including: obtaining a set of skills indexed by latent variables z∈Z by establishing intrinsic rewards in the absence of external rewards; the establishment of intrinsic rewards includes: sampling a specific latent variable z~p at the beginning of each trajectory. z And by maximizing the latent variable z and extracting the feature trajectory The mutual information between them makes the acquired skills distinguishable, enabling the completion of different behavioral actions.

[0050] Unsupervised skill learning can develop a set of skills indexed by latent variables z∈Z without external rewards. Specifically, a specific latent variable z~p is sampled at the beginning of each trajectory. z And by maximizing the potential skill z and extracting the feature trajectory Mutual information between them.

[0051] In some embodiments, by maximizing the latent variable z with the extracted feature trajectory The mutual information between them can uncover a set of potential and useful skills. This goal can be expressed by variational approximation as:

[0052]

[0053] Due to conditional distribution This is difficult to handle, so a skill discriminator is usually designed to distinguish different skills arising from state transitions. A variational approximation approach is used to learn a parametric model. We use variational approximation to obtain a lower bound for F(θ); by optimizing the lower bound of the policy parameter θ, we make the behavior generated by the policy diverse and distinguishable in the skill space.

[0054]

[0055] The skill discriminator acts as a classifier; if it fails to accurately identify a skill, the skill reward is 0. Conversely, if it accurately identifies a skill, the skill reward is logN. z The policy and skill discriminators work together to maximize the lower bound of mutual information, thereby making the behaviors generated by the policy diverse and distinguishable in the skill space.

[0056] The skill space z is modeled as a hypersphere Z = {z: ||z|| = 1}. The final sample is obtained by normalizing samples drawn from a standard Gaussian distribution, with each skill having an equal probability of being selected.

[0057]

[0058] This approach designs a bounded potential space, which can reduce the likelihood of unnatural behavior caused by potential values ​​exceeding the distribution range.

[0059] In some embodiments, the adversarial imitation learning algorithm under ill-conditioned data constraints pre-trains the underlying policy model, including: adding motion data and ill-conditioned data for imitation to the constrained adversarial imitation learning algorithm, so that the generated data is close to the imitation data while staying away from the ill-conditioned data; using a discriminator to distinguish between the generated data and expert data, and using a least-squares generative adversarial network to optimize the discriminator.

[0060] This application employs a constrained adversarial imitation learning algorithm, which, in addition to incorporating motion data for imitation, also introduces pathological data, so that the generated data is close to healthy data while avoiding pathological data, in order to generate more natural and fluid movements.

[0061] In some embodiments, a discriminator is used to distinguish between generated data and expert data, and a least-squares generative adversarial network is employed to optimize the discriminator. The optimization process is represented as follows:

[0062]

[0063] Where, d ψ Indicates the discriminator, This represents a trajectory segment that mimics the data. f represents a trajectory segment of ill-conditioned data I (s) represents the trajectory segment generated by the strategy, d M Indicates the distribution of reference health data, d M ' represents the distribution of ill-conditioned data, d π The distribution of generated data is represented; the reward function used to train the policy is as follows:

[0064] r I =max[0,1-0.25(d) ψ (f I (s))-1) 2 (5)

[0065] Where, r I For imitation rewards, imitation reward r I A bounded output between 0 and 1 is provided to simplify downstream policy learning.

[0066] In some embodiments, a progressive imitation learning algorithm is used to pre-train the underlying policy model, including: first, the robot imitating the robot's knee joint, elbow joint, and more distal joint information; then, the robot imitating the robot's hip joint, shoulder joint, and more distal joint information; and finally, the robot imitating all joints in the body to generate full-body movements.

[0067] Existing GAIL imitation learning algorithms, when generating motion data, use a discriminator to distinguish between the generated whole-body data and the imitation data in the dataset. This method largely focuses on the similarity of the whole body, ignoring the detailed information of the joints, which often has a significant impact on the robot. Therefore, this application proposes a progressive imitation learning algorithm, which consists of three steps to imitate all joints of the whole body. First, the robot imitates the knee joint, elbow joint, and more distal joints. Second, the robot imitates the hip joint, shoulder joint, and more distal joints. Finally, the robot imitates all joints to generate whole-body movements. The detailed steps are as follows: Figure 4 As shown, green represents the body parts being imitated.

[0068] In some embodiments, the formula for pre-training the underlying policy model using a progressive imitation learning algorithm is as follows:

[0069]

[0070] Where i ranges from [1,3], representing the imitation step performed at step i; and the reward function r is used to train the progressive imitation policy. I As shown below:

[0071]

[0072]

[0073] in, δ1 and δ2 represent the rewards for the first, second, and third stages of imitation learning, respectively, and the thresholds for activating the rewards for the next stage are the thresholds for the next stage.

[0074] In some embodiments, training the task policy in step S2 includes: the task policy ω(z|s,g) takes the current state s and the target g as inputs, the latent variable z as outputs, and inputs the latent variable z and the current state s into the underlying policy π(a|s,z) to obtain the action a that the robot is to perform.

[0075] Step S2 primarily involves task training. After pre-training the underlying policy π(a|s,z), the pre-trained model learns general motor skills. Based on this, by training the higher-level task policy ω(z|s,g), general skills can be selected to complete specific complex tasks. Specifically, the task policy ω(z|s,g) takes the current state s and the target g as inputs, and the latent variable z as the output. It inputs z and s into the underlying policy π(a|s,z) and outputs the action a to be performed by the robot. When the robot performs a new task, to allow it to try various skills, the higher-level policy initially uses a uniform sampling method on the latent space, gradually adopting effective skills as the training progresses.

[0076] Step S3 mainly involves the simulation setup process. The simulation experiments in this application are conducted in the Isaac Gym simulation environment, which enables end-to-end GPU acceleration, effectively speeding up the experiment. Simultaneously, we use a mainstream humanoid robot as the experimental platform to study the model; the robot structure is as follows... Figure 3 As shown.

[0077] (1) State and Action Space

[0078] Action space and state space define the sets of actions a robot takes when perceiving its environment and performing actions, respectively. A reasonable definition can help improve the efficiency and performance of robot learning, enabling it to make correct decisions in complex environments.

[0079] The robot's state space represents a complete description of the robot's current state and embodies the robot's subjective perception state. The state space selected in this application is defined in the following form:

[0080]

[0081] Where, q j and p represents the joint position and joint velocity of the j-th joint of the robot, respectively. i ,q i ,v i ,w i This represents the relative position, orientation, linear velocity, and angular velocity of the i-th joint of the robot with respect to the root node in the local coordinate system. h represents the height of the robot's root node relative to the ground, and the root node is designated as the robot's pelvis.

[0082] The dimension of the motion space is the number of degrees of freedom of the robot. By controlling each degree of freedom of the robot's joints, the robot can perform a specified action.

[0083] (2) Network Structure

[0084] This application uses the Proximal Policy Optimization (PPO) algorithm as the reinforcement learning network framework. This algorithm adopts an actor-critic architecture with a policy network and a value network. The policy network is used to output actions, while the value network is used to output action values. In this application, both networks are composed of fully connected networks.

[0085] (3) Task

[0086] Pre-training of the underlying strategy allows the robot to learn various general motor skills, which form the foundation for achieving complex skills. Further training of the upper-level strategy enables the robot to select suitable motor skills, and by combining these movements, various target tasks can be achieved. By setting a simple reward function, the robot can rely on the general motor skills acquired from the pre-trained model to achieve more complex target tasks. In subsequent experiments, this imitation learning-based bipedal robot motion control algorithm can be studied on a humanoid robot platform to perform tasks including: target path, target position, and fall recovery.

[0087] For the target route, in this task, the robot must move according to a specified speed and direction, and the target is designated as... in For the specified direction, v * The specified speed.

[0088] For a target location, the task is to move the robot to a specified position. The objective of the task is to define a target location in space and require the robot to move towards that target location. The target can be represented as...

[0089] Fall recovery is a task in which a robot performs a series of specific actions after falling, such as rolling over, kneeling, and getting up, in order to stand up again from the fallen state and continue walking.

[0090] Based on the above technical solutions, this application provides a progressive humanoid robot imitation and control algorithm under ill-conditioned data constraints. The algorithm includes the following steps: First, based on a constrained progressive imitation learning algorithm and an unsupervised reinforcement learning algorithm, a low-level policy model is pre-trained to obtain a variety of general skills; then, all parameters of the low-level policy model are frozen, and a task policy is trained to combine the various general skills to complete a specific complex task; finally, the trained task policy is verified in a simulation environment.

[0091] This application provides a progressive humanoid robot imitation and control algorithm under pathological data constraints. First, it utilizes a large dataset of unstructured motion clippings to pre-train a low-level policy, acquiring a series of reusable general skills. Then, it freezes the parameters of the low-level policy before training the task policy. Specifically, this application first proposes a constrained adversarial imitation learning approach. By introducing pathological data from patients, the generated data approximates a healthy data distribution while distancing itself from the pathological data distribution of patients. This not only accelerates training but also makes the robot learn movements more natural. Second, this application proposes a progressive imitation strategy. This approach first ensures the imitation effect of the distal joints of the human body, and then progressively imitates the root joints. Specifically, this application sets a novel reward function so that the reward for the distal joints directly affects the imitation reward for the root joints, thus balancing overall and local imitation.

[0092] Those skilled in the art will understand that the above-described embodiments are specific examples of implementing this application, and in practical applications, various changes in form and detail may be made without departing from the spirit and scope of this application. Any person skilled in the art can make their own modifications and alterations without departing from the spirit and scope of this application; therefore, the scope of protection of this application should be determined by the scope defined in the claims.

Claims

1. A progressive humanoid robot imitation and control algorithm under ill-conditioned data constraints, characterized in that, Includes the following steps: Based on constrained progressive imitation learning and unsupervised reinforcement learning algorithms, the underlying policy model is pre-trained to acquire a variety of general skills; Freeze all parameters of the underlying strategy model and train the task strategy to combine multiple general skills to complete a specific complex task; The trained task strategy was validated in a simulation environment; Before pre-training the underlying policy model, the following is also included: Obtain the dataset; the dataset includes normal gait dataset and pathological gait dataset; The normal gait dataset is a dataset of human motion segments used for imitation. ; The pathological gait dataset is a collection of patient movement segments. ; where each motion segment A sequence of states representing a specific behavior ; The adversarial imitation learning algorithm under ill-conditioned data constraints pre-trains the underlying policy model, including: In the constrained adversarial imitation learning algorithm, motion data and ill-conditioned data are added for imitation, so that the generated data is close to the imitation data while avoiding the ill-conditioned data. A discriminator is used to distinguish between generated data and expert data, and a least-squares generative adversarial network is employed to optimize the discriminator.

2. The progressive humanoid robot imitation and control algorithm under ill-conditioned data constraints according to claim 1, characterized in that, The constrained progressive imitation learning algorithm and unsupervised reinforcement learning algorithm are used to pre-train the underlying policy model to acquire a variety of general skills, including: The underlying policy model is pre-trained using unsupervised reinforcement learning, adversarial imitation learning under ill-conditioned data constraints, and progressive imitation learning in sequence to acquire a variety of general skills.

3. The progressive humanoid robot imitation and control algorithm under ill-conditioned data constraints according to claim 1, characterized in that, An unsupervised reinforcement learning algorithm is used to pre-train the underlying policy model, including: In the absence of external rewards, by establishing intrinsic rewards, a set of latent variables is obtained. The skill of indexing; the establishment of intrinsic rewards includes: sampling a specific latent variable at the beginning of each trajectory. And by maximizing latent variables With feature trajectory extraction The mutual information between them makes the acquired skills distinguishable, enabling the completion of different behavioral actions.

4. The progressive humanoid robot imitation and control algorithm under ill-conditioned data constraints according to claim 3, characterized in that, By maximizing latent variables With feature trajectory extraction The mutual information between them can uncover a set of potential and useful skills. This goal can be expressed by variational approximation as: （1） Learning a parametric model using variational approximation To obtain a variational approximation. The lower bound; by optimizing the policy parameters The lower bound of the rule makes the behavior generated by the strategy diverse and distinguishable in the skill space; （2） The skill space z is modeled as a hypersphere. The final sample was obtained by normalizing a sample drawn from a standard Gaussian distribution, with each skill having an equal probability of being selected. （3）。 5. The progressive humanoid robot imitation and control algorithm under ill-conditioned data constraints according to claim 1, characterized in that, A discriminator is used to distinguish between generated data and expert data, and a least-squares generative adversarial network is employed to optimize the discriminator. The optimization process is represented as follows: （4） in, Indicates the discriminator, This represents a trajectory segment that mimics the data. Represents trajectory segments of pathological data. This represents the trajectory fragment generated by the strategy. This indicates a reference to the distribution of health data. Indicates the distribution of ill-conditioned data. The distribution of generated data is represented; the reward function used to train the policy is as follows: （5） in, To imitate rewards, to imitate rewards A bounded output between 0 and 1 is provided to simplify downstream policy learning.

6. The progressive humanoid robot imitation and control algorithm under ill-conditioned data constraints according to claim 1, characterized in that, A progressive imitation learning algorithm is used to pre-train the underlying policy model, including: First, the robot mimics information about the robot's knee joint, elbow joint, and more distal joints; Then, the robot mimics information from the robot's hip joint, shoulder joint, and more distal joints; Finally, the robot mimics all the joints in its body to generate full-body movements.

7. The progressive humanoid robot imitation and control algorithm under ill-conditioned data constraints according to claim 6, characterized in that, The formula for pre-training the underlying policy model using a progressive imitation learning algorithm is as follows: （6） Where i takes values ​​in the range [1,3], representing the imitation steps performed in step i; The reward function r used to train the progressive imitation strategy I As shown below: （7） （8） in, , , These represent the rewards for the first, second, and third stages of imitation learning, respectively. and These are the thresholds for activating the next stage of rewards.

8. The progressive humanoid robot imitation and control algorithm under ill-conditioned data constraints according to claim 1, characterized in that, The training task strategy includes: Task Strategy The current state s and the objective g are taken as inputs, and the latent variable z is taken as the output. The latent variable z and the current state s are then input into the underlying policy. In the process, obtain the action 'a' that the robot needs to perform.

Images