A Multi-Motion Switching Control Method and System for Humanoid Robots Based on Imitation Learning

Abstract

This invention proposes a multi-motion switching control method and system for humanoid robots based on imitation learning. It uses generative adversarial networks for imitation learning and dynamically adjusts the sampling probability of each motor skill based on the performance of the humanoid robot in performing various motor skills. This enables the humanoid robot to uniformly master different motor skills, allowing it to integrate different combinations of motor skills and effectively mitigating the severity of the mode collapse problem. It also has good flexibility and scalability.

Description

Technical Field

[0001] This invention belongs to the field of imitation learning technology, and in particular relates to a method and system for controlling multiple motion switching of humanoid robots based on imitation learning. Background Technology

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] With the increasing prevalence of humanoid robots, the demand for enhanced functionality will continue to grow. One major requirement is that humanoid robots can simultaneously master multiple different motor skills to better cope with various scenarios and tasks in real-world applications. This demand has made multi-motor skill integration a recent research hotspot in the field of humanoid robot motion control. Multi-motor skill integration aims to integrate various motor skills such as standing, walking, running, and jumping onto one or more motion controllers of a humanoid robot, enabling it to flexibly switch between different motor skills.

[0004] In the field of motion control for humanoid robots, existing learning-based technologies mainly fall into two categories: reinforcement learning and imitation learning. Reinforcement learning methods are more adaptable to the form of humanoid robots. The differences between different motor skills are the main reason why existing reinforcement learning methods rely on reward engineering. For highly similar motor skills, such as walking and running, a set of similar reward functions can often be used to assist in training. However, for significantly different motor skills, such as running and jumping, different combinations of reward functions are often needed to impose different behavioral constraints on the humanoid robot during training to ensure that it learns the correct motor skills. Therefore, in multi-motor skill integration tasks for humanoid robots, these methods typically require the design of corresponding reward function combinations for each different motor skill, thus limiting their flexibility and scalability.

[0005] In recent years, imitation learning-based methods have been extensively studied and have spawned many variations. These methods can learn different motor skills by referencing different motion capture data, significantly reducing their reliance on reward engineering and providing greater flexibility. DeepMimic allows virtual characters in animation to periodically track reference motion trajectories obtained by retargeting motion capture data, enabling the controller to imitate corresponding actions within those trajectories based on the input periodic signals. AMP introduces a generative adversarial imitation learning framework, allowing a discriminator network to distinguish between generated and reference actions, thus assisting the controller in learning actions consistent with the reference action style. ASE introduces a shared parameter space, which can integrate multiple sets of different motor skills into a shared parameter space using unsupervised learning, and uses this shared parameter space as an interface for the integrated motor skills, facilitating subsequent training of corresponding high-level policies for different tasks using hierarchical reinforcement learning. Building on ASE, CALM further introduces a motion encoder to map reference action segments to corresponding information codes, thereby enabling control and switching of the integrated motor skills.

[0006] Existing imitation learning-based methods, such as DeepMimic which relies on trajectory tracking, can only periodically track trajectories from reference motor skills, and switching between different trajectories is difficult. Therefore, existing methods typically use generative adversarial imitation learning frameworks as a foundation to improve the flexibility of the trained controller. However, generative adversarial imitation learning frameworks generally suffer from mode collapse, meaning the controller may ultimately fail to learn all given reference motor skills. This problem also significantly impacts the scope and number of motor skills that existing methods can integrate, limiting their scalability.

[0007] In summary, the motion control of existing humanoid robots has at least the following problems:

[0008] 1. Reinforcement learning-based methods often rely on complex reward engineering. When learning new motor skills, corresponding reward design and adjustments are required, which greatly affects flexibility.

[0009] 2. Imitation learning-based methods are usually based on generative adversarial imitation learning frameworks, but generative adversarial imitation learning frameworks generally suffer from pattern collapse, which limits their scalability. Summary of the Invention

[0010] To overcome the shortcomings of the prior art, this invention provides a multi-motion switching control method and system for humanoid robots based on imitation learning, enabling humanoid robots to integrate different combinations of motor skills and effectively mitigating the severity of the mode collapse problem, while exhibiting good flexibility and scalability.

[0011] To achieve the above objectives, the present invention adopts the following technical solution:

[0012] In a first aspect, the present invention provides a multi-motion switching control method for humanoid robots based on imitation learning, comprising:

[0013] To obtain the body state of the humanoid robot and its target motion task;

[0014] The body state of the humanoid robot and the target motion task of the humanoid robot are input into the pre-built motion control model to obtain the motor torque to be executed by the humanoid robot;

[0015] Using the obtained motor torque to be executed by the humanoid robot, motion control is performed on the humanoid robot to achieve smooth switching between different motion skills of the humanoid robot;

[0016] The pre-constructed motion control model is a generative adversarial network model obtained through imitation learning training. During imitation learning training, the generative adversarial network model dynamically adjusts the sampling probability of each motion skill according to the performance of the humanoid robot in performing various motion skills, so that the humanoid robot can master different motion skills evenly.

[0017] Secondly, the present invention provides a humanoid robot multi-motion switching control system based on imitation learning, comprising:

[0018] The acquisition module is configured to acquire the body state of the humanoid robot and the target motion task of the humanoid robot.

[0019] The motor torque module is configured to: input the body state of the humanoid robot and the target motion task of the humanoid robot into a pre-built motion control model to obtain the motor torque to be executed by the humanoid robot; and use the obtained motor torque to be executed by the humanoid robot to perform motion control on the humanoid robot to achieve smooth switching between different motion skills of the humanoid robot.

[0020] The pre-built motion control model is a generative adversarial network model obtained through imitation learning training. During the imitation learning training, the generative adversarial network model dynamically adjusts the sampling probability of each motion skill according to the performance of the humanoid robot in performing various motion skills, so that the humanoid robot can master different motion skills evenly.

[0021] The control module is configured to: utilize the obtained motor torque to be executed by the humanoid robot to perform motion control on the humanoid robot, so as to achieve smooth switching between different motion skills of the humanoid robot.

[0022] Thirdly, the present invention provides an electronic device including a memory and a processor, and computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the method described in the first aspect.

[0023] Fourthly, the present invention provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the method described in the first aspect.

[0024] The above one or more technical solutions have the following beneficial effects:

[0025] In this invention, imitation learning is based on generative adversarial networks. The sampling probability of each motor skill is dynamically adjusted according to the performance of the humanoid robot in performing various motor skills, so that the humanoid robot can uniformly master different motor skills. This enables the humanoid robot to integrate different combinations of motor skills and effectively mitigates the severity of the pattern collapse problem, with good flexibility and scalability.

[0026] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. Attached Figure Description

[0027] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0028] Figure 1 This is a schematic diagram of the overall framework of the motion control model in Embodiment 1 of the present invention;

[0029] Figure 2 This is a motion demonstration diagram of some of the integrated motion skills of the humanoid robot in Embodiment 1 of the present invention. Detailed Implementation

[0030] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0031] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0032] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0033] Example 1

[0034] This embodiment discloses a multi-motion switching control method for humanoid robots based on imitation learning, including:

[0035] Obtain the previous state information of the humanoid robot, as well as the target motion task of the humanoid robot;

[0036] The state information of the humanoid robot at the previous moment and the target motion task of the humanoid robot are input into the pre-built motion control model to obtain the motor torque to be executed by the humanoid robot;

[0037] Using the obtained motor torque to be executed by the humanoid robot, motion control is performed on the humanoid robot to achieve smooth switching between different motion skills of the humanoid robot;

[0038] The pre-constructed motion control model is a generative adversarial network model obtained through imitation learning training. During imitation learning training, the generative adversarial network model dynamically adjusts the sampling probability of each motion skill according to the performance of the humanoid robot in performing various motion skills, so that the humanoid robot can master different motion skills evenly.

[0039] like Figure 1 As shown, this embodiment uses motor skill encoding as the conditional input to the controller network, enabling the controller network to learn different motor skills separately during training using a lesson-based learning approach. The overall framework of the motion control model is implemented based on a generative adversarial imitation learning framework and an actor-critic reinforcement learning algorithm, and includes three neural networks: a controller network, a value function network, and a conditional discriminator network.

[0040] The actor-critic reinforcement learning algorithm consists of an actor network and a critic network. The actors and critics are explained below:

[0041] The actor represents the policy function, responsible for outputting the corresponding action based on the current state input. Its goal is to learn an optimal policy to maximize cumulative reward. During training, the actor continuously interacts with the environment, tries different actions, and adjusts its policy based on the rewards from the environment's feedback, enabling it to choose better actions in the future.

[0042] The critic represents the value function and is responsible for evaluating the merits of the current policy. Its goal is to estimate the expected cumulative reward obtainable from a given state under the current policy. During training, the critic observes the interaction between the actor and the environment, calculates the value of each state, and guides the actor to adjust their policy based on changes in value.

[0043] In this embodiment, the controller network and value function network correspond to the actor network and critic network in the actor-critic reinforcement learning algorithm, respectively. Additionally, this framework requires multiple sets of human motion capture data as reference data. Each set of motion capture data corresponds to a specific movement skill the humanoid robot is expected to learn. Before training, the reference data needs to be redirected to the corresponding humanoid robot as a reference motion capture dataset. Then, for each movement skill in the reference motion capture dataset, a corresponding one-hot encoding is manually assigned to represent the type of movement skill.

[0044] At each point in the training process, the training of the overall framework of the motion control model includes the following steps:

[0045] Step 1: The overall framework samples the motion skill encoding in the reference motion capture dataset. The sampling results are combined with the current humanoid robot's body state, such as center of mass linear velocity, center of mass angular velocity, center of mass orientation, joint position, and joint angular velocity, to form observation data, which are then input into the controller network and value function network respectively.

[0046] Step 2: The controller network receives the current observation data as input and outputs the current action, i.e., the desired motor position 'a'. t ;

[0047] Step 3: The value function network outputs its predicted expected future return based on the input of the observation data at the current moment;

[0048] Step 4: The PD controller receives the desired motor position input and calculates the required motor torque τ based on the current joint state. The specific calculation formula is as follows: where k p and k d These are pre-adjusted PD parameters; the actual values ​​vary depending on the type of humanoid robot used. θ t and These represent the current position and angular velocity of the motor on the humanoid robot, respectively.t The desired motor position at the current moment is output by the controller network;

[0049] Step 5: After the simulator executes the motor torque, it returns to the body state at the next moment and updates the generated dataset with the actual motion state of the humanoid robot.

[0050] Step 6: The discriminator accepts samples from the generated dataset and the reference dataset as input, and outputs the probability that these samples belong to the reference dataset. This probability p is calculated by the formula r = -log(1-p) to obtain the style reward r, which represents the similarity between the robot's actual motion state and the reference motion state.

[0051] Step 7: All three networks in the framework use gradient descent to update their network parameters.

[0052] All three networks in the framework are multilayer perceptron (MLP) networks. They all use the backpropagation algorithm combined with the Adam optimizer to update the network parameters. The detailed update method for each network is as follows:

[0053] 1. Controller Network:

[0054] In this embodiment, the controller network can be represented as π. θ (a|s,c), where θ represents the network parameters of the controller network, s represents the input state / observation, c represents the input motor skill code, and a represents the action output by the controller network based on the input (s,c). The goal of the controller network is to find the network parameters θ that maximize the cumulative reward. * The idea behind policy gradient is to update network parameters by calculating the gradient of cumulative reward with respect to network parameters, so that the network parameters can be improved in the direction of increasing cumulative reward.

[0055] For ease of subsequent description, the cumulative reward is defined as... Where τ represents a trajectory, which includes a series of states, actions, rewards and the corresponding motor skill encoding; R(τ) is the total reward of the trajectory, and E represents the expectation.

[0056] The single update process of the controller network is as follows:

[0057] (1) Calculate the policy gradient:

[0058] The formula for calculating the policy gradient is as follows:

[0059]

[0060] Where T is the length of the trajectory. It is estimated by a value function network, and when the motor skill is encoded as c, the policy π is... θNext state s t and action a t The action value function.

[0061] (2) Update network parameters:

[0062] Backpropagation is performed based on the policy gradient, and the controller network parameters are updated using the Adam optimizer. Where α is the learning rate.

[0063] 2. Value function network

[0064] In this embodiment, the value function network can be represented as Q. ω (s,a|c), where ω represents the network parameters of the value function network. The role of the value function network is to estimate the expected cumulative reward that can be obtained by following the actions output by the current controller network, given the state, action, and motor skill encoding.

[0065] Value function networks learn using a temporal difference (TD) method. A single update process is as follows:

[0066] (1) Calculate the TD error:

[0067] The controller network interacts with the environment, collecting a series of data on states, actions, rewards, and next states. This data is combined with motor skill encoding to form experience samples (S). t ,a t ,r t ,s t+1 c). Based on empirical samples, the TD error calculation formula is as follows:

[0068] δ t =r t +γQ ω (s t+1 ,a t+1 |c)-Q ω (s t ,a t |c)

[0069] Where, r t γ represents the style reward the humanoid robot receives at the current moment, with γ being the discount factor.

[0070] (2) Update network parameters:

[0071] Backpropagation is performed based on the TD error, and the Adam optimizer is used to update the parameters of the value function network. Where α is the learning rate.

[0072] 3. Conditional Discriminator Network:

[0073] In this embodiment, the condition discriminator network can be represented as Where φ represents the parameters of the conditional discriminator network, (s t ,s t+1 Let be the motion state pair at adjacent time points, and c be the motor skill code. The function of the conditional discriminator is to determine the probability p that the input motion state pair at adjacent time points belongs to the reference dataset based on the input motor skill code. In this embodiment, the probability value p is calculated using the formula r = -log(1-p) to obtain the style reward.

[0074] The single update process of the condition discriminator network is as follows:

[0075] (1) Calculate the classification error loss:

[0076] N samples (s) are sampled from the generated dataset and the reference dataset. t ,s t+1 ) and the corresponding motor skill code c. The conditional discriminator network accepts (s t ,s t+1 c) As input, output the probability p that each sample comes from the reference dataset. For ease of description, let y = 1 represent that the sample comes from the reference dataset, and y = 0 represent that the sample comes from the generated dataset. The actual dataset from which each sample comes is known; let y' be the dataset from which it actually originates. * The classification error loss of the conditional discriminator network is calculated using cross-entropy loss:

[0077]

[0078] (2) Update network parameters:

[0079] Backpropagation is performed based on the classification error loss, and the parameters of the conditional discriminator network are updated using the Adam optimizer. Where α is the learning rate.

[0080] The conditional discriminator network is used to determine whether a motion state pair sample comes from a reference dataset or a generated dataset.

[0081] The training process of the conditional discriminator network is as follows:

[0082] 1. Randomly sample a portion of the data from both the generated dataset and the reference dataset;

[0083] 2. These samples are fed into the conditional discriminator network one by one, and the network outputs the probability that each sample comes from the reference dataset.

[0084] 3. Since it is known whether each sample comes from the reference dataset, the classification loss can be calculated based on the network output. Because there are only two categories: from the reference dataset and from the generated dataset, the commonly used cross-entropy classification loss is used.

[0085] 4. After calculating the loss, the Adam optimizer is used for backpropagation to update the network parameters.

[0086] The reinforcement learning algorithm used for training the overall framework of the motion control model is Proximal Policy Optimization (PPO). During the training process, steps 1-7 above will be repeated until the controller network converges.

[0087] In step 6, each sample in the generated dataset and the reference dataset consists of a pair of motion states of the humanoid robot at two adjacent time points (s). t ,s t+1 The dataset consists of the motion state and the corresponding motion skill encoding c. The samples in the reference dataset are randomly sampled from the reference motion capture dataset, and the probability of each motion skill being sampled is dynamically adjusted according to a dynamic skill sampling weighting method. The samples in the generated dataset are obtained from the actual motion data of the humanoid robot.

[0088] In this embodiment, the reference motion capture dataset refers to the dataset obtained by redirecting human motion capture data to a humanoid robot; while the reference dataset is a dataset composed of motion state pairs sampled from the reference motion capture dataset, where each sample is a motion state pair at adjacent time points.

[0089] The detailed construction process of the reference dataset is as follows:

[0090] Motion capture data typically consists of a series of human motion frames, ordered chronologically. Even after redirecting the motion capture data to a humanoid robot, the data composition remains unchanged. Based on this, for each motion skill in the reference motion capture dataset, a corresponding one-hot encoding is manually assigned to represent the type of motion skill. Here, the reference motion capture dataset is complete.

[0091] Randomly select a motion skill code c from the reference motion capture dataset, and randomly select a series of adjacent motion frames, i.e. motion state pairs, from the motion capture data corresponding to the motion skill. After this process, a series of samples will be obtained. These samples are composed of motion state pairs and the motion skill code c corresponding to the motion state pairs. After repeating this step multiple times, the reference dataset is completed.

[0092] During the training of the motion control model, the learning of all motor skills can be understood as an alternating process, as follows: In each round of training, the framework samples the motor skill encoding; there is a one-to-one correspondence between the motor skill encoding and the motor skill. Based on the sampling results, the framework will learn the motor skill corresponding to the sampling results in this round of training. The learning object here includes all steps of that motor skill.

[0093] Generally, humanoid robots need to undergo many rounds of training to master each motor skill, and in each round of training, the code for each motor skill has a probability of being sampled. Therefore, we can understand that the learning of all motor skills occurs alternately throughout the training process. As the training rounds progress, the humanoid robot will gradually learn each different motor skill from scratch.

[0094] This embodiment designs dynamic skill sampling weights to effectively mitigate the severity of the pattern collapse problem.

[0095] Specifically, during the training process, the average style reward for each motor skill was dynamically calculated. For a given motor skill, the higher the average style reward, the better the humanoid robot's mastery of that motor skill.

[0096] Based on the average style reward R for the i-th motor skill i Update its probability of being sampled by the frame using the following formula:

[0097]

[0098] In the formula, m represents a motor skill, and M is the set of all motor skills in the reference motion capture dataset. This method enables the controller network to dynamically adjust its learning tendency for different motor skills based on its mastery of each skill, ultimately achieving a more comprehensive and balanced mastery of each different motor skill, thereby mitigating the severity of the pattern collapse problem.

[0099] Figure 2 This embodiment uses the Unitree H1 humanoid robot to demonstrate some of its integrated motor skills, namely (1) mimicking zombie walking, (2) walking normally, and finally switching to (3) jumping forward.

[0100] To facilitate the demonstration of the technical effects of this embodiment, four different motion skill combinations 1-4 are designed, named Dataset 1, Dataset 2, Dataset 3, and Dataset 4, respectively. Datasets 1-4 contain human motion capture data, including 10, 7, 6, and 8 different motion skills, respectively. Each motion skill sequence is redirected to the UnitreeH1 humanoid robot for subsequent training and learning. The following uses Dataset 1 as an example to demonstrate the specific motion skills it contains:

[0101] The motor skills contained in Dataset 1, Table 1

[0102]

[0103] Existing technologies MultiAMP and CALM were selected as comparison objects, and two sets of experiments were designed to evaluate the ability of different methods to comprehensively learn motor skills and the severity of the impact of pattern collapse.

[0104] Experiment 1: Assessment Experiment of Motor Skill Learning Ability. Motor skill codes were randomly sampled with equal probability and used as inputs to the controller network, collecting a total of 2000 motion trajectories. Each trajectory was 200 time units long (corresponding to 5 seconds in the real world). For each motion trajectory m... * For all adjacent motion state pairs, motion matching (i.e., the following formula) is used to determine their respective motor skill categories (where D is the value of D). M That is, referencing the motion capture dataset, where m represents the motion trajectory in the reference motion capture dataset, (s t ,s t+1 () represents a pair of motion states at adjacent times in the generated dataset. (This represents a pair of motion states at adjacent times in the reference dataset);

[0105]

[0106] After obtaining the motor skill categories corresponding to all motion state pairs in each motion trajectory segment, the motor skill category with the most occurrences in each segment is determined as the motor skill category to which that segment belongs. If at least one motion trajectory is determined to be a motor skill corresponding to a certain category, the controller network is considered to have learned that motor skill. The following are the experimental results corresponding to Experiment 1, where the motor skill coverage is calculated using the following formula:

[0107]

[0108] The greater the coverage of motor skills, the more comprehensively the corresponding method enables the humanoid robot to learn the corresponding motor skills.

[0109] Table 2. Motor skill coverage corresponding to each method

[0110]

[0111] The results of Experiment 1 show that the technique in this embodiment achieves full coverage learning of motor skills on Datasets 1-4, while the comparison methods MultiAMP and CALM fail to learn all motor skills comprehensively on all datasets. The CALM method introduces an additional latent variable space and uses an unsupervised learning paradigm to train the corresponding encoder, thus increasing the difficulty of learning motor skills and resulting in its poor performance.

[0112] Experiment 2: Mode Collapse Severity Assessment Experiment. Motion skill codes were randomly sampled with equal probability and used as controller inputs, resulting in 2000 motion trajectories. Each trajectory was 200 time units long (corresponding to 5 seconds in the real world). These trajectories constituted the motion trajectory set T. The average pairwise distance (APD) of T was calculated using the following formula:

[0113]

[0114] In the formula, N represents the number of motion trajectories in T, and L represents the length of each motion trajectory segment. This represents the motion state of the humanoid robot at time t in the i-th motion trajectory. The larger the APD, the higher the diversity of motion states in the motion trajectory, which indirectly indicates a less severe pattern collapse.

[0115] The following are the experimental results for Experiment 2:

[0116] Table 3. Motion state diversity corresponding to each method

[0117]

[0118] The results of Experiment 2 show that the motor skills learned by the technology in this embodiment exhibit more diverse motor states, which indirectly demonstrates that the technology in this embodiment can effectively alleviate the pattern collapse problem.

[0119] In summary, this embodiment constructed four different combinations of motor skills and used the Unitree H1 humanoid robot to verify the effectiveness of the method. The experimental results show that the method in this embodiment enables the controller network to comprehensively learn the corresponding motor skills from a given human motion capture dataset, and significantly reduces the severity of pattern collapse. This is very helpful for the integrated application of multiple motor skills in humanoid robots and will also contribute to the future application of humanoid robots in more scenarios and tasks.

[0120] Example 2

[0121] The purpose of this embodiment is to provide a multi-motion switching control system for humanoid robots based on imitation learning, including:

[0122] The acquisition module is configured to acquire the body state of the humanoid robot and the target motion task of the humanoid robot.

[0123] The motor torque module is configured to input the body state of the humanoid robot and the target motion task of the humanoid robot into a pre-built motion control model to obtain the motor torque to be executed by the humanoid robot.

[0124] The pre-built motion control model is a generative adversarial network model obtained through imitation learning training. During the imitation learning training, the generative adversarial network model dynamically adjusts the sampling probability of each motion skill according to the performance of the humanoid robot in performing various motion skills, so that the humanoid robot can master different motion skills evenly.

[0125] The control module is configured to: use the obtained motor torque to be executed by the humanoid robot to perform motion control on the humanoid robot, and be able to achieve smooth switching between different motion skills of the humanoid robot.

[0126] In further embodiments, the following is also provided:

[0127] An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When executed by the processor, the computer instructions perform the method described in Embodiment 1. For brevity, further details are omitted here.

[0128] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0129] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0130] A computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the method described in Embodiment 1.

[0131] The method in Embodiment 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0132] A computer program product includes a computer program that, when executed by a processor, implements the method described in Embodiment 1.

[0133] The present invention also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in program modules, which execute in a device on a target real or virtual processor to perform the processes / methods described above. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules can be combined or divided among program modules as needed. The machine-executable instructions for the program modules can execute within a local or distributed device. In a distributed device, the program modules can reside in both local and remote storage media.

[0134] The computer program code used to implement the methods of the present invention may be written in one or more programming languages. This computer program code may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the computer or other programmable data processing device, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a computer, partially on a computer, as a stand-alone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server.

[0135] In the context of this invention, computer program code or related data may be carried by any suitable carrier to enable a device, apparatus, or processor to perform the various processes and operations described above. Examples of carriers include signals, computer-readable media, and the like. Examples of signals may include electrical, optical, radio, sound, or other forms of propagation signals, such as carrier waves, infrared signals, etc.

[0136] Those skilled in the art will recognize that the units and algorithm steps described in conjunction with the embodiments herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0137] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A multi-motion switching control method for humanoid robots based on imitation learning, characterized in that, include: To obtain the body state of the humanoid robot and its target motion task; The body state of the humanoid robot and the target motion task of the humanoid robot are input into the pre-built motion control model to obtain the motor torque to be executed by the humanoid robot; Using the obtained motor torque to be executed by the humanoid robot, motion control is performed on the humanoid robot to achieve smooth switching between different motion skills of the humanoid robot; The pre-built motion control model is a generative adversarial network model obtained through imitation learning training. During the imitation learning training, the generative adversarial network model dynamically adjusts the sampling probability of each motion skill according to the performance of the humanoid robot in performing various motion skills, so that the humanoid robot can master different motion skills evenly. The construction of the motion control model specifically involves: Construct a reference motion capture dataset, in which each reference motion skill corresponds to a motion skill code; The motion skill codes in the reference motion capture dataset are sampled, and the sampling results are combined with the current body state of the humanoid robot to form observation data. The observed data are input into the controller network and the value function network respectively to obtain the corresponding expected motor position and expected future return. The robot moves according to the desired motor position, returns to the body state of the humanoid robot at the next moment, and updates the generated dataset with the actual movement state of the humanoid robot. The probability that the generated dataset belongs to the reference dataset is obtained using a conditional discriminator, and the style reward is calculated. Based on style reward, expected future reward, and expected motor position, update the parameters of the controller network, and repeat the above process until the controller network converges; The sampling probability of each motor skill is dynamically adjusted based on the performance of the humanoid robot in performing various motor skills, so that the humanoid robot can master different motor skills evenly. Specifically: Calculate the average style reward for each sport skill; The sampling probability for each motion skill is obtained by summing the average style reward for each motion skill with the average style reward for all motion skills in the reference motion capture dataset. Based on the calculated sampling probability corresponding to each motor skill, the humanoid robot can uniformly master different motor skills.

2. The multi-motion switching control method for humanoid robots based on imitation learning as described in claim 1, characterized in that, The training process of the conditional discriminator is as follows: Samples are randomly sampled from both the generated dataset and the reference dataset. The probability of each motor skill being sampled in the reference dataset is dynamically adjusted based on the performance of the humanoid robot in performing each motor skill. The sampled samples are input into the conditional discriminator to obtain the probability of each sample input parameter dataset; The classification loss is calculated based on the probability of each sample input parameter dataset, and the Adam optimizer is used for backpropagation to update the network parameters of the conditional discriminator.

3. The humanoid robot multi-motion switching control method based on imitation learning as described in claim 1, characterized in that, Backpropagation is performed using a time-difference method, and the Adam optimizer is used to update the value network parameters.

4. The multi-motion switching control method for humanoid robots based on imitation learning as described in claim 1, characterized in that, The body state of the humanoid robot includes the linear velocity of the center of mass, the angular velocity of the center of mass, the orientation of the center of mass, the joint position, and the joint angular velocity.

5. The multi-motion switching control method for humanoid robots based on imitation learning as described in claim 1, characterized in that, The motor skill encoding is implemented using word vectors or one-hot encoding.

6. A multi-motion switching control system for humanoid robots based on imitation learning, characterized in that, include: The acquisition module is configured to acquire the body state of the humanoid robot and the target motion task of the humanoid robot. The motor torque module is configured to: input the body state of the humanoid robot and the target motion task of the humanoid robot into a pre-built motion control model to obtain the motor torque to be executed by the humanoid robot; the construction of the motion control model is specifically as follows: Construct a reference motion capture dataset, in which each reference motion skill corresponds to a motion skill code; The motion skill codes in the reference motion capture dataset are sampled, and the sampling results are combined with the current body state of the humanoid robot to form observation data. The observed data are input into the controller network and the value function network respectively to obtain the corresponding expected motor position and expected future return. The robot moves according to the desired motor position, returns to the body state of the humanoid robot at the next moment, and updates the generated dataset with the actual movement state of the humanoid robot. The probability that the generated dataset belongs to the reference dataset is obtained using a conditional discriminator, and the style reward is calculated. Based on style reward, expected future reward, and expected motor position, update the parameters of the controller network, and repeat the above process until the controller network converges; The pre-constructed motion control model is a generative adversarial network (GAN) model trained through imitation learning. During imitation learning training, the GAN model dynamically adjusts the sampling probability of each motion skill based on the performance of the humanoid robot in performing various motion skills, so that the humanoid robot can evenly master different motion skills. Specifically: Calculate the average style reward for each sport skill; The sampling probability for each motion skill is obtained by summing the average style reward for each motion skill with the average style reward for all motion skills in the reference motion capture dataset. Based on the calculated sampling probability corresponding to each motor skill, the humanoid robot can uniformly master different motor skills. The control module is configured to use the obtained motor torque to be executed by the humanoid robot to perform motion control on the humanoid robot, so as to achieve smooth switching between different motion skills of the humanoid robot.

7. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, perform the method described in any one of claims 1-5.

Images