Humanoid robot multi-motion switching control method and system based on imitation learning

Abstract

According to the humanoid robot multi-motion switching control method and system based on imitation learning, imitation learning is performed on the basis of a generative adversarial network, and the sampling probability of each motion skill is dynamically adjusted according to the performance condition of the humanoid robot for executing each motion skill; according to the method, the humanoid robot can uniformly master different motion skills, the humanoid robot can integrate different motion skill combinations, the severity of the mode collapse problem is effectively relieved, and good flexibility and expandability are achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field related to imitation learning, and in particular to a multi-motion switching control method and system for a humanoid robot based on imitation learning. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] As humanoid robots become more and more popular, the demand for their enhanced capabilities will continue to grow. One of the main demands is that humanoid robots be able to master multiple different motor skills at the same time to better cope with various scenarios and tasks in real-world applications. This demand has made multi-motor skill integration of humanoid robots a recent research hotspot in the field of humanoid robot motion control. Multi-motor skill integration aims to integrate multiple different motor skills such as standing, walking, running, jumping, etc. into a single or multiple motion controllers of humanoid robots, and enable them to flexibly switch between different motor skills.

[0004] In the field of motion control of humanoid robots, existing learning-based technologies mainly include two mainstream directions: reinforcement learning-based and imitation learning-based. Reinforcement learning-based methods are more adaptable to the morphology of humanoid robots. The differences between different sports skills are the main reason why existing reinforcement learning-based methods need to rely on reward engineering. For sports skills with a high degree of similarity, such as walking and running, a set of similar reward functions can often be used to assist their training and learning; while for sports skills with large differences, such as running and jumping, different reward function combinations are often required to impose different behavioral constraints on humanoid robots during training to ensure that they can learn the correct sports skills. Therefore, in the multi-sports skill integration task of humanoid robots, these methods usually need to design corresponding reward function combinations for each different sports skill, which limits the flexibility and scalability of these methods.

[0005] In recent years, imitation learning-based methods have been widely studied and have produced many variants. Imitation learning-based methods can learn different sports skills by referring to different motion capture data, which greatly reduces its dependence on reward engineering and brings it good flexibility. DeepMimic allows the virtual person in the character animation to periodically track the reference motion trajectory obtained by redirecting the motion capture data, so that the controller can imitate the corresponding actions in the motion trajectory according to the input periodic signal; AMP introduces a generative adversarial imitation learning framework, allowing the discriminator network to assist the controller's learning by distinguishing between generated actions and reference actions, so that the controller can learn actions consistent with the reference action style; ASE introduces a shared parameter space, which can integrate multiple sets of different sports skills into the shared parameter space in an unsupervised learning manner in the form of information encoding, and use the shared parameter space as the interface of the integrated sports skills, which is convenient for subsequent hierarchical reinforcement learning to train the corresponding high-level strategies for different tasks; Based on ASE, CALM further introduces an action encoder to map the reference action fragments into corresponding information encodings, thereby realizing the control and switching of the integrated sports skills.

[0006] As for existing imitation learning-based methods, since trajectory tracking-based methods such as DeepMimic can only periodically track the trajectories in the reference motion skills, and switching between different trajectories is difficult, existing methods usually use the generative adversarial imitation learning framework as a basis to improve the flexibility of the trained controller. However, the generative adversarial imitation learning framework generally has the problem of mode collapse, that is, the controller may not be able to learn all the given reference motion skills in the end. This problem also greatly affects the scope and number of motion skills that can be integrated by existing methods, limiting their scalability.

[0007] In summary, the motion control of existing humanoid robots has at least the following problems:

[0008] 1. Methods based on reinforcement learning often rely on complicated reward engineering. When learning new sports skills, corresponding reward design and adjustment are required, which greatly affects flexibility.

[0009] 2. Imitation learning-based methods are usually based on the generative adversarial imitation learning framework, but the generative adversarial imitation learning framework generally has the problem of model collapse, and its scalability is limited. Summary of the invention

[0010] In order to overcome the deficiencies of the above-mentioned prior art, the present invention provides a multi-motion switching control method and system for a humanoid robot based on imitation learning, which enables the humanoid robot to integrate different combinations of motion skills and effectively reduces the severity of the mode collapse problem, and has good flexibility and scalability.

[0011] In order to achieve the above object, the present invention adopts the following technical solution:

[0012] In a first aspect, the present invention provides a multi-motion switching control method for a humanoid robot based on imitation learning, comprising:

[0013] Obtaining the body state of the humanoid robot and the target motion task of the humanoid robot;

[0014] Inputting the body state of the humanoid robot and the target motion task of the humanoid robot into a pre-built motion control model to obtain the motor torque to be executed by the humanoid robot;

[0015] Using the obtained motor torque to be executed by the humanoid robot, the humanoid robot is controlled to achieve smooth switching between different motor skills of the humanoid robot;

[0016] Among them, the pre-constructed motion control model is a generative adversarial network model obtained through imitation learning training; during the imitation learning training, the generative adversarial network model dynamically adjusts the sampling probability of each motion skill according to the performance of the humanoid robot in executing various motion skills, so that the humanoid robot can evenly master different motion skills.

[0017] In a second aspect, the present invention provides a humanoid robot multi-motion switching control system based on imitation learning, comprising:

[0018] An acquisition module is configured to: acquire a body state of the humanoid robot and a target motion task of the humanoid robot;

[0019] a motor torque module, which is configured to: input the body state of the humanoid robot and the target motion task of the humanoid robot into a pre-built motion control model to obtain the motor torque to be executed by the humanoid robot; and use the obtained motor torque to be executed by the humanoid robot to perform motion control on the humanoid robot to achieve smooth switching between different motion skills of the humanoid robot;

[0020] The pre-built motion control model is a generative adversarial network model obtained through imitation learning training; during the imitation learning training, the generative adversarial network model dynamically adjusts the sampling probability of each motion skill according to the performance of the humanoid robot in performing various motion skills, so that the humanoid robot can evenly master different motion skills;

[0021] The control module is configured to: utilize the motor torque to be executed by the humanoid robot to control the motion of the humanoid robot, so as to achieve smooth switching between different motion skills of the humanoid robot.

[0022] In a third aspect, the present invention provides an electronic device comprising a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the method described in the first aspect is performed.

[0023] In a fourth aspect, the present invention provides a computer-readable storage medium for storing computer instructions, wherein when the computer instructions are executed by a processor, the method described in the first aspect is performed.

[0024] One or more of the above technical solutions have the following beneficial effects:

[0025] In the present invention, imitation learning is performed based on a generative adversarial network, and the sampling probability of each motor skill is dynamically adjusted according to the performance of the humanoid robot in executing various motor skills, so that the humanoid robot can evenly master different motor skills; the humanoid robot is able to integrate different combinations of motor skills, and the severity of the mode collapse problem is effectively mitigated, with good flexibility and scalability.

[0026] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0028] Figure 1 Schematic diagram of the overall framework of the motion control model in the first embodiment of the present invention;

[0029] Figure 2 This is a diagram showing some of the integrated motion skills of the humanoid robot in the first embodiment of the present invention. DETAILED DESCRIPTION

[0030] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0031] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0032] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0033] Embodiment 1

[0034] This embodiment discloses a multi-motion switching control method of a humanoid robot based on imitation learning, comprising:

[0035] Obtain the state information of the humanoid robot at the last moment and the target motion task of the humanoid robot;

[0036] Inputting the state information of the humanoid robot at the last moment and the target motion task of the humanoid robot into a pre-built motion control model to obtain the motor torque to be executed by the humanoid robot;

[0037] Using the obtained motor torque to be executed by the humanoid robot, the humanoid robot is controlled to achieve smooth switching between different motor skills of the humanoid robot;

[0038] Among them, the pre-constructed motion control model is a generative adversarial network model obtained through imitation learning training; during the imitation learning training, the generative adversarial network model dynamically adjusts the sampling probability of each motion skill according to the performance of the humanoid robot in executing various motion skills, so that the humanoid robot can evenly master different motion skills.

[0039] like Figure 1 As shown, this embodiment uses the motor skill encoding as the conditional input of the controller network, so that the controller network can learn different motor skills separately in the course learning method during the training process. The overall framework of the motion control model is based on the generative adversarial imitation learning framework and the actor-critic reinforcement learning algorithm, and contains three neural networks, namely the controller network, the value function network and the conditional discriminator network.

[0040] The actor-critic reinforcement learning algorithm consists of an actor network and a critic network. The following is an explanation of the actor and critic:

[0041] The actor represents the policy function, which is responsible for outputting the corresponding action based on the current state input. Its goal is to learn an optimal strategy to maximize the cumulative reward. During the training process, the actor continuously interacts with the environment, tries different actions, and adjusts the strategy based on the rewards fed back by the environment, so that better actions can be selected in the future.

[0042] The critic represents the value function and is responsible for evaluating the quality of the current strategy. Its goal is to estimate the expected cumulative reward that can be obtained from a certain state under the current strategy. During training, the critic calculates the value of each state by observing the interaction between the actor and the environment, and guides the actor to adjust the strategy according to the change in value.

[0043] The controller network and value function network in this embodiment correspond to the actor network and critic network in the actor-critic reinforcement learning algorithm, respectively. In addition, the framework also requires multiple sets of human motion capture data as reference data, where each set of motion capture data corresponds to the sports skills that the humanoid robot is expected to learn. Before training, the reference data needs to be redirected to the corresponding humanoid robot as a reference motion capture data set. Then, for each sports skill in the reference motion capture data set, a corresponding sports skill unique hot encoding is artificially set to indicate the type of the sports skill.

[0044] At each moment in the training process, the training process of the overall framework of the motion control model includes the following steps:

[0045] Step 1: The overall framework samples the motion skill encoding in the reference motion capture dataset. The sampling results are combined with the current humanoid robot's center of mass linear velocity, center of mass angular velocity, center of mass orientation, joint position, joint angular velocity and other body states to form observation data, which are input into the controller network and value function network respectively.

[0046] Step 2: The controller network accepts the input of the observed data at the current moment and outputs the action at the current moment, that is, the expected motor position a t ;

[0047] Step 3: The value function network outputs its predicted expected future return based on the input of the observed data at the current moment;

[0048] Step 4: The PD controller accepts the desired motor position input and calculates the motor torque τ required to be applied based on the current joint state. The specific calculation formula is: where k p and k d is a pre-adjusted PD parameter. The actual value varies depending on the type of humanoid robot used. t and Represent the current position and angular velocity of the motor on the humanoid robot, at The desired motor position at the current moment output by the controller network;

[0049] Step 5: After executing the motor torque, the simulator returns to the state of the body at the next moment, and updates the generated data set with the actual motion state of the humanoid robot;

[0050] Step 6: The discriminator accepts samples from the generated dataset and the reference dataset as input, and outputs the probability that these samples belong to the reference dataset. The probability p is calculated by the formula r = -log(1-p) to obtain the style reward r, which indicates the degree of similarity between the actual motion state of the robot and the reference motion state.

[0051] Step 7: The three networks in the framework all use the gradient descent method to update the network parameters.

[0052] The three networks in the framework are all multi-layer perceptron (MLP) networks. They all use the back propagation algorithm + Adam optimizer to update the network parameters. The detailed update method of each network is as follows:

[0053] 1. Controller network:

[0054] In this embodiment, the controller network can be represented as π θ (a|s,c), where θ represents the network parameters of the controller network, s represents the input state / observation, c represents the input motor skill encoding, and a represents the action output by the controller network based on the input (s,c). The goal of the controller network is to find the network parameters θ that can maximize the cumulative reward * The idea of ​​policy gradient is to update the network parameters by calculating the gradient of the cumulative reward with respect to the network parameters, so that the network parameters can be improved in the direction of increasing the cumulative reward.

[0055] To facilitate subsequent description, the cumulative reward is defined as Among them, τ represents a trajectory, which contains a series of states, actions, rewards and the motor skill encoding corresponding to the trajectory; R(τ) is the total reward of the trajectory, and E represents the expectation.

[0056] The process of a single update of the controller network is as follows:

[0057] (1) Calculate the policy gradient:

[0058] The calculation formula of the policy gradient is as follows:

[0059]

[0060] Where T is the length of the trajectory, is estimated by the value function network. When the motor skill is encoded as c, the strategy π θNext state t and action a t The action-value function of .

[0061] (2) Update network parameters:

[0062] Back propagation is performed according to the policy gradient, and the controller network parameters are updated using the Adam optimizer, that is, Where α is the learning rate.

[0063] 2. Value Function Network

[0064] In this embodiment, the value function network can be represented as Q ω (s,a|c), where ω is the network parameter of the value function network. The role of the value function network is to estimate the expected cumulative reward that can be obtained by following the action output by the current controller network under the conditions of given state, action and motor skill encoding.

[0065] The value function network uses the time difference (TD) method for learning. Its single update process is as follows:

[0066] (1) Calculate TD error:

[0067] The controller network interacts with the environment and collects a series of state, action, reward and next state data, which are combined with the motor skill encoding to form experience samples (s t ,a t ,r t ,s t+1 ,c). According to the empirical sample, the TD error calculation formula is as follows:

[0068] δ t =r t +γQ ω (s t+1 ,a t+1 |c)-Q ω (s t ,a t |c)

[0069] Among them, r t is the style reward obtained by the humanoid robot at the current moment, and γ is the discount factor.

[0070] (2) Update network parameters:

[0071] Back propagation is performed based on the TD error, and the Adam optimizer is used to update the parameters of the value function network, that is, Where α is the learning rate.

[0072] 3. Conditional Discriminator Network:

[0073] In this embodiment, the conditional discriminator network can be expressed as Among them, φ is the parameter of the conditional discriminator network, (s t ,s t+1 ) is a pair of motion states at adjacent moments, and c is a motion skill code. The function of the conditional discriminator is to discriminate the probability p of the motion state pair at adjacent moments belonging to the reference data set according to the input motion skill code. In this embodiment, the probability value p is calculated by the formula r = -log (1-p) to obtain the style reward.

[0074] The single update process of the conditional discriminator network is as follows:

[0075] (1) Calculate the classification error loss:

[0076] Sample N samples (s) from the generated dataset and the reference dataset t ,s t+1 ) and the corresponding motor skill code c of the sample. The conditional discriminator network accepts (s t ,s t+1 ,c) as input, output the probability p of each sample coming from the reference dataset. For the convenience of description, y = 1 means the sample comes from the reference dataset, and y = 0 means the sample comes from the generated dataset. It is known which dataset each sample actually comes from, and the dataset it actually comes from is y * . Apply the cross entropy loss to calculate the classification error loss of the conditional discriminator network:

[0077]

[0078] (2) Update network parameters:

[0079] Back propagation is performed according to the classification error loss, and the parameters of the conditional discriminator network are updated using the Adam optimizer, i.e. Among them, α is the learning rate

[0080] The conditional discriminator network is used to discriminate whether a motion state pair sample comes from the reference dataset or the generated dataset.

[0081] During the training process, the training process of the conditional discriminator network is as follows:

[0082] 1. Randomly sample a portion of samples from both the generated dataset and the reference dataset;

[0083] 2. Input these samples one by one into the conditional discriminator network, and the network outputs the probability that each sample comes from the reference data set one by one;

[0084] 3. Since it is known whether each sample comes from the reference dataset, the classification loss can be calculated based on the network output. Since there are only two categories: from the reference dataset and from the generated dataset, the commonly used cross entropy classification loss is used;

[0085] 4. After calculating the loss, use the Adam optimizer for back propagation to update the network parameters.

[0086] The reinforcement learning algorithm used for training the overall framework of the motion control model is Proximal Policy Optimization (PPO). During the training process, the above steps 1 to 7 will be repeated until the controller network converges.

[0087] Among them, each sample in the generated dataset and the reference dataset in step 6 consists of a pair of motion states of the humanoid robot at two adjacent moments (s t ,s t+1 ) and the corresponding motion skill code c of the motion state. The samples in the reference data set are randomly sampled from the reference motion capture data set, and the probability of each motion skill being sampled is dynamically adjusted according to the dynamic skill sampling weight method; the samples in the generated data set are obtained from the actual motion data of the humanoid robot.

[0088] Among them, the reference motion capture data set in this embodiment refers to the data set obtained by redirecting human motion capture data to a humanoid robot; and the reference data set is a data set composed of a set of motion state pairs sampled from the reference motion capture data set, wherein each sample is a pair of motion states at adjacent moments.

[0089] The detailed construction process of the reference dataset is as follows:

[0090] Motion capture data is usually composed of a series of human motion frames, which are sorted by time. After redirecting the motion capture data to the humanoid robot, the composition of the data remains unchanged. On this basis, for each sports skill in the reference motion capture data set, a corresponding sports skill unique hot encoding is manually set to indicate the type of the sports skill. Here, the reference motion capture data set is constructed;

[0091] A motion skill code c is randomly selected from the reference motion capture data set, and a series of motion frames at adjacent moments, i.e., motion state pairs, are randomly selected from the motion capture data corresponding to the motion skill. After this process is completed, a series of samples will be obtained, which are composed of motion state pairs and the motion skill code c corresponding to the motion state pairs. After repeating this step multiple times, the reference data set is constructed.

[0092] During the training of the motor control model, the learning of all motor skills can be understood as alternating, as follows: During each round of training, the framework samples the motor skill code; the motor skill code and the motor skill are one-to-one corresponding. Based on the sampling results, the framework will learn the motor skill corresponding to the sampling result during this round of training, and the learning objects here include all the steps in the motor skill.

[0093] Generally speaking, it takes many rounds of training for a humanoid robot to master each motor skill, and in each round of training, each motor skill code has a probability of being sampled. Therefore, we can understand that in the entire training process, the learning of all motor skills is carried out alternately. As the training rounds continue to advance, the humanoid robot will gradually learn each different motor skill from scratch.

[0094] This embodiment designs a dynamic skill sampling weight to effectively mitigate the severity of the mode collapse problem.

[0095] Specifically, during the training process, the mean of the corresponding style reward is dynamically calculated for each sports skill. For a certain sports skill, the higher the corresponding style reward mean, the better the current humanoid robot has mastered the sports skill.

[0096] According to the statistical mean value R of the style reward for the i-th sports skill i , use the following formula to update the probability of being sampled by the framework:

[0097]

[0098] In the formula, m represents the motor skill, and M is the set of all motor skills in the reference motion capture dataset. This method enables the controller network to dynamically adjust the learning tendency of different motor skills according to the mastery of each motor skill, and ultimately achieve a more comprehensive and balanced mastery of each different motor skill, thereby reducing the severity of the mode collapse problem.

[0099] Figure 2 In this embodiment, the Unitree H1 humanoid robot is used to demonstrate some of its integrated movement skills, which are (1) recovering from imitating zombie walking, (2) normal walking, and finally switching to (3) jumping forward.

[0100] In order to conveniently present the technical effects of this embodiment, this embodiment designs four different sports skill combinations 1-4, named Dataset 1, Dataset 2, Dataset 3 and Dataset 4 respectively. The contents of Datasets 1-4 are all human motion capture data, containing 10, 7, 6, and 8 different sports skills respectively. Each sports skill sequence is redirected to the UnitreeH1 humanoid robot for subsequent training and learning. The following uses Dataset 1 as an example to show the specific sports skills contained therein:

[0101] Table 1 Sports skills included in Dataset 1

[0102]

[0103] The existing technologies MultiAMP and CALM were selected as comparison objects, and two sets of experiments were designed to evaluate the ability of different methods to comprehensively learn motor skills and the severity of the impact of pattern collapse.

[0104] Experiment 1: Sports skill learning ability assessment experiment. The sports skill codes were randomly sampled with equal probability as the controller network input, and a total of 2000 motion trajectories were collected. The length of each motion trajectory was 200 unit time (corresponding to 5s in the real world). For each motion trajectory m * For all pairs of motion states at adjacent moments in the game, motion matching (i.e., the following formula) is used to determine the sports skill category to which they belong (where D M That is, the reference motion capture dataset, m represents the motion trajectory in the reference motion capture dataset, (s t ,s t+1 ) represents the motion state pairs at adjacent moments in the generated data set, represents a pair of motion states at adjacent moments in the reference data set);

[0105]

[0106] After obtaining the sports skill categories to which all the motion state pairs in each motion trajectory belong, the sports skill category with the largest number in each motion trajectory is determined as the sports skill category to which the motion trajectory belongs. If there is at least one motion trajectory that is determined to correspond to a certain category of sports skills, it is considered that the controller network has learned the sports skill. The following are the experimental results corresponding to Experiment 1, where the sports skill coverage is calculated as follows:

[0107]

[0108] The greater the motor skill coverage, the more it indicates that the corresponding method can enable the humanoid robot to learn the corresponding motor skills more comprehensively.

[0109] Table 2 Sports skill coverage corresponding to each method

[0110]

[0111] From the results of Experiment 1, it can be seen that the technology of this embodiment achieves full coverage learning of sports skills on Datasets 1-4, while the comparison methods MultiAMP and CALM cannot fully learn all sports skills on all data sets. The CALM method introduces an additional latent variable space and uses an unsupervised learning paradigm to train the corresponding encoder, which increases the difficulty of learning sports skills and leads to poor performance.

[0112] Experiment 2: Mode collapse severity evaluation experiment. The motion skill codes were randomly sampled with equal probability as controller inputs, and a total of 2000 motion trajectories were collected, each with a length of 200 unit moments (corresponding to 5s in the real world). These motion trajectories constitute the motion trajectory set T, and the average distance APD of T is calculated using the following formula:

[0113]

[0114] Where N represents the number of motion trajectories in T, L represents the length of each motion trajectory, It represents the motion state of the humanoid robot corresponding to the tth moment in the i-th motion trajectory. The larger the APD, the higher the diversity of motion states in the motion trajectory, which indirectly indicates that the degree of mode collapse is milder.

[0115] The following are the experimental results corresponding to Experiment 2:

[0116] Table 3 The diversity of motion states corresponding to each method

[0117]

[0118] From the results of Experiment 2, it can be seen that the sports skills finally learned by the technology of this embodiment have more diverse sports states, which can indirectly show that the technology of this embodiment can well alleviate the problem of mode collapse.

[0119] In summary, this embodiment constructs four different motor skill combinations and uses the Unitree H1 humanoid robot to verify the effectiveness of the method. From the experimental results, it can be seen that the method in this embodiment enables the controller network to fully learn the corresponding motor skills from a given human motion capture dataset and reduces the severity of mode collapse to a great extent, which is of great help to the integrated application of multi-motor skills of humanoid robots and will also help to apply humanoid robots to more scenarios and tasks in the future.

[0120] Embodiment 2

[0121] The purpose of this embodiment is to provide a humanoid robot multi-motion switching control system based on imitation learning, including:

[0122] An acquisition module is configured to: acquire a body state of the humanoid robot and a target motion task of the humanoid robot;

[0123] A motor torque module is configured to: input the body state of the humanoid robot and the target motion task of the humanoid robot into a pre-built motion control model to obtain the motor torque to be executed by the humanoid robot;

[0124] The pre-built motion control model is a generative adversarial network model obtained through imitation learning training; during the imitation learning training, the generative adversarial network model dynamically adjusts the sampling probability of each motion skill according to the performance of the humanoid robot in performing various motion skills, so that the humanoid robot can evenly master different motion skills;

[0125] The control module is configured to: utilize the obtained motor torque to be executed by the humanoid robot to control the motion of the humanoid robot and achieve smooth switching between different motion skills of the humanoid robot.

[0126] In further embodiments, there is also provided:

[0127] An electronic device includes a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the method described in Embodiment 1 is performed. For the sake of brevity, no further description is given here.

[0128] It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0129] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0130] A computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the method described in embodiment 1 is completed.

[0131] The method in the first embodiment can be directly embodied as a hardware processor, or a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0132] A computer program product includes a computer program, and when the computer program is executed by a processor, the method described in the first embodiment is implemented.

[0133] The present invention also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer executable instructions, such as instructions included in a program module, which are executed in a device on a real or virtual processor of the target to perform the process / method as described above. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform specific tasks or implement specific abstract data types. In various embodiments, the functions of program modules can be combined or divided between program modules as needed. Machine executable instructions for program modules can be executed in local or distributed devices. In distributed devices, program modules can be located in local and remote storage media.

[0134] The computer program code for implementing the method of the present invention can be written in one or more programming languages. These computer program codes can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the computer or other programmable data processing device, causes the function / operation specified in the flow chart and / or block diagram to be implemented. The program code can be executed completely on a computer, partially on a computer, as an independent software package, partially on a computer and partially on a remote computer or completely on a remote computer or server.

[0135] In the context of the present invention, computer program codes or related data may be carried by any appropriate carrier to enable a device, apparatus or processor to perform the various processes and operations described above. Examples of carriers include signals, computer readable media, etc. Examples of signals may include electrical, optical, radio, acoustic or other forms of propagation signals, such as carrier waves, infrared signals, etc.

[0136] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0137] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A multi-motion switching control method for a humanoid robot based on imitation learning, characterized in that: include: Obtaining the body state of the humanoid robot and the target motion task of the humanoid robot; Inputting the body state of the humanoid robot and the target motion task of the humanoid robot into a pre-built motion control model to obtain the motor torque to be executed by the humanoid robot; Using the obtained motor torque to be executed by the humanoid robot, the humanoid robot is controlled to achieve smooth switching between different motor skills of the humanoid robot; Among them, the pre-constructed motion control model is a generative adversarial network model obtained through imitation learning training; during the imitation learning training, the generative adversarial network model dynamically adjusts the sampling probability of each motion skill according to the performance of the humanoid robot in executing various motion skills, so that the humanoid robot can evenly master different motion skills.

2. The humanoid robot multi-motion switching control method based on imitation learning as claimed in claim 1, characterized in that: The construction of the motion control model is specifically as follows: Construct a reference motion capture dataset, where each reference motor skill corresponds to a motor skill code; The motion skill codes in the reference motion capture dataset are sampled, and the sampling results and the body state of the humanoid robot at the current moment constitute observation data; Inputting the observed data into the controller network and the value function network respectively to obtain the corresponding expected motor position and expected future reward; Move according to the expected motor position, return the body state of the humanoid robot at the next moment, and update the generated data set with the actual motion state of the humanoid robot; Using the conditional discriminator to obtain the probability that the generated data set belongs to the reference data set, and calculate the style reward; The parameters of the controller network are updated according to the style reward, the expected future reward and the expected motor position, and the above process is repeated until the controller network converges.

3. The humanoid robot multi-motion switching control method based on imitation learning as claimed in claim 2, characterized in that: The training process of the conditional discriminator is specifically as follows: Randomly sampling samples from both the generated data set and the reference data set, wherein the probability of each motor skill in the reference data set being sampled is dynamically adjusted according to the performance of the humanoid robot in executing each motor skill; Input the sampled samples into the conditional discriminator to obtain the probability of each sample input parameter data set; The classification loss is calculated based on the obtained probability of each sample input parameter data set, and the Adam optimizer is used for back propagation to update the network parameters of the conditional discriminator.

4. The humanoid robot multi-motion switching control method based on imitation learning as claimed in claim 2, characterized in that: The time difference method is used for back propagation, and the Adam optimizer is used to update the network parameters.

5. The humanoid robot multi-motion switching control method based on imitation learning as claimed in claim 1, characterized in that: The body state of the humanoid robot includes the linear velocity of the center of mass, the angular velocity of the center of mass, the orientation of the center of mass, the joint positions and the joint angular velocities.

6. The humanoid robot multi-motion switching control method based on imitation learning as claimed in claim 1, characterized in that: The sampling probability of each motor skill is dynamically adjusted according to the performance of the humanoid robot in performing each motor skill, so that the humanoid robot can evenly master different motor skills, specifically: Calculate the average style bonus corresponding to each sports skill; The sampling probability corresponding to each sports skill is obtained by summing the statistical mean value of the style reward corresponding to each sports skill and the mean value of the style reward corresponding to all sports skills in the reference motion capture dataset; According to the calculated sampling probability corresponding to each motor skill, the humanoid robot can evenly master different motor skills.

7. The humanoid robot multi-motion switching control method based on imitation learning as claimed in claim 1, characterized in that: The sports skill encoding is implemented in the form of word vector or one-hot encoding.

8. A humanoid robot multi-motion switching control system based on imitation learning, characterized in that: include: An acquisition module is configured to: acquire a body state of the humanoid robot and a target motion task of the humanoid robot; A motor torque module is configured to: input the body state of the humanoid robot and the target motion task of the humanoid robot into a pre-built motion control model to obtain the motor torque to be executed by the humanoid robot; The pre-built motion control model is a generative adversarial network model obtained through imitation learning training; during the imitation learning training, the generative adversarial network model dynamically adjusts the sampling probability of each motion skill according to the performance of the humanoid robot in performing various motion skills, so that the humanoid robot can evenly master different motion skills; The control module is configured to: utilize the obtained motor torque to be executed by the humanoid robot to control the motion of the humanoid robot, so as to achieve smooth switching between different motion skills of the humanoid robot.

9. An electronic device, characterized in that: The method comprises a memory and a processor and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the method according to any one of claims 1 to 7 is completed.

10. A computer-readable storage medium, characterized in that: Used to store computer instructions, which, when executed by a processor, complete the method described in any one of claims 1 to 7.

Images