Music-driven humanoid robot control method based on antagonistic imitation learning

Abstract

The invention relates to a music-driven humanoid robot control method based on antagonistic imitation learning, and the method comprises the steps: obtaining original music matched with dance demonstration, and carrying out the feature extraction of the original music; a music encoder is adopted to encode the extracted features to obtain implicit vectors of the music; inputting the implicit vector of the music into a dance generator as a condition, and generating robot dance action control information so as to construct a simulated robot dance action; the dance discriminator is used for distinguishing simulated robot dance actions and dance demonstration; alternately training the dance generator and the dance discriminator until the dance generator generates simulated robot dance actions, so that the dance discriminator cannot accurately distinguish; and obtaining a music sequence, and generating a dance action through the trained dance generator. Compared with the prior art, the method has the advantages of being capable of generating the smooth dance movement well matched with the music, and having flexibility and generalization ability and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of music-driven humanoid robots, and in particular to a music-driven humanoid robot control method based on adversarial imitation learning. Background Art

[0002] The technical principle of music-driven humanoid robots is mainly based on audio recognition technology and artificial intelligence algorithms. These technologies can collect and analyze elements such as rhythm, melody, harmony, etc. in music, and extract the emotional, style and dynamic characteristics of music. These characteristics are converted into digital signals, which in turn drive the movements, expressions and behaviors of humanoid robots.

[0003] The existing music-driven humanoid robot technologies are divided into two types: kinematics-based dance generation methods and physics-based dance generation methods, and the corresponding defects are as follows:

[0004] Kinematics-based dance generation methods have limitations that are not physically realistic:

[0005] The kinematic dance generation methods in the prior art have the problem of being physically unrealistic and are not suitable for robot motion generation. Such methods usually generate dance movements by predicting the positions of joints. Since physical simulation is not introduced in the generation process, the generated movements often violate the laws of real physics and are prone to phenomena such as floating and artifacts. This problem is particularly prominent in robot motion generation, because both simulated robots and physical robots are constrained by the laws of physics and cannot achieve movements that meet physical requirements through such generation methods.

[0006] Physics-based dance generation methods have the problems of strong data dependence and limited flexibility:

[0007] Early methods used pre-programmed action sequences to enable robots to dance to music, but this approach lacked flexibility and generalization, making it difficult for robots to adapt to changing environments and musical rhythms. Although advances in deep reinforcement learning and physical simulation have enabled robots to learn movements from human demonstrations, physics-based methods still face significant challenges. The most critical issue is how to design a reward function suitable for dynamic tasks such as music-driven dance generation. The complexity of dynamic tasks and the ambiguity of goals make it difficult to obtain effective feedback when training the controller, which in turn limits the performance of this method in real-time, diverse dance generation. Summary of the invention

[0008] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a music-driven humanoid robot control method based on adversarial imitation learning, to learn the music-driven humanoid robot control strategy, so that the robot can show smooth dance movements that are highly matched with the input music.

[0009] The purpose of the present invention can be achieved by the following technical solutions:

[0010] A music-driven humanoid robot control method based on adversarial imitation learning comprises the following steps:

[0011] Acquire original music matching the dance demonstration, and perform feature extraction on the original music;

[0012] The extracted features are encoded using a music encoder to obtain the latent vector of the music;

[0013] The latent vector of the music is input as a condition into a pre-built dance generator to generate robot dance movement control information related to the music, and simulated robot dance movements are generated through the simulation robot;

[0014] The dance discriminator is used to evaluate and distinguish the simulated robot dance movements and the corresponding dance demonstrations;

[0015] Alternately training the dance generator and the dance discriminator until the simulated robot dance movements generated by the dance generator make it impossible for the dance discriminator to accurately distinguish between the simulated robot dance movements and the corresponding dance demonstrations;

[0016] The music sequence is obtained, and after feature extraction and encoding, the latent vector of the music sequence is obtained, which is input into the trained dance generator to obtain the robot dance movement control signal to drive the robot to perform dance movements.

[0017] Furthermore, the dance generator is based on the conditional reinforcement learning strategy π(a t |s t ,z t ) to generate robot dance action control information, where a t is the action at time t, s t is the state at time t, z t is the hidden vector of the music at time t, which is used as the robot control strategy π(a t |s t ,z1), given the latent vector z of the music t and the robot's current state s t Under the condition of t ; Then, the PD controller drives the robot to perform corresponding actions according to the action and interact with the environment to obtain the reward r t , and finally the robot moves to the new state s t+1 .

[0018] Furthermore, the expression of the optimization objective of the dance generator is:

[0019]

[0020] r t (s t ,s t+1 ,z t )=-log91-D(s t ,s t+1 |z t ))

[0021] In the formula, is the objective function of the dance generator, For music dataset The expectation of sampling in represents the expectation of trajectory τ under a given strategy π and music latent vector z; to optimize the robot's action trajectory in different states, γ is the discount factor, r t (s t ,s t+1 ,z t ) is the latent vector z of the music at time t t Next, the state s at time t t Reach the next state s t+1 The reward, D(s t ,s t+1 |z t ) is the latent vector z of the music at time t t Next, the state s at time t t Reach the next state s t+1 The robot action evaluation, E(A) is the output after encoding the music clip A, and is used as the music latent vector z t .

[0022] Furthermore, the optimization objective of the dance discriminator is expressed as:

[0023]

[0024] In the formula, D(s t ,s t+1 |z t ) is the latent vector z of the music at time t t Next, the state s at time t t Reach the next state s t+1 Evaluation of robot motion, For strategy π(a t |s t ,z t ) The trajectory obtained by sampling, are the trajectories in the corresponding dance demonstration.

[0025] Furthermore, the dance discriminator is also used to determine the relevance between music and dance, and to provide feedback information to guide the learning of the dance generator.

[0026] Furthermore, the features extracted from the original music include timbre, melody and rhythm features.

[0027] Furthermore, the features extracted from the original music specifically include amplitude features, Mel-frequency cepstral coefficient features, chroma features, rhythm features and waveform features.

[0028] Furthermore, the simulation robot is built on a physical simulation platform, and physical constraints are imposed by the physical simulation platform.

[0029] Furthermore, the physical constraints include mass, collision and friction constraints on the simulation robot.

[0030] Furthermore, the simulation robot is created by an extensible markup language file, and the simulation robot is a humanoid robot with 28 degrees of freedom and 18 connecting rods, and each degree of freedom is equipped with a controller to control the movement of the joints.

[0031] Compared with the prior art, the present invention has the following advantages:

[0032] (1) High matching degree between dance movements and music: The present invention extracts the characteristic information of music, encodes it into latent vectors in latent space, and then decodes the music latent vectors into control signals such as joint torque of the robot to control the robot's movements. The dance discriminator will then identify whether the movement comes from the reference data set or the model, and give feedback information to optimize the control strategy. The optimization of the dance generator and the dance discriminator is carried out alternately. The ultimate goal is that the dance generator generates a sufficiently realistic action sequence, while the dance discriminator cannot accurately distinguish the generated robot dance movements from human demonstration dance movements. In this way, the dance generator has the ability to control the robot to show high-quality dance movements related to the input music.

[0033] (2) Physical authenticity of robot dance: The present invention models dance generation as a robot control problem. By creating a robot in a physical simulation environment and applying physical constraints to the robot, physically authentic robot dance movements are generated.

[0034] (3) End-to-end real-time generation: The present invention establishes the correlation between music and robot dance by encoding the original music information into latent vectors and then decoding the latent vectors into corresponding robot dance movements. An end-to-end interface is provided so that users can control the robot to perform dance movements that match the music well in an end-to-end manner.

[0035] (4) Flexibility and generalization capability This paper proposes a general architecture that can be easily extended to robots of any structure to generate robot dance movements related to music. By introducing human choreography logic, the robot first learns basic dance units related to music, and then arranges and combines dance units according to the input music. It can generate rich robot dances from limited music-dance data sets, greatly improving the utilization of data and solving the data dependency problem. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of a reinforcement learning principle set provided in an embodiment of the present invention;

[0037] Figure 2 A schematic flow chart of a music-driven humanoid robot control method based on adversarial imitation learning provided in an embodiment of the present invention;

[0038] Figure 3 Schematic diagram of downstream applications of a music-driven humanoid robot control method based on adversarial imitation learning provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0040] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0042] Example 1

[0043] This method involves the concepts of robot forward kinematics, inverse kinematics, reinforcement learning, generative adversarial imitation learning, and deep learning. First, a brief introduction is given:

[0044] 1. Forward kinematics and inverse kinematics

[0045] To describe an object in three-dimensional space, you need to describe its position and attitude (direction). At least 6 parameters are required, including 3 position information and 3 attitude (direction) information. The robot realizes the movement of the end through joint rotation. Therefore, solving the relationship between each joint angle and the position and attitude of the end effector is called kinematics. Knowing the angles of each joint and finding the position and attitude of the end effector is called forward kinematics. Knowing the position and attitude of the end effector and finding the angles of each joint is called inverse kinematics.

[0046] 2. Reinforcement learning, generative adversarial imitation learning

[0047] Reinforcement Learning (RL) is an important paradigm of machine learning and is widely used to learn continuous robot controllers. Reinforcement learning is usually modeled as a Markov decision process, whose goal is to maximize the cumulative reward through the interaction between the agent and the environment and the trial and error process, so as to find the optimal strategy to solve a problem. This method is applicable to almost all optimization scenarios, such as Figure 1 In the field of robot control, reinforcement learning technology can optimize control strategies through reward functions, enabling robots to learn complex skills, adapt to a variety of environments, and demonstrate higher intelligence and autonomy when performing tasks.

[0048] Generative adversarial imitation learning introduces the concept of adversarial training into the field of imitation learning, enabling robots to learn complex control strategies directly from expert datasets without explicitly defining reward functions. In this process, the robot's control strategy plays the role of a generator, while the discriminator is responsible for distinguishing whether the action comes from an expert dataset or is generated by the generator, and providing feedback to optimize the control strategy. Through the alternating optimization of the generator and the discriminator, the final generator is able to generate sufficiently realistic robot actions that the discriminator cannot distinguish the difference between it and the expert data.

[0049] 3. Deep Learning

[0050] Deep learning is an important branch of machine learning. Inspired by the biological nervous system, it processes complex data by simulating the structure of the human brain neural network. Relying on a multi-layer network structure and a large number of computing units (i.e. neurons), deep learning can effectively process massive amounts of data, learn complex relationships and patterns from them, and demonstrate superior performance in various tasks that surpass traditional methods.

[0051] 4. Construction of humanoid robot

[0052] The present invention uses an Extensible Markup Language (XML) file to create a humanoid robot with 28 degrees of freedom and 18 connecting rods. A Proportional-Derivative (PD) controller is equipped for each degree of freedom to accurately control the movement of the joints and constrain the range of joint movement according to the human physiological structure. At the same time, physical constraints such as mass, collision and friction are imposed on the robot in the physical simulation platform to ensure that the robot's movements are more realistic and in line with the laws of physics.

[0053] 5. Music feature extraction and latent space vector encoding

[0054] At each time step, a high-level feature vector containing information such as timbre, melody, and rhythm is extracted for the original music clip for subsequent training. The extracted features include 1-dimensional amplitude features, 20-dimensional Mel cepstral coefficient features, 12-dimensional chromaticity features, 1-dimensional rhythm features, and 1-dimensional waveform features. These features are what human dancers pay attention to when dancing to music. The feature extraction of music is based on music clips, corresponding to basic dance movement units. After obtaining the feature vector of the music, the music encoder will encode the feature vector into a latent vector in the latent space as an intermediate product between music and robot movements. It will be decoded into the robot's dance movements by the dance generator later to ensure the correlation between music and robot dance movements.

[0055] 6. Dance Generator

[0056] The dance generator is modeled as a conditional strategy in reinforcement learning. It takes the latent vector of the music and the current state of the robot as input information. The latent vector of the music is used as a condition to output the PD target rotation of the robot's joint degrees of freedom. The PD controller adjusts and outputs the joint torque to achieve the target rotation and control the robot to perform dance movements related to the music. The goal of the dance generator is to generate sufficiently realistic robot dance movements to deceive the dance discriminator.

[0057] 7. Dance Identifier

[0058] The dance discriminator acts as a dance teacher. It takes motion data as input and determines whether the motion data comes from a reference human demonstration or is generated by the dance generator to control the robot. It also determines the relevance of music and dance and gives feedback information to guide the learning of the dance generator to control the robot to show more realistic and music-related dance movements. The goal of the dance discriminator is to distinguish as much as possible whether the motion data comes from a reference human dance demonstration or a robot.

[0059] 8. Using adversarial imitation learning to train robot control strategies

[0060] By integrating the above technical solutions, a music-driven humanoid robot control method based on adversarial imitation learning can be obtained, which includes the following steps:

[0061] Music feature information extraction S1: obtaining original music matching the dance demonstration and performing feature extraction on the original music;

[0062] Music feature information encoding S2: Use a music encoder to encode the extracted features to obtain the latent vector of the music;

[0063] Dance generation based on music latent vector and robot current state S3: Input the music latent vector as a condition into the pre-built dance generator to generate robot dance action control information related to the music, and generate simulated robot dance actions through the simulation robot;

[0064] Identify whether the dance comes from human demonstration or is generated by the robot controlled by the dance generator S4: Evaluate and distinguish the simulated robot dance movements and the corresponding dance demonstrations through the dance identifier;

[0065] Alternating training process S5: alternately training the dance generator and the dance discriminator until the simulated robot dance movements generated by the dance generator make it impossible for the dance discriminator to accurately distinguish between the simulated robot dance movements and the corresponding dance demonstrations;

[0066] Actual usage process S6: Obtain the music sequence, perform feature extraction and encoding, obtain the latent vector of the music sequence, input it into the trained dance generator, obtain the robot dance movement control signal, and drive the robot to perform dance movements.

[0067] Detailed flow chart as follows Figure 2 shown.

[0068] This paper proposes a universal, robot-model-independent music-driven dance generation method, which is applicable to robots of any model and can generate robot dance movements end-to-end. By inputting any music, we can control the robot to perform physically realistic dance movements related to the music in real time. It has a wide range of applications in the fields of physics-based character animation, robot control, and human-computer interaction. The schematic diagram in the downstream task application is as follows: Figure 3 shown.

[0069] Specifically, the solution of the present invention is modeled as a conditional reinforcement learning problem. In the reinforcement learning problem, the agent uses a strategy π(a t |s t ) Perform action a t Interact with the environment to get rewards r t , and arrive at a new state st+1 , the goal of the agent is to learn a strategy π(a t |s t ) to maximize the expectation of the cumulative discounted reward G(τ):

[0070]

[0071] Where τ=(s0,a0,r0,…,s T-1 ,a T-1 ,r T-1 ) is the sequence of states, actions, and rewards, T is the time range of the trajectory τ, represents the possibility of trajectory τ under policy π. γ∈[0,1) is the discount factor. The reward function can be changed by t The method encourages the agent to complete different tasks.

[0072] The dance generator that controls the robot to dance is essentially a conditional reinforcement learning policy π(a t |s t ,z t ), the latent vector z of the music t is used as a conditional input to the strategy to generate robot dance action control information related to music. The goal of the dance generator is to generate sufficiently realistic robot dance movements to fool the dance discriminator:

[0073] In the formula, is the objective function of the dance generator, For music dataset The expectation of sampling in represents the expectation of trajectory τ under a given strategy π and music latent vector z; to optimize the robot's action trajectory in different states, γ is the discount factor, r t (s t ,s t+1 ,z t ) is the latent vector z of the music at time t t Next, the state s at time t T Reach the next state s t+1 The reward, D(s t ,s t+1 |z t ) is the latent vector z of the music at time t t Next, the state s at time t t Reach the next state s t+1 The robot action evaluation, E(A) is the output after encoding the music clip A, and is used as the music latent vector z t .

[0074] It can be seen that the closer the robot's dance moves are to human dance demonstrations, the higher the D(s t ,s t+1 |z t ) is larger, the reward r obtained by the corresponding robot t The bigger it is.

[0075] The goal of the dance discriminator is to distinguish as much as possible whether the dance moves come from a reference human dance demonstration or a robot. Its goal is modeled as:

[0076]

[0077] where τ π is from the strategy π(a t |s t ,z t ) sampled trajectory, It comes from the trajectory of human dance demonstrations. During the training process, the optimization of the dance generator and the dance discriminator is carried out alternately, so that the discriminator can more accurately distinguish between the generated movements and the expert demonstration movements, and the generator's ability to generate realistic movements is improved. The ultimate goal is that the dance generator generates sufficiently realistic movement sequences, while the dance discriminator cannot accurately distinguish between the generated robot dance movements and human demonstration dance movements. In this way, the dance generator has the ability to control the robot to show high-quality dance movements related to the input music.

[0078] (7) Downstream task application based on the method

[0079] In the field of computer animation and robotic control, it has always been a key task to enable simulated characters and humanoid robots to perform actions accurately. This method decodes music feature information into robot control signals, generates cross-modal dance movements related to music, and innovatively proposes a new paradigm for simulated character and robot motion control. In virtual reality scenarios, this method can bring a more realistic and immersive character interaction experience; in the field of robotics, it helps to develop more flexible and natural humanoid robots.

[0080] Furthermore, this method is applicable to music-driven dance generation of robots with arbitrary structures.

[0081] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A music-driven humanoid robot control method based on adversarial imitation learning, characterized in that: The following steps are involved: Acquire original music matching the dance demonstration, and perform feature extraction on the original music; The extracted features are encoded using a music encoder to obtain the latent vector of the music; The latent vector of the music is input as a condition into a pre-built dance generator to generate robot dance movement control information related to the music, and simulated robot dance movements are generated through the simulation robot; The dance discriminator is used to evaluate and distinguish the simulated robot dance movements and the corresponding dance demonstrations; Alternately training the dance generator and the dance discriminator until the simulated robot dance movements generated by the dance generator make it impossible for the dance discriminator to accurately distinguish between the simulated robot dance movements and the corresponding dance demonstrations; The music sequence is obtained, and after feature extraction and encoding, the latent vector of the music sequence is obtained, which is input into the trained dance generator to obtain the robot dance movement control signal to drive the robot to perform dance movements.

2. The music-driven humanoid robot control method based on adversarial imitation learning according to claim 1 is characterized in that: The dance generator is based on a conditional reinforcement learning strategy π(a t |s t ,z t ) to generate robot dance action control information, where a t is the action performed by the robot at time t, s t is the state of the robot at time t, z t is the hidden vector of the music at time t; as the robot control strategy π(a t |s t ,z t ), given the latent vector z of the music t and the robot's current state s t Under the condition of t ; Then, the PD controller drives the robot to perform corresponding actions according to the action and interact with the environment to obtain the reward r t , and finally the robot moves to the new state s t+1 .

3. The music-driven humanoid robot control method based on adversarial imitation learning according to claim 2 is characterized in that: The expression of the optimization objective of the dance generator is: r t (s t ,s t+1 ,z t )=-log(1-D(s t ,s t+1 |z t )) In the formula, is the objective function of the dance generator, For music dataset The expectation of sampling in represents the expectation of trajectory τ under a given strategy π and music latent vector z; to optimize the robot's action trajectory in different states, γ is the discount factor, r t (s t ,s t+1 ,z t ) is the latent vector z of the music at time t t Next, the state s at time t t Reach the next state s 1+1 The reward, D(s t ,s t+1 |z t ) is the state s1 at time t to the next state s under the latent vector z1 of the music at time t 1+1 The robot action evaluation, E(A) is the output after encoding the music clip A, and is used as the music latent vector z t .

4. The music-driven humanoid robot control method based on adversarial imitation learning according to claim 2 is characterized in that: The expression of the optimization objective of the dance discriminator is: In the formula, D(s t ,s t+1 |z t ) is the latent vector z of the music at time t t Next, the state s at time t t Reach the next state s t+1 Evaluation of robot motion, For strategy π(a t |s t ,z t ) The trajectory obtained by sampling, are the trajectories in the corresponding dance demonstration.

5. The music-driven humanoid robot control method based on adversarial imitation learning according to claim 1, characterized in that: The dance discriminator is also used to determine the relevance between music and dance, and to provide feedback information to guide the learning of the dance generator.

6. The music-driven humanoid robot control method based on adversarial imitation learning according to claim 1, characterized in that: The features extracted from the original music include timbre, melody and rhythm features.

7. The music-driven humanoid robot control method based on adversarial imitation learning according to claim 6 is characterized in that: The features extracted from the original music specifically include amplitude features, Mel-cephalometric coefficient features, chroma features, rhythm features and waveform features.

8. The music-driven humanoid robot control method based on adversarial imitation learning according to claim 1, characterized in that: The simulation robot is built on a physical simulation platform, and physical constraints are imposed by the physical simulation platform.

9. The music-driven humanoid robot control method based on adversarial imitation learning according to claim 8, characterized in that: The physical constraints include mass, collision and friction constraints on the simulated robot.

10. The music-driven humanoid robot control method based on adversarial imitation learning according to claim 9, characterized in that: The simulation robot is created by an extensible markup language file. The simulation robot is a humanoid robot with 28 degrees of freedom and 18 connecting rods. Each degree of freedom is equipped with a controller to control the movement of the joints.

Images