Robot motion control methods and devices

Abstract

This application provides a robot motion control method and device. The method includes inputting first historical data into a trained feature extraction network to obtain first feature data, which includes terrain features and / or behavioral features; inputting a current first action command, the first feature data, and second historical data into a trained policy network to obtain a current first action; the trained policy network is obtained by training the policy network using a reinforcement learning algorithm based on a fusion framework composed of a feature extraction network, a policy network, and an adversarial evaluation network; determining a corresponding current first torque based on the current first action; and driving the robot to execute the current first action based on the current first torque. The method of this embodiment can easily train a policy network to achieve human-like motion using only a small amount of expert dataset, improving the quality of behavioral style imitation.

Description

Technical Field

[0001] This application relates to the field of humanoid robot technology, and more particularly to a robot motion control method and device. Background Technology

[0002] Robotic anthropomorphic movement refers to the process by which robots, through specific design and technology, simulate human movement and behavior.

[0003] In related technologies, adversarial networks can be used to learn human movement styles.

[0004] However, in the process of realizing this application, the inventors discovered that the prior art has at least the following problems: the use of adversarial networks requires the separate training of an evaluation network using an expert dataset to improve the robot's humanoid movements, which places very high demands on the expert dataset. Summary of the Invention

[0005] This application provides a robot motion control method and device to improve the quality of motion style imitation and reduce reliance on expert datasets.

[0006] In a first aspect, embodiments of this application provide a robot motion control method, including:

[0007] Acquire first historical data, second historical data, and the current first action command; the first historical data includes observation data corresponding to a first preset number of time steps before the current time step, and the second historical data includes observation data corresponding to a second preset number of time steps before the current time step; the second preset number is less than the first preset number.

[0008] The first historical data is input into a trained feature extraction network to obtain the first feature data; the first feature data includes terrain features and / or behavioral features.

[0009] The current first action instruction, the first feature data, and the second historical data are input into the trained policy network to obtain the current first action; the trained policy network is obtained by training the policy network according to the reinforcement learning algorithm based on a fusion framework composed of a feature extraction network, a policy network, and an adversarial evaluation network.

[0010] Based on the current first action, a corresponding current first torque is determined, and the robot is driven to perform the current first action based on the current first torque.

[0011] In one possible design, the trained feature extraction network is a convolutional neural network; the convolutional neural network includes a first hidden layer and a second hidden layer; the first hidden layer has 42 input channels, 32 output channels, a kernel size of 6, and a stride of 5; the second hidden layer has 32 input channels, 16 output channels, a kernel size of 4, and a stride of 2; the activation function of the convolutional neural network is the ReLU activation function.

[0012] In one possible design, a fusion framework is constructed based on a feature extraction network, a policy network, and an adversarial evaluation network; based on the fusion framework, the policy network is trained according to the Proximal Policy Optimization (PPO) algorithm to obtain the trained policy network.

[0013] In one possible design, the feature extraction network includes an encoder and a decoder; the encoder includes a convolutional neural network; and the decoder includes a multilayer perceptron.

[0014] In one possible design, training the policy network based on the fusion framework using the Proximity Policy Optimization (PPO) algorithm to obtain the trained policy network includes:

[0015] For each time step, the third historical data is input into the feature extraction network to obtain the second feature data; the second feature data includes terrain features and / or behavioral features; the third historical data includes a third preset number of actions prior to the previous action;

[0016] The current second action instruction, the previous reward, the second feature data, and the fourth historical data are input into the policy network to obtain the current second action; the fourth historical data includes a fourth preset number of actions between the previous actions; the fourth preset number is less than the third preset number.

[0017] The current first action is input into the robot's controller to obtain the corresponding current second torque;

[0018] Input the current second torque into the environment model to obtain the current first state;

[0019] The current first state is input into the adversarial evaluation network to obtain the current style reward;

[0020] The current reward is determined based on a preset reward function, according to the current task reward and the style reward.

[0021] Store the current second action, the current first state, the current style reward, and the current reward in the experience replay buffer;

[0022] Sample data is sampled from the experience replay buffer; the policy network is updated based on the objective function of the PPO algorithm and the sample data.

[0023] The adversarial evaluation network is updated based on the sample data according to the preset objective function.

[0024] The feature extraction network is updated based on the sample data according to the preset loss function.

[0025] In one possible design, the objective function of the PPO algorithm is expressed as:

[0026]

[0027] in, The objective function of PPO is to find the parameter θ that maximizes the objective function. It is an estimate of the advantage function, representing the state w. t Take action a t Advantages relative to the average; w t for It is a new strategy p θ Compared to the old strategy In action a t The probability ratio, i.e. the importance sampling ratio; It is the truncation strategy ratio, where ɛ is a constant, ɛ≤0.2, and the clip function ensures that the importance sampling ratio is in the range [1-ɛ,1+ɛ].

[0028] In one possible design, the expression for the preset objective function is:

[0029]

[0030] in, Represents both the actual state and the expert reference state; The representative adversarial evaluation network output targets state pairs. The output evaluation value is closer to 1 when it is closer to the expert state, and closer to -1 when it is further away from the expert state. It refers to the difference in distribution divergence between the actual state and the expert reference state; It is the weighting coefficient.

[0031] In one possible design, the expression for the preset loss function is:

[0032]

[0033] in, These are observational data representing history. This represents the next state after the restructuring. This represents the next state; MSE (mean square error) is the mean square error. p(zt) is the posterior distribution, i.e., the conditional distribution of the latent variable z given the input x; p(zt) is the prior distribution, i.e., the marginal distribution of the latent variable z; KL is the KL divergence, which measures the approximate posterior distribution. The difference from the prior distribution p(zt).

[0034] In one possible design, the expression for the preset reward function is:

[0035]

[0036] in, As the main reward, As a reward for the task, As a style reward, ω g ω is the weight of the task reward. s Weighting for style rewards.

[0037] Secondly, embodiments of this application provide a robot motion control device, including: at least one processor and a memory;

[0038] The memory stores computer-executed instructions;

[0039] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the method described in the first aspect above and various possible designs of the first aspect.

[0040] The robot motion control method and device provided in this embodiment include acquiring first historical data, second historical data, and a current first action command. The first historical data includes observation data corresponding to a first preset number of time steps before the current time step, and the second historical data includes observation data corresponding to a second preset number of time steps before the current time step, where the second preset number is less than the first preset number. The first historical data is input into a trained feature extraction network to obtain first feature data, which includes terrain features and / or behavioral features. The current first action command, the first feature data, and the second historical data are input into a trained policy network to obtain the current first action. The trained policy network is obtained by training the policy network using a reinforcement learning algorithm based on a fusion framework composed of a feature extraction network, a policy network, and an adversarial evaluation network. Based on the current first action, a corresponding current first torque is determined, and the robot is driven to execute the current first action based on the current first torque. The robot motion control method provided in this embodiment constructs a fusion framework based on a policy network, an adversarial evaluation network, and a feature extraction network. It can easily train a policy network to achieve human-like motion by combining only a small amount of expert dataset. Furthermore, by setting up a feature extraction network, it can pre-extract feature data such as terrain features and behavioral features, and input them into the policy network, making the actions determined by the policy network more compliant and in line with human behavioral styles, thereby improving the quality of behavioral style imitation. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 A flowchart illustrating the robot motion control method provided in an embodiment of this application;

[0043] Figure 2 This is a schematic diagram illustrating an application scenario of the robot motion control method provided in the embodiments of this application;

[0044] Figure 3 A flowchart illustrating the training process of the policy network provided in this application embodiment;

[0045] Figure 4 A schematic diagram illustrating the training principle of the fusion framework for the robot motion control method provided in this application embodiment;

[0046] Figure 5 This is a schematic diagram of the structure of the robot motion control device provided in the embodiments of this application;

[0047] Figure 6 This is a schematic diagram of the hardware structure of the robot motion control device provided in the embodiments of this application. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0049] It should be noted that the robot motion control method and equipment provided in this application can be used in the field of humanoid robot technology, or in any field other than the field of humanoid robot technology. The application field of the robot motion control method and equipment provided in this application is not limited.

[0050] Robotic anthropomorphic movement refers to the process by which robots, through specific design and technology, simulate human movement and behavior.

[0051] In related technologies, human movement styles can be learned directly from human motion features. For example, Fourier Latent Dynamics (FLD) encoders can be used to learn features from human motion trajectories, obtaining a Fourier parameter space from which samples can be used to construct a reconstructed trajectory. Adversarial networks can also be used to learn human movement styles. However, the former method struggles to learn subtle changes in movement and is difficult to reproduce while maintaining balance in scenarios involving lower limb movements. The latter method requires a separate evaluation network trained using expert datasets to improve the robot's humanoid movements, placing very high demands on the quality of the expert dataset.

[0052] To address the aforementioned technical problems, the inventors of this application have discovered that a humanoid motion fusion framework based on a policy network, an adversarial evaluation network, and a feature extraction network can be easily trained using a small amount of expert data. Furthermore, by setting up a feature extraction network, terrain features, behavioral features, and other characteristic data can be pre-extracted and input into the policy network, making the actions determined by the policy network more compliant and consistent with human behavioral styles, thus improving the quality of behavioral style imitation. Based on this, embodiments of this application provide a robot motion control method.

[0053] The technical solutions of this application will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0054] Figure 1 This is a flowchart illustrating the robot motion control method provided in an embodiment of this application. Figure 1 As shown, the method includes:

[0055] 101. Obtain first historical data, second historical data, and current first action command; the first historical data includes observation data corresponding to a first preset number of time steps before the current time step, and the second historical data includes observation data corresponding to a second preset number of time steps before the current time step; the second preset number is less than the first preset number.

[0056] The executing entity in this embodiment can be a controller of a humanoid robot or a humanoid robot itself.

[0057] Specifically, the first historical data contains observation data from more time steps than the second historical data; therefore, the first historical data can also be called long historical data, and the second historical data can be called short historical data. For example... Figure 2 As shown, long historical data can include action a corresponding to time t-1. t-1 and humanoid robots perform actions a t-1 The resulting state O t ,to t-n Action a corresponding to time moment t-n and perform action a t-n The resulting state O t-n+1 n can be a positive integer greater than 50 and less than 150, such as 100. Short historical data can include the action 'a' corresponding to time t-1. t-1 and humanoid robots perform actions a t-1 The resulting state O t Action a corresponding to time tn t-n and perform action a t-n The resulting state O t-m+1 m can be a positive integer greater than or equal to 2 or less than or equal to 8, for example, 5. Where, O t Including angular velocity ω t Gravity vector g t , the position of joints throughout the body q t and the speed of all joints q t '.

[0058] The current task instruction can be an action instruction issued by the task instruction module, and may include parameters such as destination, average speed, and average angular velocity.

[0059] In this embodiment, a storage space can be set up to store the action 'a' at each time step. t-1 State Qt (This may include angular velocity ω) t Gravity vector g t , the position of joints throughout the body q t and the speed of all joints q t Data such as whether the device is standing, walking, and control commands can be extracted from this storage space for each time step. t The input consists of a policy network and a trained feature extraction network. t This includes long-term historical data, short-term historical data, and whether the data is standing or walking (e.g., standing is 0, walking is 1).

[0060] 102. Input the first historical data into the trained feature extraction network to obtain the first feature data; the first feature data includes terrain features and / or behavioral features.

[0061] Specifically, by using a trained feature extraction network to extract features from long-term historical data, terrain features or behavioral features can be obtained. By extracting representations from long-term historical data using a trained feature extraction network, the trained policy network can recognize and predict terrain environment and behavioral features through historical latent representations, resulting in smoother generated actions and behaviors. Because long-term historical data contains observation data across a sufficient number of time steps, it can more accurately predict terrain features or behavioral features, thus improving prediction accuracy.

[0062] In some embodiments, the trained feature extraction network is a convolutional neural network; the convolutional neural network includes a first hidden layer and a second hidden layer; the first hidden layer has 42 input channels, 32 output channels, a kernel size of 6, and a stride of 5; the second hidden layer has 32 input channels, 16 output channels, a kernel size of 4, and a stride of 2; the activation function of the convolutional neural network is the ReLU activation function.

[0063] Specifically, the Convolutional Neural Network (CNN) structure consists of two hidden layers. The configuration of the two hidden layers is defined in the manner of [number of input channels, number of output channels, kernel size, stride size], which are [42, 32, 6, 5] and [32, 16, 4, 2], respectively, and the ReLU activation function is used.

[0064] 103. Input the current first action instruction, the first feature data, and the second historical data into the trained policy network to obtain the current first action; the trained policy network is a fusion framework based on a feature extraction network, a policy network, and an adversarial evaluation network, and is obtained by training the policy network according to a reinforcement learning algorithm.

[0065] In this embodiment, the trained policy network consists of three layers, each containing 256 neurons using ELU (Exponential Linear Unit) as the activation function. Specifically, for the input layer: the number of neurons depends on the number of input features for the specific problem. For example, if the input features are an n-dimensional vector, then the input layer has n neurons. For the hidden layers: the first hidden layer contains 256 neurons, each using ELU as the activation function. The second hidden layer also contains 256 neurons, each also using ELU as the activation function. The main function of the hidden layers is to extract features from the input data and map them to a higher or lower dimensional space to more easily solve classification or regression problems. For the output layer: the number of neurons in the output layer depends on the output requirements of the specific problem. For example, in classification problems, the number of neurons in the output layer may equal the number of classes (using the softmax activation function); in regression problems, the output layer may have only one neuron (using no activation function or using a linear activation function). In reinforcement learning policy networks, the output layer can correspond to the probability distribution of actions or continuous values ​​of actions. ELU (Exponential Linear Unit): ELU is a non-linear activation function.

[0066] 104. Based on the current first action, determine the corresponding current first torque, and drive the robot to execute the current first action based on the current first torque.

[0067] Specifically, such as Figure 2 As shown, for the action a output by the trained policy network t-1 The torque τ is generated by the PD controller. torque This torque serves as a control command, acting on the robot's motors. The robot follows the control command and interacts with the environment to generate a new state s. t (Joint positions throughout the body q) t Linear velocity v t angular velocity ω t Height position z pos The agent will compare its style with the expert set in the constructed adversarial evaluation network, generating style rewards that, along with task rewards, will affect the policy network updates. Ultimately, the agent will improve its behavior based on style coefficients, making its actions more human-like.

[0068] The robot motion control method provided in this embodiment constructs a fusion framework based on a policy network, an adversarial evaluation network, and a feature extraction network. It can easily train a policy network to achieve human-like motion by combining only a small amount of expert dataset. Furthermore, by setting up a feature extraction network, it can pre-extract feature data such as terrain features and behavioral features, and input them into the policy network, making the actions determined by the policy network more compliant and in line with human behavioral styles, thereby improving the quality of behavioral style imitation.

[0069] Figure 3 This is a flowchart illustrating the training process of the policy network provided in an embodiment of this application. Figure 3 As shown, the method includes:

[0070] 301. Construct a fusion framework based on feature extraction network, policy network and adversarial evaluation network.

[0071] Specifically, the first step is to construct the expert dataset required for the adversarial evaluation network. This can be achieved by collecting data on each joint and key point of the human body from a motion capture device, and then mapping this data to the corresponding joints and end effector positions on the robot body. This method collects various full-body data, including walking, jumping, and squatting, as the expert dataset. Since the robot motion control method provided in this embodiment uses a fusion framework constructed from a feature extraction network, a policy network, and an adversarial evaluation network for training, the number of samples in the expert dataset does not need to be too large; for example, 200 samples are sufficient.

[0072] In this embodiment, the adversarial evaluation network consists of two MLP (Multilayer Perceptron) layers, each containing 1024 and 512 'relu' (linear rectified units) neurons as activation functions.

[0073] In some embodiments, the feature extraction network includes an encoder and a decoder; the encoder includes a convolutional neural network; and the decoder includes a multilayer perceptron.

[0074] Specifically, a Variational Autoencoder (VAE) consists of an encoder and a decoder. The encoder is based on a Convolutional Neural Network (CNN) architecture and consists of two hidden layers. The configuration of the two hidden layers is defined in the order of [number of input channels, number of output channels, kernel size, stride size], which are [42, 32, 6, 5] and [32, 16, 4, 2], respectively, and they use the ReLU activation function. The decoder is based on a Multilayer Perceptron (MLP) architecture and consists of three layers, each containing 256 neurons using 'relu' as the activation function.

[0075] 302. Based on the fusion framework, the policy network is trained according to the near-end policy optimization (PPO) algorithm to obtain the trained policy network.

[0076] Specifically, after constructing a fusion framework of CNN network, policy network, and adversarial evaluation network (AMP), expert set data and robot state data can be read to learn the action styles of the expert set. Then, based on the reinforcement learning PPO algorithm, the robot's behavior policy can be trained in an ISAAC Gym environment to learn action execution strategies. The learned execution strategy is then transformed into a model, first validated in a physical environment such as Mujoco using sim2sim, and after successful validation, converted into an ONNX model for real-world deployment and debugging, ultimately resulting in a robot capable of human-like movements.

[0077] In some embodiments, training the policy network based on the fusion framework and the Proximal Policy Optimization (PPO) algorithm to obtain the trained policy network may include: for each time step, inputting third historical data into the feature extraction network to obtain second feature data; the second feature data includes terrain features and / or behavioral features; the third historical data includes a third preset number of actions prior to the previous action; inputting the current second action instruction, the previous reward, the second feature data, and fourth historical data into the policy network to obtain the current second action; the fourth historical data includes a fourth preset number of actions between the previous actions; the fourth preset number is less than the third preset number; inputting the current first action into the robot's controller, Obtain the corresponding current second torque; input the current second torque into the environment model to obtain the current first state; input the current first state into the adversarial evaluation network to obtain the current style reward; determine the current reward based on the current task reward and the style reward according to the preset reward function; store the current second action, the current first state, the current style reward, and the current reward in the experience replay buffer; sample data from the experience replay buffer; update the policy network according to the sample data based on the objective function of the PPO algorithm; update the adversarial evaluation network according to the sample data based on the preset objective function; update the feature extraction network according to the sample data based on the preset loss function.

[0078] In this embodiment, during the update of the feature extraction network, multiple samples can be extracted from the experience replay buffer. Each sample includes long historical data for inputting the CNN encoder and the corresponding state Ot+1 (this state is the observation data obtained after the action corresponding to the long historical data is executed, and this state is used as the label value). Then, for each sample, the long historical data in the sample can be input into the feature extraction network (i.e., the VAE composed of the CNN encoder and the MLP decoder) to obtain the output of the MLP, and the loss can be calculated based on the output value and the label value.

[0079] For example, such as Figure 4 As shown, firstly, based on the CNN encoder in VAE, the long historical data a is processed. t-1 O t a t-2 O t-1 ...to learn, its output and short historical data O t-4 ~O t a t-4 ~a t-1 The control command Ct is input into the policy network at the same time. Through the extraction of the representation work of long historical data by the variational autoencoder (VAE), the agent can recognize and predict the terrain environment through the potential historical representations, and the generated actions are also more compliant.

[0080] Secondly, regarding the output action a of the policy network t-1 The torque τ is generated by the PD controller. torque As control commands, these commands act on the robot's motors, and the robot tracks these commands and interacts with the environment to generate new body state data. t The style reward is compared with the expert set in the constructed adversarial evaluation network, and this comparison is used together with the task reward to update the policy network. Ultimately, the policy network will improve its behavioral style based on the style coefficient, making its actions more like those of a human.

[0081] In some embodiments, the objective function of the PPO algorithm is expressed as:

[0082] (1)

[0083] in, The objective function of PPO is to find the parameter θ that maximizes the objective function. It is an estimate of the advantage function, representing the state w. t Take action a t Advantages relative to the average; w t for It is a new strategy p θ Compared to the old strategy In action at The probability ratio, i.e. the importance sampling ratio; It is the truncation strategy ratio, where ɛ is a constant, ɛ≤0.2, and the clip function ensures that the importance sampling ratio is in the range [1-ɛ,1+ɛ].

[0084] Specifically, the policy network employs a policy based on Markov Decision Processes (MDPs) from reinforcement learning. MDPs are the processes by which an agent (or robot) selects actions based on its current state in an environment to maximize long-term rewards. The state set (S) represents the set of all possible states the agent can be in. The action set (A) represents the set of all actions the agent can choose in each state. The state transition probability (P) describes the probability distribution of transitioning to other states after performing an action in a given state. The reward function (R) defines the reward the agent receives for performing an action in a given state.

[0085] Choosing the optimal action to obtain the maximum expected reward in a complex environment is a challenge. In this embodiment, a reinforcement learning method based on Proximal Policy Optimization (PPO) is adopted to solve the difficulty of choosing the learning rate value when the policy gradient is dealing with a continuous action space. If the learning rate is too small, the deep reinforcement learning will have poor convergence and may fail to complete the training. If the learning rate is too large, the data will be inconsistent when iterating between the old and new policies, resulting in large learning fluctuations or local oscillations. The PPO algorithm formula is shown in expression (1). It is the objective function of PPO, and the goal is to find the parameter θ that maximizes the objective function. It is an estimate of the advantage function, representing the state w. t Take action a t Advantages relative to the average. t Including short historical data (i.e. fourth historical data), the estimation of the advantage function is usually calculated by some methods, such as the Generalized Advantage (GAE). It is the probability ratio of the new policy pθ to the old policy pθk on action at. This ratio is also called the "importance sampling ratio". It is a "truncation strategy ratio" unique to PPO, in which It is a small constant, such as 0.1 or 0.2. This clip function ensures that the importance sampling ratio is within... Within this range, this can avoid excessively large policy updates.

[0086] In some embodiments, the expression of the preset objective function is:

[0087] (2)

[0088] in, Represents both the actual state and the expert reference state; The representative adversarial evaluation network output targets state pairs. The output evaluation value is closer to 1 when it is closer to the expert state, and closer to -1 when it is further away from the expert state. It refers to the difference in distribution divergence between the actual state and the expert reference state; It is the weighting coefficient.

[0089] Specifically, the update strategy of the adversarial evaluation network is mainly based on the above expression (2), which takes the data from the expert set and the robot's own state as input, and updates the network by minimizing the distance between its output value and the label. Represents both the actual state and the expert reference state; The representative adversarial evaluation network output targets state pairs. The output evaluation value is closer to 1 when it is closer to the expert state, and closer to -1 when it is further away from the expert state. This involves the difference in the distribution divergence between the actual state and the expert reference state. Ideally, this difference should not be too large, and should be addressed through... The weighting coefficient is used for adjustment, and the selection of the weighting coefficient is also based on experience.

[0090] In some embodiments, the expression for the preset loss function is:

[0091] (3)

[0092] in, It refers to observational data representing history, i.e., short-historical data. This represents the next state after the restructuring. This represents the next state; MSE (mean square error) is the mean square error. p(zt) is the posterior distribution, i.e., the conditional distribution of the latent variable z given the input x; p(zt) is the prior distribution, i.e., the marginal distribution of the latent variable z; KL is the Kullback-Leibler divergence (KL), which measures the approximate posterior distribution. The difference between the KL divergence and the prior distribution p(zt). KL divergence, also known as relative entropy or information divergence, is a method used to measure the difference between two probability distributions.

[0093] Specifically, in a Variational Autoencoder (VAE), the main function of the VAE encoder is to compress the input data into a low-dimensional representation, called a latent variable. This latent variable is a low-dimensional representation of the data in the latent space; it captures the main features of the data while removing redundant information. Through the latent variable, dimensionality reduction, reconstruction, and generation of data can be achieved, and future motion can be predicted. In a VAE, the encoder outputs not a single latent variable value, but a distribution of the latent variable. Specifically, the encoder outputs the mean (μ) and variance (σ²) of the latent variable, or their logarithmic form (logσ²). This allows sampling from this distribution in the latent space to obtain the specific values ​​of the latent variables. A VAE encoder is typically a neural network that accepts input data and outputs the mean and variance (or log-variance) of the latent variable.

[0094] The update process of a Variational Autoencoder (VAE) is as follows: First, input data: Historical observation data is fed into the encoder. Second, encoding: The encoder compresses the input data into a low-dimensional latent representation through neural network layers (CNN convolutional layers). In a VAE, this latent representation is not a single point, but rather the parameters (mean and variance) of a distribution. Third, sampling: A point is sampled from the latent distribution. Through sampling, the VAE can generate diverse outputs, not just reconstruct the input. Then, decoding: The sampled latent representation is fed into the decoder, which attempts to reconstruct the original input data through neural network layers (MLP multilayer perceptron). Next, loss is calculated: The VAE loss function consists of two parts: reconstruction error and KL divergence. Here, reconstruction error measures the difference between the reconstructed data and the predicted data, while KL divergence measures the difference between the posterior and prior distributions of the latent representation. Finally, backpropagation and parameter update: Based on the loss function, the gradient is calculated using the backpropagation algorithm, and the parameters of the encoder and decoder are updated using an optimization algorithm (Adam).

[0095] The preset loss function used to calculate the loss is shown in expression (3) above.

[0096] In some embodiments, the expression of the preset reward function is:

[0097] (4)

[0098] in, As the main reward, As a reward for the task, As a style reward, ω g ω is the weight of the task reward.s Weighting for style rewards.

[0099] Specifically, regarding the reward design, as shown in expression (4), the rewards are mainly divided into two parts: style rewards and task rewards.

[0100] Regarding task rewards, the method used in this patent mainly includes task tracking in the x and y directions and smoothing rewards. The main reward methods are as follows:

[0101] (5)

[0102] Where rv is the linear velocity tracking reward, This represents the reward coefficient, which can be adjusted based on experience. Represents task instructions. Represents actual speed.

[0103] (6)

[0104] Where rw is the angular velocity tracking reward, This represents the reward coefficient, which can be adjusted based on experience. Represents task instructions. This represents the actual angular velocity.

[0105] The robot motion control method provided in this embodiment collects data through a fusion framework composed of a feature extraction network, a policy network, and an adversarial evaluation network. Then, based on the collected data, the policy network, the adversarial evaluation network, and the feature extraction network are updated respectively. This results in a better-trained policy network and a better-trained feature extraction network with more accurate style learning, making the behavior style of the humanoid robot closer to that of a human.

[0106] Figure 5 This is a schematic diagram of the structure of the robot motion control device provided in an embodiment of this application. Figure 5 As shown, the robot motion control device 50 includes: an acquisition module 501, an input module 502, and an execution module 503.

[0107] The acquisition module 501 is used to acquire first historical data, second historical data, and current first action command; the first historical data includes observation data corresponding to a first preset number of time steps before the current time step, and the second historical data includes observation data corresponding to a second preset number of time steps before the current time step; the second preset number is less than the first preset number.

[0108] The input module 502 is used to input the first historical data into the trained feature extraction network to obtain the first feature data; the first feature data includes terrain features and / or behavioral features.

[0109] The input module 502 is further configured to input the current first action instruction, the first feature data, and the second historical data into the trained policy network to obtain the current first action; the trained policy network is obtained by training the policy network according to the reinforcement learning algorithm based on a fusion framework composed of a feature extraction network, a policy network, and an adversarial evaluation network.

[0110] The execution module 503 is used to determine the corresponding current first torque based on the current first action; and drive the robot to perform the current first action based on the current first torque.

[0111] The robot motion control device provided in this application embodiment constructs a fusion framework based on a policy network, an adversarial evaluation network, and a feature extraction network. It can easily train a policy network to achieve human-like motion by combining only a small amount of expert dataset. Furthermore, by setting up a feature extraction network, it can pre-extract feature data such as terrain features and behavioral features, and input them into the policy network, making the actions determined by the policy network more compliant and in line with human behavioral styles, thereby improving the quality of behavioral style imitation.

[0112] In some embodiments, the trained feature extraction network is a convolutional neural network; the convolutional neural network includes a first hidden layer and a second hidden layer; the first hidden layer has 42 input channels, 32 output channels, a kernel size of 6, and a stride of 5; the second hidden layer has 32 input channels, 16 output channels, a kernel size of 4, and a stride of 2; the activation function of the convolutional neural network is the ReLU activation function.

[0113] In some embodiments, the device 50 further includes: a training module (not shown) for: constructing a fusion framework based on a feature extraction network, a policy network, and an adversarial evaluation network; and training the policy network based on the fusion framework using a proximal policy optimization (PPO) algorithm to obtain the trained policy network.

[0114] In some embodiments, the feature extraction network includes an encoder and a decoder; the encoder includes a convolutional neural network; and the decoder includes a multilayer perceptron.

[0115] In some embodiments, the training module is specifically configured to: for each time step, input third historical data into the feature extraction network to obtain second feature data; the second feature data includes terrain features and / or behavioral features; the third historical data includes a third preset number of actions prior to the previous action; input the current second action instruction, the previous reward, the second feature data, and the fourth historical data into the policy network to obtain the current second action; the fourth historical data includes a fourth preset number of actions between the previous actions; the fourth preset number is less than the third preset number; input the current first action into the robot's controller to obtain the corresponding current second torque; input the current second torque into... The system is as follows: First, the current first state is obtained by inputting the environment model; second, the current first state is input into the adversarial evaluation network to obtain the current style reward; third, the current reward is determined based on the current task reward and the style reward according to a preset reward function; fourth, the current second action, the current first state, the current style reward, and the current reward are stored in an experience replay buffer; fifth, sample data is sampled from the experience replay buffer; sixth, the policy network is updated based on the objective function of the PPO algorithm according to the sample data; seventh, the adversarial evaluation network is updated based on the preset objective function and the sample data; and finally, the feature extraction network is updated based on the preset loss function and the sample data.

[0116] In some embodiments, the objective function of the PPO algorithm is expressed as:

[0117]

[0118] in, The objective function of PPO is to find the parameter θ that maximizes the objective function. It is an estimate of the advantage function, representing the state w. t Take action a t Advantages relative to the average; w t for It is a new strategy p θ Compared to the old strategy In action a t The probability ratio, i.e. the importance sampling ratio; It is the truncation strategy ratio, where ɛ is a constant, ɛ≤0.2, and the clip function ensures that the importance sampling ratio is in the range [1-ɛ,1+ɛ].

[0119] In some embodiments, the expression of the preset objective function is:

[0120]

[0121] in, Represents both the actual state and the expert reference state; The representative adversarial evaluation network output targets state pairs. The output evaluation value is closer to 1 when it is closer to the expert state, and closer to -1 when it is further away from the expert state. It refers to the difference in distribution divergence between the actual state and the expert reference state; It is the weighting coefficient.

[0122] In some embodiments, the expression for the preset loss function is:

[0123]

[0124] in, These are observational data representing history. This represents the next state after the restructuring. This represents the next state; MSE (mean square error) is the mean square error. p(zt) is the posterior distribution, i.e., the conditional distribution of the latent variable z given the input x; p(zt) is the prior distribution, i.e., the marginal distribution of the latent variable z; KL is the KL divergence, which measures the approximate posterior distribution. The difference from the prior distribution p(zt).

[0125] In some embodiments, the expression of the preset reward function is:

[0126]

[0127] in, As the main reward, As a reward for the task, As a style reward, ω g ω is the weight of the task reward. s Weighting for style rewards.

[0128] The robot motion control device provided in this application embodiment can be used to execute the above-described method embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0129] Figure 6 This is a schematic diagram of the hardware structure of a robot motion control device provided in an embodiment of this application. The device can be a humanoid robot.

[0130] Device 60 may include one or more of the following components: processing component 601, memory 602, power supply component 603, multimedia component 604, audio component 605, input / output (I / O) interface 606, sensor component 607, and communication component 608.

[0131] Processing component 601 typically controls the overall operation of device 60, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 601 may include one or more processors 609 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 601 may include one or more modules to facilitate interaction between processing component 601 and other components. For example, processing component 601 may include a multimedia module to facilitate interaction between multimedia component 604 and processing component 601.

[0132] Memory 602 is configured to store various types of data to support the operation of device 60. Examples of such data include instructions for any application or method operating on device 60, contact data, phonebook data, messages, pictures, videos, etc. Memory 602 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0133] Power supply component 603 provides power to various components of device 60. Power supply component 603 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 60.

[0134] Multimedia component 604 includes a screen that provides an output interface between the device 60 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 604 includes a front-facing camera and / or a rear-facing camera. When the device 60 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0135] Audio component 605 is configured to output and / or input audio signals. For example, audio component 605 includes a microphone (MIC) configured to receive external audio signals when device 60 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 602 or transmitted via communication component 608. In some embodiments, audio component 605 also includes a speaker for outputting audio signals.

[0136] I / O interface 606 provides an interface between processing component 601 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0137] Sensor assembly 607 includes one or more sensors for providing state assessments of various aspects of device 60. For example, sensor assembly 607 can detect the on / off state of device 60, the relative positioning of components such as the display and keypad of device 60, changes in the position of device 60 or a component of device 60, the presence or absence of user contact with device 60, the orientation or acceleration / deceleration of device 60, and temperature changes of device 60. Sensor assembly 607 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 607 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 607 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0138] Communication component 608 is configured to facilitate wired or wireless communication between device 60 and other devices. Device 60 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 608 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 608 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0139] In an exemplary embodiment, device 60 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0140] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 602 including instructions, which can be executed by a processor 609 of device 60 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0141] The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0142] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0143] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0144] This application also provides a computer program product, including a computer program, which, when executed by a processor, implements the robot motion control method executed by the above-mentioned robot motion control device.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A robot motion control method, characterized in that, include: Acquire first historical data, second historical data, and the current first action command; the first historical data includes observation data corresponding to a first preset number of time steps before the current time step, and the second historical data includes observation data corresponding to a second preset number of time steps before the current time step; the second preset number is less than the first preset number. The first historical data is input into a trained feature extraction network to obtain the first feature data; the first feature data includes terrain features and / or behavioral features. The current first action instruction, the first feature data, and the second historical data are input into the trained policy network to obtain the current first action; the trained policy network is obtained by training the policy network according to the reinforcement learning algorithm based on a fusion framework composed of a feature extraction network, a policy network, and an adversarial evaluation network. Based on the current first action, a corresponding current first torque is determined, and the robot is driven to perform the current first action based on the current first torque.

2. The method according to claim 1, characterized in that, The trained feature extraction network is a convolutional neural network; the convolutional neural network includes a first hidden layer and a second hidden layer; the first hidden layer has 42 input channels, 32 output channels, a kernel size of 6, and a stride of 5; the second hidden layer has 32 input channels, 16 output channels, a kernel size of 4, and a stride of 2; the activation function of the convolutional neural network is the ReLU activation function.

3. The method according to claim 1 or 2, characterized in that, The method further includes: A fusion framework is constructed based on feature extraction network, policy network, and adversarial evaluation network; Based on the aforementioned fusion framework, the policy network is trained using the Proximity Policy Optimization (PPO) algorithm to obtain the trained policy network.

4. The method according to claim 3, characterized in that, The feature extraction network includes an encoder and a decoder; the encoder includes a convolutional neural network; and the decoder includes a multilayer perceptron.

5. The method according to claim 3, characterized in that, The step of training the policy network based on the aforementioned fusion framework using the Proximity Policy Optimization (PPO) algorithm to obtain the trained policy network includes: For each time step, the third historical data is input into the feature extraction network to obtain the second feature data; the second feature data includes terrain features and / or behavioral features; the third historical data includes a third preset number of actions prior to the previous action; The current second action instruction, the previous reward, the second feature data, and the fourth historical data are input into the policy network to obtain the current second action; the fourth historical data includes a fourth preset number of actions between the previous actions; the fourth preset number is less than the third preset number. The current first action is input into the robot's controller to obtain the corresponding current second torque; Input the current second torque into the environment model to obtain the current first state; The current first state is input into the adversarial evaluation network to obtain the current style reward; The current reward is determined based on a preset reward function, according to the current task reward and the style reward. Store the current second action, the current first state, the current style reward, and the current reward in the experience replay buffer; Sample data is sampled from the experience replay buffer; the policy network is updated based on the objective function of the PPO algorithm and the sample data. The adversarial evaluation network is updated based on the sample data according to the preset objective function. The feature extraction network is updated based on the sample data according to the preset loss function.

6. The method according to claim 5, characterized in that, The objective function of the PPO algorithm is expressed as follows: in, The objective function of PPO is to find the parameter θ that maximizes the objective function. It is an estimate of the advantage function, representing the state w. t Take action a t Advantages relative to the average; w t for It is a new strategy p θ Compared to the old strategy In action a t The probability ratio, i.e. the importance sampling ratio; It is the truncation strategy ratio, where ɛ is a constant, ɛ≤0.2, and the clip function ensures that the importance sampling ratio is in the range [1-ɛ,1+ɛ].

7. The method according to claim 5, characterized in that, The expression for the preset objective function is: in, Represents both the actual state and the expert reference state; The representative adversarial evaluation network output targets state pairs. The output evaluation value is closer to 1 when it is closer to the expert state, and closer to -1 when it is further away from the expert state. It refers to the difference in distribution divergence between the actual state and the expert reference state; It is the weighting coefficient.

8. The method according to claim 5, characterized in that, The expression for the preset loss function is: in, These are observational data representing history. This represents the next state after the restructuring. MSE represents the next state, which is the mean squared error. It is the posterior distribution, that is, the conditional distribution of the latent variable z given the input x; It is the prior distribution, that is, the marginal distribution of the latent variable z; yes Divergence, a measure of approximate posterior distribution With prior distribution The differences.

9. The method according to claim 5, characterized in that, The expression for the preset reward function is: in, As the main reward, As a reward for the task, As a style reward, As the weight of task rewards, Weighting for style rewards.

10. A robot motion control device, characterized in that, include: At least one processor and memory; The memory stores computer-executed instructions; The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the robot motion control method as described in any one of claims 1 to 9.

Images