A method, apparatus, medium, and program product for training a robot gait generation network

Abstract

The application aims to provide a method, device, medium and program product for training a robot gait generation network, the method comprising: running a gait generation network in a simulation environment to obtain trajectory information, wherein the gait generation network comprises a policy network, a discriminator network and a value network; determining an adversarial loss based on a sampled human motion data pair and the trajectory information, determining a discriminator loss in combination with a discriminator gradient penalty, updating the discriminator network; determining a value loss based on reward information, updating the value network; updating the policy network by a policy gradient algorithm in combination with an objective function based on the reward information and the updated value network; repeating the foregoing steps until a corresponding convergence condition is met. In the training, a discriminator gradient penalty term is introduced to the discriminator network to prevent the reward signal from collapsing or fluctuating sharply, thereby avoiding problems such as AMP training oscillation and mode collapse, improving training stability, and generating more natural, stable and human-like gait behavior.

Description

[0001] This application claims priority to an earlier application (CN202610132224.0, A method, apparatus, medium and program product for training a robot gait generation network, application date 2026-01-29), the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of robot control technology, and in particular to a technique for training robot gait generation networks. Background Technology

[0003] In existing bipedal robot gait control methods, reinforcement learning (RL)-based strategies typically focus on indicators such as speed tracking, balance control, and energy consumption optimization. However, they are insufficient in terms of gait naturalness, resulting in stiff, awkward, and disjointed gait, poor coordination between the upper and lower limbs, disordered gait cycles, poor rhythmicity, and weak resistance to external disturbances.

[0004] To improve gait naturalness, Adversarial Motion Prior (AMP) is used to learn human motion distributions and uses style reward output by the discriminator to guide the policy towards human gait convergence. However, AMP training itself is unstable; discriminator learning often suffers from pattern collapse, weak reward signals, high noise, and lack of interpretability, resulting in generated gaits that still cannot reach the level of natural human gait. Summary of the Invention

[0005] One object of this application is to provide a method, apparatus, medium, and program product for training a robot gait generation network.

[0006] According to one aspect of this application, a method for training a robot gait generation network is provided, the method comprising:

[0007] Step a: Run the gait generation network in the simulation environment to obtain one or more trajectory information of the robot's movement. The gait generation network includes a policy network, a discriminant network and a value network. Each trajectory information includes the robot's state information, observation information, action information and reward information at each time step.

[0008] Step b: Sample human motion data pairs from the reference teaching database, determine the adversarial loss based on the human motion data pairs and the one or more trajectory information, and determine the corresponding discriminative loss by combining the discriminative gradient penalty, and update the discriminative network;

[0009] Step c: Based on the reward information, determine the corresponding value loss and update the value network;

[0010] Step d: Based on the reward information and the updated value network, and in conjunction with the objective function, update the policy network using the policy gradient algorithm;

[0011] Repeat the above steps ad until the corresponding convergence condition is met.

[0012] According to one aspect of this application, a computer device for training a robot gait generation network is provided, comprising a memory, a processor, and a computer program stored in the memory, characterized in that the processor executes the computer program to implement the steps of any of the methods described above.

[0013] According to one aspect of this application, a computer-readable storage medium is provided having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the steps of any of the methods described above.

[0014] According to one aspect of this application, a computer program product is provided, comprising a computer program, characterized in that, when executed by a processor, the computer program implements the steps of any of the methods described above.

[0015] According to one aspect of this application, an apparatus for training a robot gait generation network is provided, the apparatus comprising:

[0016] The module is used to run the gait generation network in a simulation environment to obtain one or more trajectory information of robot movement. The gait generation network includes a policy network, a discriminant network and a value network. Each trajectory information includes the robot's state information, observation information, action information and reward information at each time step.

[0017] The first and second modules are used to sample human motion data pairs from the reference teaching database, determine the adversarial loss based on the human motion data pairs and one or more trajectory information, and determine the corresponding discriminative loss by combining the discriminative gradient penalty, and update the discriminative network.

[0018] The first and third modules are used to determine the corresponding value loss based on the reward information and update the value network.

[0019] The first four modules are used to update the policy network based on the reward information and the updated value network, combined with the objective function, through the policy gradient algorithm.

[0020] Module 15 is used to repeat modules 11 through 14 until the corresponding convergence condition is met.

[0021] Compared with existing technologies, this application runs a gait generation network in a simulation environment to acquire one or more trajectory information of robot motion. The gait generation network includes a policy network, a discriminant network, and a value network. Each trajectory information includes the robot's state information, observation information, action information, and reward information at each time step. Human motion data pairs are sampled from a reference teaching database. Based on the human motion data pairs and the one or more trajectory information, an adversarial loss is determined, and a corresponding discriminant loss is determined by combining the discriminant gradient penalty, updating the discriminant network. Based on the reward information, a corresponding value loss is determined, updating the value network. Based on the reward information and the updated value network, the policy network is updated using a policy gradient algorithm in conjunction with the objective function. The aforementioned steps are repeated until the corresponding convergence condition is met, thus completing the gait generation network training. During training, a discriminant gradient penalty term is introduced into the discriminant network to prevent reward signal collapse or drastic fluctuations, thereby avoiding problems such as AMP training oscillations and pattern collapse, improving training stability, and generating more natural, stable, and human-like gait behavior. Furthermore, to mitigate the high-frequency fluctuations in style rewards, temporal smoothing of style rewards can be applied to prevent the policy network from being misled by high-frequency noise gradients. Additionally, a Lipshitz continuity constraint mechanism can be introduced during the policy network update process to ensure that the policy network output's response to the observed input remains continuous, bounded, and controllable. Attached Figure Description

[0022] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0023] Figure 1 This diagram illustrates a method for training a robot gait generation network according to one embodiment of the present application.

[0024] Figure 2 This diagram illustrates a device structure for training a robot gait generation network according to an embodiment of this application.

[0025] Figure 3 Exemplary systems that can be used to implement the various embodiments described in this application are shown.

[0026] The same or similar reference numerals in the accompanying drawings represent the same or similar parts. Detailed Implementation

[0027] The present application will now be described in further detail with reference to the accompanying drawings.

[0028] In a typical configuration of this application, the terminal, the device of the service network, and the trusted party all include one or more processors (e.g., a central processing unit (CPU)), input / output interfaces, network interfaces, and memory.

[0029] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash memory. Memory is an example of computer-readable media.

[0030] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PCM), programmable random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0031] The devices referred to in this application include, but are not limited to, user equipment, network equipment, or devices composed of user equipment and network equipment integrated through a network. The user equipment includes, but is not limited to, any mobile electronic product capable of human-computer interaction (e.g., via a touchpad), such as smartphones and tablets. These mobile electronic products can use any operating system, such as Android or iOS. The network equipment includes an electronic device capable of automatically performing numerical calculations and information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), and embedded devices. The network equipment includes, but is not limited to, computers, network hosts, single network servers, multiple network server clusters, or clouds composed of multiple servers. Here, a cloud consists of a large number of computers or network servers based on cloud computing, where cloud computing is a type of distributed computing, consisting of a virtual supercomputer composed of a group of loosely coupled computer clusters. The network includes, but is not limited to, the Internet, wide area network, metropolitan area network, local area network, VPN network, wireless ad hoc network, etc. Preferably, the device can also be a program running on the user equipment, network device, or a device formed by integrating user equipment and network device, network device, touch terminal, or network device and touch terminal through a network.

[0032] Of course, those skilled in the art should understand that the above-described devices are merely examples, and other existing or future devices that are applicable to this application should also be included within the scope of protection of this application, and are hereby incorporated by reference.

[0033] In the description of this application, "multiple" means two or more, unless otherwise expressly and specifically defined.

[0034] Figure 1The diagram illustrates a method for training a robot gait generation network according to an embodiment of this application, the method comprising steps S11, S12, S13, S14 and S15. In step S11, device 1 runs a gait generation network in a simulation environment to acquire one or more trajectory information of the robot's motion. The gait generation network includes a policy network, a discriminant network, and a value network. Each trajectory information includes the robot's state information, observation information, action information, and reward information at each time step. In step S12, device 1 samples human motion data pairs from a reference teaching database. Based on the human motion data pairs and the one or more trajectory information, it determines an adversarial loss and, combined with a discriminant gradient penalty, determines a corresponding discriminant loss, updating the discriminant network. In step S13, device 1 determines a corresponding value loss based on the reward information and updates the value network. In step S14, device 1 updates the policy network based on the reward information and the updated value network, combined with an objective function, using a policy gradient algorithm. In step S15, device 1 repeats steps S11-S14 until the corresponding convergence condition is met.

[0035] In some embodiments, device 1 may be a computer, server, or workstation used for model inference.

[0036] In step S11, device 1 runs a gait generation network in a simulation environment to obtain one or more trajectory information of the robot's movement. The gait generation network includes a policy network, a discrimination network, and a value network. Each trajectory information includes the robot's state information, observation information, action information, and reward information at each time step.

[0037] In some embodiments, the simulation environment can be a simulation platform such as Isaac Lab, Isaac Gym, or MuJoCo. Tests are conducted on different terrains generated by the platform. Different initial state information (e.g., different terrains, different initial robot postures, etc.) can be set using the simulation platform to run the gait generation network, obtaining a preset number of trajectory information. This collected trajectory information can then be used to update and optimize the gait generation network for the current round.

[0038] In some embodiments, the gait generation network includes a policy network, a discriminant network, and a value network. The policy network generates corresponding action information based on the robot's observation information to drive robot movement. The policy network typically consists of multiple fully connected layers or multiple fully connected layers combined with a recurrent neural network or a self-attention layer. The discriminant network outputs a style score, thereby providing style rewards to the policy network and guiding it to learn natural human movements. The discriminant network typically consists of multiple fully connected layers. The value network outputs a state value, thereby evaluating the long-term value of state information. The value network typically consists of multiple fully connected layers. Those skilled in the art should understand that the network architecture of the policy network, discriminant network, and value network described above is merely an example. This embodiment does not limit the network architecture, internal modules, hierarchy, or connection relationships of the policy network, discriminant network, and value network. Any existing or future AMP-based gait generation network can be used based on actual needs.

[0039] In some embodiments, the state information s corresponding to the corresponding time step t t This provides a complete state description of the simulation environment at time step t. The state information typically includes robot body information and environmental information. For example, robot body information includes, but is not limited to, the angles, angular velocities, and angular accelerations of each joint, and the global position, attitude, and velocity of the torso (center of mass). Environmental information includes, but is not limited to, ground geometry, friction coefficient, and information on external disturbances (such as wind speed and direction). The observation information o... t This refers to the information that the policy network can acquire at time step t. This includes the state information s. t A subset of. The observation information includes, but is not limited to, robot-related information acquired through sensors (e.g., angle, angular velocity, torso relative posture, angular velocity, linear acceleration, etc.). The motion information a t This includes, but is not limited to, the angles, angular velocities, and torques corresponding to each drive joint of the robot. The reward information is an evaluation signal fed back to the policy network. Those skilled in the art should understand that the above trajectory information is merely an example, and other existing or future information applicable to this application should also be included within the scope of protection of this application, and is hereby incorporated by reference.

[0040] In some embodiments, step S11 includes: step S111 (not shown), in a simulation environment, for each time step, based on the observation information corresponding to that time step, the action information corresponding to that time step is determined by the policy network, wherein the observation information is a subset of the state information corresponding to that time step; step S112 (not shown), based on the action information, the state information corresponding to the next time step is determined; step S113 (not shown), after determining the state information, observation information and action information corresponding to each time step in each trajectory information, for each time step, based on the motion segment corresponding to that time step, the reward information corresponding to that time step is determined by the discriminant network, wherein the motion segment includes the observation information corresponding to that time step and the next time step.

[0041] For example, for each time step, the corresponding observation information is obtained from the current simulation environment. t This observation information o t It is the status information s t A subset of. State information s t This provides a complete state description of the simulation environment at time step t. This is achieved through the policy network Π(a t |o t Output the corresponding action information a t Subsequently, based on this action information a t and the current time step's state information s t Determine the theoretical state information for the next time step. To adapt the policy network to the uncertainties of the real world, a Gaussian perturbation can be added to the previously determined theoretical state information to obtain the state information s for the next time step. t+1 This makes the trained policy network more robust to real-world perturbations. After collecting a preset amount of state, observation, and action information by running the policy network, it can be used to analyze the motion segments (i.e., continuous observation information) corresponding to each time step t. ), combined with the discriminant network to determine the corresponding reward information r t The reward information includes style rewards and task rewards. Style rewards are used to guide the policy network to learn human-like movement styles. Task rewards are used to constrain the policy network to achieve task objectives.

[0042] In some embodiments, step S113 includes: after determining the state information, observation information and action information corresponding to each time step in each trajectory information, for each time step, based on the motion segment corresponding to that time step, determining the corresponding style reward through the discriminative network; and based on the style reward, combined with the task reward, determining the reward information.

[0043] In some embodiments, based on the motion segment corresponding to each time step t The score corresponding to the motion segment is determined through the discriminant network D. Based on the score corresponding to the motion segment, and in conjunction with the corresponding style reward function, the style reward for that time step t is determined. For example, the style reward function can be designed as follows:

[0044]

[0045] Those skilled in the art should understand that the above-described style reward function is merely an example, and this embodiment does not limit it. Other existing or future style reward functions that are applicable to this application should also be included within the scope of protection of this application, and are hereby incorporated by reference.

[0046] In some embodiments, the task rewards The task reward is typically determined by a set quantifiable metric. The task reward usually includes one or more sub-reward items, which are weighted and summed to determine the corresponding task reward. In some embodiments, the task reward typically includes at least one of the following: velocity tracking reward; angular velocity tracking reward; rotational tracking reward. The velocity tracking reward is used to ensure travel efficiency, rewarding the robot for moving at a target speed. The angular velocity tracking reward is used to control the angular velocity of the torso rotation, suppressing unnecessary rotation and maintaining smooth travel. The rotational tracking reward is used to track the torso posture, keeping the body upright. In some embodiments, each of the above sub-reward items can be calculated using the Φ(x) function: Where k is the scaling factor corresponding to the corresponding sub-reward item, and x is the error value corresponding to the corresponding tracked object (for example, for speed tracking reward, it is the difference between the actual speed and the target speed). In some embodiments, in addition to the above sub-reward items, the task reward can also be calculated by combining sub-reward items such as energy efficiency reward and smoothness reward. Furthermore, the reward information corresponding to time step t can be determined based on the style reward and the task reward. Where ω is a set weight used to balance rewards. Those skilled in the art should understand that the above-described sub-reward items and corresponding calculation methods for calculating task rewards are merely examples, and this embodiment does not limit them. Other existing or future sub-reward items or calculation methods that are applicable to this application should also be included within the scope of protection of this application, and are hereby incorporated by reference.

[0047] In some embodiments, to avoid style rewards being misled by high-frequency noise, determining the corresponding style reward for each time step based on the motion segment corresponding to that time step through the discriminant network includes: for each time step, determining an initial style reward based on the motion segment corresponding to that time step through the discriminant network; and performing temporal smoothing processing on the initial style reward to obtain the corresponding style reward. That is, for the initial style reward calculated based on the aforementioned steps... In this embodiment, the smoothed style reward determined in the previous time step is also incorporated. Perform time-series smoothing and determine the smoothed style reward. Where α is the smoothing coefficient (0 < α < 1), used to control the relative weight of the initial style reward and the historical style reward at the current time step.

[0048] After collecting a preset amount of trajectory information data, including state information, observation information, action information, and reward information, the discriminant network, value network, and policy network can be updated based on this batch of collected data. The update order of the discriminant network and value network is not limited here. Preferably, the discriminant network and value network can be updated simultaneously.

[0049] In step S12, device 1 samples human motion data pairs from the reference teaching database, determines adversarial loss based on the human motion data pairs and one or more trajectory information, and determines the corresponding discriminative loss by combining the discriminative gradient penalty, and updates the discriminative network.

[0050] In some embodiments, the training method of this application is based on AMP. Within this framework, the discriminant network D evaluates the similarity between state transitions acquired from the agent and state transitions acquired from a reference teaching database. Human motion data pairs corresponding to each time step in each trajectory information can be randomly sampled from the reference teaching database. The reference teaching database is a pre-established real human motion database. Each time step's corresponding human motion data pair includes the human motion observation information for that time step sampled from the database and the next time step. Based on the human motion data pairs and the one or more trajectory information, the corresponding teaching data discriminant loss L can be calculated using the discriminant network. e Discrimination loss L based on strategy data p This leads to the determination of the adversarial loss L. AMP =0.5L e +0.5L p Furthermore, to avoid severe fluctuations in the reward signal output by the discrimination network and the occurrence of mode collapse, a discrimination gradient penalty term L can be incorporated. GP Determine the corresponding discriminant loss L D =L AMP +λGP L GP , where λ GP The weight coefficients for the discriminative gradient penalty are set. The discriminative gradient penalty term is determined based on the gradient of the discriminative network output relative to the input. The discriminative gradient penalty is used to smooth the decision boundary of the discriminative network, preventing it from becoming overly sensitive to small changes in the input data, thereby avoiding high-frequency and drastic fluctuations in style rewards. Based on the calculated discriminative loss, the parameters of the discriminative network are updated using the backpropagation algorithm and the corresponding optimizer (such as Adam, AdamW, RMSprop, etc.).

[0051] In some embodiments, determining the adversarial loss based on the human motion data pair and the one or more trajectory information includes: determining a corresponding teaching data discrimination loss based on the human motion data pair, through the discriminant network and in combination with a corresponding teaching data discrimination loss function; determining a corresponding policy data discrimination loss based on the one or more trajectory information, through the discriminant network and in combination with a corresponding policy data discrimination loss function; and determining the adversarial loss based on the teaching data discrimination loss and the policy data discrimination loss.

[0052] For example, for human motion data pairs sampled from a reference teaching database, a discriminant network can output a corresponding human motion score. Then, based on this score and the corresponding teaching data discriminant loss function, the corresponding teaching data discriminant loss can be calculated. The teaching data discriminant loss function can be designed as follows:

[0053]

[0054] in, This indicates that the data pair was obtained by sampling from the reference teaching database (Demonstrations, D) (i.e., human motion data pairs). Let E[] be the score corresponding to the human motion data pair, representing the average score error calculated for each time step of all trajectory information. Accordingly, for one or more trajectory information sets, continuous motion segments can be sampled from them. Then, the discriminant network outputs a corresponding policy motion score, and based on this score, combined with the corresponding policy data discriminant loss function, the corresponding policy data discriminant loss is calculated. The policy data discriminant loss can be designed as follows:

[0055]

[0056] in, This indicates that the data was sampled from the policy network Π. Let E[] be the score corresponding to the motion segment, and E[] represent the average score error calculated for each time step of the motion segment in all trajectory information. Then, the two losses are combined to determine the adversarial loss.

[0057] In some embodiments, the method further includes: step S16 (not shown), determining the corresponding discrimination gradient based on the human motion data pair using the discrimination network, and calculating the corresponding discrimination gradient penalty. For example, considering the stability of the reference teaching data, this embodiment uses human motion data pairs sampled from it for gradient calculation, and then calculates the discrimination gradient penalty term. Specifically, the discrimination gradient penalty is:

[0058]

[0059] In step S13, device 1 determines the corresponding value loss based on the reward information and updates the value network. For example, it can be based on the state information pairs (s) sampled from the one or more trajectory information corresponding to each time step t. t s t+1 ), and determine the corresponding predicted value V(s) through the value network V. t ), V(s) t+1 Then, combining the reward information corresponding to time step t, the corresponding value loss is determined based on the corresponding value loss function. The value loss function can be designed as follows:

[0060]

[0061] Based on the calculated value loss, the parameters of the value network are updated using the backpropagation algorithm and corresponding optimizers (such as Adam, AdamW, RMSprop, etc.).

[0062] Those skilled in the art should understand that the above-described loss calculation is merely an example, and this embodiment does not limit it. Other existing or future loss calculation methods that are applicable to this application should also be included within the scope of protection of this application, and are hereby incorporated by reference.

[0063] In step S14, device 1 updates the policy network based on the reward information and the updated value network, combined with the objective function, using a policy gradient algorithm. In some embodiments, the objective of updating the policy network is to maximize the expected cumulative reward. Typically, the objective function can be designed as follows:

[0064]

[0065] in, In policy networks Lower trajectory information The likelihood probability, where T is the time span (time range / time domain length). The discount factor set (0 < <1), This is for rewarding information. That is, the policy network maximizes... Training is then performed. During training, optimizing the objective function requires calculating the policy gradient. This can be achieved by combining appropriate gradient policy algorithms (e.g., Proximal Policy Optimization (PPO) algorithms) with the updated value network. Those skilled in the art should understand that the above-described gradient policy algorithms are merely examples, and this embodiment does not limit the scope of the application. Other existing or future gradient policy algorithms that are applicable to this application should also be included within the scope of protection of this application, and are hereby incorporated by reference.

[0066] In some embodiments, step S14 includes: updating the policy network using a policy gradient algorithm based on the reward information and the updated value network, in conjunction with the objective function, under the applied Lipschitz constraint. By introducing the Lipschitz constraint into the policy network update, the policy network output's response to the input remains continuous, bounded, and controllable. Typically, a gradient constraint is added to the aforementioned objective function to construct a constrained policy optimization problem that enforces Lipschitz continuity for optimization, i.e.:

[0067]

[0068] in, This represents a state-action pair (o) consisting of observation information and action information sampled from the policy network Π. t a t A dataset consisting of ) This represents the logarithmic probability of choosing action information a given observation information o; This represents the partial derivative with respect to the observed information o; K is the Lipschitz constant, which is an upper bound on the first-order derivative constraint. In some embodiments, to facilitate optimization using gradient-based methods, a Lagrange multiplier λ can be introduced to express the aforementioned constrained policy optimization problem as a form with a penalty term, that is, the policy network is trained by maximizing the following objective:

[0069] .

[0070] Similar to the aforementioned update process, the above formula can be combined with the corresponding gradient policy algorithm (e.g., Proximal Policy Optimization (PPO) algorithm) to optimize the policy network.

[0071] In step S15, device 1 repeats steps S11-S14 until the corresponding convergence condition is met. Steps S11-S14 illustrate one round of training in the gait generation network. After each round of training or a preset number of rounds of training, it is determined whether the corresponding convergence condition is met. If not, the next round of training continues based on the gait generation network updated in that round; otherwise, the training of the gait generation network is completed.

[0072] In some embodiments, the convergence condition includes at least one of the following:

[0073] The preset update rounds are reached. The preset update rounds can be arbitrarily set by the user based on the actual training scenario (e.g., model complexity, task complexity, computing resources, etc.), and this embodiment does not limit this.

[0074] The fluctuation range of the average total reward in consecutive rounds of updates is within a preset fluctuation range. For example, in consecutive rounds of training (such as 30 rounds, 50 rounds, 100 rounds, etc., this embodiment does not limit this), if the average total reward corresponding to the policy network no longer increases or decreases significantly, but fluctuates within a small range, and the range of its fluctuation is within the preset fluctuation range, then the current training can be considered to meet the convergence condition.

[0075] In the simulation environment, the robot's motion controlled by the gait generation network meets preset conditions. For example, these preset conditions include, but are not limited to, the robot's motion speed being within a expected speed range, the torso pitch angle being within a expected angle range, the energy consumption rate being less than a corresponding threshold, and the gait style conforming to human aesthetics.

[0076] Those skilled in the art should understand that the above convergence conditions are merely examples and this embodiment does not limit them. Other existing or future convergence conditions that may be applicable to this application should also be included within the scope of protection of this application and are hereby incorporated by reference.

[0077] In some embodiments, the method further includes: step S17 (not shown), determining current observation information based on sensor data of the target robot; and determining current action information based on the current observation information using the trained policy network, wherein the current action information is used to drive the target robot to move.

[0078] For example, the gait generation network trained through steps S11-S15 can be integrated into the main control program of the target robot. This allows the network to read current sensor data from various sensors on the target robot and determine the current observation information. Inputting this observation information into the policy network yields its output current action information. This action information can then be sent to the motor servo controller to drive the robot's joints and complete the robot's motion task.

[0079] Figure 2 This diagram illustrates a device structure for training a robot gait generation network according to an embodiment of this application. The device 1 includes a first module 11, a second module 12, a third module 13, a fourth module 14, and a fifth module 15. The first module 11 runs the gait generation network in a simulation environment to acquire one or more trajectory information of the robot's motion. The gait generation network includes a policy network, a discriminant network, and a value network. Each trajectory information includes the robot's state information, observation information, action information, and reward information at each time step. The second module 12 samples human motion data pairs from a reference teaching database. Based on the human motion data pairs and the one or more trajectory information, it determines an adversarial loss and, combined with a discriminant gradient penalty, determines a corresponding discriminant loss, updating the discriminant network. The third module 13 determines a corresponding value loss based on the reward information and updates the value network. The fourth module 14, based on the reward information and the updated value network, updates the policy network using a policy gradient algorithm in conjunction with an objective function. The fifth module 15 repeats the above steps for modules 11 to 14 until the corresponding convergence condition is met. Here, the Figure 2 The specific implementation methods corresponding to modules 11, 12, 13, 14, and 15 shown are the same as or similar to the specific embodiments of steps S11, S12, S13, S14, and S15 mentioned above, and therefore will not be repeated here. They are included here by reference.

[0080] In some embodiments, the module 11 includes a unit 111 (not shown), a second unit 112 (not shown), and a third unit 113 (not shown). In the simulation environment, the first unit 111, for each time step, determines the action information corresponding to that time step based on the observation information corresponding to that time step through the policy network, wherein the observation information is a subset of the state information corresponding to that time step; the second unit 112 determines the state information corresponding to the next time step based on the action information; after determining the state information, observation information, and action information corresponding to each time step in each trajectory information, the third unit 113, for each time step, determines the reward information corresponding to that time step based on the motion segment corresponding to that time step through the discriminant network, wherein the motion segment includes the observation information corresponding to that time step and the next time step. Here, the specific implementations of Unit 111, Unit 112 and Unit 113 are the same as or similar to the specific embodiments of steps S111, S112 and S113 mentioned above, and will not be repeated here, but are included by reference.

[0081] In some embodiments, the device 1 further includes a six-module 16 (not shown). The six-module 16, based on the human motion data pairs, determines the corresponding discrimination gradient through the discrimination network and calculates the corresponding discrimination gradient penalty. Here, the specific implementation of the six-module 16 is the same as or similar to the specific embodiment of step S16 described above, and therefore will not be repeated here, but is incorporated herein by reference.

[0082] In some embodiments, the device 1 further includes a seven-module 17 (not shown). The seven-module 17 determines current observation information based on sensor data from the target robot; based on the current observation information, it determines current action information using the trained policy network, wherein the current action information is used to drive the target robot's movement. Here, the specific implementation of the seven-module 17 is the same as or similar to the specific embodiment of step S17 described above, and therefore will not be repeated here, but is incorporated herein by reference.

[0083] Figure 3 Exemplary systems that can be used to implement the various embodiments described in this application are shown; such as Figure 3 As shown in some embodiments, system 300 can function as any of the devices described in each of the embodiments. In some embodiments, system 300 may include one or more computer-readable media having instructions (e.g., system memory or NVM / storage device 320) and one or more processors (e.g., one or more processors 305) coupled to the one or more computer-readable media and configured to execute the instructions to implement the module and thus perform the actions described in this application.

[0084] In one embodiment, the system control module 310 may include any suitable interface controller to provide any suitable interface to at least one of the processors 305 and / or any suitable device or component communicating with the system control module 310.

[0085] The system control module 310 may include a memory controller module 330 to provide an interface to the system memory 315. The memory controller module 330 may be a hardware module, a software module, and / or a firmware module.

[0086] System memory 315 can be used, for example, to load and store data and / or instructions for system 300. In one embodiment, system memory 315 may include any suitable volatile memory, such as suitable DRAM. In some embodiments, system memory 315 may include double data rate type quad synchronous dynamic random access memory (DDR4 SDRAM).

[0087] In one embodiment, the system control module 310 may include one or more input / output (I / O) controllers to provide interfaces to the NVM / storage device 320 and (one or more) communication interfaces 325.

[0088] For example, NVM / storage device 320 may be used to store data and / or instructions. NVM / storage device 320 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable (one or more) non-volatile storage devices (e.g., one or more hard disk drive (HDD), one or more optical disc (CD) drives, and / or one or more digital universal optical disc (DVD) drives).

[0089] NVM / storage device 320 may include storage resources that are physically part of a device on which system 300 is mounted, or that can be accessed by the device without necessarily being part of it. For example, NVM / storage device 320 may be accessed via a network through one or more communication interfaces 325.

[0090] One or more communication interfaces 325 may provide the system 300 with an interface to communicate over one or more networks and / or with any other suitable device. The system 300 may wirelessly communicate with one or more components of a wireless network in accordance with any of one or more wireless network standards and / or protocols.

[0091] In one embodiment, at least one of the processors 305 may be logically packaged with one or more controllers of the system control module 310 (e.g., memory controller module 330). In one embodiment, at least one of the processors 305 may be logically packaged with one or more controllers of the system control module 310 to form a system-in-package (SiP). In one embodiment, at least one of the processors 305 may be integrated with the logic of one or more controllers of the system control module 310 on the same die. In one embodiment, at least one of the processors 305 may be integrated with the logic of one or more controllers of the system control module 310 on the same die to form a system-on-a-chip (SoC).

[0092] In various embodiments, system 300 may be, but is not limited to, a server, workstation, desktop computing device, or mobile computing device (e.g., laptop computing device, handheld computing device, tablet computer, netbook, etc.). In various embodiments, system 300 may have more or fewer components and / or different architectures. For example, in some embodiments, system 300 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touchscreen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0093] In addition to the methods and devices described in the above embodiments, this application also provides a computer-readable storage medium storing computer code that, when executed, performs the method described in any of the preceding embodiments.

[0094] This application also provides a computer program product that, when executed by a computer device, performs the method described in any of the preceding claims.

[0095] This application also provides a computer device, the computer device comprising:

[0096] One or more processors;

[0097] Memory, used to store one or more computer programs;

[0098] When the one or more computer programs are executed by the one or more processors, the one or more processors cause the one or more processors to perform the method as described in any of the preceding methods.

[0099] It should be noted that this application can be implemented in software and / or a combination of software and hardware, for example, using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In one embodiment, the software program of this application can be executed by a processor to implement the steps or functions described above. Similarly, the software program of this application (including related data structures) can be stored in a computer-readable recording medium, such as RAM memory, a magnetic or optical drive, a floppy disk, or similar devices. Furthermore, some steps or functions of this application can be implemented in hardware, for example, as circuitry that cooperates with a processor to perform the various steps or functions.

[0100] Furthermore, a portion of this application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0101] Communication media include media through which communication signals containing, for example, computer-readable instructions, data structures, program modules, or other data are transmitted from one system to another. Communication media can include guided transmission media (such as cables and wires (e.g., optical fibers, coaxial cables, etc.)) and wireless (unguided transmission) media capable of propagating energy waves, such as sound, electromagnetic, RF, microwave, and infrared. Computer-readable instructions, data structures, program modules, or other data can be embodied as modulated data signals in, for example, wireless media (such as carrier waves or similar mechanisms embodied as part of spread spectrum technology). The term "modulated data signal" refers to a signal whose one or more characteristics are altered or set in a manner that encodes information in the signal. Modulation can be analog, digital, or a hybrid modulation technique.

[0102] By way of example and not limitation, computer-readable storage media may include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. For example, computer-readable storage media include, but are not limited to, volatile memories such as random access memory (RAM, DRAM, SRAM); and non-volatile memories such as flash memory, various read-only memories (ROM, PROM, EPROM, EEPROM), magnetic and ferromagnetic / ferroelectric memories (MRAM, FeRAM); and magnetic and optical storage devices (hard disks, magnetic tapes, CDs, DVDs); or other media now known or hereafter developed capable of storing computer-readable information / data for use by a computer system.

[0103] Herein, one embodiment of this application includes an apparatus comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the apparatus is triggered to run a method and / or technical solution based on the foregoing embodiments of this application.

[0104] All data collected and processed in this application have been with the user's consent or permission, and have strictly complied with legal regulations, social ethics, and the public interest. This data processing includes, but is not limited to, tag management, rule setting, and recommendation decisions. The legal regulations mentioned include, but are not limited to: 1) relevant laws and regulations of various countries or organizations regarding the protection of personal information; 2) relevant laws and regulations of various countries or organizations regarding the protection of user information; and 3) relevant laws and regulations of various countries or organizations regarding the use of personal or user information by organizations.

[0105] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within this application. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in the apparatus claims may also be implemented by a single unit or device in software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any particular order.

Claims

1. A method for training a robot gait generation network, characterized in that, The method includes: Step a: Run the gait generation network in the simulation environment to obtain one or more trajectory information of the robot's movement. The gait generation network includes a policy network, a discriminant network and a value network. Each trajectory information includes the robot's state information, observation information, action information and reward information at each time step. Step b: Sample human motion data pairs from the reference teaching database, determine the adversarial loss based on the human motion data pairs and the one or more trajectory information, and determine the corresponding discriminative loss by combining the discriminative gradient penalty, and update the discriminative network; Step c: Based on the reward information, determine the corresponding value loss and update the value network; Step d: Based on the reward information and the updated value network, and in conjunction with the objective function, update the policy network using the policy gradient algorithm; Repeat the above steps ad until the corresponding convergence condition is met.

2. The method according to claim 1, characterized in that, The gait generation network is run in a simulation environment to acquire one or more trajectory information of the robot's motion. The gait generation network includes a policy network, a discriminant network, and a value network. Each trajectory information includes the robot's state information, observation information, action information, and reward information at each time step. In the simulation environment, for each time step, based on the observation information corresponding to that time step, the action information corresponding to that time step is determined through the policy network, wherein the observation information is a subset of the state information corresponding to that time step; Based on the action information, determine the state information corresponding to the next time step; After determining the state information, observation information, and action information corresponding to each time step in each trajectory information, for each time step, based on the motion segment corresponding to that time step, the reward information corresponding to that time step is determined through the discrimination network, wherein the motion segment includes the observation information corresponding to that time step and the next time step.

3. The method according to claim 2, characterized in that, After determining the state information, observation information, and action information corresponding to each time step in each trajectory information, for each time step, based on the motion segment corresponding to that time step, the reward information corresponding to that time step is determined through the discriminant network. The motion segment includes the observation information corresponding to that time step and the next time step, including: After determining the state information, observation information, and action information corresponding to each time step in each trajectory information, for each time step, based on the motion segment corresponding to that time step, the corresponding style reward is determined through the discrimination network. Based on the style reward and combined with the task reward, the reward information is determined.

4. The method according to claim 3, characterized in that, For each time step, based on the motion segment corresponding to that time step, the corresponding style reward is determined through the discriminant network, including: For each time step, based on the motion segment corresponding to that time step, the initial style reward is determined through the discrimination network; The initial style reward is subjected to temporal smoothing to obtain the corresponding style reward.

5. The method according to claim 3 or 4, characterized in that, The task reward includes at least one of the following: Speed ​​tracking rewards; Angular velocity tracking reward; Rotation tracking reward.

6. The method according to any one of claims 1 to 5, characterized in that, The determination of adversarial loss based on the human motion data pairs and the one or more trajectory information includes: Based on the human motion data pairs, the corresponding teaching data discrimination loss is determined through the discrimination network and in conjunction with the corresponding teaching data discrimination loss function. Based on the one or more trajectory information, the corresponding policy data discrimination loss is determined through the discrimination network and in combination with the corresponding policy data discrimination loss function; The adversarial loss is determined based on the teaching data discrimination loss and the policy data discrimination loss.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: Based on the human motion data pairs, the corresponding discrimination gradient is determined through the discrimination network, and the corresponding discrimination gradient penalty is calculated.

8. The method according to any one of claims 1 to 7, characterized in that, The step of updating the policy network based on the reward information and the updated value network, combined with the objective function, using the policy gradient algorithm includes: Under the applied Lipshitz constraint, the policy network is updated using the policy gradient algorithm based on the reward information and the updated value network, combined with the objective function.

9. The method according to any one of claims 1 to 8, characterized in that, The convergence condition includes at least one of the following: Reach the preset update cycle; The average total reward fluctuates within a preset fluctuation range across multiple consecutive updates; In the simulation environment, the movement of the robot controlled by the gait generation network meets preset conditions.

10. The method according to any one of claims 1 to 9, characterized in that, The method further includes: Based on the sensor data of the target robot, determine the current observation information; Based on the current observation information, the trained policy network is used to determine the current action information, which is used to drive the target robot to move.

11. A computer device for training a robot gait generation network, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method as described in any one of claims 1 to 10.

12. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method as described in any one of claims 1 to 10.

13. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method as described in any one of claims 1 to 10.

Images