A data enhancement method based on humanoid robot reinforcement learning

Abstract

The application discloses a data enhancement method based on humanoid robot reinforcement learning, comprising the following steps: step S1: parallel training environment building; step S2: parallel training; step S3: training data acquisition; step S4: data enhancement; step S5: training task loss calculation; and step S6: strategy updating. The application proposes a data enhancement method based on humanoid robot reinforcement learning, which can significantly improve the speed of reinforcement learning training, reduce the training data required by the robot in reinforcement learning by using the data enhancement method, enhance the left-right symmetry of the robot behavior, improve the aesthetic of the action, and the proposed data enhancement method can also be applied to other robot learning methods such as imitation learning.

Description

Technical Field

[0001] This invention belongs to the field of robot control, specifically relating to a data augmentation method based on reinforcement learning for humanoid robots. Background Technology

[0002] Motion control of humanoid robots is one of the core challenges in robotics research. Due to the highly complex multi-degree-of-freedom dynamics of humanoid robots, the optimization of their motion strategies typically relies on reinforcement learning. Reinforcement learning, through a trial-and-error mechanism, enables robots to autonomously learn motion strategies in a simulation environment and ultimately transfer them to real robots for execution. However, in actual training, reinforcement learning faces problems such as high training costs, difficulty in designing reward functions, and insufficient policy generalization ability. Reinforcement learning usually requires massive amounts of simulation data to train stable policies, resulting in long training cycles and high computational resource consumption. Furthermore, designing overly stringent reward functions in reinforcement learning can lead to policies getting trapped in local optima, making it difficult to adapt to the real mechanical characteristics of robots; while overly lenient reward functions can lead to slow policy convergence or even failure to achieve the expected results. Simultaneously, when dealing with highly symmetrical tasks (such as walking and running), the left-right symmetry of the policy is difficult to guarantee, affecting the aesthetics and stability of the motion.

[0003] To address the aforementioned problems in reinforcement learning training of humanoid robots, such as low data efficiency, high training cost, and insufficient policy generalization ability, this invention proposes a data augmentation method for reinforcement learning of humanoid robots. This method enhances the left-right symmetry of robot behavior by augmenting data during the training phase. At the same time, this method can reduce the total amount of training data required, thereby increasing the training speed. Summary of the Invention

[0004] The technical problem to be solved by this invention is: to address the problems of large data requirements, long training time, and low policy generalization performance in reinforcement learning tasks for humanoid robots, by using data augmentation to enhance the left-right symmetry of robot behavior generated by reinforcement learning, reduce the total data requirements in the training phase, and improve the training speed.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a data augmentation method based on reinforcement learning of humanoid robots, comprising the following steps:

[0006] Step S1: Setting up the parallel training environment;

[0007] This involves building a simulation environment for robot training, including but not limited to defining the physical properties of the environment (such as gravity, friction, and ground type), the robot's initial state (such as position, posture, and joint angles), and defining the core layers required for parallel training, such as physical simulation, policy training, and data pipelines. It also includes defining the robot's learning methods and policy networks, such as reinforcement learning, imitation learning, and their network structures and parameters. Furthermore, it defines the task objectives, such as walking, running, and balance control. Finally, it defines the task's reward function, such as a function related to multiple factors including the robot's balance state, walking speed, and stride symmetry. Finally, it initializes all parameters required for the training process, such as the learning rate, discount factor, and momentum parameters of the training network.

[0008] Step S2: Parallel training;

[0009] The probability distribution of each action a occurring in state s at each time step is calculated using the policy network. At each time step, an action a is sampled according to the probability distribution, and the reward r for that action is calculated. Each reward in the reward list is traversed from back to front, and the cumulative reward R is calculated.

[0010] Step S3: Acquiring training data;

[0011] In the simulation environment, the robot continuously interacts with the environment and collects data from the interaction process. This data includes the robot's action tensor A, the observed environment state tensor S, and the obtained reward tensor R. This data provides practical experience for training and is used for subsequent policy optimization.

[0012] Step S4: Data Augmentation;

[0013] In reinforcement learning, the task of robot-environment interaction can be based on the MDP (Markov Decision Process) model. get:

[0014] ;

[0015] Where S, A, T, and R represent the system state, action, state transition equation, and reward, respectively.

[0016] Specifically, after the robot acquires an interactive scenario, the collected (state-action) data can be expanded:

[0017] ;

[0018] For s and a, signals can be categorized into directly symmetric signals, indirectly symmetric signals, and asymmetric signals. In directly symmetric signals, data is augmented through interchange; in indirectly symmetric signals, data is augmented through inversion; and in asymmetric signals, the original data is retained. In the context of reinforcement learning training for humanoid robots, symmetric state and motion signals include centroid information (e.g., centroid linear velocity, centroid angular velocity, body posture); joint information (e.g., joint angles, joint angular velocities); motion information (e.g., joint torques, accelerations, and velocities); and other potentially included signals.

[0019] Step S5: Calculate the training task loss;

[0020] Based on the collected and augmented data, a loss function for the training task is calculated to reflect the gap between the current policy and the target policy. Simultaneously, to ensure policy symmetry, a symmetry loss is introduced to penalize asymmetrical behaviors, thereby ensuring the consistency of movement between the left and right joints. Specifically, the loss function for the training task can be expressed as:

[0021] ;

[0022] Where T is the total number of time steps, These are the parameters of the policy network;

[0023] Step S6: Policy Update;

[0024] Based on the calculated loss value, the policy parameters are adjusted using optimization algorithms (such as SGD, Adam, etc.). By iterating through steps S2 to S6, the strategy is gradually optimized, ultimately achieving the goal of efficiently completing the task.

[0025] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0026] 1. This invention proposes a data augmentation method for reinforcement learning of humanoid robots, which can significantly improve the speed of reinforcement learning training.

[0027] 2. The data augmentation method proposed in this invention can reduce the amount of training data required for robots in reinforcement learning.

[0028] 3. The data augmentation method for reinforcement learning proposed in this invention can enhance the left-right symmetry of robot behavior and improve the aesthetics of its movements.

[0029] 4. The data augmentation method proposed in this invention can also be applied to other robot learning methods such as imitation learning. Attached Figure Description

[0030] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0031] Figure 1 Flowchart for data augmentation during the learning and training phase. Detailed Implementation

[0032] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] Example 1: The following describes a data augmentation method based on reinforcement learning for humanoid robots, comprising the following steps:

[0034] Step S1: Setting up the parallel training environment;

[0035] This involves building a simulation environment for robot training, including but not limited to defining the physical properties of the environment (such as gravity, friction, and ground type), the robot's initial state (such as position, posture, and joint angles), and defining the core layers required for parallel training, such as physical simulation, policy training, and data pipelines. It also includes defining the robot's learning methods and policy networks, such as reinforcement learning, imitation learning, and their network structures and parameters. Furthermore, it defines the task objectives, such as walking, running, and balance control. Finally, it defines the task's reward function, such as a function related to multiple factors including the robot's balance state, walking speed, and stride symmetry. Finally, it initializes all parameters required for the training process, such as the learning rate, discount factor, and momentum parameters of the training network.

[0036] Step S2: Parallel training;

[0037] The probability distribution of each action a occurring in state s at each time step is calculated using the policy network. At each time step, an action a is sampled according to the probability distribution, and the reward r for that action is calculated. Each reward in the reward list is traversed from back to front, and the cumulative reward R is calculated.

[0038] Step S3: Acquiring training data;

[0039] In the simulation environment, the robot continuously interacts with the environment and collects data from the interaction process. This data includes the robot's action tensor A, the observed environment state tensor S, and the obtained reward tensor R. This data provides practical experience for training and is used for subsequent policy optimization.

[0040] Step S4: Data Augmentation;

[0041] In reinforcement learning, the task of robot-environment interaction can be based on the MDP (Markov Decision Process) model. get:

[0042] ;

[0043] Where S, A, T, and R represent the system state, action, state transition equation, and reward, respectively.

[0044] Specifically, after the robot acquires an interactive scenario, the collected (state-action) data can be expanded:

[0045] ;

[0046] For s and a, signals can be categorized into directly symmetric signals, indirectly symmetric signals, and asymmetric signals. In directly symmetric signals, data is augmented through interchange; in indirectly symmetric signals, data is augmented through inversion; and in asymmetric signals, the original data is retained. In the context of reinforcement learning training for humanoid robots, symmetric state and motion signals include centroid information (e.g., centroid linear velocity, centroid angular velocity, body posture); joint information (e.g., joint angles, joint angular velocities); motion information (e.g., joint torques, accelerations, and velocities); and other potentially included signals. Specifically, taking joint angle data augmentation as an example, joint angles can be categorized into directly symmetric joint angles. Such as the left-right leg joint angle, and non-directly symmetrical joint angles. Examples include trunk lateral sway angle and asymmetric joint angles. Examples include trunk pitch angle. Joint angles are represented as:

[0047] ;

[0048] For directly symmetrical joint angle data, the interchange method is as follows:

[0049] ;

[0050] For non-directly symmetrical joint angle data, the inversion method is adopted as follows:

[0051] ;

[0052] For asymmetric joint angles, the following approach is adopted:

[0053] ;

[0054] Similarly, other state signals and action signals can be augmented using a similar method. This allows for the acquisition of new training data to train the policy.

[0055] Specifically, for off-policy algorithms, augmented data can be directly added to the experience pool for policy training. Taking DDPG as an example: the robot interacts with the environment according to a certain policy to obtain sample data tuples. The enhanced symmetric sample is This operation maintains the reward The data volume is doubled while the data volume remains unchanged to maintain dynamic consistency.

[0056] Subsequently, the partitioned experience pool was designed. The original interaction data and the enhanced data are stored separately, and the sampling weights are set as follows: and The sampling strategy uses weighted joint sampling from both pools. The batch sampled data obtained from the joint sampling can be represented as:

[0057] ;

[0058] Where N is the sample batch size. Prioritize raw data to maintain the authenticity of the exploration.

[0059] Specifically, for the on-policy algorithm, augmented data is added as a supplement to the original data and incorporated into the direct optimization. Taking the PPO algorithm as an example:

[0060] ;

[0061] in, for Parameter strategy, Importance weights for action inference after state reversal The expression is:

[0062] ;

[0063] Otherwise, the pruning and entropy regularization mechanisms of the PPO algorithm are retained to improve the stability and exploration capabilities of the algorithm;

[0064] Step S5: Calculate the training task loss;

[0065] Based on the collected and augmented data, a loss function for the training task is calculated to reflect the gap between the current policy and the target policy. Simultaneously, to ensure policy symmetry, a symmetry loss is introduced to penalize asymmetrical behaviors, thereby ensuring the consistency of movement between the left and right joints. Specifically, the loss function for the training task can be expressed as:

[0066] ;

[0067] Where T is the total number of time steps, These are the parameters of the policy network;

[0068] Step S6: Policy Update;

[0069] Based on the calculated loss value, the policy parameters are adjusted using optimization algorithms (such as SGD, Adam, etc.). By iterating through steps S2 to S6, the strategy is gradually optimized, ultimately achieving the goal of efficiently completing the task.

[0070] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to any specific implementation. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A data augmentation method based on reinforcement learning for humanoid robots, characterized in that, Includes the following steps: Step S1: Setting up the parallel training environment; The simulation environment for robot training is built, including defining the physical characteristics of the environment, such as gravity, friction, and ground type; the initial state of the robot, such as position, posture, and joint angles; defining the physical simulation, policy training, and data pipeline core layers required for parallel training; defining the robot learning methods and policy networks, such as reinforcement learning, imitation learning, and their network structures and parameters; and defining the objectives of the tasks, such as walking, running, and balance control tasks. And the task's reward function, such as a function related to multiple factors based on the robot's balance state, walking speed, and stride symmetry; initialize all parameters that need to be initialized in the training process, such as the learning rate, discount factor, and momentum parameter of the training network; Step S2: Parallel training; The probability distribution of each action a occurring in state s at each time step is calculated using the policy network. At each time step, an action a is sampled according to the probability distribution, and the reward r obtained for that action is calculated. Each reward in the reward list is traversed from back to front, and the cumulative reward R is calculated. Step S3: Acquiring training data; In the simulation environment, the robot continuously interacts with the environment and collects data on the interaction process. This data includes the robot's motion tensor A, the observed environmental state tensor S, and the obtained reward tensor R. This data provides practical experience for training and is used for subsequent policy optimization. Step S4: Data Augmentation; In reinforcement learning, the task of robot-environment interaction is based on MDP, Markov decision process, and model. get: ； Where S, A, T, and R represent the system state, action, state transition equation, and reward, respectively. Specifically, after the robot acquires an interactive scenario, the collected state and action data are expanded: ； For s and a, signals are categorized into directly symmetric signals, indirectly symmetric signals, and asymmetric signals. In directly symmetric signals, data is augmented by interchange; in indirectly symmetric signals, data is augmented by inversion; and in asymmetric signals, the original data is retained. In the context of reinforcement learning training for humanoid robots, symmetric state and motion signals include centroid information, such as centroid linear velocity, centroid angular velocity, and body posture; joint information, such as joint angles and joint angular velocities; and motion information, such as joint torques, accelerations, and velocities. Step S5: Calculate the training task loss; Based on the collected and augmented data, a loss function for the training task is calculated to reflect the gap between the current policy and the target policy. Simultaneously, to ensure policy symmetry, a symmetry loss is introduced to penalize asymmetrical behaviors, thereby ensuring the consistency of movement between the left and right joints. Specifically, the loss function for the training task is expressed as follows: ； Where T is the total number of time steps, These are the parameters of the policy network; Step S6: Policy Update; Based on the calculated loss value, optimization algorithms such as SGD and Adam are used to adjust the policy parameters. By iterating through steps S2 to S6, the strategy is gradually optimized, ultimately achieving the goal of efficiently completing the task.

Images