Training system and method for intermediate policy of agent

Abstract

A method for training an agent's mediation policy includes, executed by a processor: selecting a source policy and a target policy from a plurality of policies, wherein each policy drives the agent to perform multiple actions to be in multiple states, each of each state including multiple physical attributes; selecting one from each of the multiple states in the source policy and the target policy as the source state and the target state, respectively; training the mediation policy with reinforcement learning to transition the agent from the source state to the target state during a period, wherein the reinforcement learning includes an annealing function for setting multiple tolerance boundaries for the multiple physical attributes, and the multiple tolerance boundaries gradually shrink during the period.

Description

[Technical Field]

[0001] This invention relates to reinforcement learning, and in particular to a training system and method for an agent's mediation strategy. [Previous Technology]

[0002] In recent years, reinforcement learning (RL) has achieved state-of-the-art results in quadruped robot locomotion. Just a few years ago, the best controllers were handcrafted and meticulously tuned based on robot kinematic models. These traditional controllers required significant engineering effort to approximate real-world dynamics. Today, reinforcement learning-based controllers can learn optimal policies through data-driven simulations. These controllers can develop novel behaviors previously impossible and cope with the most challenging terrains and environments. RL controllers can generate new gaits with high efficiency and, with the addition of constraints, produce visually appealing gaits similar to those of traditional controllers.

[0003] A high-order task is typically decomposed into multiple low-order tasks. In this case, each low-order task is executed by an independent RL policy. However, the problem of connecting these policies arises. Since each policy is trained independently, switching from one task policy to another is neither direct nor easy.

[0004] A common approach to connecting multiple low-order policy tasks is to use a hierarchical approach, where the outputs of higher-order policies are used as weights for lower-order policies. However, this becomes problematic when new low-order policies must be added to the system. Since higher-order policies are combinations of all lower-order policies, adding a new low-order policy requires retraining the entire system. Another potential drawback of the hierarchical approach is that the number of parameters for higher-order policies increases with the number of lower-order policies. [Summary of the Invention]

[0005] In view of this, the present invention proposes a training system and method for an agent's mediation policy (also known as: composer policy) to solve the above problems.

[0006] According to an embodiment of the present invention, a method for training an agent's mediation policy includes, executed by a processor: selecting a source policy and a target policy from a plurality of policies, wherein each policy drives the agent to perform multiple actions to be in multiple states, each state including multiple physical attributes. Selecting one of each of the multiple states from the source policy and the target policy as the source state and the target state, respectively. Training the mediation policy with reinforcement learning to transition the agent from the source state to the target state during a period, wherein the reinforcement learning includes an annealing function for setting multiple tolerance boundaries of the multiple physical attributes, and the multiple tolerance boundaries gradually shrink during the period.

[0007] A training system for an agent's mediation policy according to an embodiment of the present invention includes a storage device and a processor. The storage device is used to store a plurality of instructions. The processor is electrically connected to the storage device to execute the plurality of instructions and cause a plurality of operations, the operations including: selecting a source policy and a target policy from a plurality of policies, wherein each policy is used to drive the agent to perform a plurality of actions to be in a plurality of states, each state including a plurality of physical attributes; selecting one of the plurality of states from the source policy and the target policy as the source state and the target state, respectively; training the mediation policy with reinforcement learning to transition the agent from the source state to the target state during a period, wherein the reinforcement learning includes an annealing function for setting a plurality of tolerance boundaries for the plurality of physical attributes, and the plurality of tolerance boundaries gradually shrink during the period.

[0008] In summary, the training system and method for the mediation strategy of the intelligent agent proposed in this invention have the following contributions and effects: First, it proposes a general connection mechanism of mediation strategy. As long as the target state is provided, it can transition between any two reinforcement learning strategies. Second, through the design of a reward function with shrinking boundaries, the mediation strategy can smoothly transition to a general target state.

[0009] The above description of the disclosure and the following description of the embodiments are intended to demonstrate and explain the spirit and principles of the present invention, and to provide a further explanation of the scope of the patent application of the present invention.

Implementation Method

[0011] The detailed features and advantages of the present invention are described below in the embodiments. The content is sufficient to enable anyone skilled in the art to understand the technical content of the present invention and implement it accordingly. Based on the content disclosed in this specification, the scope of the patent application, and the drawings, anyone skilled in the art can easily understand the relevant objectives and advantages of the present invention. The following embodiments further illustrate the viewpoints of the present invention, but are not intended to limit the scope of the present invention in any way.

[0012] FIG1 is a block architecture diagram of a training system for an agent's mediation strategy according to an embodiment of the present invention. As shown in FIG1, the training system 10 includes a storage device 1 and a processor 3.

[0013] The storage device 1 is used to store a plurality of instructions. In one embodiment, the storage device 1 may be implemented using at least one of the following types: flash memory, hard disk (HDD), solid-state drive (SSD), dynamic random access memory (DRAM), static random access memory (SRAM), or other non-volatile memory. However, the present invention is not limited to the above types.

[0014] The processor 3 is electrically connected to the storage device 1 to execute the plurality of instructions and cause a plurality of operations corresponding to a plurality of steps in a training method for an intelligent agent's mediation strategy according to an embodiment of the present invention. In one embodiment, the processor 3 may be at least one of the following: a personal computer, a network server, a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller (MCU), an application processor (AP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system-on-a-chip (SOC), a deep learning accelerator, or any electronic device with similar functionality. The present invention does not limit the hardware type of the processor 3.

[0015] Figure 2 is a flowchart illustrating a training method for an agent's mediation strategy according to an embodiment of the present invention. In one embodiment, the agent is a Unitree A1 quadruped robot with 12 proportional-derived (PD) controllers. These PD controllers are used to drive multiple joints of the robot, but the present invention does not limit the number of PD controllers or joints. The mediation strategy takes the agent's current state and target state as inputs to generate a new transition trajectory to drive the agent to the target state. As shown in Figure 2, the training method includes steps S1, S2, and S3.

[0016] In step S1, the processor 3 selects a source policy and a target policy from multiple policies. Each policy drives the agent to perform multiple actions to be in multiple states. Each state includes multiple physical attributes. In one embodiment, the multiple physical attributes include: position, orientation, linear velocity, angular velocity, angle and velocity of each joint, and binary foot contact indication, wherein the linear velocity can be estimated by combining inertial measurement unit (IMU) readings and leg velocity during foot contact.

[0017] In one embodiment, the multiple policies described in step S1 are prepared. In another embodiment, prior to step S1, the training method further includes the following steps: collecting multiple reference motion segments or animation data, tracking the reference motion segments in a physics simulator, and training multiple physics-based controllers as the multiple policies using a motion imitation framework. These policies can be represented as follows: where represents the action performed through the PD controller, represents the state of the agent, represents the data of the reference motion segment (e.g., including four future frames), and represents the number of policies. In the above embodiment, the motion imitation framework is implemented using Isaac Gym, and the multiple policies are trained using PPO-clip loss on a consumer-grade notebook computer equipped with an Intel 8-core i7-11800H 2.3 GHz processor and an NVIDIA RTX 3070 8GB graphics card.

[0018] In one embodiment, in order to transfer the multiple strategies from simulation to the real world and ensure that each strategy is robust enough when applied to a real-world robot, domain randomization is employed. Specifically, the mass of each joint of the robot is randomized, disturbance forces are introduced (random external forces are applied to the agent), noise is added to the sensor readings (simulating errors in sensor readings in the real world), and terrain height and friction are randomized, with detailed parameters shown in Table 1 below.

[0019] Table 1, Parameters for Domain Randomization. Uniform sampling across the range. The foot contact value represents the probability that each foot contact is zeroed out (no foot contact with the ground). parameter numerical values Strategy Mediator strategy Action noise ±0.02 ±0.02 Joint quality [75%, 125%] [95%, 105%] Differential gain [35, 65] [45, 55] proportional gain [1.0, 1.4] [0.9, 1.2] ground friction [0.1, 1.5] [0.1, 1.5] Noise - Direction ±0.05 ±0.06 Noise-Linear Speed ±0.25 ±0.25 Noise-Angular Velocity ±0.3 ±0.3 Noise - Joint Angle ±0.02 ±0.02 Noise - Joint Velocity – ±1.5 Noise - Foot Contact 20% 20%

[0020] In step S2, the processor 3 selects one of each of the multiple states in the source policy and the target policy as the source state and the target state, respectively. In one embodiment, the source state and the target state can be obtained by randomly sampling from a large number of physically feasible states or animation data, and noise is added to multiple physical attributes contained in the target state. The mediation policy will learn how to drive the agent from a certain state to any other state.

[0021] In step S3, processor 3 trains an intermediary policy using reinforcement learning to transition the agent from a source state to a target state during a period. The reinforcement learning includes an annealing function for setting multiple tolerance boundaries for the multiple physical attributes, and these tolerance boundaries gradually shrink during the period. The intermediary policy is used to generate a new transition trajectory to drive the agent to the target state. The intermediary policy can be represented as P, where P represents the agent's current state and P represents the agent's target state. The current state and the policy state also include multiple physical attributes: position, orientation, linear velocity, angular velocity, angle and velocity of each joint, and a binary foot contact indicator. The target state represents the state the agent should be in after the transition. In one embodiment, if the target state is obtained from animation data, the binary foot contact indicator can be subtracted from the multiple physical attributes included in the target state to reduce computational load.

[0022] The mediation strategy is trained through episoded reinforcement learning, and the training process is completely independent of the aforementioned multiple strategies. The goal of the mediation strategy is to match the target state. The architecture of the mediation strategy P is a two-layer feedforward neural network containing 512 and 256 hidden units, respectively. Except for the linear output layer, each layer uses an Exponential Linear Unit (ELU) as the activation function.

[0023] In one embodiment, the mediation policy receives a 208-dimensional observation vector, including the agent's three most recent states, actions over the past three timesteps, the target state, the tolerance boundary center of the current timestep, and a scalar-encoded normalized time. The time starts at 0.0 and increases to 1.0 at the end of the episode. The maximum episode length is set to 2 seconds. The output of the mediation policy is used to configure the plurality of PD controllers to control the angles of multiple joints of the agent.

[0024] Figure 3 is an exemplary schematic diagram illustrating a gradually shrinking tolerance boundary according to an embodiment of the present invention. As shown in Figure 3, the tolerance boundary B gradually shrinks from the source state to the target state. Agent A learns to avoid the gradually shrinking tolerance boundary B while moving closer to the target state. The duration of the transition is dynamic; if all physical property states are within the tolerance boundary B before the end of the period, the period will end prematurely. In fact, if the mediation strategy works well, most transitions will indeed terminate prematurely.

[0025] The reward function incentivizes agent A to remain within the tolerance boundary B for each physical attribute (such as joint position, linear velocity, angular velocity, and orientation). If agent A has all physical attributes within the tolerance boundary B (as in the current state) at the end of the period, the mediation policy receives a base reward of +1. If agent A violates any of the tolerance boundaries B (as in the current state) before the end of the period, the mediation policy receives a negative reward of -1 and the period ends. If agent A is in the target state (as in the current state) before the end of the period, and all physical attributes are within the tolerance boundary B, the mediation policy receives an additional reward of 100 greater than the base reward of 1. The above values ​​are merely examples and are not intended to limit the invention.

[0026] In one embodiment, the reward function of reinforcement learning includes a set of indicator functions that shrink hard boundaries and an energy efficiency penalty term, wherein the hard boundaries represent rules that must be followed, and penalties will be imposed if the boundaries are exceeded. The indicator functions are as shown in the following method one: (Formula 1) Where represents the annealing function, which is a linear function connecting the source state to the target state of agent A, and represents the tolerance. Annealing is performed at each time step according to Equation 2. (Formula 2)

[0027] where and represent the tolerance at the start and end of the transition, respectively, and is an exponential parameter (refer to Table 2) used to adjust the degree of contraction of the tolerance boundary B. As the tolerance boundary B anneals, the mediation strategy learns to closely match the target state. However, this does not guarantee the smoothness or energy efficiency of the trajectory. Therefore, the reward function also includes a penalty term for joint torque. In one embodiment, step S3 further includes: subtracting the weighted sum of the torques of the plurality of joints from the reward obtained from the mediation strategy. The complete reward function is shown in the following Method 3, where is the torque of the j-th joint of agent A, and is a scalar controlling the scale of the penalty: (Formula 3)

[0028] Table 2 below is an example of the detailed parameters of the tolerance boundary B and the penalty term.

[0029] Table 2, Adjustable parameters of the mediation strategy. and represent the tolerance at the start and end of the transition, respectively. element numerical values Center of mass height 0.35 0.02 direction 1 0.2 linear velocity 2.5 0.2 angular velocity 15 0.2 Joint angle 3.14 0.5 Torque – –

[0030] Figure 4 is a schematic diagram of the combination of the mediation policy with other policies. As shown in Figure 4, after training the mediation policy P, we can sequentially combine the various policies. For example, agent A may start executing a policy, and at some point, an event triggers a switch to another policy. To execute the transition between these two policies, the mediation policy P takes over the control of agent A and extracts a target state from the distribution, using its animation data. Then, the mediation policy P executes an action that makes the agent's state at the end of the transition sufficiently close to the target state so that the policy can take over control. The transition ends with the target policy taking over the agent's control. This process can be repeated indefinitely and robustly, applicable to any pair in the policy library L.

[0031] If the intermediate policy P is trained using a sufficiently large distribution (including the source and target states), then the intermediate policy P should be able to learn to make feasible transitions between any policies, even new policies. For example, if the training data includes forward and backward running movements (-1.0 m / s ~ 1.0 m / s), then during the testing phase, using a slow walking movement (0.5 m / s) can also result in a successful transition because the speed of the slow walking movement is already covered within the training range, even if the slow walking movement itself is not in the training data. Therefore, by adding new policies and using the same intermediate policy P, the policy library L can be gradually and infinitely expanded without retraining or fine-tuning.

[0032] In summary, the training system and method for the mediation strategy of the intelligent agent proposed in this invention have the following contributions and effects: First, it proposes a universal connection mechanism—the mediation strategy—which allows transitions between any two reinforcement learning strategies as long as the target state is provided. Second, through the design of a reward function with a shrinking boundary, the mediation strategy can smoothly transition to a universal target state.

[0033] Although the present invention has been disclosed above with reference to the foregoing embodiments, it is not intended to limit the present invention. Any modifications and refinements made without departing from the spirit and scope of the present invention are within the scope of patent protection of the present invention. For the scope of protection defined by the present invention, please refer to the appended claims. [Simplified Explanation of the Diagram]

[0010] Figure 1 is a block architecture diagram of a training system for an agent's mediation strategy according to an embodiment of the present invention; Figure 2 is a flowchart of a training method for an agent's mediation strategy according to an embodiment of the present invention; Figure 3 is an example schematic diagram of a gradually shrinking tolerance boundary according to an embodiment of the present invention; and Figure 4 is a schematic diagram of the combination of mediation strategy and other strategies.

Claims

1. A method for training an agent's mediation policy, comprising, executed by a processor: selecting a source policy and a target policy from a plurality of policies, wherein each of the policies drives the agent to perform a plurality of actions to be in a plurality of states, each of the states including a plurality of physical attributes; selecting one from each of the states in the source policy and the target policy as a source state and a target state; and training an mediation policy with reinforcement learning to transition the agent from the source state to the target state during a period, wherein the reinforcement learning includes an annealing function for setting a plurality of tolerance boundaries for the physical attributes, and the tolerance boundaries gradually shrink during the period.

2. The method for training the mediation policy of the agent as described in claim 1, wherein a reward function of the reinforcement learning algorithm includes: The mediation policy receives a basic reward when the physical attributes are within the tolerance boundaries at the end of the period; a negative reward when the physical attributes violate any of the tolerance boundaries before the end of the period; and an additional reward greater than the basic reward when the agent is in the target state and the physical attributes are within the tolerance boundaries before the end of the period.

3. A method for training an agent's mediation strategy as described in claim 1, wherein the agent is a robot, the robot comprising multiple joints and multiple proportional-differential controllers for driving the joints, and the physical properties include: Position, direction, linear velocity, angular velocity, and the angles and velocities of these joints.

4. The method for training the mediation policy of the agent as described in claim 3 further includes: The weighted sum of the torques of those joints is deducted from the rewards obtained from this mediation strategy.

5. A method for training an agent's mediation strategy as described in claim 3, wherein the mediation strategy is a two-layer feedforward neural network, the two layers having 512 and 256 hidden units respectively, the activation function of the feedforward neural network being an exponential linear unit, and the input to the mediation strategy including: The target state, the tolerance boundaries, and the agent's multiple historical states and actions, the output of the mediation policy is used to set the proportional-derivative controllers.

6. A training system for an agent's mediation strategy, comprising: A storage device for storing multiple instructions; The system also includes a processor electrically connected to the storage device to execute the instructions and cause a plurality of operations, the operations including: selecting a source policy and a target policy from a plurality of policies, wherein each of the policies drives the agent to perform a plurality of actions to be in a plurality of states, each of the states including a plurality of physical attributes; selecting one of the states in the source policy and the target policy as a source state and a target state, respectively; and training an intermediate policy with reinforcement learning to transition the agent from the source state to the target state during a period, wherein the reinforcement learning includes an annealing function for setting a plurality of tolerance boundaries for the physical attributes, and the tolerance boundaries gradually shrink during the period.

7. A training system for an agent's mediation policy as described in claim 6, wherein a reward function of the reinforcement learning algorithm includes: The mediation policy receives a basic reward when the physical attributes are within the tolerance boundaries at the end of the period; a negative reward when the physical attributes violate any of the tolerance boundaries before the end of the period; and an additional reward greater than the basic reward when the agent is in the target state and the physical attributes are within the tolerance boundaries before the end of the period.

8. A training system for an agent's mediation strategy as described in claim 6, wherein the agent is a robot, the robot comprising multiple joints and multiple proportional-differential controllers for driving the joints, and the physical properties include: Position, direction, linear velocity, angular velocity, and the angles and velocities of these joints.

9. The training system for the mediation strategy of the agent as described in claim 8, further comprising: The weighted sum of the torques of those joints is deducted from the rewards obtained from this mediation strategy.

10. A training system for an agent's mediation strategy as described in claim 8, wherein the mediation strategy is a two-layer feedforward neural network, the two layers having 512 and 256 hidden units respectively, the activation function of the feedforward neural network being an exponential linear unit, and the input to the mediation strategy including: The target state, the tolerance boundaries, and the agent's multiple historical states and actions, the output of the mediation policy is used to set the proportional-derivative controllers.

Images