Intelligent agent strategy generation method based on reinforcement learning and non-instantaneous computer readable medium

Abstract

The invention provides an agent strategy generation method based on reinforcement learning and a non-instantaneous computer readable medium, and the method comprises the steps: generating a first action for the execution of an agent according to an action network and a first state of the agent, and generating a first value according to a value network and the first state, obtaining a second state of the agent generated by the environment and a reward generated by a reward function, storing the first state, the first action, the first value, the second state and the reward to a buffer area, training a value network and an action network according to the buffer area, the regularization loss includes a first distance associated with the first action and a second distance associated with the first action or the first value.

Description

Technical Field

[0001] This application belongs to the field of reinforcement learning technology, and in particular to a method for generating agent policies based on reinforcement learning and a non-transient computer-readable medium. Background Technology

[0002] Reinforcement learning (RL) policies are prone to high-frequency oscillations. When no constraints or limitations are imposed on the learning or environment, RL agents often compromise other aspects in their pursuit of maximizing rewards. While achieving high performance (reward) in a task is the learning objective, other factors must be considered in certain situations. For example, high-frequency oscillations can cause wear and tear on actuators and other hardware when deploying policies on real-world hardware, which is particularly undesirable.

[0003] A simple way to reduce high-frequency oscillations is to incorporate a penalty term into the reward function. However, learning algorithms that tend to leverage reward functions may produce policies biased towards suboptimal performance for the sake of smoothness. Furthermore, reward function design is a complex problem, difficult to express from the outset for many tasks. Adding an extra penalty term with high-frequency oscillations essentially changes the original learning objective and is difficult to adjust. If the penalty weight is too large, the policy may prefer to do little to avoid large negative rewards. On the other hand, if the weight is too small, the policy may ignore it, still producing high-frequency oscillations. An ideal approach should maintain the original reward function design and eliminate the need for adding complex elements.

[0004] Another approach to reduce high-frequency oscillations is to filter the policy output actions, for example, using a low-pass filter. In the classic agent-environment framework of RL, this approach can be viewed as adding constraints to the environment rather than directly to the agent (or policy). In practice, filtering actions can lead to larger oscillations in the policy's raw output. The main drawback of using traditional filters (such as low-pass filters) is their memory nature, meaning that if the observation space does not contain past actions and observations, the policy will not be able to learn an effective model, as this violates the assumptions of Markov decision processes. Although this problem can be addressed by maintaining a history buffer for multiple steps, this requires a larger model in terms of parameters and complexity. Summary of the Invention

[0005] The purpose of this application is to provide a method for generating agent policies based on reinforcement learning and a non-transient computer-readable medium to alleviate the problem of high-frequency oscillations in deep reinforcement learning from the perspectives of network architecture and loss regularization, rather than using penalty terms in the reward function or environmental modifications (such as post-processing actions).

[0006] In a first aspect, this application provides a method for generating agent policies based on reinforcement learning, comprising multiple steps executed by a computing device. These steps include: obtaining a first state of the agent, an action network, a value network, and a reward function of the environment in which the agent resides; generating a first action for the agent to execute based on the action network and the first state, and generating a first value based on the value network and the first state; obtaining a second state of the agent generated by the environment and a reward generated by the reward function; storing the first state, the first action, the first value, the second state, and the reward in a buffer; and training the value network and the action network based on the buffer. The loss function of the action network includes a policy gradient loss and a regularization loss. The regularization loss includes a first distance and a second distance, wherein the first distance is associated with the first action, and the second distance is associated with either the first action or the first value.

[0007] Secondly, this application provides a non-transitory computer-readable medium for storing multiple instructions, which, when executed by a computing device, cause multiple operations. These operations include: obtaining a first state of an agent, an action network, a value network, and a reward function of the agent's environment; generating a first action based on the action network and the first state for the agent to execute, and generating a first value based on the value network and the first state; obtaining a second state of the agent generated by the environment and a reward generated by the reward function; storing the first state, the first action, the first value, the second state, and the reward in a buffer; and training the value network and the action network based on the buffer. The loss function of the action network includes a policy gradient loss and a regularization loss. The regularization loss includes a first distance and a second distance, where the first distance is associated with the first action and the second distance is associated with either the first action or the first value.

[0008] In summary, this application proposes a reinforcement learning-based agent policy generation method and a non-transient computer-readable medium. This invention does not rely on explicit reward / penalty terms or environmental adjustments (such as post-processing actions), but instead employs loss regularization and network architecture to encourage smooth policy learning mappings, enabling adjacent states in the input space to generate adjacent actions in the output space.

[0009] The foregoing description of the disclosure and the following description of the embodiments are intended to demonstrate and explain the spirit and principles of the present invention, and to provide a further explanation of the scope of the patent application of the present invention. Attached Figure Description

[0010] Figure 1 This is a flowchart of an agent policy generation method based on reinforcement learning according to an embodiment of the present invention.

[0011] Figure 2 This is a flowchart of regularization loss calculation according to the first embodiment of the present invention.

[0012] Figure 3 This is a flowchart of regularization loss calculation according to the second embodiment of the present invention.

[0013] Figure 4 This is a flowchart illustrating the operation of the feedforward layer of an action network according to an embodiment of the present invention; and

[0014] Figure 5 and Figure 6 The evaluation results of this invention are presented in the context of real-world robotic applications.

[0015] Component designation explanation

[0016] S1~S5, T1~T6, U1~U7, V1~V3 Steps

[0017] L1~L9 methods Detailed Implementation

[0018] The following detailed description of the features and advantages of the present invention in the embodiments is sufficient to enable anyone skilled in the art to understand the technical content of the present invention and implement it accordingly. Based on the disclosure, patent claims, and drawings in this specification, anyone skilled in the art can easily understand the related objectives and advantages of the present invention. The following embodiments further illustrate the points of the present invention in detail, but are not intended to limit the scope of the present invention in any way.

[0019] This application proposes a reinforcement learning-based agent policy generation method suitable for execution by a computing device. In one embodiment, the computing device may be at least one of the following: a personal computer, a network server, a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller (MCU), an application processor (AP), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a system-on-a-chip (SOC), a deep learning accelerator, or any electronic device with similar functionality. This invention does not limit the hardware type of the computing device. This invention also proposes a non-transitory computer-readable medium storing multiple instructions that, when executed by the computing device, cause multiple operations corresponding to the reinforcement learning-based agent policy generation method proposed in one embodiment of this invention.

[0020] Figure 1 This is a flowchart of a reinforcement learning-based agent policy generation method according to an embodiment of the present invention, including steps S1 to S5. In step S1, the computing device obtains a first state of the agent, an action network, a value network, and a reward function of the agent's environment. In step S2, the computing device generates a first action for the agent to execute based on the action network and the first state, and generates a first value based on the value network and the first state. In step S3, the computing device obtains a second state of the agent generated by the environment and a reward generated by the reward function. In step S4, the computing device stores the first state, the first action, the first value, the second state, and the reward in a buffer. In step S5, the computing device trains the value network and the action network based on the buffer, wherein the loss function of the action network includes a policy gradient loss and a regularization loss, and the regularization loss includes a first distance and a second distance, the first distance being associated with the first action, and the second distance being associated with either the first action or the first value.

[0021] The above process employs loss regularization, which adds a regularization component to the standard RL loss function instead of the reward function to reduce the frequency of action oscillations. Its general form is as follows: in It is the policy gradient loss. This is the regularization loss. Figure 2 and Figure 3 These are two implementation examples of regularization loss calculation.

[0022] Figure 2 This is a flowchart of regularization loss calculation according to a first embodiment of the present invention, including steps T1 to T6. In step T1, the computing device generates a second action based on the action network and the second state. In step T2, the computing device selects a reference state from the normal distribution of the first state. In step T3, the computing device generates a reference action based on the action network and the reference state. In step T4, the computing device calculates a first distance between the first action and the second action. In step T5, the computing device calculates a second distance between the first action and the reference action. In step T6, the computing device calculates a weighted sum of the first distance and the second distance as the regularization loss.

[0023] exist Figure 2 The process uses two regularization components, the first being the time component. It minimizes two consecutive states s t and s t+1 The distance between actions. The second is the spatial component. Its minimized state s t With state The differences between them, where the states From state s t Sampled from the nearest standard distribution, it is represented as follows:

[0024] Among them, s t Indicates the first state; s t+1 Indicates the second state; π θ (·) represents the action network, therefore π θ (S t ) indicates the first action, π θ (S t+1 ) represents the second action generated in step T1; D(·) is the distance function, therefore This represents the first distance calculated in step T4.

[0025] Corresponding to steps T2, T3, and T5, where, Indicates the reference state, N(S) t , σ) represents the reference state Let be the mean, and σ represent the normal distribution of the standard deviation, which is an adjustable hyperparameter. Spatial components. The calculation is for the second distance.

[0026] Corresponding to step T6, where λ T ,λ S These are adjustable hyperparameters.

[0027] Figure 3 This is a flowchart of regularization loss calculation according to a second embodiment of the present invention, including steps U1 to U7. In step U1, the computing device calculates the interaction result between the difference between the second state and the first state and a random variable. In step U2, the computing device generates a reference state based on the first state and the interaction result. In step U3, the computing device generates a reference value based on the value network and the reference state. In step U4, the computing device generates a reference action based on the action network and the reference state. In step U5, the computing device calculates a first distance between the first action and the reference action. In step U6, the computing device calculates a second distance between the first value and the reference value. In step U7, the computing device calculates the weighted sum of the first distance and the second distance as the regularization loss.

[0028] exist Figure 3 The process also uses two regularization components, similar to the spatial component mentioned in the first embodiment. The difference lies in that the regularization is applied to the action network π. θ Value Network V θThe output. Furthermore, the sampling distance relative to two consecutive states s. t and s t+1 The distance is determined by the parameter, rather than the predefined hyperparameter in the first embodiment. The specific calculation is as follows:

[0029] where u~U(·), where This represents the reference state generated in step U2, where u represents a random variable in the uniform distribution U(·), therefore (s t+1 -s t )·u is the interactive result obtained from step U1.

[0030] Where π θ (s t ) indicates the first action. This indicates the reference action generated in step U4. This represents the first distance calculated in step U5.

[0031] Where V θ (·) represents the value network, therefore V θ (s t ) indicates primary value. This indicates the reference value generated in step U3. This represents the second distance calculated in step U6.

[0032] Corresponding to step U7, where λ π and λ V λ represents the weights of each regularization component. V =βλ π .

[0033] λ π The lower limit is λ, and the upper limit is λ.

[0034] The second embodiment of regularization loss calculation is similar to the first embodiment, with the main difference being the sampling method. The time element in the first embodiment can be considered redundant, as nearby sampled states and two consecutive states should produce similar regularization signals. Therefore, the second embodiment does not consider the time element, which is beneficial for optimizing the outputs of the action network and value network through spatial regularization.

[0035] Figure 4 This is a flowchart of the operation of the feedforward layer of an action network according to an embodiment of the present invention, including steps V1 to V3.

[0036] In step V1, the computing device generates an output vector f(x) based on the input vector x using the first multilayer perceptron. This step is equivalent to the operation of a conventional feedforward layer.

[0037] In step V2, the computing device generates a Lipschitz value K(x) based on the input vector x using a second multilayer perceptron connected to the activation function. In one embodiment, the activation function is a Softplus function or a linear function, which is not limited by the present invention.

[0038] In step V3, the computing device performs multi-dimensional gradient normalization (MGN) based on the output vector, the gradient of the output vector, and the Lipschitz value to generate the output of the feedforward layer, as shown below:

[0039]

[0040] Where ▽f(x) is the gradient of the output vector f(x), ||▽f(x)|| is the 2-norm of the Jacobian matrix relative to the input vector x, K(x) is the Lipschitz value modeled by the feedforward network K with respect to the input x, and ε is a small positive value to avoid division by zero.

[0041] Figure 4 One implementation modifies the learning components of the action network at the architectural level to reduce the oscillation frequency of actions. Two other architecture-based implementations include Spectral Normalization (SN) and Liu-Lipschitz. It should be added that... Figure 4 The implementation examples can also be used in value networks.

[0042] Spectral normalization is most commonly used for training stable Generative Adversarial Networks (GANs). It consists of a rescaling operation that applies the spectral norm σ(W) to the layer weights. The normalized weights are... The global version of spectral normalization is applied to every layer, while the local version (Local SN) is applied only to the output layer, and the local version performs better. In one embodiment, this invention uses spectral normalization in PyTorch to implement this method, setting σ = 1.0, and no other hyperparameters are required.

[0043] The Liu-Lipschitz method was originally used to learn smooth mappings in neural distance field networks to achieve shape interpolation and extrapolation. This method restricts the Lipschitz bound of the network to the learnable parameter c of each layer.i The weights of each layer are normalized, and the layer outputs are calculated as follows:

[0044]

[0045]

[0046] in, These are the normalized weights, and σ(·) is the activation function. This method also includes a loss function element to minimize c. i The value is represented as follows:

[0047]

[0048] Where λ is an adjustable hyperparameter, N is the number of network layers, and each layer corresponds to a c. i .

[0049] The two Lipschitz-based methods mentioned above also add a term to the loss function, but this is to limit the upper limit of the network's Lipschitz value, rather than directly optimizing the differences between states and actions like loss regularization methods.

[0050] Figure 5 and Figure 6 This paper demonstrates the evaluation results of this invention applied to real-world robots, implemented using tools such as Isaac Gym, RLGames, and PyTorch. The robot's task is to track velocity vectors to achieve walking and to track target direction vectors. For this locomotion task, this invention uses a quadruped robot to conduct experiments in a real-world environment and employs Domain Randomization (DR) to ensure a successful transition from simulation to the real environment. By adding noise to different elements in the simulation, the policy can learn to execute effectively across a wider range of state distributions.

[0051] exist Figure 5 and Figure 6 In the diagram, L1 to L9 correspond to the smoothness and cumulative reward results of policies trained using different policy generation methods. L1 is the basic method of domain-free randomization; L2 is the basic method; and L3 and L4 are two implementations of regularization loss, corresponding to... Figure 2 and Figure 3 L5, L6, and L7 are three implementations of the network architecture, where L5 is a local version of spectrum normalization (Local SN), L6 is the Liu-Lipschitz method, and L7 corresponds to... Figure 4 L8 is a hybrid of L3 and L7, and L9 is a hybrid of L4 and L7. Each method is trained from scratch using a policy with 9 random seeds. Table 1 lists all hyperparameters.

[0052] Table 1 lists the hyperparameters used by each method during training.

[0053]

[0054]

[0055] exist Figure 5 and Figure 6 The evaluation metrics include Cumulative Return and Smoothness. Cumulative Return represents the sum of rewards earned at every step throughout the entire episode, as shown below:

[0056]

[0057] This metric evaluates the strategy's performance in a task. It is context-dependent and primarily used to analyze the trade-offs between smoothness and performance.

[0058] Smoothness is calculated from the spectrum obtained from the Fast Fourier Transform (FFT). Smoothness is a normalized weighted average frequency, as shown below:

[0059]

[0060] Where n is the number of frequency bands, f s It is the sampling frequency, M i and f i These represent the amplitude and frequency of the i-th frequency band, respectively. Higher smoothness indicates a large high-frequency component, while lower smoothness indicates a smoother control signal. Similar to cumulative reward, a good smoothness value varies depending on the environment but is independent of the policy control frequency.

[0061] like Figure 5 and Figure 6 As shown, each method achieves a similar level of performance in the task (evaluated by cumulative reward). Regarding smoothness, hybrid methods such as L8 and L9 consistently outperform other methods (L1–L7). Figure 5 and Figure 6 As shown, each method (L3 to L9) outperforms the basic strategies (L1 and L2) in terms of smoothness.

[0062] In summary, this application provides a reinforcement learning-based agent policy generation method and a non-transient computer-readable medium. Instead of relying on explicit reward / penalty terms or environmental adjustments (such as post-processing actions), this application employs loss regularization and network architecture to encourage smooth policy learning mappings, enabling adjacent states in the input space to generate adjacent actions in the output space. Experimental results show that the best-performing hybrid method improves smoothness by 26.8% compared to the baseline method, while the worst-case performance degradation is only 2.8%.

[0063] While the present invention has been disclosed above with reference to the foregoing embodiments, it is not intended to limit the invention. Any modifications and refinements made without departing from the spirit and scope of the invention are within the scope of patent protection of the present invention. For details regarding the scope of protection defined in the present invention, please refer to the appended claims.

Claims

1. A method for generating agent policies based on reinforcement learning, characterized in that, Multiple steps performed by a computing device, the multiple steps including: Obtain the first state of an agent, its action network, value network, and a reward function of the environment in which the agent resides; A first action is generated based on the action network and the first state for the agent to execute, and a first value is generated based on the value network and the first state; Obtain the second state of the agent generated by the environment and a reward generated by the reward function; Store the first state, the first action, the first value, the second state, and the reward in a buffer; and The value network and the action network are trained based on the buffer. The loss function of the action network includes policy gradient loss and regularization loss. The regularization loss includes a first distance and a second distance, wherein the first distance is associated with the first action and the second distance is associated with the first action or the first value.

2. The agent policy generation method based on reinforcement learning according to claim 1, characterized in that, The multiple steps also include: A second action is generated based on the action network and the second state; Select a reference state from the normal distribution of the first state; Generate a reference action based on the action network and the reference state; Calculate the first distance between the first action and the second action; Calculate the second distance between the first action and the reference action; and The weighted sum of the first distance and the second distance is calculated as the regularization loss.

3. The agent policy generation method based on reinforcement learning according to claim 2, characterized in that, The multiple steps also include: Calculate the interaction result between the difference between the second state and the first state and a random variable; A reference state is generated based on the first state and the interaction result. A reference value is generated based on the value network and the reference state; Generate a reference action based on the action network and the reference state; Calculate the first distance between the first action and the reference action; Calculate the second distance between the first value and the reference value; and The weighted sum of the first distance and the second distance is calculated as the regularization loss.

4. The agent policy generation method based on reinforcement learning according to claim 1, characterized in that, The feedforward layer of the action network includes: The first multilayer perceptron generates an output vector based on the input vector; A second multilayer perceptron, connected to the activation function, generates Lipschitz values ​​based on the input vector; and Multidimensional gradient normalization is performed based on the output vector, the gradient of the output vector, and the Lipschitz value to produce the output of the feedforward layer.

5. A non-transitory computer-readable medium for storing a plurality of instructions, said plurality of instructions causing a plurality of operations when executed by a computing device, characterized in that, The multiple operations include: Obtain the first state of an agent, its action network, value network, and a reward function of the environment in which the agent resides; A first action is generated based on the action network and the first state for the agent to execute, and a first value is generated based on the value network and the first state; Obtain the second state of the agent generated by the environment and a reward generated by the reward function; Store the first state, the first action, the first value, the second state, and the reward in a buffer; and The value network and the action network are trained based on the buffer. The loss function of the action network includes policy gradient loss and regularization loss. The regularization loss includes a first distance and a second distance, wherein the first distance is associated with the first action and the second distance is associated with the first action or the first value.

6. The non-transitory computer-readable medium according to claim 5, characterized in that, The plurality of operations further include: A second action is generated based on the action network and the second state; Select a reference state from the normal distribution of the first state; Generate a reference action based on the action network and the reference state; Calculate the first distance between the first action and the second action; Calculate the second distance between the first action and the reference action; and The weighted sum of the first distance and the second distance is calculated as the regularization loss.

7. The non-transitory computer-readable medium according to claim 5, characterized in that, The plurality of operations further include: Calculate the interaction result between the difference between the second state and the first state and a random variable; A reference state is generated based on the first state and the interaction result. A reference value is generated based on the value network and the reference state; Generate a reference action based on the action network and the reference state; Calculate the first distance between the first action and the reference action; Calculate the second distance between the first value and the reference value; and The weighted sum of the first distance and the second distance is calculated as the regularization loss.

8. The non-transitory computer-readable medium according to claim 5, characterized in that, The feedforward layer of the action network includes: The first multilayer perceptron generates an output vector based on the input vector; A second multilayer perceptron connected to the activation function generates Lipschitz values ​​based on the input vector. as well as Multidimensional gradient normalization is performed based on the output vector, the gradient of the output vector, and the Lipschitz value to produce the output of the feedforward layer.

Images