A reward optimization method and system for an automatic control system based on reinforcement learning

Abstract

This invention discloses a reward optimization method and system for automatic control systems based on reinforcement learning. The method uses the action space of the automatic control system as input to a convolutional neural network (CNN). After processing by the CNN, the input enters the actor and critic networks in the reinforcement learning process. Existing expert demonstrations are used to pre-train the network. A redefined Q-squared algorithm is then used. CAAC The value replaces the temporal difference Q used in traditional reinforcement learning algorithms. TD To train the critic network, Q CAAC The Q-value is obtained by summing the Q-values ​​calculated by Monte Carlo and temporal difference methods with different weights. The weights between the two methods decay according to the cosine function as the training process progresses. This can more effectively combine the advantages of Monte Carlo and temporal difference methods, improve the sparse reward problem in existing automatic control systems, and improve work efficiency.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and relates to a reward optimization method and system for automatic control systems based on reinforcement learning. Background Technology

[0002] Reinforcement learning has wide applications in many fields, such as autonomous driving, gaming, and robot control. However, in many real-world tasks, reward signals are not continuous and abundant, but rather quite sparse. This means that an agent receives positive feedback only at a few points in time, and remains in a rewardless state most of the time. For example, to obtain a reward in the game Montezuma's Revenge, a player must perform a series of hundreds of actions, making it one of the most challenging tasks in Atari games. Similarly, the sparse reward problem is prevalent in robot control. When robots perform complex tasks, rewards are only given upon completion of the entire task, and there may be no reward signal for most of the task's duration. This requires robots to engage in a large amount of trial and error to acquire sufficient effective data. However, in the real world, the cost of trial and error for robots is relatively high. In the field of autonomous driving, autonomous vehicles may only receive positive rewards when they successfully avoid collisions, remaining in a rewardless state for the vast majority of the time.

[0003] This sparse reward situation makes the learning process more challenging because the agent needs to spend more time and effort exploring the environment to discover state-action pairs that can yield rewards. Especially in the case of sparse rewards, the agent needs to overcome the problems of signal sparsity and feedback delay. Therefore, improving the performance of agents in reward-sparse automatic control systems has become an important research topic. Enhancing the ability to solve sparse reward problems can significantly improve the efficiency of sample utilization in reinforcement learning, which is of great significance for promoting the application of reinforcement learning.

[0004] In reinforcement learning, given an action, there exists an action-value function that depends on the current state and the most recently taken action; it represents the expected reward based on the state and the action. Assuming the agent follows a policy π, the action-value function is represented as Q. π There are currently two main methods for calculating the Q value (s,a). The first method is to use the bootstrap method to calculate the Q value using time difference. The second method is the Monte Carlo method, which executes the task for the entire round to finally calculate the Q value. The time difference method is a biased estimator but has small variance, while the Monte Carlo method is an unbiased estimator but has large variance. Summary of the Invention

[0005] The purpose of this invention is to solve the sparse reward problem in automatic control systems using reinforcement learning methods, such as controlling a robotic arm to automatically identify and grasp objects, or enabling a point mass to navigate automatically. A reward optimization method and system for automatic control systems based on reinforcement learning are designed, enabling the agent to learn intelligent behaviors comparable to humans more quickly, improving the agent's learning efficiency and the final reward obtained. The invention mainly includes improvements to existing learning methods for automatic control systems and a reinforcement learning model based on Monte Carlo reinforcement.

[0006] The reward optimization method for automatic control systems based on reinforcement learning designed in this invention includes the following steps:

[0007] Step 1: Model the automatic control system and define the three elements of reinforcement learning: state, reward, and action;

[0008] Step 2: Construct the neural network actor and critic based on the actor-critic framework in reinforcement learning algorithms;

[0009] Step 3: Pre-train the critic network using existing expert demonstration data;

[0010] Step 4: Allow the automatic control system to perform physical operations, generating a large amount of (s,a,s′,r) interactive tuple data. Store this data in the playback buffer, where the current state is s, the action taken by the existing strategy is a, the reward obtained is r, and the environment reaches the next state s′.

[0011] Step 5: Obtain data from the playback buffer to train the neural network models actor and critic built under the actor-critic framework, as follows:

[0012] Each time a fixed amount of data is extracted, the action function value Q is calculated using the temporal difference method. TD The action function value Q was calculated using the Monte Carlo method. MC Then, a weighted average of the two action function values ​​is taken to obtain the newly defined action function value. The neural network model is trained using the newly defined action function values ​​until the model converges.

[0013] Step 6: Test and evaluate the model, including the average reward obtained, the variance of the average reward, and the number of rounds to converge.

[0014] Furthermore, in step 5, the model training employs a mini-batch training method, which improves the parallel processing capabilities of the graphics card.

[0015] Preferably, the gradient calculation algorithm used in step 5 is Adam, and the gradient propagation method during training is backpropagation. The training process is round-based, meaning that the reinforcement learning process ends when a round is completed.

[0016] Furthermore, the evaluation process in step 6 is as follows:

[0017] Average reward This metric reflects the agent's performance in completing tasks; a higher value indicates greater intelligence. Here, m represents the number of training cycles in which the training curve converges; and the variance of the average reward is... This metric reflects the stability of the agent's behavior. The convergence rounds are the number of rounds required for the agent to reach a stable state, which is observed from the reward curve and reflects the speed at which the agent learns.

[0018] Based on the same inventive concept, this solution also designs an electronic device, including: one or more processors; and a storage device for storing one or more programs;

[0019] When one or more programs are executed by the one or more processors, the one or more processors implement a reward optimization method for an automatic control system based on reinforcement learning.

[0020] Based on the same inventive concept, this solution also designs a computer-readable medium storing a computer program, which, when executed by a processor, implements a reward optimization method for an automatic control system based on reinforcement learning.

[0021] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0022] (1) This invention proposes a method that combines different weights of traditional temporal difference Q-values ​​and Monte Carlo Q-values. The combined result is then used to update the Q-value of the critic network. This method outperforms classical algorithms in terms of convergence speed and stable rewards in automatic control systems.

[0023] (2) The method proposed in this invention can be easily added to existing reinforcement learning algorithms that use the actor-critic framework and is compatible with most classic algorithms.

[0024] (3) The present invention tested and verified the proposed method in three different automatic control systems with sparse reward problems, and verified the advantages of the model from three different indicators: average reward, maximum reward, and number of convergence rounds. Attached Figure Description

[0025] Figure 1 This is a diagram of the overall system structure of the present invention.

[0026] Figure 2This provides simulation environments for three different automatic control systems. Detailed Implementation

[0027] To facilitate understanding and implementation of the present invention by those skilled in the art, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0028] Example 1

[0029] The structure diagram of the overall model of the reward optimization method for automatic control systems based on reinforcement learning designed in this invention is attached. Figure 1 As shown, it mainly includes the Environment, the Agent, and the three elements of reinforcement learning: State, Reward, and Action. The execution process involves the Agent and Environment interacting, adjusting the agent's behavior based on the returned Reward, thereby training the model and optimizing the actions of the particle and the robotic arm. It includes the following steps:

[0030] Step 1: This invention uses two automatic control systems, mass navigation and robotic arm, as examples for modeling, and defines the three elements of reinforcement learning: State, Reward, and Action.

[0031] The specific implementation includes the following sub-steps:

[0032] Step 1.1: The State definition includes the position of the particle and the position of the block that the robotic arm needs to grasp. For example... Figure 2 As shown in (a), the agent's goal is to start from the black starting position on the left, bypass the green obstacle in the middle, and reach the red target point on the right. In a two-dimensional plane, the position of the point mass is represented by P = [X, Y]. Similarly, Figure 2 In (b) and (c), the robotic arm is controlled to identify and grasp the blocks, respectively. Each scene has three blocks, so a 3×2 matrix is ​​used to represent the positions of the three blocks P = [P1, P2, P3].

[0033] Step 1.2: The definition of Action represents the actions that a point mass and a robotic arm can take. The action set for the point mass is: A = {N, S, W, E}, representing the point mass moving in the four cardinal directions (north, south, east, west). The action set for the robotic arm is: A = {F, B, L, R, U, D}, representing the robotic arm moving forward, backward, left, right, up, and down.

[0034] Step 1.3: The Reward is defined as the reward obtained by the Environment from the Agent, representing the quality of the current action. In the particle navigation environment, the reward is set as follows: if the particle collides with a green obstacle, a reward of -100 is obtained; if the particle does not collide, a reward of -1 is obtained; if the particle reaches the red dot, a reward of 0 is obtained. The reward value can be defined manually for easy differentiation. In the environment where the robotic arm identifies blocks, the reward is set as follows: if the robotic arm does not identify a blue block, a reward of -100 is obtained; if the robotic arm grasps a blue block, a reward of 0 is obtained. In the system where the robotic arm sequentially grasps blocks, the reward is set as follows: if the robotic arm makes no progress, a reward of -3 is obtained; if the robotic arm grasps a block, a reward of -1 is obtained; if the robotic arm grasps blocks sequentially, a reward of 0 is obtained.

[0035] Step 2: Construct the neural network actor and critic based on the actor-critic framework in reinforcement learning algorithms;

[0036] The specific implementation process is as follows: In reinforcement learning, actor-critic is a common framework that combines two neural networks: the actor and the critic. These two networks play different roles in reinforcement learning and are responsible for different tasks. The main task of the actor network is to output the action policy, that is, the probability distribution of choosing an action given a state. The actor network determines which actions the agent takes in the environment. The actor network can be any neural network with the function of outputting parameterized policies. In this embodiment, the actor network uses three fully connected layers. The first layer takes the state dimension as input, the third layer outputs the action space dimension, and the hidden layers in the middle each have 256 neurons. The actor network receives the current state of the environment as input and outputs the probability of each possible action. The output layer uses the softmax activation function to ensure that the output is a probability distribution, allowing the agent to choose between different actions. The task of the critic network is to evaluate the quality of the actions chosen by the actor and give a value estimate for each state-action pair. This value estimate reflects the expected reward of performing an action in the current state. The critic network employs any neural network capable of supervised learning. In this embodiment, the critic network uses three fully connected layers: the first layer takes the input as a sum of state and action dimensions, the third layer outputs a dimension of 1, and there are two ReLU activation units in between. The hidden layers each have 256 neurons. The critic network receives a given input and outputs a corresponding value. If the input is a state value, it evaluates the overall expected reward in the given state; if the input is both a state value and an action value, it evaluates the expected reward of performing a specific action in the given state. During training, the actor network generates actions through interaction with the environment and updates them based on the critic's feedback to increase the probability of selecting good actions. The critic network updates the value function parameters using an approximation method to more accurately estimate state or action values.

[0037] Step 3: Pre-train the critic network using existing expert demonstration data to enable the agent to exhibit preliminary intelligent behavior.

[0038] Before the agent interacts with the environment, it is pre-trained using existing expert demonstrations. These expert demonstrations utilize interaction data generated by human manipulation of the robotic arm to train it. For the task of recognizing blocks, this invention used 50 expert demonstrations for pre-training. For the task of sequentially grasping blocks, 500 expert demonstrations were used for pre-training. The pre-trained robotic arm exhibits some intelligent behavior, but since the provided expert demonstrations are suboptimal, its behavior is not yet optimal and requires further interaction with the environment for learning.

[0039] Step 4: Under the guidance of the trained actor network, the agent continuously interacts with the environment, generating a large amount of (s,a,s′,r) interaction tuple data, which is then stored in the playback buffer.

[0040] The pre-trained agent interacts with the environment. In a scenario with state s, the agent takes action a according to the existing policy and receives a reward r. At this point, the environment reaches the next state s′, and the agent continues to take actions according to the policy until the end of the round. During this process, a large amount of interaction tuple data (s,a,s′,r) is obtained. This data is stored in a revisit buffer for use in training the neural network.

[0041] Step 5: Retrieve data from the playback buffer to train the model used in this specification, and save it;

[0042] To improve the parallelism of the graphics card, a mini-batch training approach is used, where multiple datasets are bundled together for training. Each mini-batch of data is prepared and fed into the model for training. The gradient calculation algorithm used is Adam, and the gradient is propagated backwards. The training process is round-based, meaning the reinforcement learning process ends when a round is completed. The training data needs to be trained for 500,000 rounds until the model reaches a convergent state where the loss no longer changes.

[0043] The specific training process is as follows:

[0044] Step 5.1: Initialize the critic network Q(s,a|θ) Q ) parameter θ Q actor network π(s,a|θ) π ) parameter θ π Target networks Q′ and π′.

[0045] Step 5.2: Set the number of rounds (episode) to 0, 1, ..., N;

[0046] Step 5.3: Set the time step to 0, 1, ..., T;

[0047] Step 5.4: Set the batch size to M;

[0048] Step 5.5: Retrieve a batch of data of size M from the playback buffer, and calculate the Q value Q using the time-difference method. TD =r+γQ θ′ (s t+1 ,a t+1 Here, γ represents the immediate reward gained from performing an action in the current state, and γ is the discount rate, representing the discounted present value of future rewards in the current state. The Monte Carlo method is used to calculate Q. Here, parameter i represents the i-th step, and parameter j represents the j-th trajectory. Then, a weighted average of the Q values ​​calculated by these two methods is used to obtain a newly defined Q value. Where the parameter n represents the training up to the nth round. According to Q... CAAC Train the critic network, where the loss is L = 1 / M∑ i (Q θ (s i ,a i ),Q CAAC (s i ,a i In this embodiment, M is set to 256, but other values ​​are also possible, such as 64 or 32, depending on the size of the graphics card's video memory.

[0049] Step 5.6: Train the actor network using the policy gradient method;

[0050] Step 5.7: Update network parameters;

[0051] Step 5.8: Return to step 5.2;

[0052] Step 5.9: After all training rounds are completed, save the model.

[0053] Step 6: Define evaluation metrics such as the average reward obtained, the variance of the average reward, and the number of rounds to converge to test and evaluate the model.

[0054] The specific implementation process is as follows: average reward This metric reflects the agent's performance in completing the task; a higher value indicates greater intelligence. Here, m is the number of training epochs in which the training curve converges. The variance of the average reward... This metric reflects the stability of the agent's behavior. The convergence rounds are the number of rounds required for the agent to reach a stable state, which is observed from the reward curve and reflects the speed at which the agent learns.

[0055] This invention provides a solution to the sparse reward problem in automatic control systems based on reinforcement learning. The model of this invention extends the traditional actor-critic model by incorporating an expert demonstration pre-training mechanism, which accelerates the learning efficiency of the agent. Furthermore, it assigns different weights to two different methods for calculating Q-values, enabling better utilization of reward information obtained from the environment to guide the agent's learning. In this embodiment, the expert demonstration data is sourced from an open-source human operation dataset available online.

[0056] This invention enables intelligent agents in automatic control systems that have difficulty or sparse rewards to make fuller use of existing reward information. By using reinforcement learning models to control point navigation and robotic arm object recognition and grasping, the efficiency of intelligent agents in completing tasks is improved, and they are also more adaptable to complex and ever-changing environments.

[0057] Example 2

[0058] Based on the same inventive concept, the present invention also provides an electronic device, including one or more processors; a storage device for storing one or more programs; and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in Embodiment 1.

[0059] Since the device described in Embodiment 3 of this invention is the electronic device used to implement the reward optimization method of the automatic control system based on reinforcement learning in Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this electronic device based on the method described in Embodiment 1 of this invention, and therefore will not be described again here. All electronic devices used in any method of this invention fall within the scope of protection of this invention.

[0060] Example 3

[0061] Based on the same inventive concept, the present invention also provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method described in Embodiment 1.

[0062] Since the device described in Embodiment 3 of this invention is a computer-readable medium used to implement the reward optimization method of the automatic control system based on reinforcement learning in Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this electronic device based on the method described in Embodiment 1 of this invention, and therefore will not be described again here. All electronic devices used in any method of this invention fall within the scope of protection of this invention.

[0063] It should be understood that any parts not described in detail in this specification belong to the prior art.

[0064] It should be understood that the above description of the preferred embodiments is quite detailed, but it should not be considered as a limitation on the scope of protection of this invention. Those skilled in the art, under the guidance of this invention, can make substitutions or modifications without departing from the scope of protection of the claims of this invention, and all such substitutions or modifications fall within the scope of protection of this invention. The scope of protection of this invention should be determined by the appended claims.

Claims

1. A reward optimization method for automatic control systems based on reinforcement learning, characterized in that, Includes the following steps: Step 1: Model the automatic control system and define the three elements of reinforcement learning: state, reward, and action; Step 2: Construct the neural network actor and critic based on the actor-critic framework in reinforcement learning algorithms; Step 3: Pre-train the critic network using existing expert demonstration data; Step 4: Allow the automatic control system to perform physical operations, generating a large amount of (s,a,s′,r) interactive tuple data. Store this data in the playback buffer, where the current state is s, the action taken by the existing strategy is a, the reward obtained is r, and the environment reaches the next state s′. Step 5: Obtain data from the playback buffer to train the neural network models actor and critic built under the actor-critic framework, as follows: Each time a fixed amount of data is extracted, the action function value Q is calculated using the temporal difference method. TD The action function value Q was calculated using the Monte Carlo method. MC Then, a weighted average of the two action function values ​​is taken to obtain the newly defined action function value. The neural network model is trained using the newly defined action function values ​​until the model converges. Step 6: Test and evaluate the model, including the average reward obtained, the variance of the average reward, and the number of rounds to converge.

2. The reward optimization method for automatic control systems based on reinforcement learning according to claim 1, characterized in that: The neural network actor employs a neural network with any output parameterization strategy.

3. The reward optimization method for automatic control systems based on reinforcement learning according to claim 1, characterized in that: The neural network critic can be any neural network capable of supervised learning.

4. The reward optimization method for automatic control systems based on reinforcement learning according to claim 1, characterized in that: In step 5, the model training uses mini-batch training.

5. The reward optimization method for automatic control systems based on reinforcement learning according to claim 1, characterized in that: The gradient calculation algorithm used in step 5 is Adam, and the gradient propagation method during training is backpropagation.

6. The reward optimization method for automatic control systems based on reinforcement learning according to claim 1, characterized in that: The model training process in step 5 is round-based, meaning that the reinforcement learning process ends when a round is completed.

7. The reward optimization method for automatic control systems based on reinforcement learning according to claim 1, characterized in that: Step 6, the evaluation process, is as follows: Average reward This metric reflects the level of excellence of the agent in completing the task; the higher the metric, the more intelligent the agent. Here, m is the number of rounds in which the training curve tends to converge. Variance of average reward This metric reflects the stability of the agent's behavior; The convergence rounds are the number of rounds required for an agent to reach a stable state. They are derived from the reward curve and reflect the speed at which the agent learns.

8. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs; When one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-7.

9. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by the processor, it implements the method of any one of claims 1-7.

Images