Deep reinforcement learning generalization evaluation method and application

Abstract

The invention belongs to the technical field of reinforcement learning, and particularly relates to a deep reinforcement learning generalization evaluation method and application, and the method comprises the steps: setting a reinforcement learning algorithm environment, optimizing a hyper-parameter through a total award average value of all rounds and a total award average value of a final round, and then calculating an improved progressive verification loss, training an agent to collect reward data; calculating the improved progressive check loss of each algorithm in each environment by using the reward data; according to the method, a Z test is adopted to carry out significant winning and losing statistics, winning and losing difference quantitative calculation is carried out, the winning and losing difference between two algorithms is defined, and the generalization performance of reinforcement learning is evaluated by using the winning and losing difference. And the obvious winning and losing performance of each algorithm for other algorithms can be quantitatively and intuitively determined.

Description

Technical Field

[0001] The present invention belongs to the technical field of reinforcement learning, and specifically relates to a method and application for evaluating the generalization of deep reinforcement learning. Background Art

[0002] Artificial General Intelligence (AGI), as an artificial intelligence with strong generalization and migration capabilities, has become an important development trend of the times. It should have the ability to achieve different goals, solve different problems, and perform different tasks in various contexts and environments. Among them, reinforcement learning is one of the key technologies for building AGI. The intelligent agent can optimize the behavior strategy, summarize patterns and laws through reward feedback from interaction with the environment, thereby surpassing human operation efficiency in many fields such as dynamic decision-making. However, due to the complexity and changeability of the real environment, the reinforcement learning intelligent agent that runs efficiently in the experimental environment is sometimes not applicable to the new environment. In order to ensure the wider and more efficient application of reinforcement learning in the future, and its promotion of the development of AGI technology, the generalization of the reinforcement learning method, that is, the ability of the intelligent agent to obtain rewards in different environments when the settings of the reinforcement learning agent remain unchanged, is crucial.

[0003] In order to improve the generalization, how to reasonably quantify and evaluate reinforcement learning, especially deep reinforcement learning, is an indispensable requirement. But generally speaking, since the main goal of reinforcement learning is to allow the intelligent agent to accumulate more rewards, the design idea is to interact with the environment in a certain state and adjust the selection of better actions based on the feedback reward. Therefore, most of the current research focuses on how to design algorithms to accumulate rewards more efficiently. Most mainstream and mature reinforcement learning evaluation indicators are directly related to measuring rewards, including total rewards, average rewards, human standardized scores, etc. However, there are relatively few studies on generalization evaluation.

[0004] With the development of artificial intelligence, in addition to considering the main goal of reinforcement learning, some studies have also focused on improving and evaluating the generalization of reinforcement learning algorithms. However, most of the existing generalization evaluation indicators cannot accurately and specifically quantify the generalization, or are not suitable for algorithm evaluation in general environments. Therefore, providing a complete reinforcement learning generalization evaluation framework is an urgent problem to be solved. Summary of the invention

[0005] To solve the above problems in the prior art, the present invention proposes a new method for evaluating the generalization of deep reinforcement learning, and makes improvements to reinforcement learning, including using an improved asymptotic verification loss calculation function to calculate the asymptotic verification loss, wherein the reward rt obtained by running the algorithm is normalized by the maximum reward rmax and the minimum reward rmin of the environment, thereby obtaining the cost term ci; the calculation result of the loss calculation function is input into the Z test to pick out the environments in which the algorithms have significant differences, to calculate the win-loss difference of different algorithms, and the win-loss difference is used as the input of the heat map, then the algorithm that significantly outperforms other algorithms has better generalization.

[0006] The complete technical solution of the present invention includes:

[0007] A method for evaluating the generalization of deep reinforcement learning, comprising the following steps:

[0008] (1) For different reinforcement learning algorithms, set up reinforcement learning algorithm environments. Select t tasks for each algorithm, and use z random seeds for each task, resulting in a total of k = tz environments.

[0009] (2) The average reward value of all rounds r avg，all And the total final round reward average r avg，final Optimize the hyperparameters of different reinforcement learning algorithms. By adjusting the hyperparameters, r avg，all and r avg，final Reach the maximum value;

[0010] (3) Calculate the improved asymptotic validation loss, including training agents to collect reward data; use the reward data to calculate the improved asymptotic validation loss PV for each algorithm in each environment r ;

[0011]

[0012] In the formula, n represents the fixed number of time steps for each training, c i As the consideration, s i is the state sequence in the i-th training, a i yes

[0013] In the action sequence of the i-th training, the cost term ci is calculated by the following reverse normalization formula:

[0014]

[0015] Where T is the total time steps of a task, r t (s t , a t ) is the state s at time step t t Take action a tThe reward value r max is the maximum reward of the environment, r min It is the minimum reward of the environment.

[0016] (4) Calculate the results of each pair of algorithms running in the same environment;

[0017] (5) Quantify the win-loss margin and evaluate the generalization performance of the reinforcement learning algorithm based on the results of the quantified win-loss margin calculation.

[0018] Furthermore, the reinforcement learning algorithm is a deep neural network.

[0019] Furthermore, in step (2), the total reward average of all rounds is r avg，all for:

[0020]

[0021] Where N represents the total number of all training time steps, N = nk, r i Represents the reward obtained at the i-th time step during training.

[0022] Furthermore, in step (2), the total final round reward average r avg，final for:

[0023]

[0024] Where: K is the number of environments for each reinforcement learning algorithm, r ni is the reward obtained at the ni-th time step during training.

[0025] Furthermore, the hyperparameters include a learning rate, an exploration rate, and the number of neurons in a hidden layer of the neural network.

[0026] Furthermore, in step (4), the win-loss calculation is the improved progressive verification loss p r,a and p r,b Compare according to the following formula. If the following formula is satisfied, algorithm a significantly wins over algorithm b, otherwise it loses:

[0027]

[0028] In the formula, p r,a is the improved asymptotic verification loss of algorithm a, p r,b is the improved asymptotic verification loss of algorithm b.

[0029] Furthermore, in step (4), a Z test is used to perform significant win-loss statistics.

[0030] Furthermore, in step (5), the method for quantitatively calculating the win-loss difference is as follows: based on the win-loss results obtained by each two algorithms in each environment, all the win-loss differences of each algorithm relative to other algorithms are counted, that is, the number of wins minus the number of losses, to obtain the win-loss difference: D WL =WL, where W represents the number of wins relative to other algorithms, L represents the number of losses relative to other algorithms, and D represents the number of losses relative to other algorithms. WL Larger algorithms have better reinforcement learning generalization performance.

[0031] The present invention aims at the problem of evaluating the generalization performance of reinforcement learning and proposes a new method for evaluating the generalization performance of reinforcement learning. Compared with the prior art, the present invention has the following advantages:

[0032] (1) A method for environment configuration is proposed, so that the tasks in the environment can cover various configurations such as the reward interval of the algorithm and continuous / discrete actions. Based on this environment configuration, grid search is used to optimize the hyperparameters.

[0033] (2) An improved progressive verification loss calculation function is adopted;

[0034] The traditional PV loss function is based on the context setting and lacks the state that is crucial for reinforcement learning. By adopting the improved progressive verification loss calculation function, the state can be better taken into account, and the cost term c in each round i Both reward-based states and action sequences are considered simultaneously, and their values ​​are normalized to the interval (0, 1), making subsequent Z test calculations and significant win-loss calculations more convenient;

[0035] (3) For each algorithm, a significant win-loss calculation statistic based on the improved asymptotic verification loss calculation function, namely the Z test, was used.

[0036] (4) Finally, the performance of each algorithm in all environments compared with other algorithms was summarized and scored, namely D WL , and putting the quantified results into a heat map, this method can quantitatively and intuitively determine the significant advantages and disadvantages of each algorithm over other algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is an overall flow chart of the method for evaluating the generalization of deep reinforcement learning of the present invention.

[0038] Figure 2 This is a reward curve diagram of four algorithms in the same environment of the present invention.

[0039] Figure 3 The following is a graph showing the results of running the four algorithms of the present invention in 36 environments.

[0040] Figure 4A graph showing the quantitative calculation of significant win-loss differences. DETAILED DESCRIPTION

[0041] The present invention is described in detail below in conjunction with the embodiments and drawings, but it should be understood that the embodiments and drawings are only used to exemplify the present invention and do not constitute any limitation on the protection scope of the present invention. All reasonable changes and combinations within the scope of the inventive concept of the present invention fall within the protection scope of the present invention.

[0042] like Figure 1 As shown, in response to the demand for reinforcement learning generalization evaluation, the present invention provides a new evaluation method, which sets a standard environment, calculates the rewards obtained by the algorithm by improving the progressive verification loss function, and determines the victory or defeat through the significant victory or defeat calculation formula, and statistically calculates the victory or defeat difference to evaluate the generalization performance of reinforcement learning. This method establishes a clear evaluation framework, and the comparison of different algorithms is clear and definite. The above method and framework enable the evaluation of the generalization performance of reinforcement learning to be quantitatively compared, which is convenient for improving various algorithms.

[0043] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below.

[0044] First, this implementation takes four standard deep reinforcement learning algorithms as examples. These four algorithms all integrate deep neural networks using Q-Learning, including:

[0045] DQN: This algorithm integrates Q-Learning with deep neural networks to directly map the state to the estimated value of each action;

[0046] Double DQN: A variant of DQN that uses two networks to reduce overestimation of action values;

[0047] DuelingDQN: A variant of DQN that separates the estimated value and action in a state, making it more efficient to learn states that do not require specific actions.

[0048] NoisyDQN: A variant of DQN that introduces noise to encourage exploration;

[0049] (1) A standard task environment is established for the four algorithms. Six classic deep learning RL tasks are used in the environment of this embodiment, and six random numbers are used as seeds, so there are 6*6=36 environments. The tasks use six subtasks of OpenAI's Gymnasium, including Pendulum, Mountain Car Continuous, MountainCar, Cart Pole, Acrobot, and Inverted Pendulum; three of the tasks are continuous action tasks (i.e., the action is to simulate continuous action), and three tasks are discrete action tasks; this is conducive to covering various situations and improving the calculation of the progressive verification loss PVr.

[0050] (2) In order to maximize the algorithm’s performance, grid search is used to optimize hyperparameters. During grid search, the average reward value of all rounds is r avg,all And the total final round reward average r avg,final All are recorded, as shown in the table. It can be seen that the optimization of hyperparameters can improve the consistency of the algorithm.

[0051] Table 1 r after hyperparameter optimization avg,all and r avg,final

[0052]

[0053] like Figure 2 As shown, in the reward curves of the four algorithms under the same environment, it can be seen that the rewards of the four algorithms are steadily increasing, indicating that the optimization setting of the environmental hyperparameters of this implementation method is correct.

[0054] (3) Calculate the improved progressive validation loss PVr for each algorithm in 36 environments; including training agents to collect reward data; using reward data to calculate the improved progressive validation loss of each algorithm in each environment;

[0055]

[0056] Among them, s i and a i are the state sequence and action in the i-th training respectively; the cost term c i yes:

[0057]

[0058] Among them, r max is the maximum reward of the environment, r min It is the minimum reward of the environment and is used for normalization.

[0059] (4) Use the Z test to perform significant win-loss statistics; specifically, the win-loss calculation is performed for the results of each two algorithms running in the same environment, and the improved asymptotic verification loss p is calculated. r,a and p r,b Compare according to the following formula. If the following formula is satisfied, a significantly wins over b, otherwise it loses:

[0060]

[0061] The results of running the four algorithms in 36 environments are as follows Figure 3 As shown in Figure 2, it shows the significant win-loss results of the improved progressive verification loss PVr among the four algorithms, and the red points are the significant win-loss points.

[0062] (5) Quantitative calculation of significant win-loss difference. The specific algorithm is the same as described in the previous steps.

[0063] The calculation results are as follows Figure 4 As shown, the results show that, according to the method described in the present invention, in all selected tasks, the generalization performance of the DuelingDQN algorithm is better than all the other three algorithms, and its significant winning and losing margins against the other three algorithms are 6, 4, and 14 respectively.

[0064] In summary, the method of the present invention establishes a good evaluation tool and evaluation system for the evaluation of the generalization performance of reinforcement learning. The present invention also discloses an application of a method for evaluating the generalization of deep reinforcement learning. For reinforcement learning algorithms, the method for evaluating the generalization of deep reinforcement learning is used to evaluate their generalization, and the algorithm with the best generalization is selected for application in industrial fields, including autonomous driving, robots, semiconductors, large language models, medical health, energy, telecommunications, aerospace, network security, environmental protection, chemical industry, pharmaceuticals, etc.

[0065] The following is an application of the method of evaluating the generalization of deep reinforcement learning in the present invention in the obstacle avoidance control of unmanned aerial vehicles in the aviation field, specifically including:

[0066] (1) Define the task and environment

[0067] The task is to train the drone to fly autonomously in an unknown environment, avoid obstacles and reach the target point.

[0068] The environment is a simulated environment or a real environment, such as a Gazebo simulation environment;

[0069] The state space of the UAV includes: the position, velocity, acceleration, and attitude angle of the UAV.

[0070] The drone's motion space is the control input, which includes the drone's lift, pitch, yaw, and roll. Pitch is the movement of the drone rotating around its lateral axis. When the drone tilts forward, it is called positive pitch; when it tilts backward, it is called negative pitch. Yaw is the movement of the drone rotating around its vertical axis. When the drone rotates to the left or right, it is a yaw motion. Roll is the movement of the drone rotating around its longitudinal axis. When the drone tilts to one side, it is called a roll.

[0071] Define a reward function to encourage the drone to approach the target and punish collisions or deviations from the route. Including: approaching the target point is a positive reward. Collision with obstacles is a negative reward.

[0072] (2) Selecting a Deep Reinforcement Learning Algorithm

[0073] Based on the above, the DuelingDQN algorithm with the best generalization ability is selected.

[0074] (3) Design the neural network architecture, including:

[0075] Input layer: receives the status information of the drone.

[0076] Hidden layer: Use multiple layers of fully connected layers, and each layer can use the ReLU activation function.

[0077] Output layer: Outputs the Q value corresponding to each action in the action space.

[0078] (4) Set hyperparameters, including:

[0079] Learning rate: controls how fast the model is updated.

[0080] Discount factor γ: controls the weight of future rewards.

[0081] ε-greedy strategy: ε represents the probability of taking random actions, which is used to balance exploration and exploitation.

[0082] Experience replay buffer capacity: the size of the buffer used to store past experiences.

[0083] Batch size: The number of samples drawn from the experience replay buffer each time.

[0084] Target network update frequency: The target network is used to calculate the expected reward, and its weight is periodically updated to the main network weight.

[0085] (5) Model training, including:

[0086] 1) Initialization: Create the Q network and the target network, and initialize the experience replay buffer.

[0087] 2) Training loop, including:

[0088] Observe the current status.

[0089] Actions are selected according to the ε-greedy strategy.

[0090] Perform actions and observe new states and rewards.

[0091] Store the experience (current state, action, reward, new state) in the experience replay buffer.

[0092] If the experience replay buffer is large enough, a batch of experiences is randomly selected from it for training.

[0093] Use the extracted experience to update the weights of the Q network.

[0094] Update the weights of the target network periodically.

[0095] (6) Use the trained model to perform obstacle avoidance control on the UAV.

[0096] The above applications are only some implementation methods of the present application. For those skilled in the art, several modifications and improvements can be made without departing from the creative concept of the present application, and these all belong to the protection scope of the present application.

Claims

1. A method for evaluating the generalization of deep reinforcement learning, characterized in that The steps include: (1) For different reinforcement learning algorithms, set up reinforcement learning algorithm environments. Select t tasks for each algorithm, and use z random seeds for each task, resulting in a total of k = tz environments. (2) The average reward r of all rounds avg,all And the total final round reward average r avg,final Optimize the hyperparameters of different reinforcement learning algorithms. By adjusting the hyperparameters, r avg,all and r avg,final Reach the maximum value; (3) Calculate the improved asymptotic validation loss, including training agents to collect reward data; use the reward data to calculate the improved asymptotic validation loss PV for each algorithm in each environment r ; In the formula, n represents the fixed number of time steps for each training, c i As the consideration, s i is the state sequence in the i-th training, a i is the action sequence in the i-th training, and the cost term c i It is calculated by the following reverse normalization formula: Where T is the total time steps of a task, r t (s t ,a t ) is the state s at time step t t Take action a t The reward value r max is the maximum reward of the environment, r min It is the minimum reward of the environment; (4) Calculate the results of each pair of algorithms running in the same environment; (5) Quantify the win-loss margin and evaluate the generalization performance of the reinforcement learning algorithm based on the results of the quantified win-loss margin calculation.

2. A method for evaluating the generalization of deep reinforcement learning according to claim 1, characterized in that: The reinforcement learning algorithm includes a deep neural network.

3. A method for evaluating the generalization of deep reinforcement learning according to claim 2, characterized in that: In step (2), the total reward average of all rounds is r avg,all for: Where N represents the total number of all training time steps, N = nk, r i Represents the reward obtained at the i-th time step during training.

4. A method for evaluating the generalization of deep reinforcement learning according to claim 3, characterized in that: In step (2), the total final round reward average r avg,final for: Where: K is the number of environments for each reinforcement learning algorithm, r ni is the reward obtained at the ni-th time step during training.

5. A method for evaluating the generalization of deep reinforcement learning according to claim 4, characterized in that: The hyperparameters include learning rate, exploration rate, and the number of neurons in the hidden layer of the neural network.

6. A method for evaluating the generalization of deep reinforcement learning according to claim 5, characterized in that: In step (4), the win-loss calculation is the improved asymptotic verification loss p r,a and p r,b Compare according to the following formula. If the following formula is satisfied, algorithm a significantly wins over algorithm b, otherwise it loses: In the formula, p r,a is the improved asymptotic verification loss of algorithm a, p r,b is the improved asymptotic verification loss of algorithm b.

7. A method for evaluating the generalization of deep reinforcement learning according to claim 6, characterized in that: In step (4), the Z test is used to perform significant win-loss statistics.

8. A method for evaluating the generalization of deep reinforcement learning according to claim 7, characterized in that: In step (5), the quantitative calculation method of the win-loss difference is: according to the win-loss results obtained by each two algorithms in each environment, all the win-loss differences of each algorithm relative to other algorithms are counted, that is, the number of wins minus the number of losses, and the win-loss difference is obtained: D WL =WL, where W represents the number of wins relative to other algorithms, L represents the number of losses relative to other algorithms, and D represents the number of losses relative to other algorithms. WL Larger algorithms have better reinforcement learning generalization performance.

9. Application of a method for evaluating the generalization of deep reinforcement learning, characterized in that For the reinforcement learning algorithm, the method for evaluating the generalization of deep reinforcement learning as described in any one of claims 1 to 7 is used to evaluate its generalization, and the algorithm with the best generalization is selected for application in industrial fields, including autonomous driving, robots, semiconductors, large language models, medical health, energy, telecommunications, aerospace, network security, environmental protection, chemicals, pharmaceuticals, etc.

Images