An agent reinforcement learning method and device for achieving exploration and exploitation balance

Abstract

The application provides an agent reinforcement learning method and device for realizing exploration and utilization balance, comprising the following steps: a first Bellman operator is constructed for learning and utilizing successful experience in an experience replay pool; a second Bellman operator is constructed for exploring unknown states; an exploration and utilization balance Bellman operator is generated according to the first Bellman operator and the second Bellman operator; the action actor-critic algorithm is improved by using the exploration and utilization balance Bellman operator; and the agent is subjected to reinforcement learning based on the improved action actor-critic algorithm. The application solves the exploration and utilization balance problem in the action actor-critic algorithm more effectively by designing the exploration and utilization balance Bellman operator, thereby improving the sample efficiency and generalization performance of the algorithm, and providing a feasible solution for real scene application.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning, and particularly relates to an intelligent agent reinforcement learning method and device for realizing exploration-exploitation balance. BACKGROUND

[0002] Reinforcement Learning (RL) is an important branch of artificial intelligence, and has a wide range of applications in tasks that require intelligent decision-making. Unlike traditional supervised learning, reinforcement learning focuses on how to choose the optimal strategy through continuous interaction with the environment, trial and error, and learning in the process of interaction between an intelligent agent and the environment. In reinforcement learning, the off-policy actor-critic algorithm has become a widely used algorithm, which mainly outputs actions using an actor network and evaluates the quality of the actions using a critic network to guide the learning of the network. However, this algorithm still has some problems in application, the most significant of which is the exploration-exploitation balance problem.

[0003] Currently, to solve the above problems, researchers have proposed a variety of exploration algorithms to solve the above problems, including Vallian Bellman operator, bootstrapped DQN, count-based exploration, and optimistic initialization. However, these algorithms have the problem of poor performance, specifically: Vallian Bellman operator has overestimation or underestimation when estimating Q values, resulting in poor sample efficiency of the algorithm. Bootstrapped DQN, count-based exploration, and optimistic initialization require additional assumptions, such as specific environmental assumptions or heuristic algorithms, which are difficult to generalize in different scenarios. In addition, these algorithms also have the problem of poor real scene application ability, specifically: existing off-policy actor-critic algorithms are often trained and tested in idealized scenarios, while the complexity and uncertainty of actual application scenarios are often much higher than ideal conditions, which leads to poor performance of these algorithms in actual applications, especially when applied in real scenarios, there are often significant generalization problems.

[0004] Therefore, a more efficient and universal exploration-exploitation balance algorithm is needed to improve the sample efficiency and generalization performance of the off-policy actor-critic algorithm. SUMMARY

[0005] The application provides an agent reinforcement learning method and device for realizing exploration and utilization balance, which is used for solving the problems of poor performance and poor real scene application ability of existing exploration and utilization balance algorithms, and more effectively solves the exploration and utilization balance problem in the actor-critic algorithm by designing a Bellman operator (BEE operator) for realizing exploration and utilization balance, thereby improving the sample efficiency and generalization performance of the algorithm, and providing a feasible solution for real scene application.

[0006] In a first aspect, the application provides an agent reinforcement learning method for realizing exploration and utilization balance, which comprises:

[0007] constructing a first Bellman operator for learning and utilizing successful experience in an experience replay pool;

[0008] constructing a second Bellman operator for exploring unknown states;

[0009] generating a Bellman operator for realizing exploration and utilization balance according to the first Bellman operator and the second Bellman operator;

[0010] improving an actor-critic algorithm by using the Bellman operator for realizing exploration and utilization balance;

[0011] performing reinforcement learning on an agent based on the improved actor-critic algorithm.

[0012] The agent reinforcement learning method for realizing exploration and utilization balance provided by the application records state transition information “current state-action-reward-next state” in the experience replay pool; and the expression of the first Bellman operator is:

[0013]

[0014] wherein r(s, a) is the reward of the agent performing the action a in the state s, and γ is a discount factor, is the action set of the agent, and μ(a'|s')>0 represents the probability distribution information of the agent performing the action a' in the state s' in the experience replay pool, p(s'|s, a) is the transition probability of the agent performing the action a in the state s to cause the state to transition to s', and Q(s', a') is the value function of the agent performing a' in the state s', is the value function of the agent performing the action a in the state s calculated by using the first Bellman operator. The agent reinforcement learning method for realizing exploration and utilization balance provided by the application comprises:

[0015] construct an exploration item w(s', a'|pi); wherein the w(s', a'|pi) represents an entropy value of the agent performing the action a' in the state s' under the current policy pi, and the value is negatively correlated with the historical number of times of performing the action a' in the state s';

[0016] improve the traditional Bellman operator based on the exploration item to obtain the second Bellman operator.

[0017] According to the agent reinforcement learning method for realizing exploration and utilization balance provided by the application, the expression of the second Bellman operator is:

[0018]

[0019] wherein r(s, a) is the reward of the agent performing the action a in the state s, gamma is a depreciation factor, p(s'|s, a) is the transition probability of the agent performing the action a in the state s to cause the state to transition to s', pi(a'|s') is the probability distribution of a' corresponding to the given state s' under the current policy pi, Q π (s', a') is the value function of the agent performing the action a' in the state s' under the current policy pi, is the value function of the agent performing the action a in the state s calculated by using the second Bellman operator.

[0020] According to the agent reinforcement learning method for realizing exploration and utilization balance provided by the application, the exploration and utilization balanced Bellman operator is a weighted sum of the first Bellman operator and the second Bellman operator, and the expression is:

[0021]

[0022] In the above formula, lambda is a weight parameter, is the value function of the agent performing the action a in the state s calculated by using the first Bellman operator, is the value function of the agent performing the action a in the state s calculated by using the second Bellman operator, is the value function of the agent performing the action a in the state s calculated by using the exploration and utilization balanced Bellman operator;

[0023] The expression of the improved actor-critic algorithm is:

[0024] Actor:

[0025] Critic:

[0026] In the above formula, is the value function of the agent performing the action a in the state s under the current policy pi kand a past policy sequence μ k a value function of the agent performing an action a in a state s, μ k k-1 ω(s, a | π k ) is an entropy value of the agent performing an action a in a state s under a current policy π k π k+1 is a policy of the agent in the k+1th iteration, is a value function of the agent performing an action a in a state s calculated by the Bellman operator of the exploration and exploitation balance in the kth iteration, Q(s, a) is a value function of the agent performing an action a in a state s determined by the critic network in the kth iteration, Q φ is the critic network. According to the agent reinforcement learning method for realizing the exploration and exploitation balance provided by the present application, after the current policy π k is obtained by decision, the sample formed by the corresponding state transition information of the current policy π k is regressed into the experience regression pool.

[0027] According to the agent reinforcement learning method for realizing the exploration and exploitation balance provided by the present application, the agent is reinforced by learning based on the improved actor-critic algorithm, which comprises the following steps.

[0028] For any application scenario of the agent, the current state of the agent in the application scenario and the current experience replay pool are obtained.

[0029] The current state of the agent is iteratively optimized and learned by using the current experience replay pool and the improved actor-critic algorithm, so as to obtain the corresponding optimal policy.

[0030] The any application scenario comprises a robot control application scenario, an automatic driving application scenario and an industrial power generation control application scenario.

[0031] In a second aspect, the present application provides an agent reinforcement learning device for realizing the exploration and exploitation balance, which comprises the following.

[0032] A first construction module is configured to construct a first Bellman operator for learning and utilizing the successful experience in the experience replay pool.

[0033] A second construction module is configured to construct a second Bellman operator for exploring unknown states.

[0034] A generation module is configured to generate a Bellman operator of the exploration and exploitation balance according to the first Bellman operator and the second Bellman operator.

[0035] ​An improvement module is configured to improve the actor-critic algorithm by using the Bellman operator balancing exploration and exploitation;

[0036] A reinforcement learning module is configured to perform reinforcement learning on the agent based on the improved actor-critic algorithm.

[0037] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the agent reinforcement learning method balancing exploration and exploitation according to the first aspect when executing the program.

[0038] In a fourth aspect, the present application provides a non-transitory computer readable storage medium, which stores a computer program, wherein the computer program is executable on a processor to implement the agent reinforcement learning method balancing exploration and exploitation according to the first aspect.

[0039] The agent reinforcement learning method and device balancing exploration and exploitation provided by the present application construct a first Bellman exploitation operator capable of using successful experience in historical experience and a second Bellman exploration operator capable of exploring unknown states, and then construct a BEE operator from the Bellman exploitation operator and the Bellman exploration operator. The BEE operator improves the balance between exploration of unknown states and use of historical experience, and has the advantages of high sample efficiency, strong generalization performance, strong versatility, easy implementation, and strong real scene application capability compared with the prior art. When the BEE operator is used in a reinforcement learning method using an off-policy actor-critic algorithm, the policy learning effect of the reinforcement learning can be effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0041] Figure 1 is a flowchart of the agent reinforcement learning method balancing exploration and exploitation provided by the present application;

[0042] Figure 2This is a schematic diagram of the structure of the intelligent agent reinforcement learning device that achieves a balance between exploration and utilization provided by the present invention;

[0043] Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention;

[0044] Figure label:

[0045] 310: Processor; 320: Communication interface; 330: Memory; 340: Communication bus. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0047] The technical terms involved in this invention are explained as follows:

[0048] Reinforcement Learning: Reinforcement learning is a machine learning method that learns optimal policies through continuous interaction between an agent and its environment. Its basic idea is that the agent learns optimal policies based on its current state s. t Choose an action a t And perform the action in the environment to receive a reward r. t and the next state s t+1 Intelligent beings gradually develop an expectation of cumulative rewards, Q(s). t ,a t ), by selecting the action 'a' that yields the maximum reward. t =argmax a Q(s t a) thereby learning the strategy that yields the maximum reward. Where γ is the depreciation factor, r t Let t be the reward at time t. The core of reinforcement learning lies in the process of learning the optimal policy.

[0049] The exploration-exploitation dilemma: An agent needs to maximize cumulative rewards by utilizing existing experience while exploring unknown states and action spaces. Specifically, in the exploration-exploitation dilemma, an agent needs to balance the two objectives of exploration and exploitation. Exploration refers to the agent's random exploration in unknown states and action spaces to discover new and useful information, while exploitation refers to using existing experience and knowledge to maximize cumulative rewards.

[0050] Bellman operator: The Bellman operator is a widely used operator in reinforcement learning for updating the state-value function or state-action-value function. Proposed by Richard Bellman, a renowned scholar in reinforcement learning, the Bellman operator's main idea is to update the value function of the current state (or state-action pair) to a weighted average of the current reward and the value function of the next state (or state-action pair). In value-function-based reinforcement learning algorithms, the Bellman operator is a crucial operation used to continuously update the value function, making it closer to the true value, thereby obtaining a better policy. For the state-action-value function Q(s,a), the Bellman operator is defined as follows:

[0051]

[0052] Where p(s′|s,a) represents the state transition probability. This represents the value of the action with the highest value among all possible actions taken in state s′.

[0053] The Off-policy Actor-critic (APA-critic) reinforcement learning algorithm is a widely used method. It's based on value and policy functions, learning by simultaneously updating both the actor and critic networks. The actor network outputs an action for the current state, while the critic network estimates the value of performing that action in that state. A key feature of this method is its ability to learn from previously collected data, improving sample utilization efficiency. The goal of the APA-critic reinforcement learning method is to maximize cumulative reward. The actor network learns to output the optimal action, while the critic network learns to predict the value of performing that action in the current state. Both networks learn by updating the policy and value functions.

[0054] The following is combined with Figures 1-3 The present invention describes an agent-based reinforcement learning method and apparatus for achieving a balance between exploration and utilization.

[0055] In actor-critic reinforcement learning methods, effectively achieving a balance between exploration and exploitation is a critical issue. Currently, researchers have proposed various exploration algorithms to address this problem, including the Vallian Bellman operator, bootstrapped DQN, count-based exploration, and optimistic initialization. However, these algorithms suffer from poor performance and limited real-world applicability.

[0056] The poor performance is as follows: (1) The Vallian Bellman operator is simple and easy to implement, but its performance is highly dependent on the current policy. For unexplored state-action pairs, it is difficult to accurately estimate the Q value, and overestimation or underestimation is likely to occur, resulting in poor sample efficiency of the algorithm.

[0057] (2) The bootstrapped DQN has some effect improvement, but it needs to layer the environment, and each layer uses a different network to process data, which is difficult to be universal in different scenarios.

[0058] (3) The count-based exploration has some effect improvement, but it needs to count and count the state and action, which is not suitable for high-dimensional space, so it is difficult to be universal in complex robot scenarios and tasks.

[0059] (4) The optimistic initialization has some effect improvement, but it usually needs to train a large number of Q value networks, and the training and calculation cost is very high, and it is still based on the Vallian Bellman operator for Q value estimation, and the problem of low sample utilization efficiency still exists.

[0060] The poor performance in real scene applications is as follows: The existing off-policy actor-critic algorithm is often trained and tested in idealized scenarios, while the complexity and uncertainty in actual application scenarios are often much higher than in ideal cases, which leads to poor performance of Vallian Bellman operator, bootstrapped DQN, count-based exploration and optimistic initialization in actual applications, especially in real scenarios. For example: In the walking scenario of the four-legged robot DKitty, each action will affect the future state of the robot, such as gait, stability and speed, etc., so a large number of experiments and sampling are needed to find a set of optimal action sequences. However, due to the dynamic and complex nature of the DKitty environment, a large number of experiments and sampling will result in low sampling efficiency, which in turn affects the sample efficiency of the algorithm. In addition, if the algorithm simply explores, it may go to some unreasonable actions, such as walking on unstable ground, causing the robot to fall, which further exacerbates the problem of sampling efficiency.

[0061] In summary, it is necessary to find an efficient and general exploration-exploitation balancing algorithm to improve the sample efficiency and generalization performance of the off-policy actor-critic algorithm. Solving this challenge requires studying how to find a better balance between exploration and exploitation in order to obtain more accurate Q value estimates on unexplored state-action pairs without sacrificing the sample efficiency and generalization performance of the algorithm. The solution to this challenge will help improve the practical application and promotion of the off-policy actor-critic algorithm.

[0062] Therefore, the present application provides an agent reinforcement learning method for realizing exploration-exploitation balancing, as shown in the following formula: Figure 1 The method comprises the following steps:

[0063] S11, constructing a first Bellman operator for learning and exploiting successful experiences in an experience replay pool;

[0064] S12, constructing a second Bellman operator for exploring unknown states;

[0065] S13, generating an exploration-exploitation balancing Bellman operator according to the first Bellman operator and the second Bellman operator;

[0066] S14, improving an actor-critic algorithm by using the exploration-exploitation balancing Bellman operator;

[0067] S15, performing reinforcement learning on an agent based on the improved actor-critic algorithm.

[0068] The agent reinforcement learning method for realizing exploration-exploitation balancing provided by the present application constructs a first Bellman operator (Bellman exploitation operator) capable of exploiting successful experiences in historical experiences and a second Bellman operator (Bellman exploration operator) capable of exploring unknown states, and then constructs a BEE operator from the Bellman exploitation operator and the Bellman exploration operator. The BEE operator improves the balance between exploration of unknown states and exploitation of historical experiences, and has the advantages of high sample efficiency, strong generalization performance, strong universality, easy implementation, and strong real scene application ability compared with the prior art. The BEE operator is used in a reinforcement learning method using an off-policy actor-critic algorithm, which can effectively improve the policy learning effect of reinforcement learning.

[0069] Specifically, the traditional RL algorithm only focuses on estimating the Q value of the current policy, which limits their ability to develop and improve sample efficiency using past experience. In simple terms, the state-action pair (s, a) fails to perform on the current policy, there is a successful trajectory of performing on the historical policy, but the value of the state-action pair (s, a) will be severely underestimated due to its failure to perform on the current policy, so it takes longer to respond to successful events. To address this issue, the present application proposes a balance exploration and utilization Bellman operator (BEE operator), which is an improvement based on the Bellman operator (Bellman operator) to effectively balance the relationship between exploration and utilization to solve the exploration and utilization balance problem. The BEE operator needs two sub-operators, namely the first Bellman operator (Bellman exploitation operator) and the second Bellman operator (Bellman exploration operator), the Bellman exploitation operator is mainly used to utilize the successful experience in the historical experience, and the Bellman exploration operator is mainly used to explore unknown states;

[0070] The formula of the Bellman exploitation operator is as follows:

[0071]

[0072] Wherein, r(s, a) is the reward of the agent performing action a in state s, γ is the discount factor, is the action set of the agent, μ(a'|s')>0 represents the probability distribution information of the agent performing action a' in state s' in the experience replay pool, p(s'|s, a) is the transition probability of the agent performing action a in state s to state s', and Q(s', a') is the value function of the agent performing a' in state s', is the value function of the agent performing action a in state s calculated by the first Bellman operator.

[0073] It should be noted that in order to better learn the successful experience in the historical experience, the present application constructs a constantly changing experience regression pool, the sample in the experience regression pool records the state transition information "current state-action-reward-next state", that is, the initial experience regression pool is composed of state transition samples collected in the offline stage, in the online learning stage, the state transition sample generated each time iteration will be stored in the experience regression pool to constantly update the experience regression pool.

[0074] The formula of Bellman exploration operator (hereinafter referred to as BEO) is as follows:

[0075]

[0076] wherein, π(a'|s') is a probability distribution of a' corresponding to state s' given the current policy of the agent π, Q π (s',a') is a value function of the agent performing action a' in state s' under the current policy π, is a value function of the agent performing action a in state s calculated by using the second Bellman operator, a' is a next action randomly sampled from the current policy π, s' is a next state obtained by the environment transition from (s,a), and w(s',a'|π) is an exploration term designed to encourage exploration, representing an entropy value of the agent performing action a' in state s' under the current policy π, which is negatively correlated with the historical number of times of performing action a' in state s'.

[0077] The formula of BEE operator is as follows:

[0078]

[0079] The core feature of the BEE operator is that it can effectively use successful experience in historical experience in the process of online learning, and effectively balance the relationship between exploration and exploitation by exploring new state-action pairs.

[0080] The main advantages of the BEE operator of the present application are as follows:

[0081] Efficiency: using the BEE operator to balance the trade-off between exploration and exploitation improves the sample efficiency and generalization performance of the algorithm;

[0082] Universality: the algorithm can be applied in different scenarios without additional environmental assumptions or heuristic algorithms;

[0083] Robustness: the algorithm can better utilize historical experience and has shown good performance in real and simulated scenarios.

[0084] In terms of application of the BEE operator, the BEE operator of the present application can be used in MBRL to implement the MB-BEE algorithm. The main idea of Dyna-style Model-based RL is to first learn the dynamic model of the environment using existing sample data, and then make planning and decision-making based on this. Specifically, it includes the following steps:

[0085] Data Collection: Similar to other reinforcement learning methods, Dyna-style Model-based RL first needs to collect sample data in the environment.

[0086] Model Learning: With the collected sample data, a dynamic model of the environment can be learned through some machine learning methods (such as neural networks, decision trees, etc.). This dynamic model can be used to predict the next state and reward in the environment.

[0087] Policy Optimization: Based on the learned dynamic model, Dyna-style Model-based RL uses methods similar to Model-free RL to train a value function to guide the update of the policy. (The BEE operator of the invention is applied to policy optimization)

[0088] Dyna-style Model-based RL Reference: Sutton R S. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming [M] / / Machine learning proceedings 1990. Morgan Kaufmann, 1990: 216-224.

[0089] Specifically, unlike traditional Dyna-style algorithms, the invention combines the BEE operator with the AC algorithm, not only using BEO for real data, but also using BEO for exploration in model-generated data. This can more efficiently utilize real data and model-generated data, thereby improving the sample efficiency and generalization performance of the algorithm.

[0090] The BEE-AC algorithm specifically includes:

[0091]

Action Family Actor Update

[0092] First, update the policy from {μ k , π k} to {μ k , π k+1}, where μ k = {π0, π1… π k-1}, π k+1 is the greedy policy relative to , that is

[0093]

[0094] Then extend the historical policy sequence to μk+1 , π k+1 , and add the policy π k to the policy mixture;

[0095]

Critic update

[0096] Compute

[0097] 1: First, update the V-value function in the replay buffer, and the calculation method is:

[0098]

[0099] Wherein, τ ∈ [0, 1] is a given parameter, D is an experience replay pool, I(·) is an indicator function, when Q(s, a)-V(s)≤0, I(Q(s, a)-V(s)<0) takes 1, and when Q(s, a)-V(s)>0, I(Q(s, a)-V(s)<0) takes 0;

[0100] 2: Calculate

[0101]

[0102] Calculate

[0103] Calculate the target Q value:

[0104]

[0105] Update the Q network parameters:

[0106] Wherein, λ is a weight coefficient, and B is a policy mixture.

[0107] Considering that there may be "lucky" samples in the actual application scene to cause the Q value to be overestimated or to cause oscillation, the application introduces a V-value function based on quantile regression in the calculation process of to reduce the overestimation and oscillation of the Q value estimation, and improve the stability and reliability of the algorithm.

[0108] In general, the BEE operator is the core technology of the application, which can more effectively solve the exploration-exploitation balance problem in the off-policy actor-critic algorithm, thereby improving the sample efficiency and generalization performance of the algorithm, and providing a feasible solution for real scene application.

[0109] Specifically, based on the BEE-AE algorithm, the agent is subjected to reinforcement learning, including:

[0110] for any application scenario of the agent, obtaining a current state of the agent in the application scenario and a current experience replay pool;

[0111] using the current experience replay pool and the improved actor-critic algorithm, iteratively optimizing and learning a policy of the agent in the current state to obtain a corresponding optimal policy;

[0112] Among them, the any application scenario includes: robot control application scenario, automatic driving application scenario and industrial power generation control application scenario.

[0113] Further, in the robot control application scenario, the agent state s can include the perception data of each sensor of the robot, such as motor encoder data, pictures taken by the camera, and perception data of infrared sensors, laser radars, etc. The agent action a can include motor torque instructions, motor speed instructions, and motor encoder position instructions, etc. controllable instructions. r is a reward function designed by a person according to the task performed by the robot (for example, the mechanical arm carrying task).

[0114] In the automatic driving application scenario, the agent state s can include high-precision map information, semantic segmentation information of the surrounding environment, speed, position, acceleration, and heading information of the current vehicle, and various sensor perception data, etc. The agent action a can include the strength of the vehicle brake, the strength of the throttle, the angle of the steering, the position of the next target motion point, and other motion decision controllable quantities. r is a reward function designed by a person according to the automatic driving task. For example, the longer the safe driving distance, the more stable the driving, the more fuel-efficient the car, and the greater the r.

[0115] In the industrial power generation control application scenario, the agent state s can include sensor recording data of each unit (such as the temperature and pressure of the combustion chamber, the flow, flow rate, and pressure of the liquid in the pipeline, etc.) and the proportion of chemical components in the exhaust. The agent action a can include various control quantities of the generator set (such as whether each valve is open or closed, the amplitude of the valve opening, the threshold of the pipeline flow, etc. controllable quantities). r is a reward function designed by a person according to the power generation control task, for example, the higher the power generation efficiency, the greater the r.

[0116] The second aspect describes the agent reinforcement learning device provided by the application for realizing exploration and utilization balance. The agent reinforcement learning device described below can be referred to in conjunction with the agent reinforcement learning method described above. Figure 2 A structural diagram of an agent reinforcement learning device for realizing exploration and utilization balance is provided as shown in Figure 2 The device includes:

[0117] a first Bellman operator for learning utilization of successful experiences in an experience replay pool is constructed;

[0118] a second Bellman operator for exploring unknown states is constructed;

[0119] a generating module 23 is configured to generate a Bellman operator balancing exploration and utilization according to the first Bellman operator and the second Bellman operator;

[0120] an improving module 24 is configured to improve an actor-critic algorithm by using the Bellman operator balancing exploration and utilization;

[0121] a reinforcement learning module 25 is configured to perform reinforcement learning on an agent based on the improved actor-critic algorithm.

[0122] The application provides an agent reinforcement learning device balancing exploration and utilization, which constructs a first Bellman operator (Bellman exploitation operator) capable of utilizing successful experiences in historical experiences and a second Bellman operator (Bellman exploration operator) capable of exploring unknown states, and then constructs a BEE operator by the Bellman exploitation operator and the Bellman exploration operator. The BEE operator improves the balance between exploration of unknown states and utilization of historical experiences, and has the advantages of high sample efficiency, strong generalization performance, strong universality, easy implementation, strong real scene application ability and the like compared with the prior art. The BEE operator is used in a reinforcement learning method adopting an off-policy actor-critic algorithm, and the strategy learning effect of the reinforcement learning is effectively improved.

[0123] In the above embodiments, as an optional embodiment, the sample in the experience replay pool records state transition information "current state-action-reward-next state"; the expression of the first Bellman operator is:

[0124] wherein r(s, a) is the reward of the agent performing the action a in the state s, and γ is a discount factor, μ(a'|s')>0 represents probability distribution information of the agent performing action a' in state s' in the experience replay pool, p(s'|s,a) is a transition probability of the agent performing action a in state s to cause state to transition to s', Q(s',a') is a value function of the agent performing a' in state s', is a value function of the agent performing action a in state s calculated by using the first Bellman operator.

[0125] On the basis of each of the above examples, as an optional embodiment, the second construction module comprises:

[0126] A construction unit is configured to construct an exploration term w(s',a'|π); wherein the w(s',a'|π) represents an entropy value of the agent performing action a' in state s' under the current policy π, and the value is negatively correlated with the historical number of times of performing action a' in state s';

[0127] An improvement unit is configured to improve the conventional Bellman operator based on the exploration term to obtain the second Bellman operator.

[0128] On the basis of each of the above examples, as an optional embodiment, an expression of the second Bellman operator is:

[0129]

[0130] wherein r(s,a) is a reward of the agent performing action a in state s, γ is a discount factor, p(s'|s,a) is a transition probability of the agent performing action a in state s to cause state to transition to s', π(a'|s') is a probability distribution of a' corresponding to state s' given the current policy π, Q π (s',a') is a value function of the agent performing action a' in state s' under the current policy π, is a value function of the agent performing action a in state s calculated by using the second Bellman operator.

[0131] On the basis of each of the above examples, as an optional embodiment, the exploration-exploitation balanced Bellman operator is a weighted sum of the first Bellman operator and the second Bellman operator, and an expression is:

[0132]

[0133] In the above formula, λ is a weight parameter, is a value function of the agent performing action a in state s calculated by using the first Bellman operator, BQ(s, a) is a value function of an agent performing an action a in a state s calculated by using the second Bellman operator;

[0134] An expression of the improved actor-critic algorithm is:

[0135] Actor:

[0136] Critic:

[0137] In the above formula, is a value function of an agent performing an action a in a state s under a current policy k and a past policy sequence k , μ k = {π0, π1…π k-1}, ω(s, a|π k ) is an entropy value of an agent performing an action a in a state s under a current policy k , π k+1 is a policy of the agent in the k+1th iteration, is a value function of an agent performing an action a in a state s calculated by using the exploration-exploitation balanced Bellman operator in the kth iteration, Q(s, a) is a value function of an agent performing an action a in a state s determined by the critic network in the kth iteration, Q φ is the critic network. On the basis of each of the above embodiments, as an optional embodiment, after the decision of the current policy k , the sample corresponding to the state transition information of the current policy k is regressed into the experience regression pool.

[0138] On the basis of each of the above embodiments, as an optional embodiment, the reinforcement learning module comprises:

[0139] An acquisition unit is configured to acquire a current state of the agent and a current experience replay pool in any application scenario of the agent.

[0140] A reinforcement learning unit is configured to perform iterative optimization learning of a policy of the agent in the current state by using the current experience replay pool and the improved actor-critic algorithm, to obtain a corresponding optimal policy.

[0141] The any application scenario comprises a robot control application scenario, an automatic driving application scenario, and an industrial power generation control application scenario.

[0142] A third aspect, Figure 3 An example of a schematic diagram of a physical structure of an electronic device is shown in Figure 3 As shown, the electronic device can include a processor 310, a communications interface 320, a memory 330, and a communications bus 340, wherein the processor 310, the communications interface 320, and the memory 330 can communicate with each other through the communications bus 340. The processor 310 can invoke a logical instruction in the memory 330 to execute an intelligent agent reinforcement learning method for achieving exploration-exploitation balance, which includes: constructing a first Bellman operator for learning and utilizing successful experiences in an experience replay pool; constructing a second Bellman operator for exploring unknown states; generating a Bellman operator for exploration-exploitation balance according to the first Bellman operator and the second Bellman operator; improving an actor-critic algorithm by using the Bellman operator for exploration-exploitation balance; and performing reinforcement learning on an intelligent agent based on the improved actor-critic algorithm.

[0143] In addition, the logical instruction in the memory 330 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present application or parts of the present application that essentially contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0144] In a fourth aspect, the present application further provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium, and the computer program can be executed by a processor to enable a computer to execute the intelligent agent reinforcement learning method for achieving exploration-exploitation balance provided by the above-mentioned methods. The method includes: constructing a first Bellman operator for learning and utilizing successful experiences in an experience replay pool; constructing a second Bellman operator for exploring unknown states; generating a Bellman operator for exploration-exploitation balance according to the first Bellman operator and the second Bellman operator; improving an actor-critic algorithm by using the Bellman operator for exploration-exploitation balance; and performing reinforcement learning on an intelligent agent based on the improved actor-critic algorithm.

[0145] In a fifth aspect, the present application also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the method for achieving the exploration-exploitation balance of the agent reinforcement learning method, and the method comprises: constructing a first Bellman operator for learning and utilizing successful experiences in an experience replay pool; constructing a second Bellman operator for exploring unknown states; generating a Bellman operator with exploration-exploitation balance according to the first Bellman operator and the second Bellman operator; improving an actor-critic algorithm by using the Bellman operator with exploration-exploitation balance; and performing reinforcement learning on the agent based on the improved actor-critic algorithm. The apparatus embodiments described above are merely illustrative, and the units described as separate units can or can not be physically separated, and the units shown as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Some or all of the modules can be selected to achieve the purpose of the embodiment according to actual needs. Those skilled in the art can understand and implement it without creative labor.

[0146] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0147] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some technical features; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An agent reinforcement learning method that implements an exploration versus exploitation balance, characterized in that, The method comprises: constructing a first Bellman operator for learning and utilizing successful experiences in an experience replay pool; constructing a second Bellman operator for exploring unknown states; generating a Bellman operator balancing exploration and utilization according to the first Bellman operator and the second Bellman operator; improving an actor-critic algorithm by using the Bellman operator balancing exploration and utilization; reinforcing learning of an agent based on the improved actor-critic algorithm; the Bellman operator balancing exploration and utilization is a weighted sum of the first Bellman operator and the second Bellman operator, and the expression is: ； In the above formula, is a weight parameter, is a value function of the agent performing action in state calculated using the first Bellman operator, is a value function of the agent performing action in state calculated using the second Bellman operator, is a value function of the agent performing action in state calculated using the exploration-exploitation balanced Bellman operator. the expression of the improved actor-critic algorithm is: Actor: ; Critics: ; In the above formula, is the current policy, and the past policy sequence is , the value function of the agent performing action in state , , is the current policy, , the entropy value of the agent performing action in state , is the policy of the agent in the i-th iteration, is the value function of the agent performing action in state calculated by the Bellman operator using the exploration-exploitation balance in the i-th iteration, is the value function of the agent performing action in state determined by the critic network in the i-th iteration, is the value function of the agent performing action in state , is the critic network; the reinforcing learning of the agent based on the improved actor-critic algorithm comprises: for a robot control application scenario of the agent, obtaining a current state of the agent in the application scenario and a current experience replay pool in the application scenario; performing iterative optimization learning of a policy of the agent in the current state by using the current experience replay pool and the improved actor-critic algorithm, to obtain a corresponding optimal policy; wherein the agent state s comprises motor encoder data, pictures taken by a camera, and sensor perception data; the agent action a comprises motor torque instructions, motor speed instructions, and motor encoder position instructions; and r is a reward function designed by a human according to a task performed by the robot.

2. The agent reinforcement learning method for achieving exploration-exploitation balance according to claim 1, wherein, The sample in the experience replay pool records state transition information "current state-action-reward-next state"; the expression of the first Bellman operator is: ； wherein, is a reward for the agent performing an action in a state , is a discount factor, is a set of actions for the agent, characterizes probability distribution information of the agent performing an action in a state , is a transition probability of the agent performing an action in a state causing a state to transition to , is a value function of the agent performing in a state , is a value function of the agent performing an action in a state calculated using the first Bellman operator. 3.The method of claim 1, wherein, The second Bellman operator is constructed by improving a traditional Bellman operator based on the exploration term. Build exploration items ; wherein, the The current strategy is represented as In the case of the agent in state Next action The entropy value, taking values ​​related to the state Next action The historical frequency is negatively correlated; The expression of the second Bellman operator is:

4. The agent reinforcement learning method for achieving exploration-exploitation balance according to claim 3, wherein, The device comprises: ； in, For the agent in state Next action The reward As depreciation factor, For the agent in state Next action Caused state transition The transition probability, The agent's current policy is Given a state corresponding The probability distribution, The current strategy is In the case of the agent in state Next action The value function, For an agent computed using the second Bellman operator in a state Next action The value function of . 5.The method of claim 1, wherein, after the decision has been made the current policy The corresponding state transition information is regressed into the experience regression pool.

6. An agent reinforcement learning device that realizes exploration-exploitation balance, characterized by, a first construction module configured to construct a first Bellman operator for learning and utilizing successful experiences in an experience replay pool; a second construction module configured to construct a second Bellman operator for exploring unknown states; a generation module configured to generate a Bellman operator balancing exploration and utilization according to the first Bellman operator and the second Bellman operator; an improvement module configured to improve an actor-critic algorithm by using the Bellman operator balancing exploration and utilization; a reinforcement learning module configured to reinforce learning of an agent based on the improved actor-critic algorithm; the Bellman operator balancing exploration and utilization is a weighted sum of the first Bellman operator and the second Bellman operator, and the expression is: the expression of the improved actor-critic algorithm is: ； In the above formula, is a weight parameter, is a value function of the agent performing action in state calculated using the first Bellman operator, is a value function of the agent performing action in state calculated using the second Bellman operator, is a value function of the agent performing action in state calculated using the exploration-exploitation balanced Bellman operator. the reinforcement learning module comprises: Actor: ; Critics: ; In the above formula, is the current policy, and the past policy sequence is , the value function of the agent performing action in state , , is the current policy, , the entropy value of the agent performing action in state , is the policy of the agent in the i-th iteration, is the value function of the agent performing action in state calculated by the Bellman operator of the exploration-exploitation balance in the i-th iteration, is the value function of the agent performing action in state determined by the critic network in the i-th iteration, is the value function of the agent performing action in state , is the critic network; an acquisition unit configured to, for a robot control application scenario of the agent, acquire a current state of the agent in the application scenario and a current experience replay pool in the application scenario; a reinforcement learning unit configured to perform iterative optimization learning of a policy of the agent in the current state by using the current experience replay pool and the improved actor-critic algorithm, to obtain a corresponding optimal policy; and ​ Wherein, the agent state s includes motor encoder data, pictures taken by a camera, and sensor perception data; the agent action a includes motor torque instructions, motor speed instructions, and motor encoder position instructions; r is a reward function designed by a person according to the task performed by the robot.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the agent reinforcement learning method for achieving exploration and utilization balance according to any one of claims 1-5 when executing the program.

8. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program implements the agent reinforcement learning method for achieving exploration and utilization balance according to any one of claims 1-5 when executed by the processor.

Images