Artificial intelligence subject deep reinforcement learning uncertainty estimation method

Abstract

The invention discloses an artificial intelligence subject deep reinforcement learning uncertainty estimation method. The method mainly comprises the following steps: decomposing decision uncertainty based on deep reinforcement learning into accidental uncertainty and cognitive uncertainty; calculating a basic component of each uncertainty based on Bayesian and non-Bayesian methods; and synthesizing to obtain standard uncertainty and expansion. According to the method, a traditional modeling method for uncertainty estimation of deep reinforcement learning is broken through, the uncertainty is evaluated in a Bayesian and non-Bayesian mode, and the time needed by deep reinforcement learning exploration-utilization of an artificial intelligence subject can be effectively shortened.

Description

Technical Field

[0001] The present invention relates to the field of intelligent metering and testing technology, and in particular to a decision uncertainty estimation method based on a deep reinforcement learning algorithm for an artificial intelligence subject. Background Art

[0002] Deep reinforcement learning is an artificial intelligence technology that combines deep learning perception capabilities and reinforcement learning decision-making capabilities. It enables intelligent agents (autonomous aircraft, robots, and other artificial intelligence entities) to learn optimal strategies through interaction with the environment in complex environments. It is particularly suitable for sequential decision-making problems (involving a series of ordered decision-making problems) and is widely used in fields such as autonomous driving, robot control, and games.

[0003] Its working principle is that in the process of interaction between the agent and the environment, the agent observes the state of the environment, makes selection decisions and performs actions. The environment changes accordingly and gives the agent corresponding rewards. The goal of the agent is to learn a strategy to maximize the long-term cumulative rewards. Deep reinforcement learning approximates the value function or policy function through neural networks, and continuously optimizes these functions to better predict the value of each state or state-action pair to guide the agent to make better decisions.

[0004] However, due to the complex environment, incomplete information, uncertain boundaries and other characteristics faced by intelligent agents in actual scenarios, intelligent agents based on deep reinforcement learning algorithms face problems such as sparse rewards, long interaction steps, noisy distribution, and low exploration efficiency in the process of game decision-making. As a result, the intelligent agent's observation and decision-making state space is large, the reasoning process is random, and the results are uncertain in the deep reinforcement learning decision-making process.

[0005] This uncertainty is generally divided into two different types: aleatoric uncertainty and epistemic uncertainty. Aleatoric uncertainty is caused by the randomness of the interaction between the agent and the environment. Although aleatoric uncertainty can be evaluated and modeled by methods, it cannot be reduced. Unlike aleatoric uncertainty, the epistemic uncertainty of the agent during movement can be reduced. As the number of data samples increases, the accuracy of the deep reinforcement learning model is improving, and its epistemic uncertainty is a process of continuous decline. At the same time, epistemic uncertainty is deeply related to the classic exploration-exploitation trade-off problem in deep reinforcement learning. More exploration of unknown environments and strategies during the interaction between the agent and the environment may result in a lower reward value than the known optimal strategy. Conversely, if the known strategy is used to obtain the current maximum reward, it may result in missing the strategy with potential higher rewards.

[0006] Therefore, estimating the uncertainty of deep reinforcement learning decision results to guide the next decision of the intelligent agent is of great significance for continuously improving the correctness and rationality of decisions. Summary of the invention

[0007] In response to the above-mentioned problems, the present invention discloses a method that breaks through the key technologies of uncertainty estimation in reinforcement learning, such as Bayesian inference, approximate Bayesian method, and stochastic dynamics theory, and proposes an uncertainty estimation method for deep reinforcement learning algorithms of artificial intelligence subjects, which effectively improves the efficiency of solving the exploration-exploitation balance problem, and at the same time provides an important reference basis for the accuracy of the decision-making results of deep reinforcement learning of intelligent agents.

[0008] The present disclosure provides an artificial intelligence subject deep reinforcement learning algorithm uncertainty estimation method, comprising the following steps:

[0009] S1, decomposes the uncertainty of deep reinforcement learning of artificial intelligence agents into accidental uncertainty and epistemic uncertainty;

[0010] S2, assessment of the aleatory uncertainty component based on small sample historical data and current data;

[0011] S3, assessment of the epistemic uncertainty component;

[0012] S4, synthesize the aleatory uncertainty estimate and the epistemic uncertainty estimate to obtain the synthetic standard uncertainty.

[0013] Furthermore, in step S1:

[0014] The accidental uncertainty refers to the uncertainty of data caused by the accidental randomness of the artificial intelligence subject, and its source factors include: one or more of data feature space, data diversity, data scale, data noise, and data division;

[0015] The epistemic uncertainty refers to the epistemic uncertainty caused by the randomness of the deep reinforcement learning model, including the uncertainty problems caused by the uncertainty factors in the model architecture and loss optimization stage. The uncertainty factors include: one or more of the depth and width design of the neural network, the loss function design, the local optimal characteristics of the optimization algorithm, the randomness of the strategy function, and the randomness of the reward function.

[0016] Furthermore, the specific method of step S2 includes:

[0017] S21, if deep reinforcement learning with small sample data (x s,a (k),Q s,a (k)) is the unknown statistical distribution form, where x s,a (k) represents the state-action (s, a) value pair, Qs,a (k) represents the Q value of a neural network under a given state-action (s, a) value pair. Based on the uncertainty grey estimation method, the grey correlation theory is introduced to establish a new uncertainty estimation model, as shown in the following formula:

[0018]

[0019] Among them, the standard deviation σ is used to evaluate the divergence of the agent's decision under the same conditions, that is, the uncertainty estimate; c is the gray coefficient; ε′(k) is the normalized gray correlation coefficient; x s,a (1) (k) is the current sampling accumulation sequence; is the average value of the current sampling state-action sequence (s, a)

[0020] S22, if the small sample data is in a known statistical distribution form, then on the basis of applying the new uncertainty estimation model, Bayesian information fusion is introduced to fuse the uncertainty estimation results of the current data and the historical data to obtain the final uncertainty estimation result, which specifically includes:

[0021] Assume that the initial prior mean and prior mean variance obtained from the comprehensive historical sampling data are μ0, think That is the prior density of the mean μ;

[0022] (1) Assume that the current small batch sample data is represented as: x s,a (0) ={x(k)|k=1,2,...,n}, and apply the uncertainty grayscale estimation model in step S21 to it, and obtain the standard deviation estimation value as follows:

[0023]

[0024] Among them, c is the gray coefficient, ε′(k) is the normalized gray correlation coefficient, and the current sample is considered to meet in The likelihood function is:

[0025]

[0026] (2) Based on Bayesian inference, the posterior distribution of h(μ|x)∝p(μ)L(μ|x) is:

[0027]

[0028] in,

[0029]

[0030] In the above formula It is the uncertainty estimate of the action-state (s, a) data obtained after information fusion.

[0031] Furthermore, in step S22, before each evaluation, the last posterior distribution is regarded as the prior distribution of this estimation, and then the posterior distribution is obtained in combination with the current measurement sample data, and this cycle is repeated to continuously improve the accuracy of the uncertainty estimate.

[0032] Furthermore, the step S22 further includes the following steps:

[0033] Based on the accidental uncertainty of small samples, the distribution of sample data is obtained for training reinforcement learning models:

[0034] Inverse variance weighting method, using variance information of small sample data It can learn faster in the presence of heteroscedastic noise on the labels by optimizing the parameters w of the neural network f in the loss function for N small batches of data:

[0035]

[0036] L(N,w) represents the loss function for optimizing the parameter w of the neural network f; y n Represents x n The label of , L2 is a regularization term used to limit the weight, improve the stability of the optimization process, and prevent small variance samples from dominating the loss function of small batches.

[0037] Furthermore, the specific method of step S3 includes:

[0038] S31, assume that the deep reinforcement learning algorithm of the artificial intelligence subject trains a neural network by minimizing the error between the target Y(s,a) and its prediction Q(s,a), predicting the Q-value-action pair Q in a given state π (s,a);

[0039] Among them, Y(s,a) is calculated according to the Bellman equation:

[0040] Y(s,a)=r+γQ(s′,a′)

[0041] L θ =||Y(s,a)-Q θ (s,a)|| 2 ;

[0042] S32: The estimated value Q of the critic network in the deep reinforcement learning algorithm θ (s k ,a k ) is regarded as QThe random variable sampled from (Q|s,a) has a mean of and variance

[0043] S33: Calculation The uncertainty, variance Estimation using variance networks;

[0044] To evaluate Each target network Y in the target set n predict The mean μ k,n and variance

[0045] By μ k,n and The variance σ of the Gaussian mixture Q,k (s k ′,a k ′) That is, the epistemic uncertainty is as follows:

[0046]

[0047] μ k,all represents the mean and value of all Q values.

[0048] Furthermore, the synthesis method of step S4 includes:

[0049] Assume that the accidental uncertainty estimate of deep reinforcement learning of the artificial intelligence subject is σ 数据 , the epistemic uncertainty estimate is σ 模型 , then the combined standard uncertainty σ 合成 for:

[0050]

[0051] Furthermore, the method further comprises the following steps:

[0052] S5, based on the expansion factor, the synthetic standard uncertainty is expanded to obtain the confidence interval of the artificial intelligence subject's decision.

[0053] Compared with the prior art, the beneficial effects of the present invention are: (1) it breaks through the modeling method of traditional deep reinforcement learning uncertainty estimation, changes the existing uncertainty estimation method, evaluates uncertainty in a Bayesian and non-Bayesian way, and reduces the time required for deep reinforcement learning exploration and utilization of intelligent agents; (2) improves the decision-making efficiency of intelligent agents; (3) the accuracy of decision results is continuously improved; (4) it has good versatility for the decisions of various intelligent agents. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components throughout the exemplary embodiments of the present disclosure.

[0055] Figure 1 is a flowchart of uncertainty estimation for deep reinforcement learning of an artificial intelligence agent according to the present disclosure;

[0056] Figure 2 It is a random uncertainty estimation method based on small batch sample data;

[0057] Figure 3 Estimation process for epistemic uncertainty. DETAILED DESCRIPTION

[0058] The preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0059] The present disclosure provides a method for estimating the decision uncertainty of an artificial intelligence subject based on deep reinforcement learning. The method refers to the uncertainty estimation theory, breaks through the modeling method of traditional deep reinforcement learning uncertainty estimation, changes the existing uncertainty estimation method, evaluates uncertainty in Bayesian and non-Bayesian ways, and reduces the time required for deep reinforcement learning exploration and utilization.

[0060] The main steps of this method include: establishment of a deep reinforcement learning uncertainty estimation evaluation model, estimation of aleatory uncertainty components and epistemic uncertainty components, synthesis of Bayesian and non-Bayesian uncertainty estimates, and synthesis and extended uncertainty estimation reporting.

[0061] Among them, according to the source of uncertainty, it can be divided into data (accidental) uncertainty and model (cognitive) uncertainty:

[0062] For accidental uncertainty, we mainly consider the uncertainty caused by uncertainty factors in the data sampling and data partitioning stages. The uncertainty factors include five major factors: feature space, data diversity, data scale, data noise, and data partitioning.

[0063] For cognitive uncertainty, we mainly consider the uncertainty problems caused by uncertainty factors in the model architecture and loss optimization stages. The uncertainty factors include five major factors: neural network depth and width design, loss function design, local optimal characteristics of the optimization algorithm, randomness of the strategy function, and randomness of the reward function.

[0064] The collected image information is fitted with the results to construct the most practical measurement model under the current test conditions.

[0065] Calculate each fundamental component of uncertainty based on Bayesian and non-Bayesian methods;

[0066] Finally, the synthetic uncertainty estimation result is obtained according to the synthetic formula.

[0067] In one exemplary embodiment, a synthesis method for uncertainty estimation of a deep reinforcement learning algorithm for an artificial intelligence agent is provided, such as Figure 1 As shown, it includes: constructing an agent decision uncertainty estimation and evaluation model based on the decomposition of uncertainty sources; estimating the uncertainty of each component; synthesis; and expanding the uncertainty estimation.

[0068] The main steps are further described as follows:

[0069] S101. Decomposition of sources of uncertainty

[0070] First, we determine the deep reinforcement learning algorithm to be analyzed, and decompose the sources of uncertainty of the AI ​​subject decision-making algorithm as follows:

[0071] (1) The accidental randomness of AI subjects causes data uncertainty, which mainly comes from factors such as data noise and data partitioning;

[0072] (2) The randomness of the decision-making model of the artificial intelligence subject based on deep reinforcement learning causes cognitive uncertainty. For cognitive uncertainty, the uncertainty problem caused by uncertainty factors in the model architecture and loss optimization stage is mainly considered. The uncertainty factors include five major factors: loss function design, randomness of strategy function, and randomness of reward function.

[0073] S102. Estimation of random uncertainty (i.e. data uncertainty) based on small batch sample data

[0074] like Figure 2As shown, this is the data uncertainty estimation method based on small batch samples in this embodiment. The static uncertainty estimation of small samples is based on the following two aspects: First, if the overall distribution of the current small sample data does not conform to the common distribution, the gray correlation theory is introduced on the basis of the original uncertainty gray estimation method to establish a new uncertainty estimation model; Second, if the overall distribution of the current small sample data conforms to the common distribution, the Bayesian information fusion idea is introduced on the basis of applying the new uncertainty habit gray estimation model, and the uncertainty estimation results of the current data and historical data are integrated to obtain the final uncertainty estimation result.

[0075] The specific steps for estimating uncertainty of small sample historical data and current data are as follows:

[0076] Step ①: Knowing the statistical distribution form, based on the grey uncertainty Bayesian information fusion method. Assume that the initial prior mean and prior mean variance obtained by integrating historical sampling data are It can be considered That is the prior density of the mean μ.

[0077] Step ②: Assume that the current small batch sample data can be expressed as: x s,a (0) ={x(k)|k=1,2,...,n}, and apply the uncertainty grayscale estimation model to it to obtain the data standard deviation estimation formula:

[0078]

[0079] Where c is the gray coefficient, ε′(k) is the normalized gray correlation coefficient, and it can be considered that the current sample meets in The likelihood function is:

[0080]

[0081] Step 3: Based on Bayesian inference, the posterior distribution of h(μ|x)∝p(μ)L(μ|x) is:

[0082]

[0083] in,

[0084]

[0085] In the above formula It is the uncertainty estimate of the action-state (s, a) data obtained after information fusion.

[0086] Since the Bayesian information fusion theory is inherited, before each assessment, the previous posterior distribution can be regarded as the prior distribution of this estimation, and then the posterior distribution can be obtained by combining the current measurement sample data. This cycle continues, the amount of information continues to increase, and the uncertainty estimate is more accurate.

[0087] Step ④: Accidental uncertainty fusion based on small sample inverse variance. The batch inverse variance weighted method uses additional information It can learn faster in the presence of heteroscedastic noise on the labels by optimizing the parameters w of the neural network f in the loss function for N small batches of data:

[0088]

[0089] L(N,w) represents the loss function for optimizing the parameter w of the neural network f. m Represents x n The label of , L2 is a regularization term used to limit the weight, improve the stability of the optimization process, and prevent small variance samples from dominating the loss function of small batches.

[0090] Step ⑤: Unknown statistical distribution form, based on the uncertainty gray estimation method. In reinforcement learning training, the standard deviation can be used to evaluate the degree of divergence of decisions under the same conditions. c is the gray coefficient, x (1) (k) is the current sampling accumulation sequence, is the average value of the current sampling sequence:

[0091]

[0092] S103. Estimation of epistemic uncertainty

[0093] like Figure 3 As shown in the figure, the training process of the decision-making algorithm DQN of the artificial intelligence subject and the sources of its cognitive uncertainty mainly include the randomness of the neural network, the local optimum of optimization and the randomness of the strategy.

[0094] The specific steps are as follows:

[0095] Step 1: The AI ​​agent decision algorithm DQN trains the neural network by minimizing the error between the target Y(s,a) and its prediction Q(s,a), predicting the Q value-action pair Q in a given state. π (s,a). Y(s,a) is calculated according to the Bellman equation:

[0096] Y(s,a)=r+yQ(s',a') (7)

[0097] L θ =||Y(s,a)-Q θ (s,a)||2 (8)

[0098] Step 2: The estimated value Q of the critic network in the decision algorithm DQN θ (s k ,a k ) is regarded as Q The random variable sampled from (Q|s,a) has a mean of and variance

[0099] Step 3: Calculation The uncertainty, variance Estimation is done using a variance network. The value of is the corresponding target network Y j The predicted mean μ k,i However, in order to evaluate Each target network Y in the target set n predict The mean μ k,n and variance By μ k,n and The variance σ of the Gaussian mixture Q,k (s′ k ,a′ k )as follows:

[0100]

[0101] S104. Based on the uncertainty estimation method, obtain the synthetic standard uncertainty according to the evaluation model

[0102] Based on the results of accidental uncertainty and epistemic uncertainty, the synthetic uncertainty estimation result σ is obtained based on the synthetic standard uncertainty assessment model. 合成 , the calculation formula is as follows:

[0103]

[0104] S105. Finally, the expanded uncertainty is obtained based on the expansion factor.

[0105] When the expansion factor k = 2 (confidence probability is 95%), the confidence interval of the decision of the artificial intelligence subject is (μ-2σ 合成 , μ+2σ 合成 );

[0106] When the expansion factor k = 3 (confidence probability is 99%), the confidence interval of the decision of the artificial intelligence subject is (μ-3σ 合成 , μ+3σ 合成 ).

[0107] In this embodiment, by analyzing the key influencing factors in the deep reinforcement learning algorithm and constructing a synthetic uncertainty estimation evaluation model, it is possible to accurately and effectively evaluate the accuracy and reliability of the decision-making of the artificial intelligence subject, improve the exploration-utilization efficiency through quantitative means, and at the same time provide important reference value for the credibility of the next decision of deep reinforcement learning.

[0108] The above technical scheme is only an exemplary embodiment of the present invention. For those skilled in the art, it is easy to make various types of improvements or modifications based on the application methods and principles disclosed in the present invention, and it is not limited to the method described in the above specific embodiment of the present invention. Therefore, the method described above is only preferred and does not have a restrictive meaning.

Claims

1. A method for estimating uncertainty in deep reinforcement learning of an artificial intelligence subject, comprising the following steps: S1, decomposes the uncertainty of deep reinforcement learning of artificial intelligence agents into accidental uncertainty and epistemic uncertainty; S2, assessment of the aleatory uncertainty component based on small sample historical data and current data; S3, assessment of the epistemic uncertainty component; S4, synthesize the aleatory uncertainty estimate and the epistemic uncertainty estimate to obtain the synthetic standard uncertainty.

2. The method according to claim 1, characterized in that In step S1: The accidental uncertainty refers to the uncertainty of data caused by the accidental randomness of the artificial intelligence subject, and its source factors include: one or more of data feature space, data diversity, data scale, data noise, and data division; The epistemic uncertainty refers to the epistemic uncertainty caused by the randomness of the deep reinforcement learning model, including the uncertainty problems caused by the uncertainty factors in the model architecture and loss optimization stage. The uncertainty factors include: one or more of the depth and width design of the neural network, the loss function design, the local optimal characteristics of the optimization algorithm, the randomness of the strategy function, and the randomness of the reward function.

3. The method according to claim 1, characterized in that The specific method of step S2 includes: S21, if deep reinforcement learning small sample data (xs, a(k), Q s,a (k)) is an unknown statistical distribution, where xs,a(k) represents the state-action (s,a) value pair, Q s,a (k) represents the Q value of a neural network under a given state-action (s, a) value pair. Based on the uncertainty grey estimation method, the grey correlation theory is introduced to establish a new uncertainty estimation model, as shown in the following formula: Among them, the standard deviation σ is used to evaluate the divergence of the agent's decision under the same conditions, that is, the uncertainty estimate; c is the gray coefficient; ε′(k) is the normalized gray correlation coefficient; Accumulate the sequence once for the current sample; is the average value of the current sampling state-action sequence (s, a) S22, if the small sample data is in a known statistical distribution form, then on the basis of applying the new uncertainty estimation model, Bayesian information fusion is introduced to fuse the uncertainty estimation results of the current data and the historical data to obtain the final uncertainty estimation result, which specifically includes: Assume that the initial prior mean and prior mean variance obtained from comprehensive historical sampling data are think That is the prior density of the mean μ; (1) Assume that the current small batch sample data is represented as: x s,a (0) ={x(k)|k=1,2,...,n}, and apply the uncertainty grayscale estimation model in step S21 to it, and obtain the standard deviation estimation value as follows: Among them, c is the gray coefficient, ε′(k) is the normalized gray correlation coefficient, and the current sample is considered to meet in The likelihood function is: (2) Based on Bayesian inference, the posterior distribution of h(μ|x)∝p(μ)L(μ|x) is: in, In the above formula It is the uncertainty estimate of the action-state (s, a) data obtained after information fusion.

4. The method according to claim 3, characterized in that In step S22, before each evaluation, the last posterior distribution is regarded as the prior distribution of this estimation, and then the posterior distribution is obtained by combining the current measurement sample data, and this cycle is repeated to continuously improve the accuracy of the uncertainty estimate.

5. The method according to claim 3, characterized in that: The step S22 further comprises the following steps: Based on the accidental uncertainty of small samples, the distribution of sample data is obtained for training reinforcement learning models: Inverse variance weighting method, using variance information of small sample data It can learn faster in the presence of heteroscedastic noise on the labels by optimizing the parameters w of the neural network f in the loss function for N small batches of data: L(N,w) represents the loss function for optimizing the parameter w of the neural network f; y m Represents x n The label of L2 is used as a regularization term to limit the weight, improve the stability of the optimization process, and prevent small variance samples from dominating the loss function of small batches.

6. The method according to claim 1, characterized in that The specific method of step S3 includes: S31, assume that the deep reinforcement learning algorithm of the artificial intelligence subject trains a neural network by minimizing the error between the target Y(s,a) and its prediction Q(s,a), predicting the Q-value-action pair Q in a given state π (s,a); Among them, Y(s,a) is calculated according to the Bellman equation: Y(s,a)=r+γQ(s′,a′) L θ =||Y(s,a)-Q θ (s,a)|| 2 ; S32: The estimated value Q of the critic network in the deep reinforcement learning algorithm θ (s k ,a k ) is regarded as Q The random variable sampled from (Q|s,a) has a mean of and variance S33: Calculation The uncertainty, variance Estimation using variance networks; To evaluate Each target network Y in the target set n predict The mean μ k,n and variance By μ k,n and The variance σ of the Gaussian mixture Q,k (s k ′,a k ′) That is, the epistemic uncertainty is as follows: μ k,all represents the mean and value of all Q values.

7. The method according to claim 1, characterized in that The synthesis method of step S4 comprises: Assume that the accidental uncertainty estimate of deep reinforcement learning of the artificial intelligence subject is σ 数据 , the epistemic uncertainty estimate is σ 模型 , then the combined standard uncertainty σ 合成 for:

8. The method according to any one of claims 1 to 7, further comprising the following steps: S5, based on the expansion factor, the synthetic standard uncertainty is expanded to obtain the confidence interval of the artificial intelligence subject's decision.

Images