Computer implementation method and device for reinforcement learning

Abstract

The present disclosure relates to a computer-implemented method for reinforcement learning, the method comprising: sampling one or more trajectories as a data set by one or more behavioral policies, where each of the one or more trajectories comprises a sequence of states, actions, and respective rewards to the states and actions; evaluating, via importance sampling, performance of the target policy based on an estimate obtained using the one or more trajectories; and optimizing the target policy on the dataset by maximizing the estimate of performance while constraining the deviation of the estimate from the performance on the dataset.

Description

Technical Field

[0001] Generally speaking, aspects of the present disclosure relate to artificial intelligence, and more specifically, aspects of the present disclosure relate to methods and apparatus for off-line policy reinforcement learning. Background Art

[0002] Learning to control agents directly from high-dimensional sensor inputs (e.g., vision and speech) is one of the long-standing challenges of reinforcement learning (RL). Most successful RL applications running on these domains rely on hand-crafted features. Obviously, the performance of these applications depends heavily on the quality of the feature representation.

[0003] Recent advances in deep learning have made it possible to extract high-level features from raw sensor data, leading to breakthroughs in computer vision and speech recognition. However, reinforcement learning poses some challenges from a deep learning perspective. For example, in reinforcement learning, the data distribution changes as the algorithm learns new behaviors, which can be problematic for deep learning methods that assume a fixed underlying distribution. Another issue is that most deep learning algorithms assume that data samples are independent, while in reinforcement learning, highly correlated state sequences are often encountered.

[0004] To alleviate these problems, off-policy reinforcement learning algorithms are proposed to improve the sample efficiency of deep reinforcement learning (DRL). The key idea of ​​off-policy algorithms is to reuse historical trajectories in the replay buffer, e.g., randomly sampling previous transitions to smooth the training distribution of many past behaviors, but this means that these trajectories are not sampled from the current policy. In other words, the offline policy evaluation uses trajectories generated by other policies (e.g., historical policies) to estimate the expected return of the current policy, which leads to a distribution shift.

[0005] To alleviate this problem, a standard approach in offline policy evaluation is importance sampling, a widely used Monte Carlo technique for evaluating the expected return of a policy when training data is generated by different policy actions. In practice, importance sampling usually calculates the degree of deviation between the current policy (i.e., the target policy) and the policy action.

[0006] Accurate offline policy evaluation is particularly important because it is the core of the off-policy reinforcement learning algorithm, and inaccurate offline policy evaluation may mislead the direction of the gradient, significantly slow down the training speed, and lead to poor results with high probability. Therefore, from the perspective of accuracy, it is hoped to provide an improved method for importance sampling in off-policy reinforcement learning. Summary of the invention

[0007] In order to have a basic understanding of one or more aspects according to the present disclosure, a simplified summary of these aspects is summarized below. This summary is not an exhaustive overview of all expected aspects, nor is it intended to identify the key or important elements of all aspects of the present disclosure, or to describe the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the detailed description below.

[0008] In one aspect of the present disclosure, a computer-implemented method for reinforcement learning is provided, the method comprising: sampling one or more trajectories as a dataset through one or more behavioral strategies, wherein each of the one or more trajectories comprises a sequence of states, actions, and corresponding rewards for the states and the actions; evaluating performance of a target strategy based on an estimate obtained using the one or more trajectories via importance sampling; and optimizing the target strategy on the dataset by maximizing the estimate of the performance while constraining the deviation of the estimate from the performance on the dataset.

[0009] In another aspect of the present disclosure, a computer-implemented method for reinforcement learning of a control task is provided, the method comprising: sampling one or more trajectories as a data set through one or more behavioral strategies, wherein each of the one or more trajectories comprises a sequence of states, actions, and corresponding rewards for the states and the actions; evaluating the performance of a target strategy based on an estimate obtained using the one or more trajectories via importance sampling; and optimizing the target strategy on the data set by maximizing the estimate of the performance while constraining the deviation of the estimate from the performance on the data set; wherein the data set comprises images, speech, or other sensor data, and wherein the target strategy is learned to control a task comprising controlling a robot, a vehicle, a household appliance, a power tool, a manufacturing machine, a personal assistant, or an access control system.

[0010] In another aspect of the present disclosure, there is provided an apparatus for reinforcement learning, comprising a memory and at least one processor coupled to the memory, the at least one processor being configured to: sample one or more trajectories as a data set through one or more behavioral strategies, wherein each of the one or more trajectories comprises a sequence of states, actions, and corresponding rewards for the states and the actions; evaluate performance of a target strategy based on an estimate obtained using the one or more trajectories via importance sampling; and optimize the target strategy on the data set by maximizing the estimate of the performance while constraining the deviation of the estimate from the performance on the data set.

[0011] In another aspect of the present disclosure, a computer program product for reinforcement learning is provided, the computer program product comprising a processor executable computer code for performing the following operations: sampling one or more trajectories as a data set through one or more behavioral strategies, wherein each of the one or more trajectories comprises a sequence of states, actions, and corresponding rewards for the states and the actions; evaluating performance of a target strategy based on an estimate obtained using the one or more trajectories via importance sampling; and optimizing the target strategy on the data set by maximizing the estimate of the performance while constraining the deviation of the estimate from the performance on the data set.

[0012] In another aspect of the present disclosure, a computer-readable medium storing computer code for reinforcement learning is provided, which, when executed by a processor, causes the processor to perform operations including: sampling one or more trajectories as a data set through one or more behavioral strategies, wherein each of the one or more trajectories includes a sequence of states, actions, and corresponding rewards for the states and the actions; evaluating the performance of a target strategy based on an estimate obtained using the one or more trajectories via importance sampling; and optimizing the target strategy on the data set by maximizing the estimate of the performance while constraining the deviation of the estimate from the performance on the data set.

[0013] Motivated by the observation that offline policy evaluation is biased by repeatedly using trajectories in the replay buffer for optimization and evaluation, we propose a bias-regularized importance sampling method to significantly improve the performance and sample efficiency of offline policy methods.

[0014] Other aspects or modifications and other advantages of the present disclosure will become apparent by consideration of the following detailed description and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The disclosed aspects will be described below in conjunction with the accompanying drawings, which are provided to illustrate rather than limit the disclosed aspects.

[0016] Figure 1 An example of reuse bias (or simply bias) in offline policy evaluation via importance sampling is shown through experimental results in MiniGrid according to one or more aspects of the present disclosure.

[0017] Figure 2 An exemplary flow of a method for bias regularized importance sampling (BIRIS) for controlling bias according to one or more aspects of the present disclosure is shown.

[0018] Figure 3Another exemplary flow of a method for bias regularized importance sampling (BIRIS) for controlling bias according to one or more aspects of the present disclosure is shown.

[0019] Figure 4 An example diagram of a Biased Regularized Importance Sampling (BIRIS) framework is shown in which one or more aspects of the present disclosure may be implemented.

[0020] Figure 5 and Figure 6 The experimental results of combining two common offline policy algorithms SAC (Soft Action Criteria) and TD3 (Double Delayed Deep Deterministic Policy Gradient) with the method proposed in this paper are shown respectively.

[0021] Figure 7 An example of a hardware implementation of an apparatus according to one or more aspects of the present disclosure is shown. DETAILED DESCRIPTION

[0022] The present disclosure will now be discussed with reference to some exemplary implementations. It should be understood that the discussion of these implementations is only to enable those of ordinary skill in the art to better understand and thus implement the embodiments of the present disclosure, rather than to impose any limitation on the scope of protection of the present disclosure.

[0023] Reinforcement learning problems involve learning what to do, how to map states to actions, in order to maximize a numerical reward signal. They are essentially closed-loop problems because the actions of the learning system affect its later inputs. Furthermore, in many forms of machine learning, the learner is not told which actions to take, but must discover by trial and error which actions produce the greatest reward. In uncharted territory, the learning agent must be able to learn from its own experience. In the most interesting and challenging situations, actions affect not only the immediate reward, but also the next state, and through that, all subsequent rewards. Reinforcement learning problems can include sequential decision-making tasks, such as a robot manipulating an object or a self-driving car deciding when to change lanes.

[0024] These examples may involve the interaction between an active decision-making agent and its environment, in which the agent attempts to achieve a goal despite uncertainty about its environment. This allows the agent's actions to affect the future state of the environment (e.g., the next location of a robot or vehicle), thereby affecting the options and opportunities available to the agent at later times.

[0025] In reinforcement learning, a policy defines how a learning agent should behave at a given time. In general, a policy is a mapping from perceived states of the environment to actions to take in those states. It may correspond to what is known in psychology as a set of stimulus-response rules or associations. The policy is central to the agent, as it alone is sufficient to determine what actions the agent will take. In general, the policy can be stochastic.

[0026] The reward signal defines the goal of the reinforcement learning problem. At each time step, the environment sends a single number, the reward, to the agent. The agent's only goal is to maximize the total reward it receives over the long term. The reward signal therefore defines what is a good event and a bad event for the agent. The reward sent to the agent at any time depends on the agent's current action and the current state of the agent's environment. The only way the agent can influence the reward signal is through its actions, which can have a direct effect on the reward, or an indirect effect by changing the state of the environment.

[0027] While the reward signal indicates what is good in an immediate sense, the value function specifies what is good in the long term. In general, the value of a state is the total amount of reward the agent can expect to accrue in the future, starting from that state. The reward determines the immediate, intrinsic desirability of the state of the environment, while the value indicates the long-term desirability of the state after considering the states that may follow and the rewards that can be obtained in those states.

[0028] In general, rewards are primary in a sense, while value, as a prediction of rewards, is secondary. Without rewards, there is no value, and the only purpose of estimating value is to obtain more rewards. However, when making and evaluating decisions, value is of primary concern. Action selection is based on value judgments. Actions that lead to states with the highest value, rather than the highest reward, are sought, because in the long run, these actions can obtain the largest amount of rewards. However, determining value is much more difficult than determining reward. Rewards are basically given directly by the environment, but value must be estimated and re-estimated based on the sequence of observations made by the agent throughout its lifetime. In fact, the most important component of almost all reinforcement learning algorithms is a method for efficient estimation of value.

[0029] In contrast to on-policy reinforcement learning, which generates trajectories from different policies, off-policy reinforcement learning estimates the value function of a policy given an infinite supply of trajectories generated using that policy (i.e., sequences of empirical samples of states, actions, and rewards from real or simulated interactions with the environment). That is, to estimate the value of a policy π, all we have is following another policy The trajectory of The strategy π is called the target strategy because learning its value is the goal of the learning process, and the strategy It is called a behavioral policy because it is the policy that controls the agent and generates behavior. The overall problem is called offline policy learning because it is learning a policy given only experience that is “offline” (not following) the policy. In general, the behavioral policy can be stochastic and exploratory. Therefore, off-policy reinforcement learning can improve the sample efficiency of deep reinforcement learning.

[0030] Deep learning can utilize a range of neural network architectures, including but not limited to convolutional networks, multi-layer perceptrons, restricted Boltzmann machines, and recurrent neural networks. Deep learning can extract high-level features from raw sensor data, which is beneficial for reinforcement learning using sensor data.

[0031] Importance sampling is a general technique for estimating the expected value under one distribution given samples from another distribution. Importance sampling can be used for offline policy learning by weighting the rewards according to their relative probability of occurring in trajectories under the target policy and the action policy. It is well known that importance sampling is unbiased when the target policy is independent of the samples, but has high variance that is exponentially related to the trajectory length. Therefore, many studies have focused on variance reduction for offline policy evaluation. However, little work has been devoted to examining the bias of offline policy evaluation.

[0032] In this disclosure, we focus on the bias of offline policy evaluation. Unlike the overestimation bias in Q-learning due to function approximation, the bias mentioned in this paper is caused by repeatedly using trajectories in the replay buffer for optimization and evaluation. This unavoidable estimation bias may mislead the evaluation of the policy, which may be further amplified in temporal difference learning.

[0033] While importance sampling is unbiased when the target policy is independent of the samples generated by the acting policy, this unbiasedness may no longer hold when the target policy is correlated with the samples. In other words, importance sampling may be biased in offline policy evaluation if the same batch of data from the training process is reused in the evaluation to optimize the target policy.

[0034] In this disclosure, we show both theoretically and experimentally that offline policy evaluation via importance sampling is overestimating when the same replay buffer of trajectories is reused for optimization and evaluation. Furthermore, high probability bounds on the bias are obtained, indicating that the bias cannot be eliminated simply by increasing the number of trajectory samples. Based on these observations, we propose a method for bias-regularized importance sampling to significantly improve the performance and sample efficiency of offline policy methods.

[0035] Usually, you can consider A Markov decision process (MDP) with and can represent the state space and action space respectively. For any state-action pair yes distribution on , which represents its transformation. In addition, is the reward function and γ∈(0,1) is the discount factor. The policy of any agent is modeled as a mapping π: That is, π(·|s) is a given state Next The distribution on is a dataset of trajectories.

[0036] In RL, an agent interacts with the environment by making decisions at each time step. At the beginning, the agent is assumed to be in state s0~μ(·), where μ(·) is the initial state distribution. At time step t, the agent chooses its action a t ~π(·|s t ), and arrive Receive corresponding rewards The performance of a policy π can be defined as the expectation of the discounted return over the trajectory:

[0037]

[0038] in, Representation trajectory discounted return.

[0039] Due to the trajectory distribution is unknown, and J(π) can be estimated by Monte Carlo sampling. To improve sample efficiency, the offline policy algorithm can utilize trajectories sampled by other policies via importance sampling. It is worth noting that in the replay buffer There are m trajectories in . According to the strategy Sampling, that is

[0040] For the sake of simplicity and clarity, it should be noted that and In offline policy evaluation, to estimate J(π) for a target policy π that is different from the policy that sampled the trajectories, importance sampling can be used to modify The importance sampling estimate of the performance of the target policy π can be written as:

[0041]

[0042] in, Indicates the use of strategy π to generate trajectories The probability of

[0043]

[0044] In one aspect of the present disclosure, a new concept of reuse bias (or simply bias) is proposed to measure the bias in offline policy evaluation methods. The hypothesis set of the hypothesis policy is For any offline strategy algorithm, it can be formatted as a mapping The mapping is initialized with the strategy and replay buffer As input, and output strategy

[0045] In offline policy evaluation, the importance sampling estimate defined in Eq. (2) Can be used as expected return The difference between them can be defined as the offline strategy algorithm Initialization strategy π0 and replay buffer The reuse bias (or simply bias) of , as given in this paper. For any offline policy algorithm Initialize strategy π0 and replay buffer At π0 and The reuse bias (or simply bias) on can be defined as:

[0046]

[0047] Importance sampling can be an unbiased estimator when the distribution to be estimated is independent of the sample. Independent of When , the offline policy evaluation is unbiased, i.e. However, practical offline policy algorithms can often take advantage of the replay buffer In this case, the target policy It really depends on This makes the evaluation of offline policies no longer unbiased.

[0048] In one or more aspects of the present disclosure, it is theoretically shown that offline policy evaluation via importance sampling is overestimating when the same replay buffer of a trajectory is repeatedly used for optimization and evaluation. First, assume that the offline policy algorithm satisfy With The condition with the highest estimated performance is yes In the replay buffer The optimal strategy on , that is,

[0049]

[0050] To simplify the notation, set You can resample the dataset It is independent of Therefore, having

[0051]

[0052] Consider the following facts, It is a dataset The optimal strategy on , then we can have

[0053]

[0054] In equation (7), the first equation holds because the internal part is independent of the data set.

[0055] Therefore, it is proved This means Yes An overestimation of If the equality holds, then for any have:

[0056]

[0057] In other words, for any Established.

[0058] In another aspect of the present disclosure, it can be shown that under certain assumptions, the estimated performance can even be greater than the performance of the optimal policy. Assume that the action space is discrete and the reward of any state-action pair (s, a) is not negative, i.e. And π * is the optimal strategy, It's a track The optimal strategy on , that is,

[0059]

[0060] To simplify the notation, set First, set

[0061]

[0062] Known for any and π, Then, for any satisfying of have Therefore, having

[0063]

[0064] That is to say, If the inequality takes an equal sign, then for any The state-action pairs (s, a) satisfy π * The case where (a|s)=1.

[0065] Since the equality condition in (12) may not be satisfied in non-experimental environments, reuse bias may lead to overestimating even bad policies as having higher values ​​than the optimal policy, which may lead to degradation of policy optimization and final performance.

[0066] Figure 1 An example of reuse bias (or simply bias) in offline strategy evaluation via importance sampling through experimental results in MiniGrid according to one or more aspects of the present disclosure is shown. MiniGrid includes grids of different shapes, such as Figure 1 The grids used in the experiments are 5×5, 6×6, 8×8, 16×16, and each of these grids has a discrete state space and a discrete action space. In the experiment, the policy can be initialized to be parameterized using a three-layer convolutional neural network, and 40 trajectories are sampled as the replay buffer. In the optimization phase of the experiment, the policy gradient method can be used to optimize the policy by maximizing the estimated performance obtained by importance sampling via the replay buffer. In the testing phase of the experiment, the estimated performance of the optimized policy obtained by using importance sampling via the replay buffer and the true performance of the optimized policy can be calculated. Each experiment with a different grid can be tested with 50 different random seeds. The results are reported in the form of 100 box plots. Figure 1 In each sub-figure of The distribution of In each box plot, the horizontal lines at the top 110 and bottom 120 of the box plot represent the maximum and minimum values, respectively, and the horizontal lines at the top 130 and bottom 140 of the box plot represent the first quartile and the third quartile, respectively. In addition, the dotted line and the plus sign in the box plot represent the median and the expected value, respectively.

[0067] like Figure 1As shown in , in each environment (e.g., 5×5, 6×6, 8×8, 16×16 grids), the estimated performance is higher than the true performance. This is consistent with the theoretical view according to one or more aspects of the present disclosure that using the same replay buffer to optimize and evaluate policies may lead to overestimation. In addition, the relative reuse bias (or relative bias for short) of different grids (i.e., They are 0.35, 0.89, 1.88, and 29.37, respectively. Therefore, when the grid environment is larger and more complex, the reuse bias (or simply bias) shown in these experiments is larger, which means that this overestimation phenomenon will be more serious in more complex environments.

[0068] Although importance sampling can be an unbiased estimator when the distribution to be estimated is independent of samples generated by another distribution, observations from the present disclosure indicate that there is an unavoidable and non-negligible bias in the estimate when the distribution to be estimated depends on samples generated by another distribution, which may be the case for practical off-policy algorithms. Therefore, considering that there is an unavoidable and non-negligible bias in importance sampling when the distribution to be estimated depends on samples generated by another distribution, this paper proposes a method for bias-regularized importance sampling to improve the performance and sample efficiency of off-policy methods.

[0069] In one aspect of the present disclosure, in order to control the deviation, a deviation boundary can be set. It can be considered that all trajectories are passed through a single strategy sampled, and can provide a high probability bound for the deviation. It can be assumed that for any trajectory Its return can be bounded by Then, for any offline strategy algorithm and initial strategy In the selection by the same strategy Sampled independent and identically distributed training set With a probability of at least 1-δ, the following formula holds:

[0070]

[0071] in,

[0072]

[0073] In formula (14), ∈1 (as and The KL divergence of the trajectory distribution of can indicate the similarity between them. In addition, ε2 can focus on the two strategies in the replay buffer From formula (13), it can be concluded that when the number of samples m tends to infinity, the high probability boundary will converge to Instead of zero. Therefore, the bias cannot be eliminated simply by increasing the number of trajectory samples.

[0074] In one or more aspects of the present disclosure, based on equation (13), the high probability upper bound depends on the KL divergence of the two strategies and their Therefore, controlling their divergence (especially in ), which helps in offline policy evaluation.

[0075] Figure 2 An exemplary flow of a method 200 for bias regularized importance sampling (BIRIS) for controlling bias according to one or more aspects of the present disclosure is shown. The method 200 can be used with conventional offline policy methods, such as for stochastic policies and / or deterministic policies. In box 210, one or more trajectories can be sampled as a data set by one or more behavioral policies that interact with an environment, wherein each of the one or more trajectories includes a sequence of states, actions, and corresponding rewards for the states and actions. For example, the data set can be or include a replay buffer. In box 220, the performance of the target policy can be evaluated based on an estimate obtained by using one or more trajectories via importance sampling. In box 230, the target policy on the data set can be optimized by maximizing the estimate of performance while constraining the deviation of the estimate from the performance on the data set.

[0076] In method 200 , by introducing constraints on the bias in the optimization, the negative impact of the bias as explained by the observations presented herein can be mitigated, and thus the performance of the traditional off-policy reinforcement learning algorithm can be improved accordingly.

[0077] As an example, method 200 may be formulated as:

[0078]

[0079] in,

[0080] In formula (15), can be the nominal loss function of a traditional off-policy reinforcement learning (RL) algorithm, while It can be a regularization term used to penalize deviations. For example, during optimization at block 230, a nominal loss function can be used By maximizing the performance estimate of the target policy, and by regularizing Penalize deviations to improve the target policy.

[0081] In one aspect of the present disclosure, the target strategy can be different from one or more behavioral strategies. For example, the one or more behavioral strategies can be random. For example, the target strategy can be random or deterministic.

[0082] In another aspect of the present disclosure, bias may include overestimation of the performance of the target strategy due to repeated use of the dataset for optimization and evaluation.

[0083] In another aspect of the present disclosure, during optimization at block 230, a hyperparameter α may be used in equation (15) to control the tradeoff between the standard RL loss and the regularization loss.

[0084] In one or more aspects of the present disclosure, a regularization term may be used To calculate any state-action pair of In order to control ε2, for example, the constraint on the deviation can be expressed as:

[0085]

[0086] Because ε2 can be transformed into:

[0087]

[0088] Figure 3 Another exemplary flow of a method 300 for bias regularized importance sampling (BIRIS) for controlling bias according to one or more aspects of the present disclosure is shown. Method 300 may be similar to method 200 in most processes. In box 310, one or more trajectories may be sampled as a data set by one or more behavioral policies interacting with an environment, wherein each of the one or more trajectories includes a sequence of states, actions, and corresponding rewards for the states and actions. In box 320, the performance of the target policy may be evaluated based on an estimate obtained using the one or more trajectories via importance sampling. In box 330, the target policy on the data set may be optimized by maximizing the estimate of performance while constraining the deviation of the estimate from the performance on the data set. Then, the process of method 300 may return to box 310, where the one or more behavioral policies may be updated using the historical policies previously optimized in box 330, wherein the historical policies are different from the target policy (i.e., the current policy) optimized in the latest iteration of box 330. For example, the one or more behavioral policies may include historical policies during optimization, and the target policy is the current policy during optimization. By using the history strategy as the behavior strategy, the KL divergence of ε1 can be controlled to a certain extent.

[0089] Figure 4An example diagram of a bias regularized importance sampling (BIRIS) framework 400 is shown, in accordance with one or more aspects of the present disclosure, in which method 200 or 300 may be implemented. Framework 400 may include sampling or dataset collection 410 and optimization 420. In sampling or dataset collection 410, one or more behavior policies 412 may interact with environment 411 (e.g., through actions 413 and next states and corresponding rewards 414) to sample one or more trajectories 415 to pass to optimization 420. Optimization 420 may include a nominal loss function of a conventional off-policy reinforcement learning (RL) algorithm 421 and a regularization term 422 that penalizes bias, by which a target policy 423 may be optimized to obtain an optimal policy. As an option, as shown by dashed line 424, one or more behavior policies 412 may be updated using a previously optimized policy 423 that is different from the current policy optimized in the latest iteration.

[0090] Figure 5 and Figure 6 The respective experimental results of two general offline policy algorithms SAC (Soft Action Evaluation) and TD3 (Double Delay Deep Deterministic Policy Gradient) combined with the method proposed in this article (e.g., Method 200 and / or Method 300 and / or Framework 400) are shown. Since the environment (e.g., Ant, Halfcheetah, Humanoid, and Walker2d) and the strategy are random, 10 strategies with different random seeds are trained for each algorithm in each task. In order to mitigate the impact of randomness, the mean and variance of these 10 strategies are plotted as a function of the time step of the training phase (as shown on the horizontal axis). In addition, the solid line represents the average reward of the 10 strategies, and the lighter shaded part represents their variance. Figure 5 The comparison between SAC and SAC+BIRIS was reported in Figure 6 The comparison between TD3 and TD3+BIRIS is reported in , where the vertical axis indicates the round reward. As shown in the results, SAC+BIRIS or TD3+BIRIS can learn better policies compared with SAC or TD3.

[0091] Figure 7 700 according to one or more aspects of the present disclosure. The apparatus 700 for reinforcement learning may include a memory 710 and at least one processor 720. The processor 720 may be coupled to the memory 710 and configured to execute the above reference Figure 2 , Figure 3 and Figure 4The methods 200, 300 and framework 400 described herein are described herein. The processor 720 may be a general purpose processor, or may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. The memory 710 may store input data, output data, data generated by the processor 720, and / or instructions executed by the processor 720.

[0092] The various operations, models, and networks described in conjunction with the disclosure herein may be implemented using hardware, software executed by a processor, firmware, a computer, or any combination thereof. According to one or more aspects of the disclosure, a computer program product for reinforcement learning may include a computer program for executing the above referenced Figure 2 , Figure 3 and Figure 4 The processor executable computer code of the described methods 200, 300 and framework 400. According to another embodiment of the present disclosure, a computer readable medium may store a computer code for reinforcement learning, and when the computer code is executed by a processor, the processor may execute the above reference Figure 2 , Figure 3 and Figure 4 Methods 200, 300 and framework 400 described herein. Computer readable media include non-transitory computer storage media and communication media, and communication media include any media that facilitates the transfer of computer programs from one place to another. Any connection can be appropriately referred to as a computer readable medium. Other embodiments and implementations also fall within the scope of protection of the present disclosure.

[0093] In one embodiment of the present disclosure, the target strategy can be different from one or more behavioral strategies. For example, the target strategy can be stochastic or deterministic, and one or more behavioral strategies can be stochastic. As another example, one or more behavioral strategies can be historical strategies that have been optimized in previous iterations before the latest iteration during optimization, and the target strategy can be the current strategy that has been optimized in the latest iteration during optimization. That is, except Other than the latest output of The previous output of can be used as a behavioral policy.

[0094] In one embodiment of the present disclosure, the bias may be due to reusing the dataset for optimization and evaluation and may result in the estimate having a higher value than the performance.

[0095] In one embodiment of the present disclosure, the bias may be constrained via a regularization term.

[0096] In one embodiment of the present disclosure, optimization can be performed based on a nominal loss function, a regularization term, and a hyperparameter, where the hyperparameter is used to control the balance between the nominal loss function and the regularization term. For example, the hyperparameter can have a value of 0.01, 0.05, or 0.1.

[0097] In one embodiment of the present disclosure, the methods 200 , 300 and / or framework 400 may be easily used with other off-policy reinforcement learning methods (eg, SAC, TD3) to improve the performance of these methods.

[0098] For example, for SAC+BIRIS, the following calculation can be used to control ε2:

[0099]

[0100] However, when If it is too small, the ratio may be unstable. The numerical instability caused by too low a value of can be clipped to the value using the hyperparameter β in SAC+BIRIS:

[0101]

[0102] The value of β can be -0.5, -1.0 or -2.0.

[0103] To enable any person of ordinary skill in the art to implement or use various embodiments, the above description is centered around the disclosed embodiments. Various modifications to these embodiments are apparent to those of ordinary skill in the art, and the general principles defined herein may also be applied to other embodiments without departing from the spirit or scope of the various embodiments. Therefore, the present invention is not limited to the embodiments shown in the present application, but is in accordance with the widest scope consistent with the attached claims and the principles and novel features disclosed herein.

Claims

1. A computer-implemented method for reinforcement learning, comprising: Sampling one or more trajectories as a data set through one or more behavior policies, wherein each of the one or more trajectories comprises a sequence of a state, an action, and a corresponding reward for the state and the action; evaluating, via importance sampling, a performance of a target policy based on estimates obtained using the one or more trajectories; as well as The target policy on the data set is optimized by maximizing the estimate of the performance while constraining the deviation of the estimate from the performance on the data set.

2. The computer-implemented method of claim 1 , wherein: The target policy is different from the one or more behavioral policies.

3. The computer-implemented method of claim 1 , wherein: The deviation is due to reusing the data set for the optimization and the evaluation and causes the estimate to have a higher value than the performance.

4. The computer-implemented method of claim 1 , wherein: The bias is constrained via a regularization term.

5. The computer-implemented method of claim 4, wherein: The optimization is performed based on a nominal loss function, the regularization term, and a hyperparameter, wherein the hyperparameter is used to control a balance between the nominal loss function and the regularization term.

6. The computer-implemented method of claim 5, wherein: The hyperparameter has a value of 0.01, 0.05 or 0.

1.

7. The computer-implemented method of claim 1 , wherein: The one or more behavioral strategies include historical strategies during the optimization period, and the target strategy is a current strategy during the optimization period.

8. A computer-implemented method for reinforcement learning of a control task, comprising: Sampling one or more trajectories as a data set through one or more behavior policies, wherein each of the one or more trajectories comprises a sequence of a state, an action, and a corresponding reward for the state and the action; evaluating, via importance sampling, a performance of a target policy based on estimates obtained using the one or more trajectories; as well as optimizing the target policy on the data set by maximizing the estimate of the performance while constraining the deviation of the estimate from the performance on the data set; The data set includes images, voice or other sensor data, and The target policy is learned to control a task including controlling a robot, a vehicle, a household appliance, a power tool, a manufacturing machine, a personal assistant, or an access control system.

9. An apparatus for reinforcement learning, comprising: Memory; as well as At least one processor coupled to the memory and configured to perform the computer-implemented method according to one of claims 1-8.

10. A computer program product for reinforcement learning, comprising: Processor executable computer code for performing the computer implemented method according to one of claims 1-8.

11. A computer-readable medium storing a computer code for reinforcement learning, which, when executed by a processor, causes the processor to perform the computer-implemented method according to any one of claims 1 to 8.

Images