METHOD AND SETUP FOR GUIDED OFFLINE REINFORCEMENT LEARNING

Abstract

A method for training an offline reinforcement learning network is disclosed. The method comprises: obtaining an offline reinforcement learning network, wherein the offline reinforcement learning network provides a policy for an agent to perform an action in a state of an environment; generating a guide network on a guide dataset, wherein the guide network outputs a relative importance of a policy improvement target and a policy constraint target for optimizing the offline reinforcement learning network; and updating policy parameters of the offline reinforcement learning network on an offline dataset by a policy target as a function of the policy improvement target and the policy constraint target based on their relative importance.

Description

AREA

[0001] The present disclosure relates generally to the technical field of artificial intelligence, and more particularly to offline reinforcement learning (RL) with expert guidance. BACKGROUND

[0002] Reinforcement learning (RL) is an important area of ​​machine learning that aims to solve problems such as how agents should act in different states of an environment to maximize certain types of cumulative rewards. Generally, reinforcement learning includes online RL and offline RL. Online RL learns a policy by interacting with the environment, while offline RL learns a policy by optimizing the policy based only on an offline dataset without interacting with the environment. Because offline RL learns a policy based on a previously collected dataset, it typically suffers from a distribution shift problem, which results from the discrepancy between the state-action distributions of the offline dataset and the interactions of the current policy with the test environment.In particular, after optimizing the offline dataset, the agent might encounter unvisited states or misjudge the state action values ​​during interactions with the online environment, leading to poor performance.

[0003] To mitigate this problem, previous solutions use a single trade-off between two opposing objectives for offline RL: a policy improvement objective that aims to optimize the policy according to the current value functions, and a policy constraint objective that maintains the policy's behavior around the offline dataset to prevent the policy from becoming too aggressive. Therefore, previous solutions either add an explicit policy constraint term to the policy improvement equation or implicitly constrain the policies by revising the update rules of the value functions. However, these solutions generally focus only on the global properties of the dataset, neglecting the individual characteristics of each sample.Typically, only a single trade-off is made for all data in a mini-batch or even for the entire offline dataset. Such "one-size-fits-all" trade-offs may not achieve a perfect balance for every sample and thus likely limit performance potential.

[0004] There is therefore a need to provide an improved solution for offline reinforcement learning. SUMMARY

[0005] The following is a simplified summary of one or more aspects according to the present disclosure to provide a basic understanding of such aspects. This summary is not a comprehensive overview of all aspects considered and is not intended to identify key or critical elements of all aspects, nor to delimit the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in simplified form in anticipation of the more detailed description presented below.

[0006] In one aspect of the disclosure, a method for offline reinforcement learning is disclosed. The method comprises: obtaining an offline reinforcement learning network, the offline reinforcement learning network providing a policy for an agent to perform an action in a state of an environment; generating a guidance network on a guidance dataset, the guidance network outputting a relative importance of a policy improvement objective and a policy constraint objective for optimizing the offline reinforcement learning network; and updating policy parameters of the offline reinforcement learning network on an offline dataset by a policy objective as a function of the policy improvement objective and the policy constraint objective based on the relative importance.

[0007] In another aspect of the disclosure, an apparatus for offline reinforcement learning is disclosed. The apparatus may comprise a memory and at least one processor connected to the memory. The at least one processor may be configured to maintain an offline reinforcement learning network, the offline reinforcement learning network providing a policy for an agent to perform an action in a state of an environment; generate a guidance network on a guidance dataset, the guidance network outputting a relative importance of a policy improvement objective and a policy constraint objective for optimizing the offline reinforcement learning network; and update policy parameters of the offline reinforcement learning network on an offline dataset by a policy objective as a function of the policy improvement objective and the policy constraint objective based on the relative importance.

[0008] In another aspect of the disclosure, a computer-readable medium storing computer code for offline reinforcement learning is disclosed. The computer code, when executed by a processor, may cause the processor to obtain an offline reinforcement learning network, the offline reinforcement learning network providing a policy for an agent to perform an action in a state of an environment; generate a guidance network on a guidance data set, the guidance network outputting a relative importance of a policy improvement objective and a policy constraint objective for optimizing the offline reinforcement learning network; and update policy parameters of the offline reinforcement learning network on an offline data set by a policy objective as a function of the policy improvement objective and the policy constraint objective based on the relative importance.

[0009] In another aspect of the disclosure, a computer program product for offline reinforcement learning is disclosed. The computer program product may comprise processor-executable computer code for obtaining an offline reinforcement learning network, the offline reinforcement learning network providing a policy for an agent to perform an action in a state of an environment; generating a guidance network on a guidance dataset, the guidance network outputting a relative importance of a policy improvement objective and a policy constraint objective for optimizing the offline reinforcement learning network; and updating policy parameters of the offline reinforcement learning network on an offline dataset by a policy objective as a function of the policy improvement objective and the policy constraint objective based on the relative importance.

[0010] Other aspects or variations of the disclosure will become apparent upon consideration of the following detailed description and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The following figures illustrate various embodiments of the present disclosure for illustrative purposes only. One skilled in the art will readily appreciate from the following description that alternative embodiments of the methods and structures disclosed herein may be implemented without departing from the spirit and principles of the disclosure described herein. Fig. 1 illustrates a basic structure of a reinforcement learning model according to one aspect of the present disclosure. Fig. 2 illustrates a block diagram of a method for guided offline reinforcement learning according to one aspect of the present disclosure. Fig.3 illustrates a flowchart of a method for guided offline reinforcement learning according to one aspect of the present disclosure. Fig. 4 illustrates a block diagram of an apparatus for guided offline reinforcement learning according to one aspect of the present disclosure. Fig. 5A and Fig. 5B illustrate performance comparisons for different numbers of expert samples according to one aspect of the present disclosure. DETAILED DESCRIPTION

[0012] Before embodiments of the present disclosure are explained in detail, it is to be understood that the disclosure is not limited in its application to the details of construction and feature arrangement set forth in the following description. The disclosure is capable of other embodiments and may be practiced or embodied in various ways.

[0013] Fig. Figure 1 illustrates a basic structure of a reinforcement learning model according to one aspect of the present disclosure. Reinforcement learning mainly includes agents, environments, states, actions, and rewards. As shown in Fig. 1, the agent 110 can for the current state s t of the environment 120 perform an action a for the environment 120 based on a policy. Then the environment 120 can enter a new state s t+1 , and agent 110 receives a reward r as feedback from environment 120. Agent 110 can then circularly perform new actions according to the rewards and the new states of environment 120. The goal of agent 110's policy is to maximize the accumulated rewards. This process illustrates how the agent and the environment interact through states, actions, and rewards.

[0014] Reinforcement learning allows agents to determine which actions they should perform in different environmental states to obtain the maximum reward. For example, in robot control applications, the agent can be a robot, or more specifically, a brain of the robot, and the environment can be the surroundings of the place where the robot is working. The environment can have various states, such as obstacles, inclines, depressions, etc. The robot can perform different actions, such as walking on the road, in different states based on the policy learned in its brain. If the robot performs the correct actions and avoids the obstacles, it can obtain a higher overall reward.

[0015] Reinforcement learning is usually expressed as a Markov decision process (MDP), denoted as a tuple (S, A, P, d0, R, γ), where S is the state space, A is the action space, P(st+1 | s t , a) stands for the state transition probability of the environment, s t and s t+1 belong to the state space, a belongs to the action space, the agent's policy can be based on such a probability, d0( s0 ) denotes a distribution of the initial state s0, R(s t , a, s t+1 ) defines a reward function, and γ ∈ (0,1] is a discount factor.

[0016] In contrast to online RL, which learns a policy by interacting with the environment, offline RL aims to learn the policy only through an offline dataset D={(sk,ak,s′k,rk)|k=1,2,⋯,N} to optimize, whereby a k which is in state s k the action performed in the environment, s′k which is due to a k changed state and r kthe corresponding reward. This property of offline RL provides convenience for applications in many domains where online interactions are expensive or dangerous, such as robot control, autonomous driving, and healthcare. Although various offline RL solutions exist with different training losses, the goal of most of them encompasses two opposing objectives, either explicitly or implicitly: (1) policy improvement, which aims to optimize the policy according to the current value functions; and (2) policy constraint, which adapts the policy to the behavioral policy or the distribution of the offline dataset.Offline RL must strike a trade-off between these two goals: focusing too much on the notion of policy improvement might lead the policy to enter unfamiliar territory and generate bad actions due to distributional shifts; otherwise, focusing too much on the notion of policy restriction might lead the policy to only imitate behaviors in the offline dataset and might lack generalizability to data outside the distribution.

[0017] Previous offline RL solutions include, for example, TD3+BC (Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021); its variant SAC+BC, which applies SAC (Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018) to the TD3+BC framework; CQL (Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179-1191, 2020); and IQL (Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169, 2021). The policy optimization objectives of these solutions can be unified as follows: πθ=argmaxπθE(s,a)∼DF(Lpi(Q,πθ,s,a)︸Policy improvement term,Lpc(Q,πθ,s,a)︸Policy constraint term,dc︸Constraint degree), where π: S → A is a policy with trainable parameters θ, Q(s, a): S × A → R is a state-action value function that estimates the expected sum of discounted rewards after performing action a in state s. In addition, L pi(·) and L pc(·) for policy improvement and policy restriction, and F(·) is a trade-off function between L pi(·) and L pc(·) . d c ∈ R is a degree of restriction: a larger d c would encourage greater policy constraints, making the policy more conservative; otherwise, the policy would place more emphasis on the notion of policy improvement and thus tend to be more aggressive.

[0018] An ideal balance between improvement and constraint for offline RL is to focus more on policy constraints for samples that resemble expert behavior and more on policy improvement for data that resemble random behavior. Furthermore, many online RL methods demonstrate that expert demonstrations, even in small numbers, improve policy performance, but current offline RL methods do not fully utilize expert data.

[0019] In the present disclosure, an offline RL method is provided that can determine an adaptive trade-off between policy improvement and policy restriction for different samples with the guidance of only a few expert data.

[0020] Fig.Figure 2 illustrates a block diagram of a method for guided offline reinforcement learning (GORL) according to one aspect of the present disclosure. An offline dataset 210 may contain a massive amount of data and may be used to train an RL network 220. The RL network 220 may provide a policy for an agent. A guidance dataset 230 may consist of a few expert data obtained from expert demonstrations or behaviors and may be used to train a guidance network 240. The guidance network 240 may be used to guide the training of the RL network 220 by the policy. The GORL method may alternate between updating the guidance network on the guidance dataset in a MAML (Model-Agnostic Meta-Learning)-like manner and training the RL agent on the offline dataset with the guidance of the updated guidance network.

[0021] The GORL method is a plug-in approach and can adaptively and consistently evaluate the relative importance of policy improvement and policy constraint for each piece of data. The GORL method reveals a theoretically guaranteed optimization direction for the agent and is easily implemented in most offline RL solutions. The GORL method can achieve significant performance improvements over several state-of-the-art offline RL solutions with the guidance of only a few expert data points.

[0022] In the GORL method, the offline data set can be D={(sk,ak,s′k,rk)|k=1,2,⋯,N}, and the master data set can be called G={(ak,ak,s′k,rk)|k=1,2,⋯,M} where M << N, and E(sk,ak,s′k,rk)∼G[rt]>E(sk,ak,s′k,rk)∼D[rt]. For example, D is a large offline dataset containing suboptimal or even random trajectories of policies, and G is a guideline dataset with a small amount of optimal data as collected from expert or near-expert policies.

[0023] For simplicity, a training objective of the GORL method can be formulated similarly to TD3+BC as follows: πθ=argmaxπθEk∼unif{1,N}[Q(sk,πθ(sk))︸Policy improvement term−Bw(Lpc(ak,πθ(sk)))︸Constraint degree⋅Lpc(ak,πθ(sk))︸Policy constraint term] where unif {1, N} denotes a uniform distribution on {1,2,···, N} and L pc(·) stands for a term of the policy constraint, e.g., (π θ (s k ) - a k ) 2 in TD3+BC. The control network B w : R → R with parameters w takes a policy constraint term L pc as input and outputs a degree of constraint. It should be noted that in

[0024] In contrast to the typical framework of existing offline solutions (see equation 1), which have a fixed degree of restriction (e.g. d c ) for all data, in the GORL procedure the data transmitted by the control network B w generated levels of constraints for different state / action pairs (s k , a k) may vary. In equation (2), the guidance network may take the policy constraint objective as input and outputs a constraint degree for the policy constraint objective. The constraint degree varies for different samples in the offline dataset and the guidance dataset. For example, the guidance network may output a higher relative importance of the policy constraint objective compared to the policy improvement objective for samples that correspond to (i.e., are similar to) expert behavior in the offline dataset, and may output a higher relative importance of the policy improvement objective compared to the policy constraint objective for samples that correspond to (i.e., are similar to) random behavior in the offline dataset.

[0025] Fig.3 illustrates a flowchart of a method 300 for training a guided offline reinforcement learning network according to one aspect of the present disclosure. The method 300 may be implemented by a computer. The computer may be any computing device, such as cloud servers, distributed computing units, etc. At block 310, the method 300 may include obtaining an offline reinforcement learning network. The offline reinforcement learning network may provide a policy for an agent to perform an action in a state of an environment. The offline reinforcement learning network is used for robot control, autonomous driving, or healthcare.In robot control applications, an agent (such as the robot or the robot's brain) can perform corresponding actions (such as walking) under different environmental conditions (such as road conditions) based on the policy in the robot's brain provided by a learned RL network. Similarly, in autonomous driving applications, the central controller of a car (i.e., an agent) can control the car, e.g., turning left or braking (i.e., actions) under different traffic conditions (i.e., environment) based on a policy learned by RL. In healthcare applications, an agent (such as an automated robotic arm for surgery) can make decisions about the type of treatment, drug dosage, or check-up time (i.e., actions) at a given time according to the current health status and previous treatment history of a patient (i.e.,the environment) based on a policy learned by RL.

[0026] In one embodiment, in block 310, the offline reinforcement learning network may be obtained by initializing an offline reinforcement learning network with random policy parameters. In another embodiment, an offline reinforcement learning network trained based on the previous or even the current offline RL methods may be obtained and used as an initial offline reinforcement learning network for further optimization. The offline reinforcement learning network may, for example, be trained based on TD3+BC, SAC+BC, IQL, CQL, etc. Although in Fig.3, the method 300 may also include any other known steps for training an offline RL network. For example, the method 300 may also include updating the state / action value function (such as the Q function in Equation 1 or 2) of the offline RL network with the offline dataset.

[0027] At block 320, method 300 may include generating a guidance network from a guidance dataset. The guidance network may evaluate a relative importance of a policy improvement objective and a policy constraint objective to optimize the offline reinforcement learning network. Generating a guidance network at block 320 may include initializing a guidance network with random guidance parameters or updating the guidance network's guidance parameters on the guidance dataset based on updated policy parameters. At block 330, method 300 may include updating policy parameters of the offline reinforcement learning network on an offline dataset by a policy objective as a function of the policy improvement objective and the policy constraint objective based on the relative importance.The updating step may comprise updating the policy parameters toward maximizing the policy objective as a function of the policy improvement objective and the policy constraint objective based on the relative importance output by the guidance network with the updated guidance parameters.

[0028] The steps in block 320 and block 330, as well as one or more other steps of method 300 (if necessary), may alternatively and circularly be performed multiple times to better balance the policy improvement and policy constraint terms and then optimize the offline RL network policy.

[0029] In one embodiment, during the initial circular steps in block 320, the method 300 may include initializing a guidance network with random guidance parameters, and in block 330, the method 300 may include updating the policy parameters θ with a gradient descent step on the offline dataset D as follows, based on the initial B w(t) (t=1) include: θ^(t)(w(t)=θ(t))−αθ1nθ∑k=1nθ[−∇θQ(sk,πθ(sk))|θ(t)+Bw(t)(Lpc(ak,πθ(t)(sk))).∇θLpc(ak,πθ(sk))|θ(t)],(sk,ak)∼D where n θ and α θ the mini-batch size or learning rate for the policy π θ are.

[0030] Then, in the next circular steps, in block 320, the method 300 may update the control parameters of the control network B w on the master data set G, that is (s k , a k ) ~ G, based on the updated θ ^(w) as follows: w(t+1)=w(t)−αw1nw∑k=1nw[−∇wQ(sk,πθ(t)(w(t))(sk))|w(t)+∇wLpc(ak,πθ(t)(w(t))(sk))|w(t)] where n w the mini-batch size and α w the step size for the control network B w In particular, the policy parameters θ^((t))(w (t) ) the updated parameters in equation 3 related to w.

[0031] In block 330, the method 300 may update the policy π θ of the offline reinforcement learning network on an offline dataset by moving the policy parameters θ in the direction of maximizing the policy objective in Equation 2 as follows: θ(t+1)=θ(t)−αθ1nθ∑k=1nθ[−∇θQ(sk,πθ(sk))|θ(t)︸Policy improvement gradient+Bw(t+1)(L pc(ak,πθ(t)(sk)))⋅∇θLpc(ak,πθ(sk))|θ(t)︸Policy Constraint Gradient],(sk,ak)∼D. where w (t+1) in equation 5 of w (t)in equation 3. In equation 5 it can be clearly observed that the control network B w the relative update steps of the policy improvement and policy restriction gradient for each data pair (s k , a k ) in the mini-batches.

[0032] The pseudocode of the disclosed plug-in framework, i.e., guided offline reinforcement learning (GORL), is presented below in Algorithm 1.

[0033] The GORL plug-in approach disclosed here can be applied to various past or future offline RL methods, including TD3+BC, SAC+BC, IQL, CQL, etc. To implement GORL on offline RL methods, one of the most important tasks is to find the corresponding policy constraint term. Such a constraint term may be explicit in some methods (e.g., TD3+BC and SAC+BC), while it is much more implicit in others (e.g., CQL and IQL).

[0034] In one embodiment of the implementation of GORL on TD3+BC, the procedures in Algorithm 1 can be followed and L pc(ak,πθ ) by (π θ (s k ) - a k ) 2 The pseudocode of TD3+BC with GORL is shown in Algorithm 2 as follows. Table 1 shows the hyperparameters of TD3+BC with GORL on Gym Locomotion and Adroit robot manipulation tasks in the D4RL benchmark dataset. Table 1 Hyperparameter Value TD3 hyperparameters optimizer Adam Critic learning rate 3e-4 Actor learning rate 3e-4 Mini-batch size 256 Discount factor 0,99 Target refresh rate 5e-3 Policy noise 0,2 Clipping of guidelines (-0,5, 0,5) Rush Update frequency 2 the directive TD3 architecture Critics Hidden Dim. 256 Critics Covert Layers 2 Critic activation function ReLU n Actor Hidden Dim. 256 Actor Hidden Layers Actor Activation Function 2 ReLU TD3+BC- λ 2,5 / 0,1 Hyperparameter State normalization True GORL hyperparameters Guide network learning rate 1e-5 Update frequency 500 of the control network size of the 200 Key data Size of the master data 20 Mini-batches Control network architecture Concealed dim. 100 Hidden layers 1 Activation function Sigmoid Weight normalization 0 / 20 ng

[0035] In another embodiment of the implementation of GORL on SAC+BC, which is a natural extension of TD3+BC, the replacement of TD3 by SAC as the target of policy optimization is done as follows: πθ=argmaxπθEk∼unif{1,N}[Q(sk,a˜k)−αlog πθ(a˜k|sk)︸maximizing entropy term−Bw((a˜k−ak)2)⋅(a˜k−ak)2], where (s k , a k ) ~ D, α ∈ R is a constant, a ~ k is an action that results from π θ(·|sk) is sampled using a reparameterization trick, and πθ(a~k|sk) denotes the probability that π θ a ~ k in state s kThe Q-function optimization of SAC+BC is the same as that of SAC. By adding the entropy maximization term to Equations 3, 4, and 5, GORL can be applied to SAC+BC using the methods in Algorithm 1. The pseudocode of SAC+BC with GORL is presented below in Algorithm 3.

[0036] Table 2 below shows the hyperparameters of SAC+BC with GORL on the Locomotion / Adroit dataset. Table 2 Hyperparameter Value SAC hyperparameters optimizer Adam Critic learning rate 3e-4 Actor learning rate 3e-4 Mini-batch size 256 Discount factor 0,99 Target refresh rate 5e-3 Alpha target 0,2 Update frequency of the 2 Directive SAC architecture Critics Hidden Dim. 256 Critics Hidden Layers 2 Critic activation function ReLU Actor Hidden Dim. 256 Actor Hidden Layers 2 Actor activation function ReLU SAC+BC- λ 2,5 / 0,1 Hyperparameter State normalization True GORL hyperparameters Guide network learning rate le-5 Update frequency of the 500 control network Size of the guide data 200 Size of the master data mini- 20 Batches Control network architecture Concealed dim. 100 Hidden layers 1 Activation function Sigmoid Weight normalization 0 / 20

[0037] In another embodiment of the GORL on IQL implementation, the goal of the policy update in IQL can be: πθ=argmaxπθE(s,a)∼D[exp(β(Q(s,a)−V(s)))logπθ(a|s)], where V(s) is an approximator of E a[Q(s,a)] The reason behind equation 7 is that when an action a k has an advantage, ie Q(s, a k ) > E a [Q(s, a)] = V(s k ), the term exp (β(Q(s, a k) - V(s))) is greater than its expected value E a[exp(β(Q(s,a)-v(s)))] . Therefore, after updating with equation 7, it is more likely that π θ opt for a k decides than for other actions. It is evident that the scalar β, together with Q(s, a) - V(s), controls the extent to which π θ accepts action a in state s. Based on the above analysis, equation 7 can be reformulated as follows: πθ=argmaxπθEk∼unif{1,N}[exp(β−k(Q−(sk,ak)−V(sk)))logπθ(ak|sk)],(sk,ak)∼D.

[0038] Compared to equation 7, equation 8 has a different scalar β k for each data pair(s) k , a k ). However, it should be noted that exp (β k (Q(s k , a k ) - V(s k ))) = (exp(Q(s k , a k ) - V(s k ))) βk It is difficult to find an optimal β k end-to-end, because β kis the exponent of (exp(Q(s k , a k ) - V(s k ))).

[0039] To optimize β k To enable this, equation 8 can be further modified to: πθ=argmaxπθEk∼unif{1,N}[βkexp(Q(sk,ak)−V(sk))logπθ(ak|sk)],(sk,ak)∼D where β k with exp(Q(s k , a k ) - V(s k )), which is much easier to optimize.

[0040] GORL can be implemented on IQL based on Algorithm 1 with the new objective (Equation 9). B w is used to calculate β k to generate by log π θ (a ~ k | s k ) is used as input. The pseudocode of IQL with GORL is shown in Algorithm 4 as follows.

[0041] Table 3 below shows the hyperparameters of SAC+BC with GORL on the Locomotion / Adroit dataset. Table 3 Hyperparameter Value IQL hyperparameters optimizer Adam Policy learning rate 3e-4 Mini-batch size 256 Dropout rate 0,0 / 0,1 beta 3 / 0,5 quantile 0,7 GORL hyperparameters Guide network learning rate le-5 Update frequency 200 / 1 nz of the control network Size of the guide data 200 Size of the master data 20 Mini-batches Control network architecture Concealed dim. 100 Hidden layers 1 Sigmoi Activation function d Weight normalization 0 / 1

[0042] In another embodiment of the implementation of GORL on CQL, the goal of the policy update in CQL can be: πθ=argmaxπθE(s,a)∼D[Q(s,a)−logπθ(a|s)], where Q(s, a) is a conservative approximation of the state / action value. The policy constraint objective is implicit during conservative Q-learning. The more conservative Q-value represents the stronger policy constraint. In this case, GORL can be implemented according to Algorithm 1, with the new policy update objective as follows: πθ=argmaxπθE(s,a)∼D[Bw(Q(s,a))⋅Q(s,a)−log πθ(a|s)].

[0043] The pseudocode of CQL with GORL is shown in Algorithm 5 as follows.

[0044] Table 4 below shows the hyperparameters of CQL with GORL on the Locomotion / Adroit dataset. Table 4 Hyperparameter Value optimizer Adam CQL hyperparameters Policy learning rate le-4 Mini-batch size 256 Lagrange Threshold -1,0 Min. q-weighting 5,0 / 1,0 Min. q version 3 / 2 GORL hyperparameters Guide network learning rate le-5 Update frequencies 200 / z of the control network 100 Size of the guide data 200 Size of the master data 20 Mini-batches Control network architecture Concealed dim. 100 Hidden layers 1 Activation function Sigmoid Weight normalization 0 / 1 tion

[0045] Fig. 4 illustrates a block diagram of a device 400 according to one aspect of the present disclosure. The device 400 may include a memory 410 and at least one processor 420. In one embodiment, the device 400 may be used to train an offline reinforcement learning network. The processor 420 may be coupled to the memory 410 and configured to perform the method 300 as described above with reference to Fig.3. In another embodiment, the device 400 may be used for a trained offline reinforcement learning network. The processor 420 may be coupled to the memory 410 and configured to implement an offline reinforcement learning network trained by performing the method 300. The processor 420 may be a general-purpose processor, an artificial intelligence processor, or may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. The memory 410 may store the input data, output data, data generated by the processor 420, and / or instructions executed by the processor 420.

[0046] The Guided Offline Reinforcement Learning (GORL) in this disclosure is a general training framework compatible with most offline RL methods. The GORL method can learn a sample-adaptive intensity of the policy constraint under the guidance of only a few high-quality data (i.e., expert data). In particular, GORL can exert a weak constraint on “random-like” samples in the offline dataset, and can exert a strong constraint on “expert-like” samples in the offline dataset. During guided learning, each sample can be assigned a different weight, and the weights differ throughout the training. In particular, when fed with relatively high-quality samples (i.e., samples similar to expert behavior), the agent may be prone to imitation learning; otherwise, when encountering low-quality samples (i.e.,(e.g., on samples that resemble random behaviors), it may choose to deviate slightly from the distribution of those samples. Such adaptive weightings aim to reach the full potential of each sample, resulting in higher performance compared to fixed weighting.

[0047] The rationality of GORL's update mechanism and the near-optimality of GORL's guidance are described below.

[0048] Using the chain rule, equation 4 can be reformulated as follows: w(t+1)=w(t)+αθαwnθ∑k1=1nθ Ck1∂Bw(Lk1policy(θ(t)))∂w|w(t), where Ck1=(1nw∑k2=1nw ∂Lk2guide(θ)∂θ|θ−(t)(w(t)))τ︸Conductivity gradient average∂Lk1policy(θ)∂θ|θ(t),

[0049] The loss of the directive Lk1policy(θ)=Lpc(ak1,πθ(sk1)) with (sk1,ak1)∼D, and the conduction loss Lk2guide(θ)=−Q(sk2,πθ(sk2))+Lpc(ak2,πθ(sk2)) with (sk2,ak2)∼G.

[0050] It can be observed that in equation 12 a larger C k1 the control network B w would lead to a greater degree of restriction for the loss of the relevant directive Lk1policy(θ(t)) Furthermore, it should be noted that in equation 13 C k1 an inner product between the mean of the guide gradient and the gradient of the guideline ∂Lk1policy(θ)∂θ Therefore, B would w those Lk1policy(θ) assign larger weights to gradients whose gradients are close to the mean of the leading gradient. The advantages are twofold: (1) The policy would align its update directions closer to the mean of the leading gradient, whose reliability is theoretically guaranteed; (2) The policy could also benefit from a wealth of information about the environment provided by a large amount of data in the offline dataset D, which is scarce in G due to its sparse dataset.

[0051] To show that the mean value of the conduction gradient ∂Lguide∂θ^=1nw∑k2=1nw∂Lk2guide(θ)∂θ|θ^(t)(w(t)) in equation 13 is suitable for guiding the offline training processes, the guidance gradient obtained on n expert guidance data can be defined as ∂L1:nguide(θ)∂θ be referred to. Formal, ∂L1:nguide(θ)∂θ:=1n∑k=1n ∂Lkguide(θ)∂θ

[0052] As the number of leading data tends to infinity, the mean of the leading gradient reaches its optimal form: ∂L*guide∂θ^:=limn→∞∂L1:nguide(θ^)∂θ^=limn→∞Ek∼unif{1,n}[∂Lkguide(θ^)∂θ^],

[0053] Where unif {1, n} is the uniform distribution on {1, 2,···,n}. It can be proved that as n increases, the guidance gradient on n expert guidance data ∂L1:nguide∂θ to the optimal conduction gradient ∂Liguide∂θ with a probability of rate ≥1n will converge.

[0054] In other words, when the guidance dataset G has enough expert data, the mean of the guidance gradient in Equation 13 approaches the optimal gradient and thus provides reliable guidance for the offline RL algorithms.

[0055] Compared to a vanilla scheme that simply mixes expert demonstrations with the offline dataset, guided training makes better use of the limited high-quality data. Fig. Figure 5A illustrates performance comparisons between the mixed regimen (e.g., vanilla) and the instructed regimen for a different number of expert trials. In Fig. 5A, the horizontal axis is the number of expert trials, the vertical axis is the percentage difference between these two schedules, the gray bars correspond to the instructed schedule (denoted as D(e)→D), and the black bars correspond to the mixed schedule (denoted as D(e)+D). As in Fig.As shown in Figure 5A, a fairly small amount of expert data, e.g., a hundred or several hundred (the size of the offline dataset is typically 1 million), can be sufficient for the guidance dataset G to generate a sufficiently good mean of the guidance gradient in Equation 13. When the amount of expert data is small, the guided scheme consistently outperforms the mixed scheme.

[0056] Fig. Figure 5B shows the result of a policy trained on a pure expert dataset (denoted as “D(e)”) with different dataset scales. In Fig. 5B, the horizontal axis is the number of expert samples, and the vertical axis is the normalized score. It is obvious that the guideline scores remain quite low until the number of expert samples reaches 10. 4 achieved, which shows that a large amount of training data is necessary for offline RL.

[0057] Out of Fig. 5A and Fig. 5B shows that the limited expert data itself cannot generate a satisfactory agent because the training samples are insufficient; and the GORL method can generate reliable guidance for offline RL with only a few expert samples.

[0058] The various operations, modules, and networks described herein in connection with the disclosure may be implemented in hardware, processor-executed software, firmware, or any combination thereof. According to one embodiment of the disclosure, a computer program product for training an offline reinforcement learning network may include processor-executable computer code for performing the operation described above with reference to Fig.3 described method 300. According to one embodiment of the disclosure, a computer program product for an offline reinforcement learning network may comprise processor-executable computer code, when executed by a processor, that causes the processor to implement the offline reinforcement learning network trained by performing the method 300. According to another embodiment of the disclosure, a computer-readable medium may store computer code for training an offline reinforcement learning network, wherein the computer code, when executed by a processor, may cause the processor to implement the method described above with reference to Fig.3. According to another embodiment of the disclosure, a computer-readable medium may store computer code for an offline reinforcement learning network, where the computer code, when executed by a processor, may cause the processor to implement the offline reinforcement learning network trained by performing the method 300. Computer-readable media includes both non-transitory, computer-readable storage media and communication media, including any media that supports the transfer of a computer program from one location to another. Any connection may be referred to as a computer-readable medium. Other embodiments and implementations are within the scope of the disclosure.

[0059] The foregoing description of the disclosed embodiments is provided to enable one skilled in the art to make or use the various embodiments. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the scope of the various embodiments. Thus, the claims are not intended to be limited to the embodiments shown herein, but are to be accorded the broadest scope consistent with the following claims and the principles and novel features disclosed herein. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited non-patent literature

[0000] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021

[0017] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel and Sergey Levine. Soft actor-critic: Off policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018

[0017] Aviral Kumar, Aurick Zhou, George Tucker und Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179-1191, 2020

[0017] Ilya Kostrikov, Ashvin Nair und Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169, 2021

[0017]

Claims

[1] A computer-implemented method for training an offline reinforcement learning network, comprising: Maintaining an offline reinforcement learning network, the offline reinforcement learning network providing a policy for an agent to perform an action in a state of an environment; Generating a guidance network on a guidance dataset, wherein the guidance network outputs a relative importance of a policy improvement objective and a policy constraint objective for optimizing the offline reinforcement learning network; and Updating policy parameters of the offline reinforcement learning network on an offline dataset by a policy objective as a function of the policy improvement objective and the policy restriction objective based on relative importance. [2] The computer-implemented method of claim 1, wherein the offline reinforcement learning network is used for robot control, autonomous driving, or healthcare. [3] The computer-implemented method of claim 1, wherein the guidance network takes the policy constraint target as input and outputs a constraint degree for the policy constraint target, the constraint degree being different for different samples in the offline data set and the guidance data set. [4] The computer-implemented method of claim 1, wherein the guidance network outputs a higher relative importance of the policy constraint objective compared to the policy improvement objective for high-quality samples in the offline dataset, and outputs a higher relative importance of the policy improvement objective compared to the policy constraint objective for low-quality samples in the offline dataset. [5] The computer-implemented method of claim 1, wherein the lead dataset includes hundreds of high-quality samples collected from expert behavior. [6] The computer-implemented method of claim 1, wherein obtaining an offline reinforcement learning network comprises: Initializing an offline reinforcement learning network with random policy parameters. [7] The computer-implemented method of claim 6, further comprising: updating a value function of the offline reinforcement learning network on a mini-batch of offline data sampled from the offline data set. [8] The computer-implemented method of claim 7, wherein generating a guidance network comprises: Initializing a control network with random control parameters. [9] The computer-implemented method of claim 8, wherein updating policy parameters of the offline reinforcement learning network comprises: Update the policy parameters with a gradient descent step on the mini-batch of offline data based on the relative importance output by the guidance network with the random guidance parameters. [10] The computer-implemented method of claim 9, wherein generating a guidance network further comprises: updating guidance parameters of the guidance network on a mini-batch of guidance data sampled from the guidance data set, based on the updated policy parameters. [11] The computer-implemented method of claim 10, wherein updating policy parameters of the offline reinforcement learning network further comprises: Updating the policy parameters toward maximizing the policy objective as a function of the policy improvement objective and the policy constraint objective based on the relative importance output by the guidance network with the updated guidance parameters. [12] Apparatus for training an offline reinforcement learning network, comprising: a memory; and at least one processor coupled to the memory and configured to perform the computer-implemented method of any one of claims 1 to 11. [13] A computer-readable medium storing computer code for training an offline reinforcement learning network, the computer code, when executed by a processor, causing the processor to perform the computer-implemented method of any one of claims 1 to 11. [14] A computer program product for training an offline reinforcement learning network, comprising: processor-executable computer code for performing the computer-implemented method according to any one of claims 1 to 11. [15] Device for an offline reinforcement learning network, comprising: a memory; and at least one processor coupled to the memory and configured to implement the offline reinforcement learning network trained by executing the computer-implemented method of any one of claims 1 to 11. [16] A computer-readable medium storing computer code for an offline reinforcement learning network, the computer code, when executed by a processor, causing the processor to implement the offline reinforcement learning network trained by performing the computer-implemented method of any one of claims 1 to 11. [17] A computer program product for an offline reinforcement learning network, comprising: processor-executable computer code for implementing the offline reinforcement learning network trained by performing the computer-implemented method of any one of claims 1 to 11.

Images