Method and apparatus for guided offline reinforcement learning

Abstract

The invention discloses a method for training an offline reinforcement learning network. The method comprises the steps that an off-line reinforcement learning network is obtained, and the off-line reinforcement learning network provides a strategy for an intelligent agent to take actions in an environment state; generating a bootstrap network on a bootstrap data set, the bootstrap network outputting a relative importance of a policy improvement target and a policy constraint target for optimizing the offline reinforcement learning network; and updating policy parameters of the offline reinforcement learning network on an offline data set through a policy target as a function of the policy improvement target and the policy constraint target based on the relative importance.

Description

Technical Field

[0001] The present disclosure relates generally to the field of artificial intelligence technology, and more particularly, to offline reinforcement learning (RL) with expert guidance. Background Art

[0002] Reinforcement learning (RL) is an important field of machine learning, which aims to solve the problem of how an agent should take actions in different states of the environment in order to maximize a certain cumulative reward. Typically, reinforcement learning includes online RL and offline RL. Online RL learns strategies by interacting with the environment, while offline RL learns strategies by optimizing strategies based only on offline datasets without any interaction with the environment. Since offline RL learns strategies on previously collected datasets, it usually suffers from the distribution shift problem caused by the gap between the state-action distribution of the offline dataset and the interaction between the current strategy and the test environment. Specifically, after optimizing on the offline dataset, the agent may encounter unvisited states or misestimated state-action values ​​during interaction with the online environment, resulting in poor performance.

[0003] To alleviate this problem, existing solutions adopt a single compromise between two conflicting objectives of offline RL, namely, the policy improvement objective that aims to optimize the policy according to the current value function, and the policy constraint objective that keeps the behavior of the policy around the offline dataset to avoid the policy being too aggressive. Therefore, previous solutions either add explicit policy constraint terms to the policy improvement equation or implicitly constrain the policy by modifying the update rule of the value function. However, these solutions usually only focus on the global characteristics of the dataset, but ignore the individual characteristics of each sample. Typically, they only make a single trade-off for all data in a small batch or even the entire offline dataset. This "one-size-fits-all" trade-off may not achieve a perfect balance for each sample, which may limit the performance potential.

[0004] Therefore, there is a need to provide an improved scheme for offline reinforcement learning. Summary of the invention

[0005] A simplified summary of one or more aspects of the present disclosure is presented below to provide a basic understanding of these aspects. This summary is not an extensive overview of all contemplated aspects, and is neither intended to identify key or important elements of all aspects, nor to delineate the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to a more detailed description presented later.

[0006] In one aspect of the present disclosure, a method for offline reinforcement learning is disclosed. The method includes: obtaining an offline reinforcement learning network, wherein the offline reinforcement learning network provides a strategy for an agent to take actions under the state of an environment; generating a guidance network on a guidance data set, wherein the guidance network outputs a relative importance of optimizing a policy improvement target and a policy constraint target of the offline reinforcement learning network; and updating a policy parameter of the offline reinforcement learning network on the offline data set through a policy target that is a function of the policy improvement target and the policy constraint target based on the relative importance.

[0007] In another aspect of the present disclosure, a device for offline reinforcement learning is disclosed. The device may include a memory and at least one processor coupled to the memory. The at least one processor may be configured to: obtain an offline reinforcement learning network, wherein the offline reinforcement learning network provides a strategy for an agent to take actions under the state of an environment; generate a guidance network on a guidance data set, wherein the guidance network outputs a relative importance of optimizing a policy improvement target and a policy constraint target of the offline reinforcement learning network; and based on the relative importance, update the policy parameters of the offline reinforcement learning network on the offline data set through a policy target that is a function of a policy improvement target and a policy constraint target.

[0008] In another aspect of the present disclosure, a computer-readable medium storing computer code for offline reinforcement learning is disclosed. The computer code, when executed by a processor, can cause the processor to: obtain an offline reinforcement learning network, wherein the offline reinforcement learning network provides a strategy for an agent to take actions under the state of an environment; generate a guidance network on a guidance data set, wherein the guidance network outputs a relative importance of a policy improvement target and a policy constraint target for optimizing the offline reinforcement learning network; and based on the relative importance, update the policy parameters of the offline reinforcement learning network on the offline data set through the policy target as a function of the policy improvement target and the policy constraint target.

[0009] In another aspect of the present disclosure, a computer program product for offline reinforcement learning is disclosed. The computer program product may include a processor executable computer code for obtaining an offline reinforcement learning network, wherein the offline reinforcement learning network provides a strategy for an agent to take actions under the state of an environment; generating a guidance network on a guidance data set, wherein the guidance network outputs a relative importance of optimizing a policy improvement target and a policy constraint target of the offline reinforcement learning network; and based on the relative importance, updating the policy parameters of the offline reinforcement learning network on the offline data set through the policy target as a function of the policy improvement target and the policy constraint target.

[0010] Other aspects or variations of the present disclosure will become apparent by consideration of the following detailed description and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The following drawings depict various embodiments of the present disclosure for illustrative purposes only. Those skilled in the art will readily appreciate from the following description that alternative embodiments of the methods and structures disclosed herein can be implemented without departing from the spirit and principles of the present disclosure described herein.

[0012] Figure 1 The basic structure of a reinforcement learning model according to one aspect of the present disclosure is shown.

[0013] Figure 2 A block diagram of a method for guided offline reinforcement learning according to one aspect of the present disclosure is shown.

[0014] Figure 3 A flowchart of a method for guided offline reinforcement learning according to one aspect of the present disclosure is shown.

[0015] Figure 4 A block diagram of an apparatus for guided offline reinforcement learning according to one aspect of the present disclosure is shown.

[0016] Figure 5A and Figure 5B A performance comparison for different numbers of expert samples according to one aspect of the present disclosure is shown. DETAILED DESCRIPTION

[0017] Before explaining any embodiment of the present disclosure in detail, it should be understood that the application of the present disclosure is not limited to the construction details and feature arrangements set forth in the following description. The present disclosure is capable of other embodiments and can be practiced or carried out in various ways.

[0018] Figure 1 The basic structure of a reinforcement learning model according to one aspect of the present disclosure is shown. Reinforcement learning mainly includes an agent, an environment, a state, an action, and a reward. Figure 1 As shown, for the current state s of the environment 120 t , the agent 110 can perform action a on the environment 120 based on the policy. Then, the environment 120 can transition to a new state s t+1 , and the reward r is given to the agent 110 as feedback from the environment 120. Subsequently, the agent 110 can cyclically perform new actions according to the reward and the new state of the environment 120. The goal of the strategy of the agent 110 is to maximize the accumulated reward. This process shows how the agent and the environment interact through states, actions, and rewards.

[0019] Through reinforcement learning, agents can determine what actions they should take to get the maximum reward when they are in different states of the environment. For example, in a robot control application, the agent can be a robot or particularly the brain of the robot, and the environment can be the environment where the robot works. The environment can have various states, such as obstacles, slopes, potholes, etc. The robot can take different actions in different states based on the strategy learned in its brain, such as to walk on a road. If the robot takes the right action and bypasses the obstacle, it can get a higher total reward.

[0020] Reinforcement learning is usually expressed as a Markov decision process (MDP), represented as a tuple in is the state space, is the action space, P(s t+1 ∣s t ,a) represents the state transition probability of the environment, s t and t+1 belongs to the state space, a belongs to the action space, and the agent’s strategy can be based on such probabilities, d0(s0) represents the distribution of the initial state s0, R(s t ,a,s t+1 ) defines the reward function, and γ∈(0,1] is the discount factor.

[0021] Unlike online RL, which learns policies by interacting with the environment, offline RL aims to learn policies only through offline datasets. To optimize the strategy, where a k is the state of the environment k The action taken in k It is due to a k The state of the conversion, and r k is the corresponding reward. This property of offline RL brings convenience to applications in many fields where online interactions are expensive or dangerous, such as robotic control, autonomous driving, and healthcare. Although there are various offline RL schemes with different training losses, the objectives of most of them explicitly or implicitly include two conflicting objectives: (1) policy improvement, which aims to optimize the policy according to the current value function; and (2) policy constraints, which keep the policy around the distribution of behavioral policies or offline datasets. Offline RL must balance these two objectives: if too much focus is placed on the policy improvement term, the policy may enter unfamiliar areas and produce bad behaviors due to distribution shift; in addition, excessive focus on the policy constraint term may cause the policy to only imitate the behavior in the offline dataset and may lack the ability to generalize to out-of-distribution data.

[0022] For example, previous offline RL schemes may include: TD3+BC (Scott Fujimoto and Shixiang ShaneGu. A minimalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021); its variant SAC+BC, which applies SAC (Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off policy maximum entropy deep reinforcement learning with a stochastic actor. International conference on machine learning, pp. 1861–1870. PMLR, 2018) to the TD3+BC framework; CQL (Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33: 1179–1191, 2020); and IQL (Ilya Kostrikov, Ashvin Nair, and Sergey Levine.Offline reinforcement learning with implicit q-learning.arXiv preprint arXiv:2110.06169,2021). The policy optimization objectives of these schemes can be unified as:

[0023]

[0024] in is a policy with trainable parameters θ, is the state-action value function, which estimates the expected sum of discounted rewards after taking action a in state s. In addition, L pi (·) and L pc (·) represents the policy improvement term and policy constraint term, and F(·) is L pi (·) and L pc (·) is a trade-off function between is the degree of constraint: the larger the d c Stronger policy constraints will be encouraged, and thus the policy becomes more conservative; otherwise the policy will place more emphasis on policy improvements, and thus tend to be more aggressive.

[0025] The ideal improvement-constraint balance for offline RL is to focus more on policy constraints for samples that resemble expert behavior, but to emphasize policy improvement more for data that resembles random behavior. In addition, many online RL methods have demonstrated that even a small amount of expert demonstrations will benefit policy performance, but current offline RL methods do not fully utilize expert data.

[0026] In this disclosure, we provide an offline RL method that can determine the adaptive trade-off between policy improvement and policy constraints for different samples with only a small amount of expert data as guidance.

[0027] Figure 2 A block diagram of a method for guided offline reinforcement learning (GORL) according to one aspect of the present disclosure is shown. Offline dataset 210 can contain a large amount of data and can be used to train RL network 220. RL network 220 can provide a policy for the agent. Guided dataset 230 can consist of a small amount of expert data similar to expert demonstrations or behaviors and can be used to train guided network 240. Guided network 240 can be used to guide the training of the policy of RL network 220. The GORL method can alternate between updating the guided network on the guided dataset in a manner similar to MAML (model-agnostic meta-learning) and training the RL agent on the offline dataset with the guidance of the updated guided network.

[0028] The GORL method is a plug-in method and can adaptively and end-to-end evaluate the policy improvement and relative importance of policy constraints for each data. The GORL method points out the theoretically guaranteed optimization direction of the agent and can be easily implemented on most offline RL schemes. With only a small amount of expert data, the GORL method can achieve significant performance improvements on many state-of-the-art offline RL schemes.

[0029] In the GORL method, the offline dataset can be represented as And the bootstrap dataset can be expressed as Where M<<N, and For example, is a large offline dataset containing trajectories from suboptimal or even random policies, and is a bootstrapped dataset with a small amount of optimal data, such as collected by an expert or near-expert strategy.

[0030] For simplicity, similar to TD3+BC, the training objective of the GORL method can be formulated as:

[0031]

[0032] Among them, unif{1,N} represents the uniform distribution on {1,2,…,N}, and L pc (·) represents a policy constraint, such as (π θ (s k )-a k ) 2 . A guided network with parameter w The policy constraint L pc As input, and outputs the degree of constraint. Note that this is different from assigning a fixed degree of constraint to all data (e.g., d c ) is different from the typical framework of existing offline solutions (as shown in Equation 1), in the GORL method, the guided network The degree of constraint generated can vary with different state-action pairs (s k ,a k ). In equation (2), the guidance network may take the policy constraint target as input and output the degree of constraint of the policy constraint target. The degree of constraint varies for different samples in the offline dataset and the guidance dataset. For example, the guidance network may output a higher relative importance of the policy constraint target than the policy improvement target of the sample corresponding to (i.e., approximating) the expert behavior in the offline dataset, and the guidance network may output a higher relative importance of the policy improvement target than the policy constraint target of the sample corresponding to (i.e., approximating) the random behavior in the offline dataset.

[0033] Figure 3A flow chart of a method 300 for training a guided offline reinforcement learning network according to one aspect of the present disclosure is shown. The method 300 may be implemented by a computer. The computer may be any computing device, such as a cloud server, a distributed computing entity, etc. In box 310, the method 300 may include obtaining an offline reinforcement learning network. The offline reinforcement learning network may provide an agent with a strategy for taking actions in an environmental state. The offline reinforcement learning network is used for robot control, autonomous driving, or healthcare. In robot control applications, an agent (e.g., a robot or a robot's brain) may take corresponding actions (e.g., walking) in different states of the environment (e.g., road conditions) based on the strategy in the robot's brain provided by the learned RL network. Similarly, in autonomous driving applications, the central control unit of a car (i.e., an agent) may drive the car, such as turning left or braking (i.e., actions), in different traffic conditions (i.e., environments) based on the strategy learned through RL. In healthcare applications, an agent (such as an automated robotic arm for surgery) can make decisions such as type of treatment, medication dosage, or time for an exam (i.e., action) at a certain point in time based on a policy learned via RL, depending on the patient’s current health state and previous treatment history (i.e., environment).

[0034] In one embodiment, in block 310, an offline reinforcement learning network may be obtained by initializing the offline reinforcement learning network with random policy parameters. In another embodiment, an offline reinforcement learning network trained based on a previous or even current offline RL method may be obtained and used as an initial offline reinforcement learning network for further optimization. For example, the offline reinforcement learning network may be trained based on TD3+BC, SAC+BC, IQL, CQL, etc. Although Figure 3 Although not shown in FIG. 3 , method 300 may also include any other well-known steps for training an offline RL network. For example, method 300 may also include updating a state-action value function (e.g., a Q function in Equation 1 or 2) of the offline RL network using the offline dataset.

[0035] In block 320, method 300 may include generating a guided network on a guided data set. The guided network may evaluate the relative importance of a policy improvement objective and a policy constraint objective to optimize the offline reinforcement learning network. Generating the guided network in block 320 may include initializing the guided network with random guided parameters, or updating guided parameters of the guided network on the guided data set based on the updated policy parameters. In block 330, method 300 may include updating policy parameters of the offline reinforcement learning network on the offline data set based on the relative importance, using the policy objective as a function of the policy improvement objective and the policy constraint objective. The updating step may include updating the policy parameters in a direction that maximizes the policy objective as a function of the policy improvement objective and the policy constraint objective based on the relative importance evaluated using the updated guided parameters.

[0036] The steps in box 320 and box 330 and one or more other steps of method 300 (if necessary) can be performed multiple times alternately and cyclically to better balance the policy improvement term and the policy constraint term, and then optimize the policy of the offline RL network.

[0037] In one embodiment, during the initial loop step, at block 320, method 300 may include initializing the bootstrap network using random bootstrap parameters, and at block 330, method 300 may include initializing the bootstrap network based on the initial bootstrap parameters. The following is the offline dataset The policy parameters θ are updated using a gradient descent step:

[0038]

[0039]

[0040] Among them, n θ and α θ The strategies are θ The mini-batch size and learning rate are

[0041] Then, in a subsequent loop step, in block 320, method 300 may include: In the bootstrap dataset On, that is Update Boot Network The boot parameters are as follows:

[0042]

[0043] Among them, n w is the mini-batch size, α w Is the boot network In particular, the policy parameters is the update parameter related to w in Equation 3.

[0044] In block 330, method 300 may include updating a policy π of the offline reinforcement learning network on the offline dataset by moving the policy parameters θ in a direction that maximizes the policy objective in Equation 2 θ , as shown below:

[0045]

[0046] Where w in Equation 5 (t+1) can be different from w in Equation 3 (t) It can be clearly observed in Equation 5 that the guided network Control each data pair (s k ,a k ) is a relative update step of the policy improvement gradient and policy constraint gradient.

[0047] The pseudo code of the disclosed plugin framework, namely Guided Offline Reinforcement Learning (GORL), is presented in Algorithm 1 below.

[0048]

[0049] The GORL plug-in method of the present disclosure can be applied to various previous or future offline RL methods, including TD3+BC, SAC+BC, IQL, CQL, etc. In order to implement GORL on offline RL methods, one of the most important tasks is to find the corresponding policy constraints. Such constraints can be explicit in some methods (e.g., TD3+BC and SAC+BC) and more implicit in other methods (e.g., CQL and IQL).

[0050] In one embodiment of implementing GORL on TD3+BC, the process in Algorithm 1 can be followed by using (π θ (s k )-a k ) 2 Replace L pc (a k ,π θ ). The pseudo code of TD3+BC using GORL is presented in Algorithm 2 below.

[0051]

[0052] Table 1 below shows the hyperparameters of TD3+BC with GORL for Gym locomotion and Adroit robotic manipulation tasks in the D4RL benchmark dataset.

[0053]

[0054] Table 1

[0055] In another embodiment of implementing GORL on SAC+BC, which is a natural extension of TD3+BC, TD3 is replaced by SAC as the policy optimization objective as follows:

[0056]

[0057] in, is a constant, is obtained by reparameterization from π θ (·∣s k ) sampling action, and Represents π θ In status k Next Select The Q function optimization of SAC+BC is the same as that of SAC. By adding the maximization entropy term to Equations 3, 4, and 5, GORL can be applied to SAC+BC using the process in Algorithm 1. The pseudo code of SAC+BC using GORL is presented in Algorithm 3 below.

[0058]

[0059] Table 2 below shows the hyperparameters of SAC+BC using GORL on the locomotion / adroit dataset.

[0060]

[0061] Table 2

[0062] In another embodiment of implementing GORL on IQL, the policy update target in IQL can be:

[0063]

[0064] Where V(s) is The reason behind Equation 7 is that if an action a k is beneficial, that is Then exp(β(Q(s,a k )-V(s))) will be greater than its expected value Therefore, after updating with Equation 7, π θ More likely to choose a k Instead of other actions. It can be seen that the scalar β, together with Q(s,a)-V(s), controls π θ The degree to which action a is accepted in state s. Based on the above analysis, Equation 7 can be restated as:

[0065]

[0066] Compared with Equation 7, Equation 8 is k ,a k ) assign different scalar β k However, note that Because β k is(exp(Q(s k ,a k )-V(s k ))) index, so it is difficult to find the end-to-end optimal β k .

[0067] In order to make β k The optimization becomes possible, and Equation 8 can be further changed to:

[0068]

[0069] Among them, β k Multiply by exp(Q(s k ,a k )-V(s k )), which is easier to optimize.

[0070] GORL can be implemented on IQL based on Algorithm 1 modified with a new objective (Equation 9). The pseudo code of IQL with GORL is presented in Algorithm 4 below.

[0071]

[0072] Table 3 below shows the hyperparameters of SAC+BC using GORL on the locomotion / adroit dataset.

[0073]

[0074] Table 3

[0075] In another embodiment of implementing GORL on CQL, the policy update target in CQL can be:

[0076]

[0077] Where Q(s,a) is a conservative approximation of the state-action value. The policy constraint objective is implicitly included during conservative Q learning. More conservative Q values ​​indicate stronger policy constraints. In this case, GORL can be implemented following Algorithm 1, where the new policy update objective is as follows:

[0078]

[0079] The pseudo code for CQL with GORL is presented in Algorithm 5 below.

[0080]

[0081] Table 4 below shows the hyperparameters of CQL with GORL on the locomotion / adroit dataset.

[0082]

[0083] Table 4

[0084] Figure 4 4 shows a block diagram of an apparatus 400 according to one aspect of the present disclosure. The apparatus 400 may include a memory 410 and at least one processor 420. In one embodiment, the apparatus 400 may be used to train an offline reinforcement learning network. The processor 420 may be coupled to the memory 410 and configured to execute the above-referenced Figure 3 The method 300 described. In another embodiment, the apparatus 400 can be used for a trained offline reinforcement learning network. The processor 420 can be coupled to the memory 410 and configured to implement the offline reinforcement learning network trained by executing the method 300. The processor 420 can be a general-purpose processor, an artificial intelligence processor, or can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. The memory 410 can store input data, output data, data generated by the processor 420, and / or instructions executed by the processor 420.

[0085] Guided Offline Reinforcement Learning (GORL) in the present disclosure is a general training framework compatible with most offline RL methods. The GORL method can learn the policy constraint strength of sample adaptation under the guidance of only a small amount of high-quality data (i.e., expert data). Specifically, GORL can impose weak constraints on "random-like" samples in offline datasets, and can impose strong constraints on "expert-like" samples in offline datasets. During guided learning, each sample can be assigned a different weight, and the weight changes during training. Specifically, when fed with relatively high-quality samples (i.e., samples similar to expert behavior), the agent may tend to imitate learning; otherwise, when encountering low-quality samples (i.e., samples similar to random behavior), it may choose to deviate slightly from the distribution of these samples. This adaptive weight attempts to realize the full potential of each sample, resulting in higher performance compared to fixed weights.

[0086] The rationality of GORL’s update mechanism and the near-optimality of guidance from GORL are described below.

[0087] Using the chain rule, Equation 4 can be reformulated as:

[0088]

[0089] in

[0090]

[0091] Here, the loss of the strategy in Guided Loss in

[0092] It can be observed that in Equation 12, the larger Will encourage the guidance of the network For the corresponding strategy loss Output a greater degree of constraint. Further note that in Equation 13, is the average of the guided gradients and the policy gradient Therefore, will be those whose gradients are close to the mean of the guided gradients The benefits are twofold: (1) the policy will keep its update direction closer to the guided gradient average whose reliability is theoretically guaranteed; (2) the policy can also enjoy the same The large amount of data in the is scarce due to its small data size.

[0093] To prove that the guided gradient average in Equation 13 is eligible to guide the offline training process, and the guided gradient obtained on n expert guided data can be expressed as formal,

[0094]

[0095] If the number of guided data approaches infinity, the guided gradient average reaches its optimal form:

[0096]

[0097] Where unif{1,n} is a uniform distribution on {1,2,…,n}. It can be shown that as n increases, n experts guide the data The guided gradient on Converges to the optimal guided gradient in probability

[0098] In other words, when bootstrapping the dataset With enough expert data, the guided gradient average in Equation 13 will approximate the optimal gradient, thus providing reliable guidance for offline RL algorithms.

[0099] Compared to the vanilla scheme of simply mixing expert demonstrations with offline datasets, guided training makes better use of limited high-quality data. Figure 5A The performance comparison between the hybrid scheme (such as vanilla) and the guided scheme on different numbers of expert samples is shown. Figure 5A In , the horizontal axis is the number of expert samples, the vertical axis is the percentage difference between the two solutions, the gray bar corresponds to the guided solution (denoted as D(e)→D), and the black bar corresponds to the hybrid solution (denoted as D(e)+D). Figure 5A As shown, a very small amount of expert data (e.g., one hundred or a few hundred (the size of an offline dataset is typically 1 million)) may be sufficient to bootstrap the dataset. A sufficiently good guided gradient average is generated in Equation 13. When the amount of expert data is small, the guided scheme is always better than the hybrid scheme.

[0100] Figure 5B Results are shown for policies trained on expert-only datasets (denoted as “D(e)”) with different dataset scales. Figure 5B In the example, the horizontal axis is the number of expert samples and the vertical axis is the normalized score. Obviously, the score of the strategy remains quite low until the number of expert samples reaches 10. 4 , which shows that offline RL requires a large amount of training data.

[0101] from Figure 5A and 5B As can be seen in Figure 2, limited expert data itself cannot produce a satisfactory agent due to insufficient training samples; the GORL method can generate reliable guidance for offline RL using only a small number of expert samples.

[0102] The various operations, modules, and networks described in conjunction with the disclosure herein may be implemented in hardware, software executed by a processor, firmware, or any combination thereof. According to an embodiment of the present disclosure, a computer program product for training an offline reinforcement learning network may include a computer program for executing the above reference Figure 3 According to an embodiment of the present disclosure, a computer program product for an offline reinforcement learning network may include a processor executable computer code, which, when executed by a processor, enables the processor to implement an offline reinforcement learning network trained by executing method 300. According to another embodiment of the present disclosure, a computer readable medium may store a computer code for training an offline reinforcement learning network, which, when executed by a processor, enables the processor to perform the above reference Figure 3Method 300 described. According to another embodiment of the present disclosure, a computer readable medium may store a computer code for an offline reinforcement learning network, which, when executed by a processor, may enable the processor to implement an offline reinforcement learning network trained by executing method 300. Computer readable media include both non-transitory computer storage media and communication media, including any media that facilitates the transfer of a computer program from one place to another. Any connection may be appropriately referred to as a computer readable medium. Other embodiments and implementations are within the scope of the present disclosure.

[0103] The foregoing description of the disclosed embodiments is provided to enable those skilled in the art to make or use the various embodiments. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the scope of the various embodiments. Therefore, the claims are not intended to be limited to the embodiments shown herein, but to be consistent with the broadest scope consistent with the following claims and the principles and novel features disclosed herein.

Claims

1. A computer-implemented method for training an offline reinforcement learning network, comprising: Obtaining an offline reinforcement learning network, wherein the offline reinforcement learning network provides a strategy for the agent to take actions under the state of the environment; Generating a guided network on a guided data set, wherein the guided network output is used to optimize the relative importance of a policy improvement objective and a policy constraint objective of the offline reinforcement learning network; and Based on the relative importance, a policy parameter of the offline reinforcement learning network is updated on the offline dataset via a policy objective that is a function of the policy improvement objective and the policy constraint objective.

2. The computer-implemented method of claim 1 , wherein: The offline reinforcement learning network is used for robotic control, autonomous driving or healthcare.

3. The computer-implemented method of claim 1 , wherein: The guidance network takes the policy constraint target as input and outputs a constraint degree of the policy constraint target, wherein the constraint degree varies for different samples in the offline dataset and the guidance dataset.

4. The computer-implemented method of claim 1 , wherein: The guided network outputs a higher relative importance of the policy constraint target compared to the policy improvement target of high-quality samples in the offline dataset, and the guided network outputs a higher relative importance of the policy improvement target compared to the policy constraint target of low-quality samples in the offline dataset.

5. The computer-implemented method of claim 1 , wherein: The bootstrap dataset consists of hundreds of high-quality samples collected from expert behaviors.

6. The computer-implemented method of claim 1 , wherein: The obtaining of the offline reinforcement learning network comprises: Initialize an offline reinforcement learning network with random policy parameters.

7. The computer-implemented method of claim 6, further comprising: The value function of the offline reinforcement learning network is updated on a small batch of offline data sampled from the offline dataset.

8. The computer-implemented method of claim 7, wherein: The generation guidance network comprises: Initialize the bootstrap network with random bootstrap parameters.

9. The computer-implemented method of claim 8, wherein: The updating of the strategy parameters of the offline reinforcement learning network includes: The policy parameters are updated using a gradient descent step on the mini-batch of offline data based on the relative importance output by the bootstrap network with the random bootstrap parameters.

10. The computer-implemented method of claim 9, wherein: The generation guidance network also includes: Based on the updated policy parameters, the guidance parameters of the guidance network are updated on a mini-batch of guidance data sampled from the guidance dataset.

11. The computer-implemented method of claim 10, wherein: The updating of the strategy parameters of the offline reinforcement learning network further includes: Based on the relative importance output by the guided network with updated guided parameters, the policy parameters are updated in a direction that maximizes the policy objective as a function of the policy improvement objective and the policy constraint objective.

12. An apparatus for training an offline reinforcement learning network, comprising: Memory; as well as At least one processor coupled to the memory and configured to perform the computer-implemented method according to one of claims 1-11.

13. A computer readable medium storing computer code for training an offline reinforcement learning network, the computer code, when executed by a processor, causing the processor to perform the computer-implemented method according to one of claims 1-11.

14. A computer program product for training an offline reinforcement learning network, comprising: Processor executable computer code for performing the computer implemented method according to one of claims 1-11.

15. An apparatus for offline reinforcement learning network, comprising: Memory; as well as At least one processor coupled to the memory and configured to implement an offline reinforcement learning network trained by executing the computer-implemented method according to one of claims 1-11.

16. A computer-readable medium storing computer code for an offline reinforcement learning network, which, when executed by a processor, causes the processor to implement an offline reinforcement learning network trained by performing a computer-implemented method according to one of claims 1-11.

17. A computer program product for an offline reinforcement learning network, comprising: Processor-executable computer code for implementing an offline reinforcement learning network trained by performing a computer-implemented method according to one of claims 1-11.

Images