Learning device and learning method

Abstract

To provide a learning device and a learning method that can reduce the problems in reinforcement learning.SOLUTION: A learning device includes: a dataset acquisition unit configured to acquire a dataset including state information and action information on which a policy is to be learned; a discrete latent variable estimation unit configured to estimate a discrete latent variable representing characteristics of features from the state information and the action information; an optimal action learning unit configured to learn an optimal action using the state information and the discrete latent variable; a value function estimation unit configured to learn an action value from the state information and the action information; and an identification unit configured to identify a discrete latent variable that maximizes the action value using a result from the optimal action learning unit and a result from the value function estimation unit.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a learning device and a learning method. [Background technology]

[0002] Reinforcement learning has achieved remarkable results in various applications, but most of these successes have been achieved in online learning environments where a reinforcement learning agent interacts with the environment during the learning process. For example, in reinforcement learning, a predictive model is generated using multiple input parameters (see, for example, Patent Document 1).

[0003] In reinforcement learning, interactions with the environment require time and computational costs. Therefore, offline reinforcement learning, also known as batch reinforcement learning, has been attracting attention in order to reduce the number of interactions (see, for example, Non-Patent Document 1). In offline reinforcement learning, the goal is to learn an optimal policy from a dataset collected through an arbitrary and unknown process. Recent research has shown that offline reinforcement learning can significantly reduce the number of interactions with the environment required to achieve satisfactory performance. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2020-14841 [Non-patent literature]

[0005] [Non-Patent Document 1] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, et al., “AWAC: Accelerating Online Reinforcement Learning with Offline Datasets”, Machine Learning (cs.LG); Robotics (cs.RO); Machine Learning (stat.ML), arXiv:2006.09359 [cs.LG], 2006 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the performance of policies obtained by conventional offline reinforcement learning algorithms is highly dependent on the quality of the given dataset.Recent research has shown that offline reinforcement learning has issues such as approximation errors and extrapolation errors in the value function due to the lack of online interaction with the target environment.

[0007] The present invention has been made in consideration of the above problems, and aims to provide a learning device and a learning method that can alleviate the problems in reinforcement learning. [Means for solving the problem]

[0008] (1) In order to achieve the above object, a learning device according to one aspect of the present invention comprises: a dataset acquisition unit that acquires a dataset consisting of state information and action information that are the subject of policy learning; a discrete latent variable estimation unit that estimates discrete latent variables that represent characteristic properties from the state information and the action information; an optimal action learning unit that learns optimal actions using the state information and the discrete latent variables; a value function estimation unit that learns action value from the state information and the action information; and an identification unit that identifies the discrete latent variable that maximizes action value using the results of the optimal action learning unit and the value function estimation unit.

[0009] (2) In order to achieve the above object, a learning method according to one embodiment of the present invention includes: an acquisition step of acquiring a dataset consisting of state information and action information from which a policy is to be learned; an estimation step of estimating discrete latent variables representing the characteristics of the dataset from the state information and the action information included in the dataset; a first learning step of learning an optimal action using the state information and the estimated discrete latent variables; a second learning step of learning an action value from the state information and the action information; and an identification step of identifying the discrete latent variable that maximizes the action value using the results learned in the first learning step and the results learned in the second learning step.

[0010] (3) Furthermore, the learning method according to one aspect of the present invention may include a value function updating step of inputting the identified discrete latent variables into the second learning step to update a value function; a latent variable action updating step of inputting the updated value function into the estimation step and the first learning step to update the discrete latent variables and the optimal action; and a third learning step of repeating the value function updating step and the latent variable action updating step to learn the discrete latent variables and the optimal action.

[0011] (4) Furthermore, in a learning method according to one aspect of the present invention, when the learned policy is executed, not all of the first learning steps are activated, but the discrete latent variables according to the situation may be estimated, and sub-policies corresponding to the estimated discrete latent variables may be sequentially selected and activated.

[0012] (5) In the learning method according to one aspect of the present invention, z is the discrete latent variable, z' is the next discrete latent variable, s is a state, s' is the next state, and Q w is the estimate of the Q-value parameterized by the vector w, y is the target value, r is the reward in learning, γ is the discount coefficient, θ is a vector representing the parameters of the policy, φ is a vector representing the parameters of the post-hoc secretion model, and (z ~)' is the estimated discrete latent variable f π is a function that quantifies the performance of the policy π, and l cvae is a variational lower bound, a is an action, and the estimation step calculates the latent variables using

number

number

number

number

[0013] According to (1) to (5), the challenges in reinforcement learning can be alleviated. According to (1) to (5), learning performance can be improved by learning discrete variables and mixed strategies. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram for explaining an overview of reinforcement learning. [Figure 2] FIG. 1 is a diagram for explaining a model used in an embodiment. [Figure 3] FIG. 1 is a diagram illustrating an example of the configuration of a learning device according to an embodiment. [Figure 4] FIG. 10 is a diagram illustrating an example of a data set used in the embodiment. [Figure 5]10 is a flowchart illustrating an example of an outline of a learning process according to the embodiment. [Figure 6] 10 is a flowchart of an example of a learning process procedure according to the embodiment. [Figure 7] 10 is a flowchart of an example of a processing procedure when estimating behavior using a trained model according to an embodiment. [Figure 8] FIG. 10 is a diagram illustrating an example of an algorithm used during learning according to the embodiment. [Figure 9] FIG. 10 is a diagram showing the differences between the methods used for comparison. [Figure 10] FIG. 10 is a diagram showing the results of evaluating the influence of the number of dimensions of a discrete latent variable. [Figure 11] FIG. 10 is a diagram illustrating a comparison between V2AE, which is the method of the present embodiment, and a baseline method in the Mujoco task. [Figure 12] FIG. 10 is a diagram showing a comparison between V2AE, which is the method of the present embodiment, and a baseline method in the Kitchen and Adroit tasks. [Figure 13] FIG. 10 is a diagram showing an example of visualization of state-action pairs in the Pen Human-v0 task. [Figure 14] This figure shows the activation of sub-policies in the pen-human-v0 task, at the 20th, 40th, 60th, and 80th time steps. [Figure 15] FIG. 10 shows the activation of sub-policies in the pen-human-v0 task, illustrating the action values ​​of each sub-policy in each state. [Figure 16] FIG. 10 is a diagram illustrating normalized scores and values ​​of the critic loss function during learning using V2AE, which is a method of the embodiment, and AWAC, which is a method of the comparative example. [Figure 17] Figure 1 shows the results of the first episode of sub-policy activation in the pen-human-v0 task. [Figure 18] Figure 1 shows the results of the second episode of sub-policy activation in the pen-human-v0 task. [Figure 19]Figure 10 shows the results of the third episode of the pen-human-v0 task, showing the activation of sub-policies. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the drawings used in the following description, the scale of each component is appropriately changed so that each component can be recognized. In all the drawings for explaining the embodiments, the same reference numerals are used for components having the same functions, and repeated explanations will be omitted. Furthermore, in this application, "based on XX" means "based on at least XX," and includes cases where it is based on other elements in addition to XX. Furthermore, "based on XX" is not limited to cases where XX is used directly, but also includes cases where it is based on XX that has been calculated or processed. "XX" is any element (for example, any information).

[0016] [Reinforcement learning overview] First, an overview of reinforcement learning will be explained. Figure 1 is a diagram for explaining the overview of reinforcement learning. As shown in Figure 1, in reinforcement learning, for example, a "state" is acquired from the environment, and the acquired "state" and "reward" are input to a policy. Then, in reinforcement learning, the policy estimates an "action" based on the input "state" and "reward." Then, in reinforcement learning, the estimated "action" is performed in the environment, and the state is acquired again.

[0017] [Model Description] In this embodiment, the following three models are used for reinforcement learning. FIG. 2 is a diagram for explaining the model used in this embodiment. Reference symbol g11 is an image showing the input and output of an encoder, which is a first model. As shown by reference symbol g11, the encoder receives a state and an action as input, estimates a discrete latent variable (e.g., [0,0,1,0]) for the given state and action, and outputs it. In the following description, the "discrete latent variable" will also be referred to as a "latent variable."

[0018] Symbol g12 is an image showing the input and output of the lower-level policy, which is the second model. As shown in symbol g12, the lower-level policy receives estimated latent variables and a state as input, and estimates and outputs the optimal action for the given state and latent variables.

[0019] Symbol g13 is an image showing the input and output of the action-value function, which is the third model. As shown in symbol g13, the action-value function receives inputs of a state and an action, and calculates the value of the state and action for the given state and action. Estimate and output the action value.

[0020] [Example of learning device configuration] Next, an example of the configuration of the learning device 1 that performs learning will be described. 3 is a diagram showing an example of the configuration of a learning device according to this embodiment. As shown in FIG. 3, the learning device 1 includes, for example, an acquisition unit 11, a storage unit 12, a discrete latent variable estimation unit 13, an optimal behavior learning unit 14, a value function estimation unit 15, an identification unit 16, and a processing unit 17.

[0021] The acquisition unit 11 acquires a dataset consisting of a reward and state information and action information that are targets for policy learning. If the dataset does not include a reward, it is necessary to recalculate only the reward according to the task to be learned.

[0022] The storage unit 12 stores the data set. The storage unit 12 stores the program used by the processing unit 117 for processing, the predetermined number of times, etc.

[0023] The discrete latent variable estimation unit 13 includes a first model (encoder) 131. The discrete latent variable estimation unit 13 estimates a discrete latent variable that represents the nature of a feature from the state information and the behavior information.

[0024] The optimal behavior learning unit 14 includes a second model (lower-level policy) 141. The optimal behavior learning unit 14 learns optimal behavior by using state information and discrete latent variables to estimate the behavior with the second model 141.

[0025] The value function estimation unit 15 includes a third model (action value function) 151. The value function estimation unit 15 estimates an action value from state information and action information, and learns the action value by updating the third model 151.

[0026] The identification unit 16 uses the results of the optimal behavior learning unit and the value function estimation unit to identify the discrete latent variables that maximize the behavior value.

[0027] At the start of learning, the processing unit 17 initializes the first model 131, the second model 141, and the third model 151. The processing unit 17 extracts some of the sets of state s, action a, next state s', and reward r from the data set.

[0028] [Dataset example] Next, an example of a data set will be described. 4 is a diagram showing an example of a dataset used in this embodiment. As shown in FIG. 4, the dataset is made up of a set of four elements: a state s, an action a, a next state s', and a reward r.

[0029] [Example of learning process procedure] Next, an example of the procedure of the learning process will be described. First, an outline of the learning process will be described using Fig. 5 while referring to Fig. 2. Fig. 5 is a flowchart of an example of the outline of the procedure of the learning process according to this embodiment.

[0030] (Step S1) The acquisition unit 11 acquires in advance a data set made up of state information and action information that are targets for learning a policy, and stores the acquired data set in the storage unit 12 (acquisition step).

[0031] (Step S2) The processing unit 17 initializes the first model 131, the second model 141, and the third model 151.

[0032] (Step S3) The discrete latent variable estimation unit 13 estimates a discrete latent variable representing the nature of the feature using the state information and behavior information included in the dataset and the first model 131 (estimation step).

[0033] (Step S4) The optimal behavior learning unit 14 uses the state information, the estimated discrete latent variables, and the second model 141 to learn optimal behavior (first learning step).

[0034] (Step S5) The value function estimation unit 15 uses the state information, the action information, and the third model 151 to learn the action value (second learning step).

[0035] (Step S6) The identification unit 16 uses the results learned in step S4 and the results learned in step S5 to identify the discrete latent variable that maximizes the action value (identification step).

[0036] Next, an example of a learning procedure including model updating will be described. 6 is a flowchart showing an example of the procedure of the learning process according to this embodiment. The learning device 1 performs learning for each action by the following process. The learning device 1 also repeats the following process a predetermined number of times.

[0037] (Step S11) The acquisition unit 11 acquires a data set in advance and stores the acquired data set in the storage unit 12 (acquisition step).

[0038] (Step S12) The processing unit 17 initializes the first model 131, the second model 141, and the third model 151.

[0039] (Step S13) The processing unit 17 extracts some of the pairs of state s, action a, next state s', and reward r from the data set. The processing unit 17 selects a number of pairs, for example, about 256 to 1024 pairs.

[0040] (Step S14) The identification unit 16 identifies the latent variable z that maximizes the action value for the state s.

[0041] (Step S15) The value function estimation unit 15 uses the latent variable z identified in step S13 to train and update the third model 151 (action value function).

[0042] (Step S16) The discrete latent variable estimation unit 13 trains and updates the first model 131 (encoder) that estimates latent variables corresponding to the state s and the action a.

[0043] (Step S17) The discrete latent variable estimation unit 13 estimates the latent variable z corresponding to the state s and the action a using the first model 131.

[0044] (Step S18) The optimal behavior learning unit 14 uses the state s and the latent variables estimated in step S16 to learn and update the second model 141 (lower-level policy).

[0045] The state s selected in step S13 continues to be used in steps S14 to S18. In the process, the same operation is performed simultaneously on multiple s, and each model is updated based on the results.

[0046] [Example of processing procedure when using] Next, an example of a processing procedure for estimating behavior using a trained model will be described. FIG. 7 is a flowchart of an example of a processing procedure when estimating behavior using a trained model according to this embodiment.

[0047] (Step S21) The learning device 1 determines a latent variable that maximizes the action value for the observed state using the trained first model 131.

[0048] (Step S22) The learning device 1 determines an action using the trained second model 141 based on the determined latent variables.

[0049] [Explanation of learning method] The principles of the learning method used in this embodiment will be described below. First, we consider reinforcement learning under a Markov decision process (MDP) defined by the following set of equations (1).

[0050]

number

[0051] In the following description, in formula (1), the first is represented by S, the second by A, and the third by P. S is the state space, A is the action space, and P(s t+1 |s t ,a t ) is the transition probability density, r(s,a) is the reward function, γ is the discount factor, and d(s0) is the probability density of the initial state.

[0052] The strategy π(a|s) in the following equation (2) is defined as the conditional probability density function of the action for a given state. Note that the double-underlined letter R is the set of all actions.

[0053]

number

[0054] The goal of reinforcement learning is to identify a strategy that maximizes the expected cumulative discounted reward in equation (3):

[0055]

number

[0056] where R t is expressed as the following equation (4).

[0057]

number

[0058] Also, the Q function Q π (s, a) is the expected reward when starting from state s, taking action a, and following policy π under a given Markov decision process. In offline reinforcement learning, we assume a dataset of the following equation (5) consisting of states, actions, and rewards collected using an unknown policy.

[0059]

number

[0060] The goal of offline reinforcement learning is to obtain a policy that maximizes the expected reward using a dataset D.

[0061] Here, we formulate the offline reinforcement learning problem as follows: Given a dataset D (Equation (5)), the goal of the learning process is to obtain a policy π that maximizes the reward obtained by interacting with the environment without interacting with the environment. In offline reinforcement learning, the expected reward is evaluated with respect to the states stored in a given dataset. Therefore, the objective function is given by the following equation (6):

[0062]

number

[0063] In equation (6), f π is a function that quantifies the performance of the policy π. In reinforcement learning, there are several choices for f, as shown in Reference 1. For example, the TD3 method (see e.g. Reference 2) uses the action function f π (s,a)=Q π (s, a) is adopted, and the A2C (Advantage Actor Critic) method uses the advantage function f π (s,a)=A π (s, a) is adopted. The A2C method is a variant of A3C, and is a method that removes the asynchronous element from A3C (Asynchronous Actor Critic).

[0064] Reference 1; John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel, “High-dimensional continuous control using generalized advantage estimation”, In Proceedings of the International Conference on Learning Representations (ICLR), 2016.

[0065] Reference 2; Scott Fujimoto and Shixiang Shane Gu, “A minimalist approach to offline reinforcement learning”, Advances in Neural Information Processing Systems (NeurIPS), 2021.

[0066] Other previous studies have adopted calculations involving exponential functions, which are expressed by the following equations (7) or (8):

[0067]

number

[0068]

number

[0069] Without loss of generality, we assume that the objective function is given by Equation (6). Previous studies often propose objective functions that add a regularization term to learn a policy, but in this embodiment, we derive the objective function as a variational lower bound of (1), which is a simple objective function for offline reinforcement learning.

[0070] [Mixed Strategy] In this embodiment, a model of the following equation (9) that can be expressed by a distribution having multiple peaks is introduced: The model given by the following equation (9) is a mixed model of policies.

[0071]

number

[0072] In equation (9), z is a discrete latent variable, π(s|a) is a higher-level policy that determines the latent variable, and π(s|a,z) is a lower-level policy that determines the action for given s and z. We assume that the lower-level policy π(s|a,z) is a deterministic policy. Therefore, the lower-level policy determines the action for given s and z as a=μ θ (s,z) is determined definitively. Note that μ θ (s,z) is parameterized by the vector θ. Furthermore, the higher-level policy π(s|a) determines the latent variables as shown in the following equation (10).

[0073]

number

[0074] In equation (10), Q w (s,z) is the estimate of the Q value parameterized by the vector w.

[0075] [Learning Mixture Policies by Maximizing Variational Lower Bounds] Here, for any s and a, f π When (s, a)>0, the variational lower bound of log(J(π)) can be obtained using Jensen's inequality as shown in the following equations (11) to (13).

[0076]

number

[0077]

number

[0078]

number

[0079] The second version of equation (13) is independent of the policy π. Therefore, to maximize the lower bound of the change in J(π), we need to maximize the following equation (14):

[0080]

number

[0081] f π (s,a)=exp(A π When (s, a)) is adopted and the strategy is Gaussian, the resulting algorithm is equivalent to AWAC (Non-Patent Document 1). To adopt a mixed strategy using discrete latent variables, the objective function of Equation (13) can be further analyzed to obtain the following Equation (15):

[0082]

number

[0083] In equation (15), D KL Since (q(z|s,a)||p(z|s,a))>0, we can transform the variational lower bound as used in conditional VAE (see Reference 3) to obtain the following equation (16). Note that p and q are probabilities. Note that q Φ (z|s i ,a i ) term indicates that q is parameterized by Φ (variation parameter). Also, π θ (a i The term |si,z) indicates that π is parameterized by θ.

[0084]

number

[0085] Reference 3; Kihyuk Sohn, Honglak Lee, and Xinchen Yan, “Learning structured output representation using deep conditional generative models”, In Advances in Neural Information Processing Systems(NeurIPS), 2015.

[0086] In previous studies, it is often assumed that z is statistically independent from s. That is, in previous studies, p(z|s)=p(z),p(z|s) is often used. In contrast, in the framework of this embodiment, the upper policy π θ However, the behavior of (z|s) needs to be expressed. θ Since it is difficult to accurately express (z|s), in this embodiment, it is approximated by the softmax distribution given by the following equation (16).

[0087]

number

[0088] Since we adopted double-clipped Q-learning (see Reference 4), we obtain the following equation (18).

[0089] Reference 4; Scott Fujimoto, Herke van Hoof, and David Meger, “Addressing function approximation error in actor-critic methods”, In Proceedings of the International Conference on Machine Learning (ICML), pages 1587-1596, 2018.

[0090]

number

[0091] Here, the second term in equation (16) is approximated as the mean square error, as in standard implementations of VAE. Based on equations (13) and (16), in this embodiment, a mixed deterministic strategy is trained by maximizing the following equation (19): In equation (19), θ is a vector representing the parameters of the strategy, φ is a vector representing the parameters of the model of the posterior distribution, and f π is a function that quantifies the performance of the policy π, and l cvae is a (conditional) variational lower bound and a is an action.

[0092]

number

[0093] This objective function can be viewed as a weighted maximum likelihood. Traditional offline reinforcement learning methods, such as BCQ (see Reference 5) and FischerBRC (see Reference 6), utilize variational autoencoders (VAEs) to obtain dataset-constrained policies. The latent variables learned by the VAEs used in these methods are based on the density of state-action pairs in a given dataset.

[0094] Reference 5; Scott Fujimoto, David Meger, and Doina Precup, “Off-policy deep reinforcement learning without exploration”, In Proceedings of the International Conference on Machine Learning (ICML), pages 2052-2062, 2019. Reference 6; Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum, “Offline reinforcement learning with fisher divergence critical regularization”, In Proceedings of the International Conference on Machine Learning (ICML), 2021.

[0095] In contrast, the approach of this embodiment learns latent variables that maximize the lower bound of the objective function. Therefore, the meaning of the latent variables learned in this embodiment is different from that of conventional methods. Furthermore, while conventional methods learn continuous latent variables, the approach of this embodiment learns discrete latent variables.

[0096] Furthermore, the approach of this embodiment can be considered as dividing the state-action space by learning discrete latent variables. In addition, the conventional method TD3-BC (see Non-Patent Document 1) recommends a strategy to imitate the behavior contained in a given dataset regardless of the quality of that behavior.

[0097] However, in offline reinforcement learning, a given dataset may contain samples obtained by a wide variety of actions, and it is inappropriate to force a policy to reproduce any action in the dataset. Therefore, in this embodiment, the measure π θ (a|s,z) encourages the imitation of state-action pairs with the same value of z. Therefore, in this embodiment, the strategy π θ (a|s,z) is not constrained to imitate actions with the same value of z.

[0098] The objective function of this embodiment is composed of terms that reconstruct state-action pairs with adaptive weights, and does not include terms that lead to extrapolation, such as the following equation (20) in the conventional TD3-BC method. Therefore, in this embodiment, actions are sampled and evaluated within the distribution of the given data, and actions that deviate from the distribution of the given data are not evaluated.

[0099]

number

[0100] [Estimating the Q function for mixed policies] Next, we explain the estimation method of the Q function for the mixed strategy. Since the mixed policy is adopted in Equation (9), the estimation of the Q function is based on an operator, which is slightly different from the standard Bellman operator. In the framework of this embodiment, the training of the critic is based on the operator of the following Equation (21):

[0101]

number

[0102] In equation (21), T z The operator is called the latent-max-Q operator. The following first and second theorems that support the algorithm of this embodiment can be proved as follows.

[0103] I. The First Theorem In the table format settings, T z The operator is L ∞ is the contraction operator in the norm. Therefore, T z By repeated application of the operator, any initial Q-function will converge to a unique fixed point.

[0104] II. Second Theorem The unique fixed point obtained in the first theorem is called Q zThen, the latent variable z is selected as follows (22), and the policy to output the action given by μ(s,z) is defined as π z In this case, Q z is π z is the Q function corresponding to

[0105]

number

[0106] Here, we will explain the proof of the second theorem. Rearranging equation (21) gives the following equation (23).

[0107]

number

[0108] Thus, from the definition, Q z is T z It is a unique fixed point of

[0109] Based on the second theorem, the Q function is estimated by applying the latent-max-Q operator. In this embodiment, as described above, double-clipped Q-learning is adopted. In this way, when a dataset D is given, the Critic learns by minimizing the following equation (24).

[0110]

number

[0111] For J=1,2, the target value is calculated as follows:

[0112]

number

[0113] [implementation] Hereinafter, the method of this embodiment will be referred to as Value-Weighted Variational Auto-Encoder (V2AE). This algorithm is summarized in FIG. FIG. 8 is a diagram showing an example of an algorithm for learning according to this embodiment. In FIG. 8, the processing indicated by reference symbol g21 corresponds to the processing at step S12 in FIG. 6. The processing indicated by reference symbol g22 corresponds to the processing at step S13 in FIG. 6. The processing indicated by reference symbol g23 corresponds to the processing at step S14 in FIG. 6, and calculates latent variables using the following equation (26). The processing indicated by reference symbol g24 corresponds to the processing at step S15 in FIG. 6, and calculates a target value y using the following equation (27), and updates the critic to minimize the following equation (28). The processing indicated by reference symbol g25 corresponds to the processing at step S15 in FIG. 6, and updates the actor and posterior distribution so as to maximize the following equation (29). Note that in equation (26), (z ~ )' is the symbol used to consider the max for all possible discrete latent variables, and is the next estimated latent variable.

[0114]

number

[0115]

number

[0116]

number

[0117]

number

[0118] As described above, in this embodiment, the discrete latent variable estimation unit 13 calculates the latent variables using equation (26), the value function estimation unit 15 calculates the target value y using equation (27), updates the third model of the action value function by updating critic to minimize the following equation (28), and updates the first model by updating actor and posterior distribution to maximize equation (29). The algorithm shown in FIG. 8 is an example and is not limited to this.

[0119] As in TD3, the actor is updated once after the Critic is solved. The algorithm uses dinterval = 2. For modeling the discrete latent variables, the Gumbel-softmax method (see, for example, Reference 7) is used. The state normalization used in TD3+BC is also used.

[0120] Reference 7; Eric Jang, Shixiang Gu, and Ben Poole, “Categorical reparameterization with gumbel-softmax”, In Proceedings of the International Conference on Learning Representations (ICLR), 2017.

[0121] As a result of preliminary experiments, f in Eq. (19) π (s, a)=exp(βA π We found that the scaling coefficient has a nontrivial effect on performance when (s, a)), and the optimal value of β varies for each task. Therefore, to avoid changing the scaling parameter for each task, we normalized the advantage function as shown in the following equation (26).

[0122]

number

[0123] In equation (26), D batchis a mini-batch sampled from a given dataset D, and is a constant, here set to α=10. In the V2AE method of this embodiment, the policy is given as a mixture of deterministic policies. The sub-policies are selected deterministically as shown in Equation (10). Therefore, the mixed policy in this framework is deterministic. If a deterministic policy is used, the critic may overfit to a narrow peak. In contrast, because the policy of this embodiment is deterministic, we also adopted a method called target policy smoothing, which is used in TD3. As a result, the target value of equation (25) is corrected as shown in the following equation (27).

[0124]

number

[0125] In equation (27), ε clip is given by the following equation (28).

[0126]

number

[0127] In equation (28), the constant c defines the range of the noise.

[0128] [evaluation] Next, an example of the results of confirming the learning effect of the mixed policy according to the method of this embodiment and the effect of the Critic dropout layer will be described. A workstation and a physical simulator were used for the evaluation.

[0129] First, the method of this embodiment was evaluated using the benchmark task of D4RL (see Reference 8). As baselines, TD3-BC, CQL (see Reference 9), AWAC (see Non-Patent Document 1), easyBCQ (see Reference 10), and EDAC (see Reference 11) were evaluated. Note that in the implementation of AWAC, state normalization and double-clipped Q-learning were used, as in TD3+BC, and dominance function normalization was also performed.

[0130] Reference 8; Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine, “D4rl: Datasets for deep data-driven reinforcement learning”, arXiv, 2020 Reference 9; Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine, “Conservative q-learning for offline reinforcement learning”, In Advances in Neural Information Processing Systems(NeurIPS), 2020. Reference 10; David Brandfonbrener, William F. Whitney, Rajesh Ranganath, and Joan Bruna, “Offline rl without off-policy evaluation”, In Advances in Neural Information Processing Systems (NeurIPS), 2021. Reference 11; Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song, “Uncertainty-based offline reinforcement learning with diversified q-ensemble”, In Advances in Neural Information Processing Systems (NeurIPS), 2021.

[0131] In this way, the difference between the evaluation results of AWAC and the V2AE of the method of this embodiment represents the difference in the model of the strategy. Note that double clipping Q-learning is also used in easyBCQ. Figure 9 shows the differences between the methods used in the comparison. Note that for the evaluation, we reran the baseline method on the D4RL-v0 dataset. Note that the results for the Kitchen and Adroit tasks of the EDAC method are omitted.

[0132] First, we evaluated the effect of the dimensionality of the discrete latent variables. Figure 10 shows the results of evaluating the effect of the dimensionality of the discrete latent variable. The evaluation in Figure 10 shows the average normalized score over the past 10 test episodes and five seeds, and the performance after 1 million updates. The horizontal axis is the absolute value of the latent variable z, and the vertical axis is the average normalized score.

[0133] Graph g101 is the average normalized score for "walker2d-expert" in the D4RL-v0 dataset. Graph g102 is the average normalized score for "walker2d-medium-expert" in the D4RL-v0 dataset. Graph g103 is the average normalized score for "walker2d-medium" in the D4RL-v0 dataset. Graph g104 is the average normalized score for "walker2d-medium-replay" in the D4RL-v0 dataset. As shown in Figure 10, the absolute value of the latent variable |Z|=8 consistently showed satisfactory performance, so we adopted |Z|=8 in the following evaluations.

[0134] A comparison between the V2AE method of this embodiment and the baseline method is shown in Figures 11 and 12. Note that the D4RL-v0 dataset is also used in the following evaluation. Fig. 11 is a diagram showing a comparison between V2AE, which is the method of this embodiment, and baseline methods in the Mujoco task. In Fig. 11, HCheetah is an abbreviation for Half Cheetah. For Half Cheetah, Hopper, Walker2d, etc., see Non-Patent Document 1. The results in Fig. 11 also show the average normalized scores of the past 10 test episodes and five seeds.

[0135] FIG. 12 shows a comparison of the V2AE method of the present embodiment with a baseline method for the Kitchen and Adroit tasks. In FIG. 12, the Kitchen task is abbreviated as "Kitchen." and the human task is abbreviated as "Hum." Furthermore, "complete," "partial," and "mixed" represent the difficulty of the Mujoco task, with "complete" being the most difficult and "mixed" being the least difficult. Furthermore, "pen," "Hammer," "door," and "relocate" represent datasets (see, for example, Reference 12). The results in FIG. 12 are for the Kitchen task and the Adroit task, and represent the average normalized scores and five seeds over the past 10 test episodes. For kitchen-complete-v0 and *-human-v0, the data points are approximately 10,000, so the performance after 10,000 updates is shown. For these datasets, the data points are approximately 10,000, so the performance after 10,000 updates is shown. For the other datasets, the performance after 1,000,000 updates is shown.

[0136] Reference 12; Wenxuan Zhou, Sujay Bajracharya, David Held, “PLAS: Latent Action Space for Offline Reinforcement Learning”, 4th Conference on Robot Learning (CoRL 2020), Cambridge MA, USA, 2020

[0137] As shown in Figure 11, in the Mujoco task, the V2AE of the method of this embodiment achieved performance comparable to that of the latest offline reinforcement learning methods, TD3+BC and EDAC. Furthermore, as shown in Figure 12, the superiority of the V2AE of the method of this embodiment is more evident in the Kitchen task and Adroit task. The V2AE of the method of this embodiment clearly outperforms the baseline method in these tasks. The difference between AWAC and V2AE is due to the influence of differences in the expression of the policy. The V2AE of the method of this embodiment showed performance equal to or better than AWAC.

[0138] 11 and 12, it can be seen that the use of a mixed policy in reinforcement learning is effective. In particular, the V2AE of the method of this embodiment showed the best performance in the Adroit task and the Kitchen task.

[0139] [Visualization of learned latent variables] Next, we will explain an example of visualizing the learned latent variables. Figure 13 shows an example of visualizing state-action pairs in the pen-human-v0 task. In Figure 13, the shading of the circles indicates the values ​​of the latent variables. Image g151 shows the state-action pairs in the pen-human-v0 task. φ This is an example of visualizing the distribution of latent variables sampled from (s, a). Image g152 shows the distribution of latent variables sampled from (s, a) with z=arg max{Q w This is an example of visualizing the distribution of latent variables given by {(s,μ(s,z))}.

[0140] The dimension of the state-action pairs was reduced using t-SNE

[36] . The distribution of latent variable values ​​indicates how the state-action space is divided. As part of the objective function, we minimize the KL divergence DKL(q(z|s,a)||p(z|s)). Therefore, the samples generated from q(z|s,a) and p(z|s) should be similar.

[0141] Figures 14 to 16 show the activation of sub-policies in the pen-human-v0 task. The task involves holding a pen at the correct angle. Figure 14 shows the activation of sub-policies in the pen-human-v0 task, illustrating the states at the 20th, 40th, 60th, and 80th time steps. Figure 15 shows the activation of sub-policies in the pen-human-v0 task, illustrating the action values ​​of each subpolicy in each state. In Figures 14 and 15, images g201 and g251 represent the state at the 20th time step, images g202 and g252 represent the state at the 40th time step, images g203 and g253 represent the state at the 60th time step, and images g203 and g254 represent the state at the 80th time step. In Figure 15, the horizontal axis represents the latent variable z (0 to 7, 8 combinations), and the vertical axis represents Q(s, a, z)-min. z The value of (Q(s,a,z)).

[0142] Previous research on option frameworks has reported that a problem with existing methods is that only some options are activated, and the remaining options may not be useful. In contrast, as shown in Figure 15, at the 20th time step, the latent variable z = 4 was the largest, at the 40th time step, the latent variable z = 5 was the largest, at the 60th time step, the latent variable z = 3 was the largest, and at the 80th time step, the latent variable z = 3 was the largest. As such, in this embodiment, as shown in Figure 15, the value of each sub-policy changes over time, indicating that various sub-policies are activated during execution. As such, the method of this embodiment can solve the problems of conventional methods.

[0143] Next, the estimation error of the function will be described. FIG. 16 is a diagram showing the normalized score and the value of the critic loss function during learning using V2AE, which is the method of this embodiment, and AWAC, which is the method of the comparative example. In FIG. 16, the horizontal axis represents the time step (1e6), and the vertical axis represents the normalized score. In addition, in graphs g301 to g304, line g311 represents V2AE, which is the method of this embodiment, and line g312 represents AWAC, which is the comparative example. In addition, graph g301 represents the normalized score and the value of the critic loss function for halfcheetah-medium-v0. Graph g302 represents the normalized score and the value of the critic loss function for halfcheetah-medium-v0. Graph g303 represents the normalized score and the value of the critic loss function for walker2d-medium-replay-v0. Graph g304 represents the normalized score and the value of the critic loss function for walker2d-medium-replay-v0. As the critic loss, the value of the critic loss given by Equation (24) is plotted every 5000 updates.

[0144] Previous studies have shown that the estimation error of the Q function accumulates with repeated learning. As shown in Fig. 15, it can be seen that the error in function approximation accumulates in the conventional method AWAC, which is a comparative example of the HalfCheetah medium-v0 task. In contrast, Fig. 15 shows that the function approximation error is clearly smaller in the method V2AE of this embodiment, and the policy performance is improved compared to the comparative example of the method AWAC.

[0145] The difference between the AWAC method of the comparative example and the V2AE method of this embodiment is the representation of the policy. Therefore, these results suggest that by using a mixed policy as in this embodiment and learning a mixed policy, the problem of accumulating estimation errors in the Q function is alleviated, and learning performance is improved.

[0146] As described above, in this embodiment, the V2AE method is used for learning mixed policies. The V2AE method of this embodiment can be interpreted as an approach that divides the state-action space by learning discrete latent variables and learns corresponding sub-policies in each region. Evaluation results show that the approach of this embodiment can reduce extrapolation errors in offline reinforcement learning. Furthermore, it has been confirmed that the V2AE method of this embodiment exhibits the best performance in several D4RL benchmark tasks.

[0147] In this way, when a policy learned by the method of this embodiment is executed, not all sub-policies are activated, but rather discrete latent variables according to the situation are estimated, and the corresponding sub-policies are sequentially selected and activated. As a result, according to this embodiment, if a dataset contains samples of varying quality, some of the latent variables will be associated with low-performing behavioral samples, and the performance of the corresponding sub-policies will also be low, so such sub-policies will not be activated at runtime. Conversely, according to this embodiment, information on high-performing behavioral samples in the dataset is actively utilized.

[0148] According to this embodiment, learning performance has been confirmed to be improved by learning discrete variables and mixed policies. Specifically, it has been confirmed that the method outperforms existing methods in several existing benchmark tasks called Datasets for Deep Data-Driven Reinforcement Learning (D4RL). The problem of accumulated estimation errors in the value function and extrapolation errors have been reduced.

[0149] In the following description, an example of performing reinforcement learning offline will be described, but the method and configuration of this embodiment can also be applied online.

[0150] <Additional Notes> [Proof of the first theorem] Here, we will explain the proof of the first theorem. The operator T given by the following equation (29) z Think about it.

[0151]

number

[0152] T z To prove the contractibility of (30), the infinity norm given by the following equations (31) and (32) is used.

[0153]

number

[0154]

number

[0155] [Additional results on latent conditional policy activation] Figure 17 shows the results of the first episode of sub-policy activation in the pen-human-v0 task. Figure 18 shows the results of the second episode of sub-policy activation in the pen-human-v0 task. Figure 19 shows the results of the third episode of sub-policy activation in the pen-human-v0 task. Note that in Figures 17 and 18, the horizontal and vertical axes of graphs g405-g408, g415-g418, and g425-g428 are the same as in Figure 15. In Figures 17 and 18, images g401-g404, g411-g414, and g421-g424 show the states of the hand and object in the task. In Figures 17-18, the horizontal axes of images g409, g419, and g429 represent sampling time. Images g409, g419, and g429 represent the largest changes in the latent variable z at episodes 20, 40, 60, and 80. Figures 17-19 also use the same policy trained for 10,000 updates.

[0156] As shown in Figures 17-19, the target pose of the object is different in each episode, and different sub-policies are activated to achieve the given goal. This qualitative result supports the claim that different behaviors are encoded in each sub-policy.

[0157] [Hyperparameters and implementation details] Below, we will explain the hyperparameters and implementation details used in the evaluation. For the evaluation, we used the implementations of TD3+BC, CQL, and EDAC by the authors of each paper. In addition, easyBCQ and AWAC were independently implemented to ensure a fair comparison with the V2AE method of this embodiment. Double clipping Q-learning is used in the implementation of easyBCQ and AWAC.

[0158] In the method V2AE of this embodiment, both the upper policy π(z|s) and the lower policy π(a|s,z) are deterministic, so the policies are deterministic. Therefore, the state value function is given by the following equation (32).

[0159]

number

[0160] Therefore, the advantage function is given by the following equation (33).

[0161]

number

[0162] In updating the policy, the target actor in the second term of the following equation (34) was used. Therefore, in the implementation of the method of this embodiment, the advantage function is approximated as follows:

[0163]

number

[0164] A program for implementing all or part of the functions of the learning device 1 of the present invention may be recorded on a computer-readable recording medium, and the program may be loaded into a computer system and executed to perform all or part of the processing performed by the learning device 1. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer system" also includes a WWW system equipped with a homepage provision environment (or display environment). The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. The term "computer-readable recording medium" also includes devices that retain a program for a certain period of time, such as volatile memory (RAM) within a computer system that acts as a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line.

[0165] The program may also be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may also be a program that realizes part of the above-mentioned functions. Furthermore, the program may be a so-called differential file (differential program) that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0166] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0167] 1... learning device, 11... acquisition unit, 12... memory unit, 13... discrete latent variable estimation unit, 14... optimal behavior learning unit, 15... value function estimation unit, 16... identification unit, 17... processing unit, 131... first model, 141... second model, 151... third model

Claims

1. a dataset acquisition unit that acquires a dataset including state information and action information from which a policy is to be learned; a discrete latent variable estimation unit that estimates a discrete latent variable representing a characteristic property from the state information and the behavior information; an optimal behavior learning unit that learns optimal behavior using the state information and the discrete latent variables; a value function estimation unit that learns an action value from the state information and the action information; an identification unit that identifies the discrete latent variable that maximizes the action value using the result of the optimal action learning unit and the result of the value function estimation unit; A learning device comprising:

2. A step executed by a computer, an acquisition step of acquiring a dataset consisting of state information and action information from which a policy is to be learned; an estimation step of estimating discrete latent variables representing the characteristics of the dataset from the state information and the behavior information included in the dataset; a first learning step of learning an optimal behavior using the state information and the estimated discrete latent variables; a second learning step of learning an action value from the state information and the action information; an identification step of identifying the discrete latent variable that maximizes the action value using the results learned in the first learning step and the results learned in the second learning step; Learning methods including.

3. A step executed by a computer, a value function updating step of updating a value function by inputting the identified discrete latent variables into the second learning step; a latent variable action update step of inputting the updated value function into the estimation step and the first learning step to update the discrete latent variables and the optimal action; a third learning step of repeating the value function updating step and the latent variable action updating step to learn the discrete latent variables and the optimal action; The learning method of claim 2 , comprising:

4. When the learned policy is executed, not all of the first learning steps are activated, but the discrete latent variables according to the situation are estimated, and sub-policies corresponding to the estimated discrete latent variables are sequentially selected and activated. The learning method according to claim 2 or 3.

5. z is the discrete latent variable, z' is the next discrete latent variable, s is a state, s' is the next state, and Q ｗ is the estimate of the Q-value parameterized by the vector w, y is the target value, r is the reward in learning, γ is the discount coefficient, θ is a vector representing the parameters of the policy, φ is a vector representing the parameters of the model of the posterior distribution, and (z ～ )' is the estimated discrete latent variable, and f π is a function that quantifies the performance of the policy π, and l ｃｖａｅ is the variational lower bound, a is the action, The estimation step calculates latent variables using the following formula: [Equation 1] The value function updating step calculates a target value y using the following formula: [Equation 2] The value function updating step updates the row value function by updating a critic that minimizes [Equation 3] The latent variable behavior update step updates the first model by updating the Actor and the posterior distribution so as to maximize the following equation: [Equation 4] The learning method according to claim 3 .

Images