Training device, training method, and recording medium

Abstract

Provided is a training device, wherein a generation means generates a value estimation model of a teacher model from the state and action data of the teacher model. An acquisition means acquires the next state as an execution result of an action determined by using the policy of a student model. An estimation means uses the next state and the value estimation model to estimate a state value or an action value in the next state. A policy update means updates the policy of the student model by using the state value or the action value.

Description

Learning device, learning method, and recording medium

[0001] The present disclosure relates to imitation learning in reinforcement learning.

[0002] In reinforcement learning, a method has been proposed that uses imitation learning to learn policies. Imitation learning is a technique for learning policies. A policy is a model that determines the next action for a given state. Among imitation learning methods, interactive imitation learning learns policies by referring to a teacher model rather than action data. Several methods have been proposed for interactive imitation learning, including a method that uses the teacher's policy as the teacher model and a method that uses the teacher's value function as the teacher model. Furthermore, even in methods that use the teacher's value function as the teacher model, there are methods that use a state value, which is a function of the state, as the value function, and a method that uses an action value, which is a function of the state and the action.

[0003] As an example of interactive imitation learning, Non-Patent Document 1 proposes a method for learning a policy by introducing a parameter k that discards a specific reward when calculating the sum of expected discounted rewards, and simultaneously shaping the reward using a teacher model.

[0004] Wen Sun, J. Andrew Bagnell, Byron Boots, "Truncated Horizon Policy Search: Combining Reinforcement Learning & Imitation Learning", ICLR 2018

[0005] Interactive imitation learning utilizes online feedback from a teacher model, so the applicable teacher model is limited. In particular, a value function for the teacher model is required for efficient learning.

[0006] One objective of the present disclosure is to provide a method for interactive imitation learning that can be performed if offline data of a teacher model is available.

[0007] In one aspect of the present disclosure, a learning device comprises: a generation means for generating a value estimation model of a teacher model from state and action data of the teacher model; an acquisition means for acquiring a next state as a result of executing an action determined using a policy of a student model; an estimation means for estimating a state value or action value of the next state using the next state and the value estimation model; and a policy update means for updating the policy of the student model using the state value or the action value.

[0008] In another aspect of the present disclosure, a learning method is executed by a computer, and includes generating a value estimation model for the teacher model from state and action data of the teacher model, obtaining a next state as a result of executing an action determined using a student model's policy, estimating a state value or action value for the next state using the next state and the value estimation model, and updating the student model's policy using the state value or action value.

[0009] In yet another aspect of the present disclosure, a recording medium records a program that causes a computer to execute the following processes: generate a value estimation model for the teacher model from state and action data of the teacher model; obtain the next state as a result of executing an action determined using the student model's policy; estimate a state value or action value for the next state using the next state and the value estimation model; and update the student model's policy using the state value or action value.

[0010] According to the present disclosure, it is possible to provide a method of interactive imitation learning that can be implemented if offline data of a teacher model is available.

[0011] It is a block diagram showing the hardware configuration of the learning device. It is a block diagram showing the functional configuration of the learning device. It is a flowchart of the student model learning process by the learning device. It is a block diagram showing the functional configuration of another learning device. It is a flowchart of the process by another learning device.

[0012] Preferred embodiments of the present disclosure will be described below with reference to the drawings. <Explanation of Principle> (1) Imitation Learning In reinforcement learning problems, imitation learning utilizes information from a teacher model, which serves as a model, to learn a student model that seeks a policy. In this case, the teacher model can be anything, such as a human, an animal, or an algorithm. Behavioral cloning, a typical imitation learning method, is a method that simply involves supervised learning of historical data on the states and behaviors of the teacher model, and is therefore weak when there is little or no data. For this reason, when the learned student model is actually run, the deviation from the teacher model amplifies over time, making it useful only for short-term problems.

[0013] Interactive imitation learning solves the above problem by providing online feedback from the teacher model to the student during learning instead of the teacher model's historical data. However, because interactive imitation learning requires online feedback from the teacher model, i.e., feedback at any time, the applicable teacher models are limited. In particular, a value function for the teacher model is required for efficient learning.

[0014] Therefore, the method of this embodiment (hereinafter also referred to as "this method") generates an action value function based on offline data of a teacher model, and uses the generated action value function to estimate the action value or state value of the teacher model to perform interactive imitation learning. As a result, this method can perform interactive imitation learning based on offline data of the teacher model without requiring a value function of the teacher model.

[0015] (2) Explanation of Terms Before explaining the method of this embodiment, the following explanation will be given of related terms. (2-1) Objective Function of Reinforcement Learning and Optimal Policy As the objective function of reinforcement learning, the sum of expected discounted rewards J[π] shown in formula (1) is typically used.

[0016] In equation (1), the following reward function r is the expected value of reward r obtained when action a is performed in state s.

[0017] The following discount factor γ is a coefficient used to discount future reward values ​​when evaluating them in the present.

[0018] The optimal policy shown below is a policy that maximizes the objective function J.

[0019]

[0020] (2-2) Value function The value function is the objective function J[π] in the initial state s ０ and initial action a ０ The value function represents the sum of expected discounted rewards to be obtained in the future when that state or action is taken. The state value function and action value function are expressed by the following equations (2) and (3). Note that when entropy regularization is introduced into the objective function J[π], the state value function and action value function are expressed by the following equations (2x) and (3x).

[0021]

[0022] The optimal value function is given by the following equation:

[0023]

[0024] (3) Interactive Imitation Learning Interactive imitation learning is a technique in which online feedback from a teacher model is given to a student model during learning, instead of historical data from the teacher model. Examples of interactive imitation learning include the following.

[0025] (3-1) The teacher should take action a in each state s. e = π ｅ (s) teaching type An example of this type of technique is DAgger. ｅ is a teacher strategy.

[0026] (3-2) A type in which a teacher teaches the value of an action a in each state s. Examples of this type of method include AggreVate, AggreVaTeD, and the method of Non-Patent Document 1 (hereinafter referred to as "THOR (Truncated HORizon Policy Search)"). Specifically, AggreVate and AggreVateD are based on the teacher's action value Q ｅ (s, a) is a teaching method, and THOR is the teacher's state value V ｅ It is a method of teaching (s).

[0027] Details of THOR are described in the following document (Non-Patent Document 1), which is incorporated herein by reference: Wen Sun, J. Andrew Bagnell, Byron Boots, "Truncated Horizon Policy Search: Combining Reinforcement Learning & Imitation Learning", ICLR 2018

[0028] (4) Method of this embodiment (4-1) Generation of action value function Q This method generates an action value function Q of a teacher model from offline data of the state and action of the teacher model. The action value function Q is a function for estimating the action value or state value of the teacher model, and is an example of a value estimation model of the present disclosure. Then, this method estimates the state value or action value of the teacher model using the obtained action value function Q, and performs interactive imitation learning. As a result, this method enables interactive imitation learning even without a value function of the teacher model.

[0029] Specifically, this method uses IQ-Learn (inverse soft Q learning) technology to generate an action-value function Q from offline data on the state and actions of a teacher model. IQ-Learn is a technology used in the field of inverse reinforcement learning that reads a reward function r from offline data on the state and actions of a teacher model, and this method uses this technique to generate an action-value function Q from offline data on the state and actions of a teacher model.

[0030] The IQ-Learn method is described in the following document, which is incorporated herein by reference: IQ-Learn: Inverse soft-Q Learning for Imitation, Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, Stefano Ermon, arXiv:2106.12142v4

[0031] Below, we will explain a method for generating the action value function Q. In the framework of maximum entropy inverse reinforcement learning, which is the mainstream of inverse reinforcement learning, the reward function r is interpreted as the solution to the optimization problem shown in the following equation.

[0032]

[0033] In equation (4), π ｅ is the teacher's policy, Ψ[r] is the regularization term for inverse reinforcement learning, and J[r, π] is the objective function of maximum entropy reinforcement learning, expressed as follows:

[0034]

[0035] This method uses J[r, π ｅ ] is the teacher model's policy π ｅ The estimation is performed from samples from the teacher model, i.e., from offline data of the teacher model's states and actions.

[0036] However, Equation (4) is a dual optimization problem of minimizing the reward function r and maximizing the policy π at the same time, and this optimization has a high computational cost and tends to make learning unstable. Therefore, this method uses the aforementioned IQ-Learn technique to convert the variables (r, π) into variables (Q, π), thereby replacing the optimization problem of Equation (4) with a dual optimization problem of minimizing the action value function Q and maximizing the policy π at the same time.

[0037] Furthermore, this method analytically finds the maximization of the policy π by solving the following equation, and finally replaces equation (4) with a single optimization problem for the action value function Q.

[0038]

[0039] Then, in this method, the solution Q obtained by solving the optimization problem to minimize this action value function Q is used as the action value function Q of the teacher model. ｅ In this way, the present method obtains the action-value function Q from offline data of the teacher model's states and actions.

[0040] (4-2) Application to Interactive Imitation Learning Next, a method for applying the action-value function Q obtained as described above to interactive imitation learning will be explained.

[0041] (4-2-1) Application to interactive imitation learning of the type that teaches teacher behavioral values. The teacher’s behavioral values ​​Q in the student’s state s, such as AggreVate and AggreVateD, ｅ For interactive imitation learning of the type that teaches (s, a), this method inputs the state s and action a at each time into the action value function Q obtained as described above to calculate the action value function Q(s, a), and the obtained value is the action value Q ｅ (s, a) is given to the student model.

[0042] (4-2-2) Application to interactive imitation learning of teaching teacher state value. The teacher state value V in the student state s, such as THOR, ｅ For interactive imitation learning of the type that teaches (s), this method uses the action value function Q obtained as described above to calculate the teacher's state value V ｅ Calculate (s).

[0043] When the set of actions A that an action a can take is a discrete set, this method specifically executes all actions a∈A, calculates the sum of the values ​​of the following relational expression using the action value function Q, and calculates this value as the teacher's state value V ｅ (s) is given to the student model.

[0044]

[0045] Note that when the action set A is a continuous set, unlike when the action set A is a discrete set, it is not possible to calculate the sum of the values ​​of the relational expression (7) by specifically executing all actions a∈A. When the action set A is a continuous set, IQ-Learn is executed using a method called Actor-Critic, which alternately updates the action value Q and the policy π while simultaneously learning. Therefore, this method alternately updates the action value Q and the policy π using the Actor-Critic method, and calculates the state value V(s) of the teacher model using the obtained policy π and the following relational expression:

[0046]

[0047] Since the above equation (8) is an equation for the expected value of the policy π, this method generates sample data using the policy π obtained by the Actor-Critic method and offline data on the state and actions of the teacher model, and approximates equation (8) with the sample data to obtain the state value V(s) of the teacher model. The state s and action a included in the offline data of the teacher model can be used as sample data. Alternatively, action a can be generated by inputting state s into the policy π obtained by the Actor-Critic method, and this can be used as sample data. The two types of sample data mentioned above, i.e., offline data of the teacher model and sample data generated using the policy π, can be mixed in any ratio. While this method simultaneously learns the action value Q and the policy π, it is expected to be computationally more efficient than solving the dual optimization problem shown in the aforementioned equation (4).

[0048] First Embodiment Next, a description will be given of a learning device according to the first embodiment. The learning device 100 according to the first embodiment is a device that learns a student model using the above-described technique.

[0049] 1 is a block diagram showing the hardware configuration of a learning device 100 according to the first embodiment. As shown in the figure, the learning device 100 includes an interface (I / F) 11, a processor 12, a memory 13, a recording medium 14, and a database (DB) 15.

[0050] The I / F 11 inputs and outputs data to and from external devices. For example, when the agent based on reinforcement learning of this embodiment is applied to an autonomous vehicle, the I / F 11 acquires the outputs of various sensors attached to the vehicle as the state of the environment, and outputs actions to various actuators that control the running of the vehicle.

[0051] The processor 12 is a computer such as a CPU (Central Processing Unit) that executes a pre-prepared program to control the entire learning device 100. The processor 12 may be a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an MPU (Micro Processing Unit), an FPU (Floating Point number Processing Unit), a PPU (Physics Processing Unit), a TPU (Tensor Processing Unit), a quantum processor, a microcontroller, or a combination thereof. The processor 12 executes the student model learning process described below.

[0052] The memory 13 is configured by a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The memory 13 is also used as a working memory while the processor 12 is executing various processes.

[0053] Recording medium 14 is a non-volatile, non-transitory recording medium such as a disk-shaped recording medium or semiconductor memory, and is configured to be detachable from learning device 100. Recording medium 14 records various programs executed by processor 12. When learning device 100 executes various processes, the programs recorded on recording medium 14 are loaded into memory 13 and executed by processor 12.

[0054] DB15 stores data used for learning by learning device 100. For example, DB15 stores data related to the teacher model used for learning, specifically offline data on the state and behavior of the teacher model. DB15 also stores data such as sensor output that indicates the state of the target environment and is input via I / F11.

[0055] (Functional Configuration) Figure 2 is a block diagram showing the functional configuration of the learning device 100. The learning device 100 learns a student model through interaction with the environment and the teacher's value estimation unit 20. First, the learning device 100 generates an action value function Q of the teacher model using offline data on the state and behavior of the teacher model using the method described above. The teacher's value estimation unit 20 uses the generated action value function Q.

[0056] The learning device 100 generates an action a based on the policy π of the current student model and inputs it to the environment. Next, the learning device 100 acquires a state s and a reward r for the action a from the environment. In the case of interactive imitation learning that does not use the reward r, the learning device 100 acquires only the state s from the environment.

[0057] Next, the learning device 100 inputs the state s acquired from the environment to the teacher's value estimation unit 20. The teacher's value estimation unit 20 uses the action value function Q to calculate the state value V of the teacher model. ｅ or Action Value Q ｅ Specifically, when interactive imitation learning of the type that teaches the state value of the teacher model is performed, the teacher value estimation unit 20 calculates an estimated value of the state value V of the teacher model. ｅ In addition, when interactive imitation learning of the type that teaches the behavioral value of the teacher model is performed, the teacher's value estimation unit 20 calculates an estimated value of the behavioral value Q of the teacher model. ｅ Then, the learning device 100 calculates an estimate of the state value V of the acquired teacher model. ｅ or Action Value Q ｅ Using the estimated value of , interactive imitation learning is performed to update the policy π.

[0058] (Student Model Learning Process) Figure 4 is a flowchart of the student model learning process by the learning device 100. This process is realized by the processor 12 shown in Figure 1 executing a program prepared in advance and operating as each element shown in Figure 2.

[0059] First, the learning device 100 acquires offline data of the state and action of the teacher model, and learns the action-value function Q of the teacher model using the method described above (step S11).

[0060] Next, the learning device 100 calculates the student model's current policy π ｔ Act based on a ｔ is output and input to the environment, and the next state s ｔ＋１ is acquired (step S12).

[0061] Next, the learning device 100 calculates the state s ｔ＋１ is input to the teacher's value estimation unit 20, and the state value V of the teacher model is calculated using the action value function Q. ｅ (s ｔ＋１ ) or Action Value Q ｅ (s ｔ＋１ , a) are estimated (step S13).

[0062] Next, the learning device 100 performs interactive imitation learning using the estimated state value or action value of the teacher model, and calculates the student model's policy π ｔ to π ｔ＋１ (step S14).

[0063] Next, learning device 100 determines whether learning has ended (step S15). Specifically, learning device 100 determines whether a predetermined learning end condition has been met. If learning has not ended (step S15: No), the process returns to step S12, and steps S12 to S14 are repeated. On the other hand, if learning has ended (step S15: Yes), the learning process for the student model ends.

[0064] [Effects] According to the method of this embodiment, similar to existing interactive imitation learning, it is possible to efficiently train a student model by utilizing information from a teacher model. Furthermore, unlike existing interactive imitation learning, the method of this embodiment allows learning even from a teacher model that only has offline data on states and actions. Because offline data on states and actions is typically easier to obtain than the value function of a teacher model, it is beneficial to be able to use a teacher model that only has offline data. Furthermore, by utilizing information from a wider range of teacher models, such as models of humans and animals, models constructed using learning or optimization methods other than reinforcement learning, and reinforcement-learned models that do not learn a value function, it is possible to reduce search from scratch and efficiently learn with less data.

[0065] In interactive imitation learning, there are methods that can efficiently learn from a non-optimal teacher model while achieving performance that exceeds that of the teacher. For example, the above-mentioned THOR and the method proposed by the present inventor based on THOR and described in Japanese Patent Application No. 2022-180115 are examples of such methods. The method of this embodiment can be combined with these methods, enabling interactive imitation learning using a non-optimal teacher model that only has offline data on states and actions.

[0066] 4 is a block diagram showing the functional configuration of a learning device according to Embodiment 2. As shown in the figure, a learning device 70 includes a generating unit 71, an acquiring unit 72, an estimating unit 73, and a policy updating unit 74.

[0067] 5 is a flowchart of processing by the learning device of the second embodiment. The generation means 71 generates a value estimation model for the teacher model from data on the state and action of the teacher model (step S71). The acquisition means 72 acquires the next state as a result of executing the action determined using the policy of the student model (step S72). The estimation means 73 estimates the state value or action value of the next state using the next state and the value estimation model (step S73). The policy update means 74 updates the policy of the student model using the state value or action value (step S74).

[0068] According to the learning device 70 of the second embodiment, if offline data of the teacher model is available, interactive imitation learning becomes possible.

[0069] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0070] (Supplementary Note 1) A learning device comprising: a generation means for generating a value estimation model of a teacher model from state and action data of the teacher model; an acquisition means for acquiring a next state as a result of executing an action determined using a policy of a student model; an estimation means for estimating a state value or action value of the next state using the next state and the value estimation model; and a policy update means for updating the policy of the student model using the state value or the action value.

[0071] (Supplementary Note 2) The learning device according to Supplementary Note 1, wherein the generating means generates the value estimation model from state and behavior data of the teacher model using inverse soft Q learning.

[0072] (Supplementary Note 3) The learning device according to Supplementary Note 1, wherein the policy update means updates the policy of the student model through interactive imitation learning.

[0073] (Supplementary Note 4) The learning device according to Supplementary Note 3, wherein the estimation means estimates an action value of a next state when the interactive imitation learning uses an action value of a teacher model.

[0074] (Supplementary Note 5) The learning device according to Supplementary Note 4, wherein the estimation means calculates an action value at each time using the value estimation model, and the policy update means updates the policy of the student model using the calculated action value.

[0075] (Supplementary Note 6) The learning device according to Supplementary Note 3, wherein the estimation means estimates a state value of a next state when the interactive imitation learning uses a state value of a teacher model.

[0076] (Appendix 7) A learning device as described in Appendix 6, wherein the estimation means calculates a state value at each time using the value estimation model and a predetermined relational expression, and the policy update means updates the policy of the student model using the calculated state value.

[0077] (Appendix 8) A learning method executed by a computer, which generates a value estimation model for the teacher model from data on the state and action of the teacher model, obtains the next state as a result of executing an action determined using the student model's policy, estimates the state value or action value of the next state using the next state and the value estimation model, and updates the student model's policy using the state value or action value.

[0078] (Appendix 9) A recording medium having recorded thereon a program that causes a computer to execute the following process: generate a value estimation model for the teacher model from data on the state and action of the teacher model; obtain the next state as a result of executing an action determined using the policy of the student model; estimate the state value or action value of the next state using the next state and the value estimation model; and update the policy of the student model using the state value or action value.

[0079] Although the present disclosure has been described above with reference to the embodiments and examples, the present disclosure is not limited to the above-described embodiments and examples. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure.

[0080] 12 Processor 20 Teacher value estimation unit 71 Generation means 72 Acquisition means 73 Estimation means 74 Policy update means

Claims

1. A learning device comprising: a generation means for generating a value estimation model of a teacher model from state and action data of a teacher model; an acquisition means for acquiring a next state as a result of executing an action determined using a policy of a student model; an estimation means for estimating a state value or action value of the next state using the next state and the value estimation model; and a policy update means for updating the policy of the student model using the state value or action value.

2. The learning device according to claim 1, wherein the generating means generates the value estimation model from state and action data of the teacher model using inverse soft-Q learning.

3. The learning device according to claim 1, wherein the policy update means updates the policy of the student model by interactive imitation learning.

4. The learning device according to claim 3, wherein said estimation means estimates an action value of a next state when said interactive imitation learning uses an action value of a teacher model.

5. A learning device according to claim 4, wherein the estimation means calculates an action value at each time using the value estimation model, and the policy update means updates the policy of the student model using the calculated action value.

6. The learning device according to claim 3, wherein said estimation means estimates a state value of a next state when said interactive imitation learning uses a state value of a teacher model.

7. The learning device described in claim 6, wherein the estimation means calculates a state value at each time using the value estimation model and a predetermined relational expression, and the policy update means updates the policy of the student model using the calculated state value.

8. A learning device that is executed by a computer, generates a value estimation model of a teacher model from state and action data of a teacher model, obtains the next state as a result of executing an action determined using a policy of a student model, estimates a state value or action value of the next state using the next state and the value estimation model, and updates the policy of the student model using the state value or action value.

9. A recording medium having recorded thereon a program that causes a computer to execute the following processes: generate a value estimation model of a teacher model from state and action data of a teacher model; obtain the next state as a result of executing an action determined using a student model's policy; estimate a state value or action value of the next state using the next state and the value estimation model; and update the student model's policy using the state value or action value.

Images