Inverse reinforcement learning device, inverse reinforcement learning method and program

Abstract

To provide a reverse reinforcement learning device, a method and a program for achieving efficiency of learning of a compensation scheme.SOLUTION: A reverse reinforcement learning device (10) comprises: a measures determination part (111) for determining measures (π) of a behavior on the basis of a locus (Sg) of a given behavior; and a compensation determination part (112) for determining a compensation (r) applied from an environment with respect to a behavior of an agent, so as to maximize an expected compensation (J) calculated from the measures (π) and a state worth (V). The compensation determination part (112) mixes the state worth (Vt) with an additional worth (Rt) being calculated using the compensation (r) so as to approximate the state worth (Vt), for calculating the state worth (V), and updates the compensation (r) so as to maximize the expected compensation (J) with respect to the calculated state worth (V).SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an inverse reinforcement learning device, an inverse reinforcement learning method, and a program. [Background technology]

[0002] Reinforcement learning has traditionally been used to accomplish given tasks. Reinforcement learning is a method of evaluating the behavior of an agent in an environment where a task is given, based on rewards given by the environment, and learning a policy that maximizes the cumulative reward of a series of actions. For example, reinforcement learning has been applied to games, motor control, and automatic driving control of vehicles (see Patent Documents 1 and 2). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2018-63602 [Patent Document 2] Japanese Patent Publication No. 2020-144483 Summary of the Invention [Problem to be solved by the invention]

[0004] Meanwhile, inverse reinforcement learning (IRL) is used to model the skills of experts. InIRL learns a reward system based on the trajectory of an expert's actions, determining which actions should be evaluated and rewarded. However, reward systems are generally complex. A method that can efficiently learn a reward system that approximates the expert's reward system has been sought.

[0005] The present invention aims to improve the efficiency of learning reward systems. [Means for solving the problem]

[0006] One aspect of the present invention is an inverse reinforcement learning device (10) that determines a reward (r) to be given by an environment for an agent's behavior based on a given behavior trajectory (Sg). The inverse reinforcement learning device (10) includes a policy determination unit (111) that determines an action policy (π) from the trajectory (Sg), and a reward determination unit (112) that determines the reward (r) so as to maximize an expected reward (J) calculated from the policy (π) and a state value (V) of the environment, as shown in the following formula (1). The reward determination unit (112) determines a state value (V) that evaluates a state (s) at time t, as shown in the following formula (2): t ) to the state value (V t ) is calculated using the reward (r) to approximate the added value (R t ) to calculate the state value (V), and update the reward (r) so that the expected reward (J) is maximized for the calculated state value (V).

number

[0007] Another aspect of the present invention is an inverse reinforcement learning method for determining a reward (r) given by an environment for an agent's behavior based on a given behavior trajectory (Sg). The inverse reinforcement learning method includes the steps of: determining an action policy (π) from the trajectory (Sg); and determining the reward (r) so as to maximize an expected reward (J) calculated from the policy (π) and a state value (V) of the environment, as shown in the above formula (1). The step of determining the reward (r) involves using a state value (V) that evaluates a state (s) at time t, as shown in the above formula (2). t ) to the state value (Vt ) is calculated using the reward (r) to approximate the added value (R t ) to calculate the state value (V), and updating the reward (r) so that the expected reward (J) is maximized for the calculated state value (V).

[0008] Another aspect of the present invention is a program for causing a computer to execute an inverse reinforcement learning method for determining a reward (r) to be given by the environment for an agent's action based on a given action trajectory (Sg). The inverse reinforcement learning method includes the steps of: determining an action policy (π) from the trajectory (Sg); and determining the reward (r) so as to maximize an expected reward (J) calculated from the policy (π) and a state value (V) of the environment, as shown in the above formula (1). The step of determining the reward (r) involves using a state value (V) that evaluates a state (s) at time t, as shown in the above formula (2). t ) to the state value (V t ) is calculated using the reward (r) to approximate the added value (R t ) to calculate the state value (V), and updating the reward (r) so that the expected reward (J) is maximized for the calculated state value (V). [Effects of the Invention]

[0009] According to the present invention, learning of a reward system can be made more efficient. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram showing a configuration of an inverse reinforcement learning device according to an embodiment of the present invention. [Figure 2] 10 is a flowchart illustrating an inverse reinforcement learning process. [Figure 3] FIG. 10 is a diagram showing an example of the trajectory of an expert's actions. [Figure 4] FIG. 2 is a diagram illustrating a state transition model. DETAILED DESCRIPTION OF THE INVENTION

[0011] An inverse reinforcement learning device, an inverse reinforcement learning method, and a program according to an embodiment of the present invention will be described below with reference to the drawings. The following description is an example (typical example) of the present invention, and the present invention is not limited thereto.

[0012] FIG. 1 shows the configuration of an inverse reinforcement learning device 10 according to one embodiment of the present invention. The inverse reinforcement learning device 10 includes a CPU (Central Processing Unit) 11 and a storage unit 12. The inverse reinforcement learning device 10 may further include an operation unit 13, a display unit 14, and a communication unit 15.

[0013] The CPU 11 executes an inverse reinforcement learning process, which will be described later, by reading and executing a program from the storage unit 12. In the inverse reinforcement learning process, the CPU 11 functions as a strategy determination unit 111 and a reward determination unit 112.

[0014] The policy determination unit 111 determines a policy for action from the given expert's action trajectory. The reward determination unit 112 determines a reward from the policy determined by the policy determination unit 111 and the state value of the environment so as to maximize the expected reward from a series of actions.

[0015] The storage unit 12 stores programs that can be read by the CPU 11, data used to execute the programs, etc. As the storage unit 12, for example, a recording medium such as a hard disk can be used.

[0016] The operation unit 13 is a keyboard, a mouse, etc. The operation unit 13 accepts an operation from a user and outputs the operation content to the CPU 11.

[0017] The display unit 14 is a display etc. The display unit 14 displays an operation screen, a processing result of the CPU 11, etc. in accordance with a display instruction from the CPU 11.

[0018] The communication unit 15 is an interface for communicating with an external computer via a network.

[0019] The inverse reinforcement learning device 10 can determine the reward (r) to be given from the environment from the trajectory of the action of the expert to be imitated by inverse reinforcement learning processing. In this embodiment, the reward (r) is defined as a neural network having a parameter (θ) as shown in equation (4). The parameter refers to a weight or bias set in the neural network. (4) r = r(θ)

[0020] FIG. 2 is a flowchart of the inverse reinforcement learning process. First, the policy determination unit 111 acquires a group of trajectories (Sg) of the expert's actions given together with the environment (step S1). The policy determination unit 111 may acquire the group of trajectories (Sg) from the storage unit 12 or from an external device on the network. The trajectory (Sg) is expressed as a set of environmental states (s) transitioned by a series of actions, as shown in equation (5). (5) Sg={(s0,s1,...,s n )}

[0021] Figure 3 shows an example of a behavioral trajectory (Sg). The trajectory L1 shown in Figure 3 is the path taken by the expert when moving through the maze from the start point Ps to the goal point Pg. The maze consists of an area 30 of multiple blocks, and movement can be made one block at a time. Movement may be blocked by walls placed between the blocks. Here, the area 30 is a given environment, and each block corresponds to a state(s) of the environment.

[0022] The policy determination unit 111 generates a state transition model from this trajectory (Sg). The state transition model is a distribution of transition probabilities from a certain state (s) of the environment to the next state (s). For example, the state transition model is generated as a state transition matrix in which the transition probabilities are tabulated. The policy determination unit 111 determines a policy (π) based on this state transition model (step S2). The policy (π) is a probability distribution of the action (a) selected in each state (s).

[0023] FIG. 4 is a diagram illustrating a state transition model. In the maze described above, the blocks on the trajectory L1 have a high value of the state (s). In Fig. 4, the circles placed on each block represent the value of the state (s), and the higher the density of the circles, the higher the value of the state (s). The strategy determination unit 111 can determine the transition probability from each block (state) to the next block (state) so that the probability of transitioning to a block on the trajectory L1 becomes higher.

[0024] Next, the reward determination unit 112 determines a reward (r) from the determined policy (π) so that the expected reward (J) is maximized. The expected reward (J) is the cumulative reward that can be expected to be obtained in one episode. An episode is a process from the initial state (s0) of the environment to the final state (s e ) The expected reward (J) is calculated using the policy (π) and state value (V) as shown in equation (1).

[0025]

number

[0026] Above, E[] represents a function that outputs the expected value in []. π(s|a) represents the strategy (π) for selecting an action (a) in state (s). V(s) represents the state value (V) that evaluates state (s).

[0027] In this embodiment, the state value (V) is defined as shown in equation (2). As shown in equation (2), the state value (V) is a state value (V) that evaluates the state (s) at time t.t ), plus added value (R t ) is calculated by mixing the added value (R t ) is the state value (V t ) is calculated using the reward (r) to approximate

[0028]

number

[0029] Above V t represents the state value (V) that evaluates the state (s) at time t. R t represents the added value at time t. P , τ D and τ I are coefficients between 0 and 1. E represents the discount rate of the reward (r) given to each action (a), and 0<γ E ≦1. t e is the final state (s e ) represents the time in the equation (3). P The term containing the coefficient τ D The term containing the differential term, the coefficient τ I The term containing is called the integral term.

[0030] Coefficient τ in the proportional term P multiplied by r * represents the reward (r) given to the action (a) in the state (s) at time t in one episode. For example, when t=5, the reward determination unit 112 can use the reward (r) given to the action (a) in the state (s5) in the proportional term.

[0031] In the differential term, the coefficient τ Ddr / dt, which is multiplied by , represents the differential value of the reward (r) granted within a certain time period from a state before time t to state (s) at time t in one episode. By adding the differential term, it is possible to determine the state value (V) taking into account the change in reward (r) over time. For example, when t=5, the reward determination unit 112 can use the differential value of the reward (r) granted from state (s3) to state (s5) in steps 3 to 5 as the differential term.

[0032] In the integral term, the coefficient τ I The integral of r multiplied by represents the cumulative value of the reward (r) given during one episode. In this cumulative value, the reward (r) for the action (a) in each state is discounted by γ E It is discounted by the discount rate γ E is(t e -t) and the final state (s e ), the smaller the discount rate of the reward (r).

[0033] When determining the remuneration (r), the remuneration determination unit 112 first calculates the added value (R t ) is optimized (step S3). Specifically, the reward determination unit 112 updates the parameter (θ) so as to maximize the expected reward (J), thereby updating the reward (r) shown in equation (4). The updated reward (r) is used to calculate the added value (R t ) to calculate the optimized value added (R t ) is obtained.

[0034] Next, the remuneration determination unit 112 determines the optimized added value (R t ) in equation (2), t ) is optimized (step S4). This optimization uses the updated reward (r) to calculate the state value (V t ) is updated.

number

[0035] s trepresents the state of the environment (s) at time t. t+1 is the state (s t ) represents the state (s) one step after the transition from r t+1 is the state (s t ) is the reward (r) given by the environment according to the action (a) in the state (s). α is the learning rate and satisfies 0<α≦1. γ is the discount rate and satisfies 0<γ≦1. max is the time it takes to t+1 It represents a function that outputs the maximum value of the state value (V) of the next state that can be transitioned from the state

[0036] State Value (V t If the expected reward (J) shown in equation (1) does not converge after the optimization of the added value (R t ) (Step S3) and the state value (V t ) (Step S4) are alternately repeated. As a result, the reward (r) is optimized so that the expected reward (J) is maximized. When the expected reward (J) converges, the reward (r) also converges. In this case (Step S5: YES), the inverse reinforcement learning process ends.

[0037] In this way, the remuneration determination unit 112 determines the added value (R t ) update, that is, update of reward (r) and update of state value (V t ) and iteratively update the reward (r). This iteration is possible because, as shown in equation (2), the state value (V t ) term and the reward (r) to calculate the state value (V t ) is an approximate added value (R t ) and define the state value (V). According to equations (2) and (3), it is easy to calculate the reward (r) that maximizes the expected reward (J), making it possible to improve the efficiency of inverse reinforcement learning.

[0038] In the inverse reinforcement learning process, the reward determination unit 112 determines the added value (R t) can be mixed. For example, if they are mixed equally, you can adjust τ = 0.5. The original state value (V t ) rather than added value (R t If you want to increase the ratio of τ, you can adjust it to τ=0.7.

[0039] The reward determination unit 112 determines whether the added value (R t It is preferable to reduce the ratio (τ) of mixing the reward (r). The reward determination unit 112 can ultimately reduce the ratio (τ) to 0. By reducing the ratio (τ), the learning time of the reward (r) can be shortened while the added value (R t ) can be converged to the same result as when it is not used.

[0040] The reward determination unit 112 may monotonically decrease the rate (τ), or may arbitrarily determine the degree to which the rate (τ) is decreased relative to the number of updates. Furthermore, the reward determination unit 112 may temporarily increase the rate (τ) during the process of decreasing it.

[0041] In addition, the remuneration determination unit 112 determines each coefficient τ P , τ D and τ I By adjusting τ, the ratio of the proportional term, the differential term, and the integral term can be adjusted, and it is also possible to set the ratio to 0. For example, τ P =0, τ D =0, τ I By setting =1, only the integral term, i.e., the cumulative reward, is used as the state value (V t If you want to emphasize the state (s) at time t, you can use τ P If you want to consider time changes, set τ D =1 to add the derivative term.

[0042] As described above, according to this embodiment, a policy (π) is determined from a given trajectory (Sg), and the state value (V) is updated so that the expected reward (J) for this policy (π) is maximized. At this time, as shown in equation (2), the state value (V) at time t ist ) and the added value (R t ) in a predetermined ratio (τ) to calculate the state value (V).

[0043] This allows us to maximize the expected reward (J) by calculating the added value (R t ) and state value (V t ) can be updated repeatedly. t Since the reward (r) is updated by updating J, the reward (r) can be optimized by maximizing the expected reward (J). Equation (2) simplifies calculations and makes it easy to optimize even complex rewards (r), making it possible to improve the efficiency of inverse reinforcement learning.

[0044] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments and various modifications are possible within the scope of the present invention. For example, the value added (R t ) was defined, but the reward (r) was used to calculate the state value (V t ) can be approximated, but is not limited to this.

[0045] Since the state value (V) is calculated by the reward (r), instead of dr / dt in equation (3), we use dV / dt or V(s t )-V(s) may be used. dV / dt represents the differential value of the state value (V) within a certain time period from the state before time t to the state (s) at time t. Also, V(s t )-V(s) represents the change in state value (V) from the state before time t to state (s) at time t.

[0046] Furthermore, the reward (r) is defined by a neural network, but may also be defined as a reward function that is approximated by a linear function.

[0047] The inverse reinforcement learning device 10 can be used in various technical fields, and the technical fields are not particularly limited. For example, the inverse reinforcement learning device 10 can be used for automatic driving control that determines a vehicle's driving path while avoiding hazards, motor drive control, game character control, etc.

[0048] Furthermore, a recording medium may be provided that stores a program for causing a computer to execute the inverse reinforcement learning method of the present invention. The recording medium is not particularly limited as long as it is a recording medium that can be read by a computer such as a CPU, and semiconductor memories, magnetic disks, optical disks, etc. may be used. [Explanation of symbols]

[0049] 10 Reinforcement learning device, 11 CPU, 111 Policy determination unit, 112 Reward determination unit, 12 Memory unit

Claims

1. In an inverse reinforcement learning device (10) that determines a reward (r) given by an environment for an agent's action based on a given action trajectory (Sg), A strategy determination unit (111) that determines an action strategy (π) from the trajectory (Sg); a reward determination unit (112) that determines the reward (r) so that an expected reward (J) calculated from the policy (π) and the state value (V) of the environment is maximized as shown in the following formula (1), The remuneration determination unit (112) As shown in the following formula (2), the state value (V ｔ ) to the state value (V ｔ ) is calculated using the reward (r) to approximate the added value (R ｔ Calculate the condition value (V) by mixing Update the reward (r) so that the expected reward (J) is maximized for the calculated state value (V). An inverse reinforcement learning device (10). [Equation 1] [E represents a function that outputs an expected value. π(s|a) represents a strategy (π) for selecting an action (a) in a state (s). V(s) represents a state value (V) that evaluates the state (s). V ｔ represents the state value (V) that evaluates the state (s) at time t. ｔ represents the added value at time t. τ is R ｔ represents the mixing ratio of τ, and satisfies 0≦τ≦1.

2. As shown in the following formula (3), the reward determination unit (112) calculates the added value (R) by adding the reward (r) given for the action (a) in the state (s) at time t in one episode, the differential value of the reward (r) given from the state before time t in the one episode to the state (s) at time t, and the cumulative value of the reward (r) given during the one episode. ｔ ) to calculate The inverse reinforcement learning device (10) according to claim 1. [Equation 2] [τ Ｐ , τ Ｄ and τ Ｉ represents a coefficient between 0 and 1. ＊ represents the reward (r) given to the action (a) in the state (s) at time t. dr / dt represents the differential value of the reward (r) given from the state before time t to the state (s) at time t. γ Ｅ represents the discount rate, and 0 < γ Ｅ ≦1. ｅ represents the time at the end of one episode.

3. The remuneration determination unit (112) The reward (r) is updated so as to maximize the expected reward (J), thereby ｔ ) and optimize The updated reward (r) is used to calculate the state value (V ｔ ) to update the state value (V ｔ ) and optimize The added value (R ｔ ) and the state value (V ｔ ) and repeat the process until the reward (r) converges. An inverse reinforcement learning device (10) according to claim 1 or 2.

4. The remuneration determination unit (112) determines the added value (R ｔ Adjust the mixing ratio (τ) An inverse reinforcement learning device (10) according to any one of claims 1 to 3.

5. The reward determination unit (112) determines the added value (R ｔ ) to reduce the mixing ratio (τ) An inverse reinforcement learning device (10) according to any one of claims 1 to 4.

6. The remuneration determination unit (112) Define the reward (r) as a neural network with parameters (θ), Update the reward (r) by updating the parameter (θ) so that the expected reward (J) is maximized. An inverse reinforcement learning device (10) according to any one of claims 1 to 5.

7. In a computer-based inverse reinforcement learning method, a reward (r) given by the environment for an agent's action is determined based on a given action trajectory (Sg), determining a course of action (π) from said trajectory (Sg); and determining the reward (r) so as to maximize the expected reward (J) calculated from the policy (π) and the state value (V) of the environment, as shown in the following formula (1): The step of determining the reward (r) comprises: As shown in the following formula (2), the state value (V ｔ ) to the state value (V ｔ ) is calculated using the reward (r) to approximate the added value (R ｔ Calculating the state value (V) by mixing updating the reward (r) so that the expected reward (J) is maximized for the calculated state value (V). Inverse reinforcement learning methods. [Equation 3] [E represents a function that outputs an expected value. π(s|a) represents a strategy (π) for selecting an action (a) in a state (s). V(s) represents a state value (V) that evaluates the state (s). V ｔ represents the state value (V) that evaluates the state (s) at time t. ｔ represents the added value at time t. τ is R ｔ represents the mixing ratio of τ, and satisfies 0≦τ≦1.

8. A program for causing a computer to execute an inverse reinforcement learning method for determining a reward (r) to be given from the environment for an agent's action based on a given action trajectory (Sg), The inverse reinforcement learning method includes: determining a course of action (π) from said trajectory (Sg); and determining the reward (r) so as to maximize the expected reward (J) calculated from the policy (π) and the state value (V) of the environment, as shown in the following formula (1): The step of determining the reward (r) comprises: As shown in the following formula (2), the state value (V ｔ ) to the state value (V ｔ ) is calculated using the reward (r) to approximate the added value (R ｔ Calculating the state value (V) by mixing updating the reward (r) so that the expected reward (J) is maximized for the calculated state value (V). program. [Equation 4] [E represents a function that outputs an expected value. π(s|a) represents a strategy (π) for selecting an action (a) in a state (s). V(s) represents a state value (V) that evaluates the state (s). V ｔ represents the state value (V) that evaluates the state (s) at time t. ｔ represents the added value at time t. τ is R ｔ represents the mixing ratio of τ, and satisfies 0≦τ≦1.

Images