Reinforcement learning device, reinforcement learning method, and program

Abstract

To enhance learning efficiency of reinforcement learning.SOLUTION: A device (10) for reinforcement learning repetitively performs an episode until a task is accomplished in given environment and learns actions of an agent such that a cumulative value of a series of actions in one episode is maximized. The device (10) for reinforcement learning includes: an action selection unit (111) which selects the actions of the agent; a calculation treatment unit (112) which computes a value (Q) of the selected actions by using a reward (R); and a task control unit (113) which defines one or more subtasks from the task and defines a second reward (M) which is given according to accomplish degree of the subtasks. The calculation treatment unit (112) acquires the second reward (M) defined for the selected actions and the first reward (r) given by the environment and computes a reward (R) by adding the second reward (M) to the first reward (r).SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a reinforcement learning device, a reinforcement learning method, and a program. [Background technology]

[0002] Conventionally, reinforcement learning has been used to achieve a given task. Reinforcement learning is a method of learning an agent's behavior so as to maximize the cumulative value of a series of actions of the agent in an environment in which a task is given. For example, reinforcement learning has been applied to games and motor control, automatic driving control of vehicles, and the like (see Patent Documents 1 and 2). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-63602 A [Patent Document 2] JP 2020-144483 A Summary of the Invention [Problem to be solved by the invention]

[0004] In general reinforcement learning, the agent's actions are selected randomly or probabilistically, and trial and error is repeated. Since the learning time until the cumulative value converges to the maximum value tends to be long, there is a demand for more efficient learning.

[0005] In particular, in an environment where an agent does not receive a reward for its actions until it completes a task, the learning time is likely to be long. The value of an agent's actions is calculated based on the reward given by the environment for that action, and the calculated value becomes a guideline for selecting the agent's subsequent actions. However, if no reward is given during the process of completing a task, the value of the actions during that time does not change, and no guideline for selecting an action is obtained, leading to increased trial and error.

[0006] An object of the present invention is to improve the learning efficiency of reinforcement learning. [Means for solving the problem]

[0007] One aspect of the present invention is a reinforcement learning device (10) that repeatedly executes an episode of selecting an action of an agent until a task is accomplished in a given environment, and learns the action of the agent so as to maximize the cumulative value of a series of actions in one episode. The reinforcement learning device (10) includes an action selection unit (111) that repeatedly selects an action of the agent, a calculation processing unit (112) that calculates a value (Q) of the selected action using a reward (R) granted to the selected action each time the action is selected, and a task control unit (113) that defines one or more subtasks from the task and defines a second reward (M) to be granted depending on the degree of achievement of the subtask. The calculation processing unit (112) obtains the second reward (M) defined for the selected action and a first reward (r) granted from the environment, and calculates the reward (R) by adding the second reward (M) to the first reward (r).

[0008] Another aspect of the present invention is a reinforcement learning method for repeatedly selecting an action of an agent until a task is accomplished in a given environment, and learning the action of the agent so as to maximize the cumulative value of a series of actions in one episode. The reinforcement learning method includes a step of repeatedly selecting an action of the agent, a step of calculating a value (Q) of the selected action using a reward (R) given to the selected action each time the action is selected, and a step of defining one or more subtasks from the task and defining a second reward (M) to be given according to the degree of accomplishment of the subtask. The step of calculating the value (Q) includes a step of acquiring the second reward (M) defined for the selected action and a first reward (r) given from the environment, and a step of calculating the reward (R) by adding the second reward (M) to the first reward (r).

[0009] Another aspect of the present invention is a program for causing a computer to execute a reinforcement learning method for repeatedly selecting an action of an agent until the agent accomplishes a task in a given environment, and learning the action of the agent so as to maximize the cumulative value of a series of actions in one episode. The reinforcement learning method includes a step of repeatedly selecting an action of the agent, a step of calculating a value (Q) of the selected action using a reward (R) given to the selected action each time the action is selected, and a step of defining one or more subtasks from the task and defining a second reward (M) to be given according to the degree of achievement of the subtask. The step of calculating the value (Q) includes a step of acquiring the second reward (M) defined for the selected action and a first reward (r) given from the environment, and a step of calculating the reward (R) by adding the second reward (M) to the first reward (r). Effect of the Invention

[0010] According to the present invention, it is possible to improve the learning efficiency of reinforcement learning. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram showing a configuration of a reinforcement learning device according to an embodiment of the present invention. [Diagram 2] 1 is a flowchart showing a reinforcement learning process executed in a reinforcement learning device. [Diagram 3] FIG. 2 is a diagram showing a game area as an example of an environment. [Figure 4] FIG. 13 illustrates an example of a subtask. [Diagram 5] FIG. 13 is a diagram showing an example of a second remuneration table. [Figure 6] FIG. 13 is a diagram showing another example of the second remuneration table. [Figure 7] FIG. 13 is a diagram showing an example of a value table. [Figure 8] FIG. 13 illustrates an example of a table of values ​​calculated for each subtask. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of a reinforcement learning device, a reinforcement learning method, and a program according to the present invention will be described with reference to the drawings. The following description is an example (representative example) of the present invention, and the present invention is not limited thereto.

[0013] FIG. 1 shows a configuration of a reinforcement learning device 10 according to an embodiment of the present invention. The reinforcement learning device 10 includes a CPU (Central Processing Unit) 11 and a storage unit 12. The reinforcement learning device 10 may further include an operation unit 13, a display unit 14, and a communication unit 15.

[0014] The CPU 11 executes a reinforcement learning process, which will be described later, by reading and executing a program from the storage unit 12. In the reinforcement learning process, the CPU 11 functions as a behavior selection unit 111, a calculation processing unit 112, and a task control unit 113.

[0015] The action selection unit 111 selects an action of the agent in an environment in which a task is given. The calculation processing unit 112 calculates the value of the action selected by the action selection unit 111 using a reward to be given to the action. The task control unit 113 defines a subtask from the given task.

[0016] The storage unit 12 stores programs that can be read by the CPU 11, tables used to execute the programs, etc. As the storage unit 12, for example, a recording medium such as a hard disk can be used.

[0017] The operation unit 13 is a keyboard, a mouse, etc. The operation unit 13 accepts an operation by a user and outputs the content of the operation to the CPU 11.

[0018] The display unit 14 is a display etc. The display unit 14 displays an operation screen, a processing result of the CPU 11, etc. according to a display instruction from the CPU 11.

[0019] The communication unit 15 is an interface for communicating with an external computer via a network.

[0020] In the reinforcement learning device 10, the CPU 11 can determine a strategy for accomplishing a given task by reinforcement learning. In this embodiment, an example of Q-learning, which is one type of reinforcement learning, will be described.

[0021] (General reinforcement learning method) In Q-learning, an agent is placed in an environment where a task is given. An agent is a subject of action. The agent determines the state of the environment (s t ) one action (a) is selected from multiple possible actions (a). The selected action (a) affects the state of the environment (s t ) is the state (s t+1 )

[0022] Each action (a) of an agent is associated with a value (Q) that evaluates the action (a). The value (Q) is calculated by the action value function Q expressed by the following formula (10).

number

[0023] In equation (10), s t represents the state of the environment (s) at time t. t+1 is the state (s t ) represents the state (s) one step after the transition due to the action (a) in t+1 is the state (s t ) depending on the action (a). α represents the learning rate and satisfies 0<α≦1. γ represents the discount rate and satisfies 0<γ≦1. maxQ(s t+1 ,a) is the state (s t+1 ) the value Q(s t+1 ,a) represents a function that outputs the maximum value.

[0024] The environment changes from the initial state (s0) to the final state (s e The agent continues to take action (a) until it transitions from the initial state (s0) to the final state (s e ) is called an episode. By repeatedly performing episodes, the value (Q) of the action (a) in each state (s) is calculated and updated in sequence. The series of actions that has the greatest cumulative value after a certain number of episodes is learned as the optimal action for a given environment.

[0025] In general Q-learning, episodes must be repeated and trial and error must be carried out until the cumulative value converges to near the maximum value. This can lead to long learning times. This is especially true in environments where rewards (r) are not given until a task is accomplished. Value (Q) is weighted by reward (r) and serves as a guide for the selection of the agent's subsequent actions (a). However, if the reward (r) given until the task is accomplished is 0, the value (Q) of action (a) will not change until the task is accomplished, leading to increased trial and error.

[0026] In contrast, the reinforcement learning device 10 of this embodiment defines a subtask that is a part of a given task, and grants a reward according to the action (a) for achieving the subtask. In other words, the reinforcement learning device 10 uses not only the reward (r) given by the environment but also a reward according to the achievement degree of the subtask to calculate the value (Q). The value (Q), which serves as a guideline for selecting the action (a) even until the task is accomplished, is weighted, thereby reducing trial and error, thereby improving the efficiency of learning. Hereinafter, the conventional reward (r) given by the environment is referred to as the first reward (r), and the reward added to this first reward (r) is referred to as the second reward (M).

[0027] (Reinforcement learning method according to this embodiment) 2 shows a flow of the reinforcement learning process in the reinforcement learning device 10. This reinforcement learning process is executed by the CPU 11 reading a program from the storage unit 12. A game will be described below as an example of a task given to the reinforcement learning process. Fig. 3 shows a 6x6 block game area 30 given as the game environment.

[0028] 3, each block is identified by a block number (ij). i is a number from 0 to 5 that represents a row of the game area 30. j is a number from 0 to 5 that represents a column of the game area 30. For example, the block in the second row and first column is represented as block (10).

[0029] A brush 20 is placed on the game area 30. The brush 20 can move one block at a time from the current block (ij) up, down, left, or right, and clean the block (ij) after the movement. The color of the cleaned block (ij) changes from black to white.

[0030] The task of this game is to change the color of all blocks (ij) from black to white. The shorter the total moving distance of the brush 20, the more efficiently the task can be accomplished. The first reward (r) given from the game area 30 when all blocks (ij) have changed to white is 100 points. While there are black blocks (ij), the first reward (r) given for any action (a) is 0 points. Information on the first reward (r) is given together with the game area 30 and stored in the memory unit 12.

[0031] In this task, the game area 30 is the given environment. In the initial state (s0) of the environment, the number of white blocks is 0 and the remaining number of black blocks is 36. The action (a) of moving the brush 20 causes a transition in the state (s) of the environment, i.e., the color state (s) of the blocks (ij). The game ends when the task is accomplished, so the final state (s e ) is a state in which all blocks are white, meaning there are 36 white blocks and no black blocks remaining.

[0032] The reinforcement learning device 10 repeatedly selects an action (a) of an agent that moves the brush 20 as a game player and calculates its value (Q). In this way, the reinforcement learning device 10 searches for an action (a) that maximizes the cumulative value of the value (Q) calculated for a series of actions (a) until the color of all blocks (ij) changes from black to white.

[0033] First, the task control unit 113 of the CPU 11 defines one or more subtasks from a given task (hereinafter, referred to as a main task) (step S1). A subtask is a part of the main task. In other words, the achievement of a subtask results in the achievement of part of the main task.

[0034] FIG. 4 shows an example of a subtask. In this example, the task control unit 113 defines three subtasks from a main task of changing the color of a game area 30 of 6×6 blocks, and assigns IDs 01, 02, and 03 to them. The subtask with ID=01 is to change the color of an area 31 of 4×2 blocks. The subtask with ID=02 is to change the color of an area 32 of 2×4 blocks. The subtask with ID=03 is to change the color of an area 33 of 3×4 blocks.

[0035] A part of each subtask may overlap with a part of another subtask. In the example of Fig. 4, area 32 of subtask ID=02 partially overlaps with area 33 of subtask ID=03. Note that multiple subtasks without overlaps may be defined by dividing the main task into several parts.

[0036] By the definition of the subtasks, the environment has a state (s) in the main task and a state (m) in each subtask. The state (s) has multiple elements that represent the state of the main task, and s = {x1, x2, . . . , x n}. State (m) has multiple elements that represent the states of the subtasks, and m={y1, y2, . . . , y n Each element can be determined arbitrarily. For example, element x of the state (s) of the main task n includes the block numbers (ij) of the blocks that have turned white due to the movement of the brush 20 in the game area 30, the number of black blocks remaining, etc. Also, the element y n includes the block numbers (ij) that have changed to white, the number of remaining black blocks, etc., within the area of ​​each subtask.

[0037] For example, in the game area 30 shown in Fig. 4, the main task has block numbers 00 and 01 that have changed to white, and the number of remaining black blocks is 34 (s). The subtask with ID=01 has no block numbers that have changed to white in area 31, and the number of remaining black blocks is 8 (m). The subtask with ID=03 has no block numbers that have changed to white in area 33, and the number of remaining black blocks is 12 (m). The subtask with ID=02 has block numbers 00 and 01 that have changed to white in area 32, and the number of remaining black blocks is 6 (m).

[0038] Next, the task control unit 113 defines a second reward (M) to be given to the agent's action (a) according to the achievement level of the subtask (step S2). The task control unit 113 stores the defined second reward (M) in the storage unit 12.

[0039] The task control unit 113 can define the second reward (M) independently for each subtask. The second reward (M) for each subtask may have the same definition or may have different definitions.

[0040] FIG. 5 shows an example of a table T31 for holding a second reward (M) defined for a subtask with ID=01. In FIG. t ,m) and (s t+1 ,m) indicates only the number of remaining black blocks in area 31 among the elements indicating the state.

[0041] In table T31, a certain state (s t ,m) to state (s t+1 For an action (a) that transitions to the state (s , m) and changes the color of the block, a second reward (M) of 3 points is associated. For example, in the state (s t ,m) to state 7 (s t+1 Action (a) that transitions to step (m) has a second reward (M) associated with it of 3 points.

[0042] On the other hand, an action (a) that does not change the number of remaining blocks is associated with a second reward (M) of 0 points. According to this definition, every time the number of remaining black blocks in the area 31 is reduced by 1 due to the action (a) of moving the brush 20, 3 points of the second reward (M) are awarded.

[0043] In this embodiment, the definition of the second reward (M) for the subtask with ID=02 is the same as that for the subtask with ID=01. Therefore, the configuration of table T32 of the second reward (M) for the subtask with ID=02 is the same as table T31.

[0044] FIG. 6 shows an example of a table T33 for holding a second reward (M) defined for a subtask with ID=03. In FIG. t ,m) and (s t+1 ,m) indicates only the remaining number of black blocks in area 33 among the elements indicating the state of the subtask. In table T33, the remaining number of black blocks in area 33 is 1 (s t ,m) to the 0 state (s t+1 A second reward (M) of 36 points is associated with an action (a) that transitions to the next action (m), while a second reward (M) of 0 points is associated with any other action (a).

[0045] According to this definition, the second reward (M) given to the action (a) of changing part of area 33 to white is 0 points. The second reward (M) of 36 points is given to the action (a) of changing the last black block to white. In other words, the second reward (M) is not given every time the color of each block changes, but is given all at once when the color of all blocks changes.

[0046] 5 and 6 show an example of the definition of the second reward (M), and the method of defining the second reward (M) is not limited to this. Other definition methods can be adopted depending on the contents of the task.

[0047] When the definition of the subtask and the second reward (M) is completed, the action selection unit 111 initializes the game and starts the episode (step S3). The initialization resets the environment to the initial state (s0). That is, the color of all blocks changes from white to black, and the remaining number of black blocks is reset to 36.

[0048] The action selection unit 111 determines whether the agent is in the current state (s t One action (a) is selected from among the actions (a) that can be taken in the game area 30 in the state (s, m) (step S4). t ,m) to state (s t+1 ,m).

[0049] For example, as shown in Fig. 3, in the initial state (s0), the action (a) of placing the brush 20 on the block (00) is selected. Because the color of the block (00) changes to white, the environment changes from a state (s0,m) in which the number of remaining black blocks is 36 to a state (s1,m) in which the number of remaining black blocks is 35.

[0050] In this embodiment, the selection of the action (a) is performed probabilistically by the ε-greedy method. Specifically, the action selection unit 111 selects the state (s t The action selection unit 111 randomly selects one of the actions (a) that can be taken in the state (s , m). t Among the possible actions (a) in the next state (s t+1 ,m) select the action (a) with the largest value (Q) of the action. In other words, maxQ(s t+1 ,m,a) is selected.

[0051] maxQ(s t+1If an action (a) that gives the highest Q (m,a) is always selected, learning may stagnate. This is because eliminating an action (a) with low value (Q) may result in failing to select a subsequent action (a) with higher value (Q). The ε-greedy method deliberately selects random actions (a), including actions (a) with low value (Q), with a certain probability. This expands the possibilities for selecting actions, making learning more efficient. The method for selecting the action (a) is not limited to the ε-greedy method, but other methods such as the softmax method can be adopted depending on the purpose.

[0052] When the action (a) is selected, the calculation processing unit 112 calculates the state (s t , m) calculates the value (Q) of the action (a) selected (step S5). The following formula (1) shows the action value function Q used to calculate the value (Q) in this embodiment. R in formula (1) is the state (s t ,m) is the reward given to the action (a) selected in the step (a), m. The following formula (2) shows the reward function used to calculate the reward (R).

[0053]

number

[0054] In formula (2), r t+1 is the state (s t represents the first reward (r) associated with the action (a) in M(s t+1 ,m) is the state (s t ) represents the second reward (M) associated with the action (a) in the equation (10). τ represents a coefficient satisfying 0≦τ≦1. The definitions of γ, α, and max are the same as those in the equation (10).

[0055] First, the calculation processing unit 112 acquires the first reward (r) and the second reward (M) to be used in the calculation of the value (Q). The calculation processing unit 112 calculates the reward (R) by adding the second reward (M) to the acquired first reward (r). When the second reward (M) is granted from multiple subtasks, the calculation processing unit 112 adds them together and uses them in the calculation of the reward (R).

[0056] For example, the first reward (r) given by the environment due to the action (a) of moving the brush 20 from block (00) to block (01) is 0 points. On the other hand, due to this action (a), the remaining number of black blocks in area 32 changes from 7 to 6, and a second reward (M) of 3 points is obtained from table T31. The second reward (M) obtained from the subtasks in areas 32 and 33, where the color of the blocks does not change, is 0 points. Therefore, the reward (R) is calculated as 3 points.

[0057] Next, the calculation processing unit 112 calculates the value (Q) using the calculated reward (R) as shown in formula (1). The calculation processing unit 112 stores the calculated value (Q) in the storage unit 12.

[0058] FIG. 7 shows an example of a table Tq that holds a value (Q) in the storage unit 12. In the table Tq, each action (a) is associated with a calculated value (Q). Also, each action (a) is associated with a state (s t+1 ), the block number (ij) that has turned white is associated with the remaining number n of black blocks in the game area 30. These are written in each time an action (a) is selected and a value (Q) is calculated.

[0059] The selected action (a) changes the environment to a final state (s e ) (step S6: NO), the action selection unit 111 selects the state (s t+1 ,m) to the current state (s t, m) (step S7). Then, the processes of steps S4 and S5 are repeated. That is, the final state (s e ), the selection of an action (a) and the calculation of the value (Q) of that action are repeated. As a result, the calculated values ​​(Q) are stored in table Tq one by one.

[0060] The selected action (a) changes the environment to a final state (s e ) (step S6: YES), one episode ends. If a certain number of episodes have been performed (step S8: YES), the reinforcement learning process ends. The certain number can be set arbitrarily.

[0061] On the other hand, if the certain number of episodes has not been performed (step S8: NO), the process returns to step S3 and a new episode is started. In other words, the processes of steps S3 to S7 are repeated until the certain number of episodes have been performed, and the value (Q) is updated based on a series of actions during the performed episodes.

[0062] In this way, the second reward (M) given according to the degree of achievement of the subtask increases the reward (R) and the calculated value (Q) also increases. While the value (Q) does not change due to the first reward (r), the value (Q) changes due to the second reward (M), and the action (a) can be evaluated. This reduces trial and error and allows the system to efficiently learn the action (a) that can accomplish the task.

[0063] When calculating the reward (R), the calculation processing unit 112 can adjust the rate (τ) of adding the second reward (M) by changing the value of the coefficient τ. The larger the rate (τ), the higher the rate of the second reward (M) added to the first reward (r). The calculation processing unit 112 can increase the rate (τ) when prioritizing the efficiency of learning by the second reward (M), and can decrease the rate (τ) when prioritizing learning by trial and error.

[0064] It is preferable that the calculation processing unit 112 reduces the ratio (τ) as the number of times the episode is performed increases. Although the granting of the second reward (M) induces the agent's action (a) to accomplish the subtask, the selection of new actions (a) decreases, and the agent's action (a) tends to be biased toward the action (a) to which the second reward (M) is granted. Therefore, while the number of episodes is small, the learning can be made more efficient by increasing the ratio (τ) of the second reward (M). After a certain number of episodes have been performed, the ratio (τ) of the second reward (M) can be reduced and the agent can act randomly, making it possible to learn actions (a) with higher value (Q), and improving the efficiency of learning.

[0065] The calculation processing unit 112 can eventually reduce the rate (τ) to 0. This allows the rate to converge to a result similar to that of normal Q-learning. The calculation processing unit 112 may monotonically reduce the rate (τ), or may temporarily increase it during the process of reducing it.

[0066] As described above, according to this embodiment, episodes are repeatedly performed to learn an agent's action (a) so that the cumulative value of a series of actions of the agent in one episode is maximized. In such reinforcement learning, the action selection unit 111 repeatedly selects an agent's action (a). Each time an action (a) is selected, the calculation processing unit 112 calculates the value (Q) of the action (a) using the reward (R) given to the selected action (a).

[0067] In normal Q-learning, the reward (R) used to calculate value (Q) is only the first reward (r) as shown in equation (10). The first reward (r) is a constant that is predefined in the environment. In an environment where a high first reward (r) is given only when the color of all blocks changes, the first reward (r), which is an indicator of action (a), cannot be obtained until that point. Since trial and error must be repeated until the color of all blocks changes, the number of episodes required to maximize value (Q) increases, and learning takes time.

[0068] In contrast, the task control unit 113 in this embodiment defines a subtask from the main task and a second reward (M) that is granted according to the achievement level of the subtask. The calculation processing unit 112 obtains a reward (R) by adding the second reward (M) to the first reward (r). The reward (R) weighted by the addition of the second reward (M) increases the value (Q) of a series of actions that accomplish the subtask. Therefore, the agent can be guided to an action (a) that finally accomplishes the main task while accomplishing the subtask.

[0069] Induction of action (a) reduces trial and error. It also accelerates the maximization of value (Q). This shortens the learning time, thereby improving the learning efficiency of reinforcement learning.

[0070] The reinforcement learning device 10 can be used to determine various measures, and the technical field is not particularly limited. For example, the reinforcement learning device 10 can be used for automatic driving control that determines a vehicle's driving route while avoiding dangerous objects, motor drive control, game character control, and the like.

[0071] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments. Various modifications are possible within the scope of the present invention, and some modifications are listed below. Each modification may be combined.

[0072] (Variation 1) The task control unit 113 can select whether to enable or disable each subtask. The task control unit 113 can switch between enabling and disabling the subtask for each episode, and can also switch between enabling and disabling the subtask within one episode depending on the environmental state (s).

[0073] For example, the task control unit 113 can select to disable the subtask with ID 02 and enable the subtasks with IDs 01 and 03 until the color of half of the game area 30 has changed in one episode. In this case, in the first half of the game, the action (a) of changing the color of areas 31 and 33 is more likely to be selected than the block of area 32. This is because the value (Q) of the action (a) of changing the color of areas 31 and 33, which can obtain the second reward (M), is higher than that of the action (a) of changing the color of areas 31 and 33, which can obtain the second reward (M). Therefore, the action (a) can be scheduled to change the color of areas 31 and 33 first, and then change the color of area 32.

[0074] When enabling or disabling a subtask is selected, the calculation processing unit 112 may calculate a reward (R) by adding up all of the second rewards (M) of each enabled subtask, and use the reward (R) to calculate the value (Q).

[0075] Alternatively, the calculation processing unit 112 may first calculate the value (Q) for each subtask, and then calculate the average value of the values ​​(Q) of each activated subtask as the value (Q) of the action (a). Specifically, the calculation processing unit 112 calculates the value (Q) when only the subtask with ID=01 is activated, the value (Q) when only the subtask with ID=02 is activated, and the value (Q) when only the subtask with ID=03 is activated. That is, the three values ​​(Q) are calculated using only the second reward (M) granted from each subtask.

[0076] FIG. 8 shows examples of tables Tq1 to Tq3 of values ​​(Q) calculated for each subtask. Table Tq1 holds a value (Q) calculated using only the second reward (M) of the subtask with ID=01. Similarly, table Tq2 holds a value (Q) calculated using only the second reward (M) of the subtask with ID=02. Table Tq3 holds a value (Q) calculated using only the second reward (M) of the subtask with ID=03.

[0077] The calculation processing unit 112 calculates the average value of the values ​​(Q) of the activated subtasks among the three values ​​(Q). For example, when the subtasks with IDs 01 and 02 are activated and the subtask with ID 03 is deactivated, the calculation processing unit 112 calculates the average value of the values ​​(Q) stored in tables Tq1 and Tq2. This average value is stored in table Tq in association with the selected action (a).

[0078] (Variation 2) The behavior selection unit 111 may select the enabling or disabling of the subtask as one of the behaviors (a) of the agent, thereby making it possible to learn a schedule of the behavior (a) for accomplishing the subtask.

[0079] The task control unit 113 can define a second reward (M) for the action (a) of selecting whether to enable or disable each subtask. The second reward (M) makes it possible to learn the schedule of the action (a) more efficiently.

[0080] (Variation 3) The task control unit 113 can define the second reward (M) based on information about the environment given in advance. For example, if the environmental state(s) to be avoided or the state(s) to be passed through are known in advance, the calculation processing unit 112 can define the second reward (M) based on that information.

[0081] An example will be described in which traps are placed in some blocks (ij) of the game area 30. When the brush 20 reaches a trap, the game ends and the task fails. When the position information of this trap is given in advance, the task control unit 113 can define a second reward (M) for the action (a) of moving the brush 20 to the block (ij) in which this trap is placed that is lower than other actions (a), such as -100 points.

[0082] When such a low-value second reward (M) is defined, the reward (R) calculated by equation (2) becomes smaller when the action (a) that leads to the state (s) is selected, and as a result, the value (Q) also becomes smaller. This allows the agent's action (a) to be induced to avoid traps.

[0083] Depending on the control content using reinforcement learning, not only is it necessary to have an action (a) that can accomplish the task efficiently, but also an action (a) with low risk. For example, even if the action (a) accomplishes the task via the shortest route, it is better to avoid an action (a) that approaches a high-risk point. In this case, the risk of the learned action can be reduced by defining the second reward (M) as described above.

[0084] (Variation 4) The second reward (M) in the above embodiment is a constant defined together with the subtask. However, the second reward (M) may be a variable that is updated according to the expected value of the reward (R) to be granted by the future action (a).

[0085] The following formula (3) shows an example of a reward function M(s,m) used to calculate the updated second reward (M). In formula (3), R e represents the expected value of the reward (R) that is given when the subtask is completed. e ) is calculated by formula (4). Each time one or more actions (a) are selected, the calculation processing unit 112 can calculate an updated second reward (M) and update tables T31 to T33. The updated second reward (M) is used to calculate the reward (R).

[0086]

number

[0087] For example, the current environment is in a state (s) where the color of all blocks in area 31 has changed, but the colors of areas 32 and 33 have not changed. In this state (s), no matter what action (a) is selected, the first reward (r) obtained is 0 points. In addition, since the subtask with ID 01 has already been achieved, the expected value (R) of the reward (R) obtained in the future by achieving this subtask is 0 points. e ) is worth 0 points.

[0088] On the other hand, by accomplishing the subtasks with IDs 02 and 03, the cumulative value of the second reward (M) defined for each subtask is expected as the reward (R). According to formulas (3) and (4), the second reward (M) given to the action (a) of accomplishing the subtasks with IDs 02 and 03 is higher than that of the subtask with ID 01. As a result, the value (Q) of the action (a) increases, and it is possible to induce the action (a) of the agent to accomplish the subtasks with IDs 02 and 03.

[0089] When calculating the updated second reward (M), the calculation processing unit 112 changes the value of the coefficient λ to calculate the expected value (R e The larger the ratio (λ), the smaller the expected value (R e ) is accelerated. Therefore, by increasing the ratio (λ), the calculation processing unit 112 can more quickly reflect the results of the actual action (a) in the subsequent action (a).

[0090] (Variation 5) The second reward (M) is calculated by using the reward function M γ (s,m). According to formula (5), the second reward (M) after the update is the expectation value (E) of the reward (R) obtained until each action (a) selected in one episode is reached, and is calculated by adding up each reward (R). Here, the reward (R) given to each action (a) is multiplied by a coefficient γ E is discounted by.

[0091]

number

[0092] Gamma E represents the discount rate of the reward (R) given to each action (a), and 0<γ E ≦1. Discount rate γ E Since is raised to the power of (et), the final state (s e The closer the state approaches the initial state (s0), the smaller the discount rate of the reward (R). This reward function M γ According to (s,m), the final state (s e ) is more likely to be guided to past actions close to the target. Therefore, it is suitable for an environment in which the first reward (r) is given to the final action (a) that achieves the main task. Also, γ E (e-t) If is smaller than 1, the action (a) with a high expected value (E) is more likely to be selected. Therefore, the cumulative reward (R e ) is obtained, taking into account the time average E can be set.

[0093] For example, state (s t ,m) to state (s t+3 ,m), the expected value (E) of each reward (R) until each action (a) is reached is calculated as follows: M γ (s,m)=γ E 3 R(s t ,m)+γ E 2 R(s t+1 ,m) +γ E R(s t+2 ,m)+R(s t+3 ,m)

[0094] (Variation 6) The task control unit 113 may set a difference in the second reward (M) between each subtask. This makes it easier to control the order in which the subtasks are completed. For example, the task control unit 113 can define the second reward (M) for each subtask so that the second reward (M) for the subtask with ID 01 is greater than the subtasks with IDs 02 and 03. In this case, the action (a) is induced so that the subtask with ID 01 is completed before the subtasks with IDs 02 and 03.

[0095] (Variation 7) If there is a part of a subtask that should be completed first, task control unit 113 can define this part so that it overlaps with a part of another subtask. For example, when the color of a block where the areas of multiple subtasks overlap, such as blocks (12) and (13) in Fig. 4, is changed, the second reward (Q) of each subtask is used to calculate the value (Q). As a result, the value (Q) of the action (a) of moving the brush 20 to a block where the areas overlap is likely to be higher than the action (a) of moving to a block where there is no overlap. Therefore, it is possible to induce the action (a) so that the color of the block where the areas overlap changes first.

[0096] In the above embodiment, the storage unit 12 stores the tables T31 to T33, Tq, and Tq1 to Tq3, but these may be stored in an external device such as a server. By communicating with the external device through the communication unit 15, the tables T31 to T33, Tq, and Tq1 to Tq3 can be downloaded or uploaded.

[0097] Although an example of Q-learning has been described, the present invention can be applied to any reinforcement learning that calculates the value of an agent's action using a reward. For example, the present invention can be applied to SARSA, Markov decision process (MDP), or DQN (Deep Q-Network), etc.

[0098] In addition, a recording medium may be provided that stores a program for causing a computer to execute the reinforcement learning method of the present invention. The recording medium is not particularly limited as long as it is a recording medium that can be read by a computer such as a CPU, and semiconductor memories, magnetic disks, optical disks, etc. may be used. [Explanation of symbols]

[0099] 10 reinforcement learning device, 11 CPU, 111 action selection unit, 112 calculation processing unit, 113 task control unit, 12 storage unit

Claims

1. A reinforcement learning device (10) that repeatedly performs an episode of selecting an agent's behavior until a task is accomplished in a given environment, and learns the agent's behavior so as to maximize the cumulative value of a series of actions in one episode, an action selection unit (111) for repeatedly selecting an action of the agent; a calculation processing unit (112) that calculates a value (Q) of the selected action using a reward (R) given to the selected action each time the action is selected; A task control unit (113) that defines one or more subtasks from the task and defines a second reward (M) to be granted according to the degree of achievement of the subtask; The calculation processing unit (112) obtains the second reward (M) defined for the selected action and a first reward (r) provided by the environment, and calculates the reward (R) by adding the second reward (M) to the first reward (r). A reinforcement learning device (10).

2. The calculation processing unit (112) adjusts the rate (τ) of adding the second reward (M). A reinforcement learning device (10) according to claim 1.

3. The calculation processing unit (112) reduces the rate (τ) of adding the second reward (M) as the number of times the episode is performed increases. A reinforcement learning device (10) according to claim 2.

4. The task control unit (113) defines the second reward (M) independently for each of the subtasks. A reinforcement learning device (10) according to any one of claims 1 to 3.

5. The task control unit (113) selects whether to enable or disable the subtask. A reinforcement learning device (10) according to any one of claims 1 to 4.

6. The action selection unit (111) selects enabling or disabling the subtask as one of the actions of the agent. A reinforcement learning device (10) according to any one of claims 1 to 4.

7. A reinforcement learning method for repeatedly selecting an action for an agent in a given environment until the agent accomplishes a task, the method comprising: Iteratively selecting an action for the agent; Each time the action is selected, a value (Q) of the selected action is calculated using a reward (R) given to the selected action; defining one or more subtasks from the task and defining a second reward (M) to be given according to the degree of achievement of the subtask; The step of calculating the value (Q) comprises: Obtaining the second reward (M) defined for the selected behavior and a first reward (r) provided by the environment; and calculating the reward (R) by adding the second reward (M) to the first reward (r). Reinforcement learning methods.

8. A program for causing a computer to execute a reinforcement learning method for repeatedly executing an episode of selecting an action of an agent until the agent accomplishes a task in a given environment, and learning the action of the agent so as to maximize a cumulative value of a series of actions in one episode, comprising: The reinforcement learning method includes: Iteratively selecting an action for the agent; Each time the action is selected, a value (Q) of the selected action is calculated using a reward (R) given to the selected action; defining one or more subtasks from the task and defining a second reward (M) to be given according to the degree of achievement of the subtask; The step of calculating the value (Q) comprises: Obtaining the second reward (M) defined for the selected behavior and a first reward (r) provided by the environment; and calculating the reward (R) by adding the second reward (M) to the first reward (r). program.

Images