Reinforcement learning device, reinforcement learning method, and program

Abstract

To achieve efficiency of reinforcement learning.SOLUTION: A reinforcement learning device (10) comprises: a behavior selecting part (111) for selecting a behavior according to measures (π); a calculation processing part (112) for calculating a behavior worth Q using a compensation (r); and a measure update part (113) for updating the measures (π) so that the expected compensation (J) is maximized using the behavior worth Q. The measure update part (113) defines the expected compensation (J) as a neural network as shown in a formula (1) and then updates a parameter (θ) of the expected compensation, calculates a grade of the expected compensation (J) in update time, as shown in a formula (2), then calculates an added worth (R) in the formula (2) using the compensation (r) so as to approximate the behavior worth Q.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a reinforcement learning device, a reinforcement learning method, and a program. [Background technology]

[0002] Conventionally, reinforcement learning has been used to achieve a given task. For example, reinforcement learning has been applied to games, motor control, automatic driving control of vehicles, and the like (see Patent Documents 1 and 2). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-63602 A [Patent Document 2] JP 2020-144483 A Summary of the Invention [Problem to be solved by the invention]

[0004] Actor-critic is known as one of the algorithms in reinforcement learning. Actor-critic is a method of learning a policy of action that maximizes the cumulative value of a series of actions of an agent in a given task environment.

[0005] In the early stages of learning, the strategy that will produce high value is not yet determined, so trial and error is repeated. It tends to take a long time for the learning process to converge to the maximum value.

[0006] The present invention aims to improve the efficiency of reinforcement learning. [Means for solving the problem]

[0007] One aspect of the present invention is a reinforcement learning device (10) that learns a policy (π) so as to maximize an expected reward (J) calculated from the policy (π) and an action value (Q) in an environment in which a task is given. The reinforcement learning device (10) includes an action selection unit (111) that selects an action of an agent placed in the environment according to the policy (π), a calculation processing unit (112) that calculates an action value (Q) of the selected action using a reward (r) given by the environment for the selected action, and a policy update unit (113) that updates the policy (π) using the calculated action value (Q) so as to maximize the expected reward (J). The policy update unit (113) defines the expected reward (J) as a neural network having a parameter (θ) as shown in the following formula (1), updates the parameter (θ), and updates the policy (π) so that the expected reward (J) is maximized for the calculated action value (Q), and updates the gradient (∇) of the expected reward (J) used to update the parameter (θ). θ As shown in the following formula (2), the measure (π) and the action value (Q) mixed with the added value (R) are calculated, and the added value (R) is calculated using the reward (r) so as to approximate the action value (Q).

number

[0008] Another aspect of the present invention is a reinforcement learning method for learning a policy (π) so as to maximize an expected reward (J) calculated from the policy (π) and an action value (Q) in an environment in which a task is given. The reinforcement learning method includes the steps of: selecting an action of an agent placed in the environment according to the policy (π); calculating an action value (Q) of the selected action using a reward (r) given by the environment for the selected action; and updating the policy (π) using the calculated action value (Q) so as to maximize the expected reward (J). The step of updating the policy (π) includes the steps of defining the expected reward (J) as a neural network having a parameter (θ) as shown in the above formula (1), updating the parameter (θ), and updating the policy (π) so as to maximize the expected reward (J) for the calculated action value (Q); and θ The method includes a step of calculating the reward (r) using the policy (π) and the action value (Q) mixed with the added value (R) as shown in the above formula (2), and a step of calculating the added value (R) using the reward (r) so as to approximate the action value (Q).

[0009] Another aspect of the present invention is a program for causing a computer to execute a reinforcement learning method for learning a policy (π) so as to maximize an expected reward (J) calculated from the policy (π) and an action value (Q) in an environment in which a task is given. The reinforcement learning method includes the steps of: selecting an action of an agent placed in the environment according to the policy (π); calculating an action value (Q) of the selected action using a reward (r) given by the environment for the selected action; and updating the policy (π) using the calculated action value (Q) so as to maximize the expected reward (J). The step of updating the policy (π) includes the steps of defining the expected reward (J) as a neural network having a parameter (θ) as shown in the above formula (1), updating the parameter (θ) so as to maximize the expected reward (J) for the calculated action value (Q), and updating the policy (π) so as to maximize the expected reward (J) for the calculated action value (Q); and θ The method includes a step of calculating the reward (r) using the policy (π) and the action value (Q) mixed with the added value (R) as shown in the above formula (2), and a step of calculating the added value (R) using the reward (r) so as to approximate the action value (Q). Effect of the Invention

[0010] According to the present invention, reinforcement learning can be made more efficient. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram showing a configuration of a reinforcement learning device according to an embodiment of the present invention. [Diagram 2] 1 is a flowchart showing a reinforcement learning process. [Diagram 3] FIG. 1 is a diagram illustrating a game as an example of a task. [Figure 4] 1 is a flowchart showing a reinforcement learning process in a multi-agent system. [Diagram 5]FIG. 2 is a schematic diagram illustrating a relationship between a main policy and a sub-policy. [Figure 6] FIG. 13 is a schematic diagram illustrating a relationship between a main measure and a sub-measure in a modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of a reinforcement learning device, a reinforcement learning method, and a program according to the present invention will be described with reference to the drawings. The following description is an example (representative example) of the present invention, and the present invention is not limited thereto.

[0013] FIG. 1 shows a configuration of a reinforcement learning device 10 according to an embodiment of the present invention. The reinforcement learning device 10 includes a CPU (Central Processing Unit) 11 and a storage unit 12. The reinforcement learning device 10 may further include an operation unit 13, a display unit 14, and a communication unit 15.

[0014] The CPU 11 executes a reinforcement learning process, which will be described later, by reading and executing a program from the storage unit 12. In the reinforcement learning process, the CPU 11 functions as an action selection unit 111, a calculation processing unit 112, and a policy update unit 113.

[0015] The action selection unit 111 selects an action of the agent according to a policy (π) in an environment in which a task is given. The calculation processing unit 112 calculates an action value (Q) for evaluating the action selected by the action selection unit 111 using a reward (r) to be given to the action. The policy update unit 113 updates the policy (π) using the calculated action value (Q).

[0016] The storage unit 12 stores programs that can be read by the CPU 11, tables used to execute the programs, etc. As the storage unit 12, for example, a recording medium such as a hard disk can be used.

[0017] The operation unit 13 is a keyboard, a mouse, etc. The operation unit 13 accepts an operation by a user and outputs the content of the operation to the CPU 11.

[0018] The display unit 14 is a display etc. The display unit 14 displays an operation screen, a processing result of the CPU 11, etc. according to a display instruction from the CPU 11.

[0019] The communication unit 15 is an interface for communicating with an external computer via a network.

[0020] The reinforcement learning device 10 can determine a policy for accomplishing a given task by reinforcement learning. In this embodiment, an example of actor-critic, which is one of the algorithms of reinforcement learning, will be described. The actor-critic is a method of selecting an action of an agent in a given environment according to a policy (π), and updating the policy (π) based on an action value (Q) that evaluates the action.

[0021] The state (s) of the environment changes depending on the action (a) of the agent. The policy (π) is the probability distribution of the action (a) selected in each state (s). e ), a series of actions (a) of an agent up to state (s) is called an episode. For an action (a) that transitions state (s), a reward (r) is given from the environment. In the actor-critic, the policy (π) is updated to maximize the expected reward (J). The expected reward (J) is the cumulative reward expected to be obtained in one episode.

[0022] The expected reward (J) is calculated by the policy (π) and the action value (Q) as shown in formula (1). In this embodiment, the expected reward (J) is defined as a neural network having a parameter θ. The parameter refers to a weight or bias set in the neural network.

number

[0023] J(θ) represents the expected reward output by a neural network with parameter θ. s represents a state, and a represents an action. π(s|a,θ) represents a strategy for selecting an action (a) in state (s), and is updated as parameter θ is optimized. Q(s,a) represents the action value (Q) that evaluates the action (a) selected in state (s).

[0024] FIG. 2 shows the flow of the reinforcement learning process in the reinforcement learning device 10. First, the policy update unit 113 initializes the environment given together with the task (step S1), thereby resetting the environment to the initial state (s0).

[0025] Next, the action selection unit 111 selects an action (a) for the agent from among the actions (a) that can be taken in the current state (s) according to the policy (π) (step S2). The policy (π) at the start of learning may be an arbitrary probability function that is initially set.

[0026] The calculation processing unit 112 calculates an action value (Q) for the selected action (a) (step S3). The action value (Q) is calculated as shown in formula (4). The calculation processing unit 112 associates the calculated action value (Q) with the action (a) and stores it in the storage unit 12. This is called a Q table.

number

[0027] s t represents the state of the environment (s) at time t. t+1 is the state (s t ) represents the state (s) one step after the transition due to the action (a) in t+1 is the state (s t ) depending on the action (a). α represents the learning rate and satisfies 0<α≦1. γ represents the discount rate and satisfies 0<γ≦1. maxQ(s t+1 ,a) is the state (st+1 ) the value Q(s t+1 ,a) represents a function that outputs the maximum value.

[0028] The environment after the action is the final state (s e If the environment does not reach the final state (s) (step S4: NO), the selection of the action (a) (step S2) and the calculation of the action value (Q) (step S3) are repeated. As a result, the action value (Q) is stored in the Q table. Then, the environment is e ), one episode ends (step S4: YES).

[0029] If the predetermined number of episodes have not been performed yet (step S5: NO), steps S1 to S4 are repeated. That is, the environment is initialized and a new episode is started. By repeating the episodes, the Q table is updated. If the predetermined number of episodes have been performed (step S5: YES), the policy update unit 113 updates the policy (π) (step S6).

[0030] When updating the policy (π), first, the policy update unit 113 optimizes the action value (Q) in formula (1). The policy update unit 113 may obtain the action value (Q) stored in the Q table as the optimized action value (Q).

[0031] The policy update unit 113 calculates the action value (Q φ ) and update the parameter φ so as to minimize the loss function L shown in equation (5), thereby optimizing the action value (Q).

number

[0032] Next, the policy update unit 113 determines a policy (π) that maximizes the expected reward (J) for the optimized action value (Q). The expected reward (J) is maximized by using the gradient (∇ θThis is done by updating the parameter (θ) using

[0033]

number

[0034] In this embodiment, the gradient of the expected reward (∇ θ J) is defined as shown in equation (2). As shown in equation (2), the gradient (∇ θ In the calculation of J, the action value (Q) mixed with the added value (R) is used, rather than using the optimized action value (Q) as it is. The added value (R) is calculated using the reward (r) to approximate the action value (Q), as shown in equation (3).

[0035]

number

[0036] t, t P , τ D and τ I Each represents a coefficient between 0 and 1. E represents the discount rate of the reward (r) given to each action (a), and 0<γ E ≦1. t e is the final state (s e In equation (3), the coefficient τ P The term containing the coefficient τ D The term containing the differential term, the coefficient τ I A term containing is called an integral term.

[0037] Coefficient τ in the proportional term P multiplied by r * represents the reward (r) given to the action (a) in the state (s) at time t in one episode. For example, when t=5, the policy update unit 113 can use the reward (r) given to the action (a) in the state (s5) in the proportional term.

[0038] In the differential term, the coefficient τ Ddr / dt, which is multiplied by, represents the differential value of the reward (r) given within a certain time from a state before time t to the state (s) at time t in one episode. By adding the differential term, it is possible to approximate the action value (Q) taking into account the time change of the reward (r). For example, when t=5, the policy update unit 113 can use the differential value of the reward (r) given from the state (s3) to the state (s5) of steps 3 to 5 as the differential term.

[0039] In the integral term, the coefficient τ I The integral of r multiplied by represents the cumulative value of the reward (r) given during one episode. In this cumulative value, the reward (r) for the action (a) in each state is discounted by γ E It is discounted by the discount rate γ E is (t e -t) and the final state (s e ), the smaller the discount rate of reward (r).

[0040] The policy update unit 113 updates each coefficient τ P , τ D and τ I By adjusting τ, the ratio of the proportional term, the differential term, and the integral term can be adjusted, and it is also possible to set the ratio to 0. For example, τ P = 0, τ D = 0, τ I By setting τ = 1, only the integral term, i.e., the cumulative reward, can be mixed into the action value (Q). If you want to emphasize the state (s) at time t, set τ P If you want to consider time changes, set τ D = 1 to add the derivative term.

[0041] By mixing such added value (R), the action value (Q) can be weighted, accelerating the maximization of expected reward (J). The learning time required to converge to the policy (π) that maximizes expected reward (J) is shortened, improving learning efficiency.

[0042] In addition, added value (R) increases the action value (Q) for a series of actions that had a high cumulative value in past episodes, and conversely, decreases the action value (Q) for a series of actions that had a low cumulative value. Therefore, a policy (π) can be determined that avoids actions (a) that have led to failure in the task in the past and selects actions (a) that lead to success in the task. This makes it possible to learn a low-risk policy (π).

[0043] The policy update unit 113 can adjust the ratio (τ) of mixing the added value (R) into the action value (Q). It is preferable that the policy update unit 113 reduces the ratio (τ) of mixing the added value (R) as the number of updates of the policy (π) increases. The policy update unit 113 can ultimately reduce the ratio (τ) to 0. By reducing the ratio (τ), it is possible to shorten the learning time while converging to a learning result similar to that when only the normal action value (Q) is used.

[0044] The policy update unit 113 may monotonically decrease the rate (τ), or may arbitrarily determine the rate of decrease with respect to the number of updates. Moreover, the policy update unit 113 may temporarily increase the rate (τ) in the process of decreasing it.

[0045] (Example of application to multi-agent systems) The efficiency of reinforcement learning using the above-mentioned added value (R) can also be applied to multi-agent systems, which are systems that execute tasks using multiple agents.

[0046] A game will be described below as an example of a task given to the reinforcement learning process. Fig. 3 shows a game area given as the game environment. A stone 31, five targets 32, and two brushes 33 are arranged in the game area.

[0047] When the game starts, the stone 31 is thrown out horizontally. The stone 31 slides along the floor surface. The stone 31 gradually slows down, and the game ends when the stone 31 stops moving. During the game, the floor surface can be polished with the brush 33. When the area around the direction of movement of the stone 31 is polished, the direction of movement of the stone 31 changes to that direction. The greater the force with which the floor surface is polished, the less the reduction in the movement speed can be suppressed.

[0048] The task of this game is to make the stone 31 hit the target 32. The player operates the brush 33 and guides the stone 31 to the target 32 ​​by adjusting the direction (angle with respect to the horizontal direction) and force with which the floor surface is brushed. For each target 32 ​​hit, 20 points are awarded as a reward for the game. For example, when the stone 31 moves along the path L1, two targets 32 are hit, and a reward of 40 points is obtained. When the stone 31 moves along the path L2, there is no target 32 ​​to hit, and therefore the reward is 0 points.

[0049] When the task of this game is executed by a multi-agent system, the reinforcement learning device 10 assigns one agent to one brush 33 from among multiple agents. The reinforcement learning device 10 repeatedly selects the operation contents of the brush 33, such as the direction and force of brushing the floor surface, the moving direction and moving speed of the brush 33, as actions (a) of the two agents and calculates the action value (Q). In this way, the reinforcement learning device 10 finds a policy (π) that maximizes the expected reward (J) by hitting as many targets 32 as possible.

[0050] FIG. 4 shows the flow of the reinforcement learning process when determining a policy (π) in a multi-agent system. First, the policy update unit 113 updates the main policy (π k ) and sub-strategies (π j ) is defined (step S21). k ) is a strategy for achieving the task, and the sub-strategies (π j) may be a policy for achieving the subtasks into which the task is divided. At the start of learning, any probability function is used to determine the main policy (π k ) or sub-strategy (π j ) may be defined as

[0051] Sub-strategy (π j ) outputs the agent's action (a) for the state (s) of the environment. This action (a) is expressed as a meta-state (m ij ) is output as the main policy (π k ) is the state (s) and the meta-state (m ij ) action (a). In other words, the main policy (π k ) is a function of not only the state (s) of the environment but also the sub-policies (π j Based on the next action (a) of each agent selected by (a), the actual agent's action can be selected.

[0052] In this embodiment, the state (s) of the environment is divided into states (s) shared among the agents and states (s) not shared among the agents. (i) The shared state (s) may be the positions of the stones 31 and the target 32 ​​in the above game example, the movement speed of the stones 31, etc. This state (s) may be classified into the main policy (π k ) and sub-strategies (π j ) are input to both the unshared state (s (i) ) is a state that is specific to each agent, such as the fatigue state of each agent. (i) ) is the sub-policy (π j ) only.

[0053] Sub-strategy (π j ) outputs an action (a), that is, a meta-state (m ij ) information content of the main policy (π k For example, in the above game, the position of the stone 31 is the state of the environment (s) and the sub-policy (π j ) and the main policy (π k) is input to the sub-policy (π j ) outputs the direction and force of polishing the floor as the agent's action (a). On the other hand, the main policy (π k ) outputs not only the direction and force with which the agent polishes the floor as the agent's action (a), but also the direction, distance, and speed of the agent's movement.

[0054] Therefore, meta-states (m ij ) can be communicated. Agents can share the general movements of each agent.

[0055] Meta state (m ij ) is a sub-policy (π j ) can include information related to the action (a) output from the sub-policy (π j ) action (a) is output, but the agent is unable to execute the action due to trouble or fatigue; otherwise, it is set to 1; otherwise, it is set to 0.

[0056] In addition to error information, meta status (m ij ) is the sub-strategy (π j ) is assigned to the agent (i), or the sub-policy (π j The identification information (i) may include flag information indicating whether the meta state (m ij ) can be used to determine which agent's action is the action of the sub-policy (π j The flag information indicating the validity of the sub-policy (π j ) can be easily enabled or disabled.

[0057] The policy update unit 113 can also define a reward to be given to each agent for their action (a) in addition to the reward given by the environment. For the sake of distinction, the reward given by the game, i.e., the environment, is called the first reward (r), and the newly defined reward is called the second reward (A i The second reward (A i ) is a sub-policy (π j It is preferable that the reward is defined so that the reward is given according to the degree of achievement of the subtask by the first reward (r) and the second reward (A i ) is stored in the storage unit 12.

[0058] The policy update unit 113 independently calculates a second reward (A i ) can be defined. j ) second reward (A i ) may have the same definition or different definitions. For example, the second reward A0 of the agent with i=0 may be 10 points given for each hit on the target 32, and the second reward A1 of the agent with i=1 may be 3 points given when the distance between the stone 31 and the target 32 ​​approaches a certain distance.

[0059] Next, the action selection unit 111 initializes the game environment and starts the game (step S22). The action selection unit 111 acquires the current state (s) of the environment and selects the sub-policies (π j ) and the main policy (π k ) for each agent. The action selection unit 111 selects the sub-policy (π j The action selection unit 111 selects an action (a) for each agent in the state (s) of the environment according to the meta state (m ij ) (step S23).

[0060] At this time, the action selection unit 111 selects each sub-policy (π j ) output action (a), along with the agent's identification information (i) and sub-policy (π jThe flag information indicating the validity of the meta state (m ij ) Also, if the agent is unable to execute the selected action, the action selection unit 111 outputs the error information as a meta state (m ij ) can be output.

[0061] Next, the action selection unit 111 selects the meta state (m ij ) and obtain the main policy (π k ) The action selection unit 111 inputs each main policy (π k ) according to the current environment state(s) and meta state(m ij ) selects an action (a) for each agent in the target group (step S24).

[0062] Each agent is a machine such as a robot or a work vehicle, and each machine has a sub-policy (π j ) and the main policy (π k When an action is selected based on the meta-state (m ij ) to other agents, and the meta-states (m ij ) is received.

[0063] In this way, the meta-state (m ij ) communication allows each agent to roughly share their next action (a). This allows the main policy (π k ) can select an agent's action (a) based not only on the state of the environment (s) but also on the next action (a) of other agents. This makes it easier for agents to cooperate with each other, enabling more efficient actions.

[0064] Meta state (m ij ) contains the agent's error information, the main policy (π kBased on the error information, the system can select an action (a), such as switching agents. Even if one of the agents is unable to act, the system can continue the task as a multi-agent system.

[0065] Meta state (m ij ) is the sub-policy (π j ) contains the identity (i) of the assigned agent, then the main policy (π k ) is a sub-policy (π j ) can identify which agent's action (a) is selected. Therefore, the main policy (π k ) enables the selection of efficient action (a). For example, it is possible to select action (a) to allow agents with different characteristics or agents located close to each other to work together.

[0066] In a typical multi-agent system, the behavior of each agent is determined by one policy for each agent. However, this behavior contains a large amount of information, and sharing it between agents requires a long time for communication. In this embodiment, first, a sub-policy (π j ) determines the general action, and the main policy (π k ) determines more detailed actions. Since general actions contain less information, the meta-state (m ij ) and can communicate with each other as the main policy (π k ) can be easily communicated to

[0067] Figure 5 shows the main policy (π k ) and sub-policy (π j ) is a schematic diagram illustrating the relationship. Agent i=0 has sub-policy π 0j and the main policy π 0k is assigned. Based on the state s of the environment, the sub-policy π 0j The action selected by is in meta-state m 0j The main policy π 0k In meta state m 0jis input, and the meta-state m 1j The meta state m 0j and m 1j Based on the state of the environment s, the main policy π 0k In the example, the agent selects an action a0 and outputs it. This action a0 causes a transition in the state s of the environment, and the state s after the transition is the main policy π 0k and sub-policy π 0j and the next action is selected.

[0068] Each agent chooses the main policy (π k When the action (a) selected by the action selection unit 111 is executed, the state (s) of the environment is transitioned by the action (a). t+1 ) from the environment. The calculation processing unit 112 also obtains a first reward (r) to be given for the action (a) from the environment. i ) is defined, the calculation processing unit 112 receives the second reward (A i ) is also acquired (step S25).

[0069] The environment after the action is the final state (s e ) (step S26: NO), the calculation processing unit 112 calculates the action value (Q) for the action (a) of each agent. The calculation processing unit 112 writes the calculated action value (Q) in the Q table and updates it (step S27).

[0070] In the multi-agent system, the calculation processor 112 calculates the main policy (π k ) and sub-policy (π j ) and calculate the action value (Q) separately. As a result, the main policy (π k ) and sub-policy (π j ) are created for each agent, and are stored in the storage unit 12. k ) for the first action value (Q), the sub-policy (π j )'s action value (Q) is sometimes called the second action value (Q).

[0071] Normally, if an attempt is made to manage the action value (Q) of each agent in a multi-agent system using a single Q table, the size of the Q table becomes large. For example, if the number of states (s) is represented by n and the number of agents by i, the size of the Q table becomes n^i. In contrast, according to this embodiment, the size of the Q table is small. j ), the size of the entire Q table is n+j^i, which is much smaller.

[0072] Main Strategy (π k ) and sub-strategies (π j ), the action value (Q) is calculated as shown in equation (7).

number

[0073] s represents the state of the environment shared by all agents. t and t+1 in s represent the states at times t and t+1, respectively. r t+1 represents the first reward given for action (a). m ij is the sub-policy (π j )

[0074] In addition, when calculating the action value (Q), the second reward (A i Specifically, the first reward (r t+1 ) as shown in equation (8), the second reward (A i ) in a given ratio (σ) to generate the first reward (r t+1 ) can be used. The second reward (A i ) tends to accelerate the maximization of action value (Q) and shorten learning time. (8) r t+1 =(1-σ)r t+1 +σA i

[0075] The policy update unit 113 updates the main policy (π k ) and sub-strategies (π j ) is updated (step S28).

[0076] In a multi-agent system, the sub-policies (π j ) and the main policy (π k ) and the action value (Q), the expected reward (J) is calculated. To maximize this expected reward (J), the sub-policy (π j ) and the main policy (π k ) are updated respectively.

number

[0077] The gradient of the expected reward when updating the parameter (θ) (∇ θ The added value (R) in equation (2a) is the same as the added value (R) defined by equation (3).

number

[0078] Main Strategy (π k ) is updated in the same way as described above. That is, the action value (Q) is optimized so that the expected reward (J) shown in equation (1a) is maximized. The Q table is k ) and each sub-policy (π j ), but the policy update unit 113 uses the main policy (π k ) is updated using the main policy (π k ) for the Q-table. Then, we use the gradient of the expected reward (∇ θ J), the main policy (π k ) is updated.

[0079] Sub-strategy (π j ), the sub-policy (πj ), except for using the Q-table for the main policy (π k ) can be updated in the same way.

[0080] In addition, the main policy (π k ) and each sub-policy (π j ) may be defined as a neural network having a parameter φ, and each action value (Q) may be optimized by updating the parameter φ so as to minimize the loss function L shown in equation (5).

[0081] Sub-strategy (π j ) the second reward (A i ) is defined, the main policy (π k ) and sub-strategies (π j ) update with the second reward (A i ) can be used. In this case, the gradient of the expected reward (∇ θ J) is defined as shown in equation (2b). As shown in equation (2b), the gradient (∇ θ The optimized action value (Q) is not used as is to calculate the added value (R * ) is used as the action value (Q). * As shown in equation (3a), the value of the action (Q) is approximated by the first reward (r) to the added value (R), and the value of the second reward (A i ) is calculated.

[0082]

number

[0083] Above π j (s|a,θ) is a sub-policy (π j ) and the sub-policy (π j ) is represented as π k (s,m ij |a,θ) is the state of the environment (s) and the meta-state (m ij) selects the main policy (π k ) and the main policy (π k ) is represented as Q(s,m ij , a) is either a first action value (Q) that evaluates an action (a) in state (s), or a second action value (Q) that evaluates an action (a) in state (s) and a meta-state (m ij ) represents the second action value (Q) that evaluates the action in

[0084] In equation (3a), δ represents the delta function δ(x). When x≠0, δ(x)=0, and when x=0, δ(x)=1. R represents the added value calculated as shown in equation (3). M ij is the meta state (m ij ) is a sub-policy (π j ) is assigned to an agent with identity (i). j ) if valid, M ij =1, M if disabled ij = 0. That is, the sub-policy (π j ) is valid (R * ) is the sum of the added value (R) calculated by formula (3) and the second reward (A i ) is added.

[0085] In this way, the added value (R) is supplemented by a second reward (A i By weighting the action value (Q) by adding π k ) or sub-strategy (π j ) is shortened, improving learning efficiency.

[0086] After updating, the processes in steps S23 to S25 are repeated. k ) and sub-strategies (π j ) is updated. The final state (s e ) (step S26: YES), the reinforcement learning process ends.

[0087] As described above, according to this embodiment, an action value (Q) is calculated for an action selected according to a policy (π). The policy (π) is updated by updating a parameter (θ) so that an expected reward (J) is maximized for the calculated action value (Q). The gradient (∇ θ J(θ)) is calculated from the policy (π) and the action value (Q), which is mixed with added value (R).

[0088] The value added (R) is calculated using the reward (r) to approximate the action value (Q). Such value added (R) weights the action value (Q), accelerating the maximization of the expected reward (J). This shortens the learning time to converge to a policy (π) that maximizes the expected reward (J), improving the efficiency of reinforcement learning.

[0089] In the multi-agent system of this embodiment, the main policy (π k ) and sub-policy (π j ) is defined. j ) selects the general action of each agent, and this general action is expressed as the meta-state (m ij ) as the main policy (π k ) is input to the sub-policy (π j ) contains little information. Therefore, the main policy (π k ) in the meta state (m ij ) can be shared. The main policy (π k ) gives the state (s) of the environment and the meta-state (m ij ) is output, which makes it possible to select an efficient action that takes into account the actions of other agents.

[0090] In a multi-agent system, the main policy (π k ) and sub-strategies (π j) can improve the efficiency of reinforcement learning.

[0091] Second Reward (A i ) is defined, the second reward (A i ) is added to the added value (R * ) is mixed into the action value (Q). Therefore, the sub-policy (π j ) or main strategy (π k ) is likely to take less time to converge.

[0092] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications are possible within the scope of the present invention.

[0093] For example, in the above embodiment, the sub-policy (π j ) and the main policy (π k ), but each agent has one common main policy (π k ) can also be defined. In this case, one main policy (π k ) is the state of the environment (s) and the meta-state (m ij The action selection unit 111 outputs the action of each agent for the meta state (m ij ) and use this one main policy (π k ) in the action selection unit 111. k ) can be used to select the actions of each agent.

[0094] Figure 6 shows the main policy (π k ) for each sub-policy (π j ) and explain its relationship with Agent i=0 has sub-policy π 0j and agent i=1 is assigned sub-policy π 1j Each agent is assigned one main policy π k is assigned. 0j and π 1jThen, the action selected for the state s of the environment is a meta-state m 0j and m 1j These are the main policies π k , and the action a0 of the agent i=0 and the action a1 of the agent i=1 are the main policy π k is output from

[0095] Main Strategy (π k ), the calculation processing unit 112 can calculate, for example, the sum or average value of the action value (Q) for the action of each agent as the action value (Q) for the main policy.

[0096] In addition, the added value (R) is defined by equation (3), but is not limited to this as long as the action value (Q) can be approximated using the reward (r).

[0097] Since the action value (Q) is calculated by the reward (r), dQ / dt or Q(s) is used instead of dr / dt in equation (3). t ,a)-Q(s,a) may be used. dQ / dt represents the differential value of the action value (Q) within a certain time from a state before time t to the state (s) at time t. Also, Q(s t ,a)-Q(s,a) represents the change in action value (Q) from a state before time t to the state (s) at time t.

[0098] In the above embodiment, the sub-policy (π j ) is invalid, the agent’s sub-policy (π j ) and the main policy (π k ) when the second reward (A i ) is not added to the value added (R). However, in other embodiments, j ) is invalid, the main policy (π k When updating A i ) may be added to the added value (R).

[0099] Meta state (m ij) is not limited to the above-mentioned information, and may be an embodiment including information that may be shared, such as the agent's position information.

[0100] In the above embodiment, when there is a single agent, the policy (π) is updated for each episode, and when there are multiple agents, the sub-policy (π j ) and the main policy (π k In either case, the timing of the update is not limited to these, and the strategy may be updated for one or more actions in an episode, or for one or more episodes.

[0101] In the above embodiment, the storage unit 12 stores the Q table, but the Q table may be stored in an external device such as a server. The Q table can be downloaded or uploaded by communicating with the external device via the communication unit 15.

[0102] The reinforcement learning device 10 can be used to determine various measures, and the technical field is not particularly limited. For example, the reinforcement learning device 10 can be used for automatic driving control that determines a vehicle's driving route while avoiding dangerous objects, motor drive control, game character control, and the like.

[0103] A recording medium may be provided that stores a program for causing a computer to execute the reinforcement learning method of the present invention. The recording medium is not particularly limited as long as it is a recording medium that can be read by a computer such as a CPU, and semiconductor memories, magnetic disks, optical disks, etc. may be used. [Explanation of symbols]

[0104] 10 reinforcement learning device, 11 CPU, 111 action selection unit, 112 calculation processing unit, 113 policy update unit, 12 storage unit

Claims

1. A reinforcement learning device (10) that learns a policy (π) so as to maximize an expected reward (J) calculated from the policy (π) and an action value (Q) in an environment in which a task is given, an action selection unit (111) that selects an action of an agent placed in the environment according to the policy (π); a calculation processing unit (112) for calculating an action value (Q) of the selected action using a reward (r) given by the environment for the selected action; a policy update unit (113) that updates the policy (π) using the calculated action value (Q) so as to maximize the expected reward (J); The policy update unit (113) The expected reward (J) is defined as a neural network having a parameter (θ) as shown in the following formula (1), and the parameter (θ) is updated to update the policy (π) so that the expected reward (J) is maximized for the calculated action value (Q); The gradient (∇) of the expected reward (J) used to update the parameter (θ) θ J(θ)) is calculated using the policy (π) and the action value (Q) mixed with the added value (R) as shown in the following formula (2): Calculate the added value (R) using the reward (r) to approximate the action value (Q). A reinforcement learning device (10). [0010] [π(s|a, θ) represents the strategy (π) for selecting an action (a) in a state (s). Q(s, a) represents the action value (Q) for evaluating the action (s) selected in a state (s). ∇ θ J(θ) represents the gradient of expected reward (J). R represents added value. τ represents a coefficient that satisfies 0≦τ≦1.

2. The policy update unit (113) calculates the added value (R) that approximates the action value (Q) using the reward (r) as shown in the following formula (3): A reinforcement learning device (10) according to claim 1. [0025] [τ Ｐ , τ Ｄ and τ Ｉ Each of r represents a coefficient between 0 and 1. ＊ represents the reward (r) given to the action (a) in the state (s) at time t. dr / dt represents the differential value of the reward (r) given from the state before time t to the state at time t. γ Ｅ represents the discount rate, and 0 < γ Ｅ ≦1. ｅ represents the time at the final state.

3. The policy update unit (113) adjusts the mixing ratio (τ) of the added value (R). A reinforcement learning device (10) according to claim 2.

4. The policy update unit (113) reduces the ratio (τ) of mixing the added value (R) as the number of updates of the policy (π) increases. A reinforcement learning device (10) according to claim 3.

5. When the task is executed by a plurality of agents, the policy update unit (113) updates a sub-policy (π ｊ ) and the main policy (π ｋ ) and The sub-strategy (π ｊ ) expresses the agent's behavior with respect to the state (s) of the environment in terms of the meta-state (m ｉｊ ) and The main strategy (π ｋ ) is the state of the environment (s) and the meta-state (m ｉｊ ) and outputting the action of the agent with respect to the The sub-strategy (π ｊ The agent's action output by the main policy (π ｋ ) has less information than the agent's behavior output by The action selection unit (111) selects the sub-policy (π ｊ The action selected according to the meta-state (m ｉｊ ) to the other agent, and the meta-state (m ｉｊ ) and the meta state (m ｉｊ ) to the main strategy (π ｋ ) and select the action of the agent according to A reinforcement learning device (10) according to any one of claims 1 to 4.

6. The calculation processing unit (112) calculates the main strategy (π ｋ A first action value (Q) for evaluating the action of each agent selected according to the sub-policy (π ｊ a second action value (Q) for evaluating the action of each agent selected according to The policy update unit (113) updates the main policy (π ｋ ) and update the sub-policy (π ｊ ) to update A reinforcement learning device (10) according to claim 5.

7. The policy update unit (113) A second reward (A) given in addition to the reward (r) given by the environment for the agent's behavior. ｉ ) and The main strategy (π ｋ ) or the sub-strategy (π ｊ ) when updating the expected reward gradient (∇ θ J(θ)) is calculated by the sub-strategy (π ｊ ) and the main strategy (π ｋ ) and added value (R ＊ ) is calculated by the first action value (Q) or the second action value (Q), As shown in the following formula (3a), the reward (r) and the second reward (A ｉ ) to approximate the second action value (Q), ＊ ) to calculate A reinforcement learning device (10) according to claim 6. [0030] [π j (s|a) is a sub-policy (π ｊ ) is represented as π ｋ (s, m ｉｊ |a) represents the state (s) of the environment and the meta-state (m ｉｊ ) is the main policy (π ｋ ) represents Q(s,m ｉｊ , a) is a first action value (Q) that evaluates an action (a) in a state (s), or a first action value (Q) that evaluates an action (a) in a state (s) and a meta-state (m ｉｊ ) represents the second action value (Q) that evaluates the action in the state of the sigma-based ... ｉｊ is each sub-policy (π ｊ ) represents one of the meta-states that indicates the effectiveness of the sub-policy (π ｊ ) is valid, M ｉｊ = 1, M if invalid ｉｊ = 0. A ｉ represents the second reward.

8. The meta state (m ｉｊ ) is the sub-strategy (π ｊ ) includes the identity (i) of the agent to whom the A reinforcement learning device (10) according to any one of claims 5 to 7.

9. The meta state (m ｉｊ ) contains the error information of the agent A reinforcement learning device (10) according to any one of claims 5 to 8.

10. A reinforcement learning method for learning a policy (π) so as to maximize an expected reward (J) calculated from the policy (π) and an action value (Q) in an environment in which a task is given, comprising: selecting an action for an agent placed in the environment according to the policy (π); Calculating an action value (Q) of the selected action using a reward (r) given by the environment for the selected action; Using the calculated action value (Q), update the policy (π) so as to maximize the expected reward (J); The step of updating the policy (π) comprises: A step of defining the expected reward (J) as a neural network having a parameter (θ) as shown in the following formula (1), and updating the policy (π) by updating the parameter (θ) so that the expected reward (J) is maximized for the calculated action value (Q); The gradient (∇) of the expected reward (J) used to update the parameter (θ) θ A step of calculating the value of the action (Q) mixed with the policy (π) and the added value (R) as shown in the following formula (2); Calculating the added value (R) using the reward (r) to approximate the action value (Q). Reinforcement learning methods. [0045] [π(s|a, θ) represents the strategy (π) for selecting an action (a) in a state (s). Q(s, a) represents the action value (Q) for evaluating the action (s) selected in a state (s). ∇ θ J(θ) represents the gradient of expected reward (J). R represents added value. τ represents a coefficient that satisfies 0≦τ≦1.

11. A program for causing a computer to execute a reinforcement learning method for learning a policy (π) so as to maximize an expected reward (J) calculated from the policy (π) and an action value (Q) in an environment in which a task is given, the program comprising: The reinforcement learning method includes: selecting an action for an agent placed in the environment according to the policy (π); Calculating an action value (Q) of the selected action using a reward (r) given by the environment for the selected action; Using the calculated action value (Q), update the policy (π) so as to maximize the expected reward (J); The step of updating the policy (π) comprises: A step of defining the expected reward (J) as a neural network having a parameter (θ) as shown in the following formula (1), and updating the policy (π) by updating the parameter (θ) so that the expected reward (J) is maximized for the calculated action value (Q); The gradient (∇) of the expected reward (J) used to update the parameter (θ) θ A step of calculating the value of the action (Q) mixed with the policy (π) and the added value (R) as shown in the following formula (2); Calculating the added value (R) using the reward (r) to approximate the action value (Q). program. [0050] [π(s|a, θ) represents the strategy (π) for selecting an action (a) in a state (s). Q(s, a) represents the action value (Q) for evaluating the action (s) selected in a state (s). ∇ θ J(θ) represents the gradient of expected reward (J). R represents added value. τ represents a coefficient that satisfies 0≦τ≦1.

Images