Reinforcement learning device, reinforcement learning method, and program

Abstract

This reinforcement learning device learns a policy for determining the next action of an agent on the basis of the state of the environment and a reward acquired from the action of the agent, and executes: a value function update process for updating a state value function by using a risk parameter pertaining to a risk of the reward; a correction coefficient calculation process for calculating a correction coefficient by using the risk parameter and a first advantage function obtained by using the state value function; and a policy learning process for calculating a second advantage function by multiplying the first advantage function by the correction coefficient, and updating the policy by using the calculated second advantage function.

Description

Reinforcement learning device, reinforcement learning method, and program

[0001] The present disclosure relates to a reinforcement learning device, a reinforcement learning method, and a program.

[0002] Reinforcement learning is a well-known technique. Reinforcement learning is a technique for learning a policy, which is a guideline for determining an agent's next action, based on the state observed from the environment and the reward that can be obtained from the agent's action results. In reinforcement learning, the policy is updated to increase the expected value of the reward in response to changes in the environment caused by the agent's actions.

[0003] Furthermore, a technique called risk-sensitive reinforcement learning (hereinafter referred to as "risk-sensitive reinforcement learning") has been proposed, which considers the variance of rewards as a risk in reinforcement learning. In risk-sensitive reinforcement learning, the expected value of rewards is Q and the variance of rewards is σ ２ , where the risk preference / aversion parameter is β, Q + βσ ２ With this configuration, risk-sensitive reinforcement learning can reduce and avoid risks (or increase and prefer risks when parameter β>0) while increasing the expected value of rewards.

[0004] As an example of risk-sensitive reinforcement learning, for example, Non-Patent Document 1 describes a method for predicting the expected value Q of a reward and the variance σ of the reward. ２ Furthermore, as another example of risk-sensitive reinforcement learning, there is a technique described in Non-Patent Document 2. Non-Patent Document 2 proposes that by assuming that the distribution of rewards is a normal distribution and using a new estimation value update method, Q+βσ can be estimated with a single estimator. ２ Non-Patent Document 2 discloses, through numerical experiments, that even when the reward distribution is not a normal distribution, deviations will occur, but learning that estimates the magnitude of risk correctly to a certain extent is possible.

[0005] Another well-known example of a reinforcement learning technique is the actor-critic technique. Actor-critic combines value function-based and policy gradient-based approaches, learning both the value function and the policy. In actor-critic, the actor selects and executes an action based on the policy. The critic then observes the state and reward obtained by that action from the environment. Some actor-critic techniques evaluate an advantage function A(s, a) = Q(s, a) - V(s), which indicates how superior the action is, and update the policy based on the advantage function A(s, a). Here, Q(s, a) is the action value function, and V(s) is the state value function. s is the state, and a is the action.

[0006] Jain, Arushi, et al., "Variance penalized on policy and off policy actor critic," 2021Deletang, Gregoire, et al., "Model-Free Risk Sensitive Reinforcement Learning," 2021

[0007] However, even if the above-mentioned Actor-Critic technique is applied to the technique described in Non-Patent Document 2, it is difficult to appropriately evaluate measures in cases where risk is preferred or averse.

[0008] The present disclosure has been made in consideration of the above-mentioned problems, and an exemplary purpose thereof is to provide a technology that enables learning in reinforcement learning with more appropriate evaluation of policies.

[0009] A reinforcement learning device according to an exemplary aspect of the present disclosure is a reinforcement learning device that learns a policy for determining an agent's next action based on the state of the environment and a reward obtained from the agent's action, and includes: a value function update means that updates a state value function using a risk parameter related to the risk of the reward; a correction coefficient calculation means that calculates a correction coefficient using the risk parameter and a first advantage function obtained using the state value function; and a policy learning means that calculates a second advantage function by multiplying the correction coefficient by the first advantage function and updates the policy using the calculated second advantage function.

[0010] A reinforcement learning method according to one exemplary aspect of the present disclosure is a reinforcement learning method in which at least one processor learns a strategy for determining an agent's next action based on the state of the environment and a reward obtained from the agent's action, the method including the processor updating a state value function using a risk parameter related to the risk of the reward, calculating a correction coefficient using the risk parameter and a first advantage function obtained using the state value function, calculating a second advantage function by multiplying the correction coefficient by the first advantage function, and updating the strategy using the calculated second advantage function.

[0011] A program according to an exemplary aspect of the present disclosure is a program that causes a computer to perform reinforcement learning to learn a policy for determining an agent's next action based on the state of the environment and the reward obtained from the agent's action, and causes the computer to perform the following: a value function update process that updates a state value function using a risk parameter related to the risk of the reward; a correction coefficient calculation process that calculates a correction coefficient using the risk parameter and a first advantage function obtained using the state value function; and a policy learning process that calculates a second advantage function by multiplying the correction coefficient by the first advantage function and updates the policy using the calculated second advantage function.

[0012] According to an exemplary aspect of the present disclosure, an exemplary effect is achieved in that a technology can be provided that enables learning in reinforcement learning with more appropriate evaluation of policies.

[0013] FIG. 1 is a block diagram showing a configuration of a reinforcement learning device according to the present disclosure. FIG. 2 is a flow diagram showing the flow of a reinforcement learning method according to the present disclosure. FIG. 3 is a block diagram showing a configuration of an information processing device according to the present disclosure. FIG. 4 is a block diagram showing an example of the functional configuration and processing flow of an information processing device according to the present disclosure. FIG. 5 is a flow diagram showing the flow of a reinforcement learning method according to the present disclosure. FIG. 6 is a diagram showing the results of a simulation performed by an information processing device according to the present disclosure. FIG. 7 is a block diagram showing the configuration of a computer functioning as a reinforcement learning device and an information processing device according to the present disclosure.

[0014] The following are examples of embodiments of the present invention. However, the present invention is not limited to the exemplary embodiments shown below, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, embodiments obtained by appropriately omitting some of the technical means employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, the effects mentioned in the exemplary embodiments shown below are examples of effects expected in the exemplary embodiments, and do not define the scope of the present invention. In other words, embodiments that do not exhibit the effects mentioned in the exemplary embodiments shown below may also be included in the scope of the present invention.

[0015] [First Exemplary Embodiment] A first exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. This exemplary embodiment is a basic form for each of the exemplary embodiments described below. Note that the scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise. Furthermore, each technical means shown in the drawings referenced to explain this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise.

[0016] <Configuration of Reinforcement Learning Device> The configuration of the reinforcement learning device 1 will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the reinforcement learning device 1. The reinforcement learning device 1 learns a policy for determining the next action of an agent based on the state of the environment and a reward obtained from the agent's action. As shown in Fig. 1, the reinforcement learning device 1 includes a value function update unit 11, a correction coefficient calculation unit 12, and a policy learning unit 13.

[0017] The value function update unit 11 updates the state value function using a risk parameter related to the risk of reward. The correction coefficient calculation unit 12 calculates a correction coefficient using the risk parameter and a first advantage function obtained using the state value function. The policy learning unit 13 calculates a second advantage function by multiplying the first advantage function by the correction coefficient, and updates the policy using the calculated second advantage function.

[0018] <Effects of the Reinforcement Learning Device> As described above, the reinforcement learning device 1 is a reinforcement learning device that learns a policy for determining the next action of an agent based on the state of the environment and the reward obtained from the agent's action, and includes a value function update unit 11 that updates a state value function using a risk parameter related to the risk of the reward, a correction coefficient calculation unit 12 that calculates a correction coefficient using the risk parameter and a first advantage function obtained using the state value function, and a policy learning unit 13 that calculates a second advantage function by multiplying the first advantage function by the correction coefficient and updates the policy using the calculated second advantage function. Therefore, the reinforcement learning device 1 has the effect of enabling learning that more appropriately evaluates policies in reinforcement learning.

[0019] <Flow of Reinforcement Learning Method> The flow of reinforcement learning method S1 will be described with reference to Fig. 2. Fig. 2 is a flow diagram showing the flow of reinforcement learning method S1. Reinforcement learning method S1 is a reinforcement learning method in which at least one processor learns a policy for determining the next action of an agent based on the state of the environment and a reward obtained from the agent's action. As shown in Fig. 2, reinforcement learning method S1 includes a value function update process S11, a correction coefficient calculation process S12, and a policy learning process S13.

[0020] In a value function update process S11, at least one processor updates a state value function using a risk parameter related to the risk of reward. In a correction coefficient calculation process S12, at least one processor calculates a correction coefficient using the risk parameter and a first advantage function obtained using the state value function. In a policy learning process S13, at least one processor calculates a second advantage function by multiplying the first advantage function by the correction coefficient, and updates the policy using the calculated second advantage function.

[0021] <Effects of Reinforcement Learning Method> As described above, reinforcement learning method S1 is a reinforcement learning method for learning a policy for determining the next action of an agent based on the state of the environment and the reward obtained from the agent's action, and includes the following configuration: the processor updates a state value function using a risk parameter related to the risk of the reward, calculates a correction coefficient using the risk parameter and a first advantage function obtained using the state value function, calculates a second advantage function by multiplying the correction coefficient by the first advantage function, and updates the policy using the calculated second advantage function. Therefore, reinforcement learning method S1 has the effect of enabling learning that more appropriately evaluates policies in reinforcement learning.

[0022] Second Exemplary Embodiment A second exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same functions as those described in the above exemplary embodiment will be denoted by the same reference numerals, and their description will be omitted as appropriate. The scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs. Furthermore, each technical means shown in each drawing referenced to describe this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs.

[0023] <Overview of Information Processing Device> An information processing device 1A according to the present disclosure is a device that performs reinforcement learning to sequentially update a policy π for determining a next action a of an agent based on an environmental state s and a reward x obtained from the agent's action a. The information processing device 1A is an example of a reinforcement learning device according to the present disclosure. The information processing device 1A performs risk-sensitive reinforcement learning that takes into account the variance of the reward x as a risk. That is, the information processing device 1A calculates the expected value of the reward as Q and the variance of the reward as σ. ２, and the risk parameter related to risk is β, Q + βσ ２ The information processing device 1A can be used for, for example, controlling a mobile robot, scheduling production and inventory management, simulating trades and games, and the like.

[0024] <Configuration of Information Processing Device> The configuration of the information processing device 1A will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the configuration of the information processing device 1A. The information processing device 1A includes a control unit 10A, a storage unit 20A, a communication unit 30A, and an input / output unit 40A.

[0025] (Communication Unit) The communication unit 30A communicates with devices external to the information processing device 1A via a communication line. While the specific configuration of the communication line does not limit the present exemplary embodiment, examples of the communication line include a wireless local area network (LAN), a wired LAN, a wide area network (WAN), a public line network, a mobile data communication network, or a combination thereof. The communication unit 30A transmits data supplied from the control unit 10A to other devices, and supplies data received from other devices to the control unit 10A.

[0026] (Input / Output Unit) Input / output devices such as a keyboard, a mouse, a display, a printer, and a touch panel are connected to the input / output unit 40A. The input / output unit 40A receives various types of information input to the information processing device 1A from the connected input devices. Furthermore, under the control of the control unit 10A, the input / output unit 40A outputs various types of information to connected output devices. Examples of the input / output unit 40A include an interface such as a USB (Universal Serial Bus).

[0027] (Storage Unit) The storage unit 20A includes an episode storage unit 121 and a risk parameter storage unit 122. The episode storage unit 121 stores various data related to episodes of reinforcement learning by the information processing device 1A. As an example, the episode storage unit 121 stores the state s, reward x, and action a of each step. The episode storage unit 121 also stores the risk-corrected reward v and discounted reward sum r estimated by the reward estimation unit 111, which will be described later.

[0028] The state s is a state observed from the environment. For example, in a task of autonomous play of a computer game, the state s includes the state of an object that affects the progress of the game. As another example, in a task of selecting a transport operation of a transport device, the state s includes the position, movement direction, speed, angular velocity, and path position of the transport device, or the position and speed of a static obstacle or a dynamic obstacle. As another example, in a task of selecting an excavation operation of an excavator, the state s includes the attitude and position of the excavator excavating the soil, the shape of the soil to be excavated, or the amount of soil in the excavator bucket. However, the state s is not limited to the above examples and may be other states. The state s may also include environmental conditions such as temperature or weather.

[0029] Action a is an action of an agent in reinforcement learning, and examples thereof include autonomous play control of a computer game, excavation operation control of an excavator, or transport operation control of a transport device. However, the actions are not limited to these examples and may be other actions. Reward x is a reward obtained by the agent taking an action. Examples of reward x include a value added or subtracted depending on the outcome of a game, the score of the game, the amount of excavation by an excavator, the time required for excavation, the time required for transport, or whether or not an obstacle is encountered during transport. However, reward x is not limited to these examples and may be other actions.

[0030] The risk-corrected reward v and the discounted reward sum r are information obtained by the estimation process of the reward estimation unit 111, which will be described later. The risk-corrected reward v is a reward that takes risk into account, and is calculated from the state s using the state value function V(s). The discounted reward sum r is a cumulative reward obtained by discounting future rewards by a discount rate γ (0≦γ<1). The estimation process of the risk-corrected reward v and the discounted reward sum r by the reward estimation unit 111 will be described later.

[0031] The risk parameter storage unit 122 stores a risk parameter β relating to the variance (risk) of rewards. If one wishes to avoid risk by suppressing the variance of rewards obtained, a negative value is set for the risk parameter β. On the other hand, if one wishes to prefer actions that involve greater risk, a positive value is set for the risk parameter β. In other words, the smaller the value of the risk parameter β, the smaller the risk can be, i.e., the more the variance of rewards can be suppressed.

[0032] (Control Unit) Control unit 10A controls each unit of information processing device 1A in an integrated manner. Control unit 10A includes reward estimation unit 111, correction coefficient calculation unit 112, and policy learning unit 113. Reward estimation unit 111 is an example of reward estimation means according to the present disclosure. Correction coefficient calculation unit 112 is an example of correction coefficient calculation means according to the present disclosure. Policy learning unit 113 is an example of policy learning means, value function update means, output means, and display means according to the present disclosure.

[0033] 4 is a block diagram showing an example of the functional configuration and processing flow of the information processing device 1A. Note that the unidirectional arrows in FIG. 4 simply indicate the direction of the flow of a certain signal (data), and do not exclude bidirectionality.

[0034] (Reward Estimation Unit / State Value Function) The reward estimation unit 111 acquires the reward x and the state s from the simulator 2. The reward estimation unit 111 also estimates the risk-corrected reward v from the state s using the state value function V(s). The state value function V(s) is the expected value of the cumulative reward when the initial state is s under the policy π. As an example, the state value function V(s) is updated using the following equation (1): Here, α is the learning rate, and δ is the TD error. The TD error δ is calculated, for example, by the following equation (2). where R(s) is the reward emitted in state s and γ is the discount factor.

[0035] In addition, in the formula (1), the correction coefficient ξ β (δ) is a correction coefficient used to update the state value function V(s), and is calculated using, for example, a sigmoid function. β (δ) is, for example, is.

[0036] In addition, the remuneration estimation unit 111 calculates the discounted remuneration sum r. For example, after one episode ends, the remuneration estimation unit 111 calculates r ｔ +γr ｔ＋１ +γ ２ r ｔ＋２ +... to calculate the sum of discounted rewards r.

[0037] (Correction Coefficient Calculation Unit, Correction Coefficient) The correction coefficient calculation unit 112 calculates a correction coefficient ξ using the risk-corrected reward ν and the discounted reward sum r estimated by the reward estimation unit 111, and the state s. β Calculate the correction coefficient ξ (A'). β (A') is a correction coefficient for correcting the advantage function A(s, a) described later in consideration of risk. β (A') is the correction coefficient ξ used to update the state value function V(s) described above. β For example, the correction coefficient calculation unit 112 calculates the correction coefficient ξ from the pre-correction advantage function A′ (described later) using a sigmoid function including the risk parameter β as a coefficient. β Calculate the correction coefficient ξ β As an example, (A') is expressed by the following formula.

[0038] The pre-correction advantage function A' is a function that indicates how superior the action taken is, calculated using the risk-corrected reward v. The pre-correction advantage function A' is an example of a first advantage function according to the present disclosure.

[0039] (Specific Example 1 of Pre-correction Advantage Function) As an example, the pre-correction advantage function A' is the difference (r-V) between the sum of discounted rewards r up to the last step in the episode and the estimated value V of the risk-corrected reward.

[0040] (Specific Example 2 of Pre-correction Advantage) In addition, as an example, the pre-correction advantage function A′ is calculated by dividing the sum of discounted rewards in k steps by r ｋ , discount rate is γ, and the state after the transition of step t is s ｔ , state s ｔ The state value function of V(s ｔ ), it is expressed by the following formula: ｋ +γ ｋ V (S ｔ＋ｋ ) -V(s ｔ )

[0041] (Specific Example 3 of Pre-Correction Advantage) Furthermore, as one example, the pre-correction advantage function A' may be a double-discounted sum of TD errors. The double-discounted sum is a double-discounted sum used in the GAE (Generalized Advantage Estimator) method.

[0042] (Policy Learning Unit) The policy learning unit 113 updates the state value function V(s) using the risk parameter β. As an example, the policy learning unit 113 updates the state value function V(s) using the above-mentioned equation (1).

[0043] Furthermore, the policy learning unit 113 uses the correction coefficient ξ calculated by the correction coefficient calculation unit 112 β The advantage function A(s, a) is updated using (A') and the risk-corrected reward v. The advantage function A(s, a) is an example of a second advantage function according to the present disclosure. The policy learning unit 113 also updates the policy π(s, a) using the updated advantage function A(s, a).

[0044] (Advantage Function) The advantage function A(s, a) is a function that indicates how superior the action taken is. The advantage function A(s, a) is the difference between the action value function Q(s, a) and the state value function V(s). The advantage function A(s, a) can also be estimated as the actual reward sum minus the estimated reward sum, and the desired policy (a more preferable policy / preferred action) has a larger reward sum than the estimate, i.e., a positive advantage. On the other hand, when the best policy / accurate reward estimate has already been obtained, the advantage function A(s, a) will be zero or negative on average. In this case, a zero or negative advantage function A(s, a) means that there is no better action (i.e., it is the best action), so the ultimately preferable state is one in which the advantage function A(s, a) is zero or negative.

[0045] For example, the policy learning unit 113 calculates the advantage function A(s, a) using the pre-correction advantage function A′ according to the following equation: A(s, a)=ξ β (A'(s, a))·(A'(s, a)) In other words, in this example, the advantage function A(s, a) is calculated by converting the uncorrected advantage function A' using a sigmoid function into a correction coefficient ξ, which is a value in the range of 0 to 1. β It is calculated by multiplying (A') by the uncorrected advantage function A'.

[0046] (Specific Example 1 of Advantage Function) The advantage function A(s, a) is expressed by the following formula, for example: A(s, a)=ξ β (r-V)·(r-V), where r is the sum of discounted rewards up to the last step in the episode. V is the estimated value of the risk-corrected reward. That is, in this example, the advantage function A(s, a) is calculated by adding a correction coefficient ξ calculated from the uncorrected advantage function A'=r-V. β It is obtained by multiplying (r−V) by the uncorrected advantage function A′=r−V.

[0047] (Specific Example 2 of Advantage Function) When the advantage function A(s, a) of the above specific example 1 is generalized (not to the end, but up to k steps), the following equation is obtained: A(s, a)=ξ β (r ｋ +γ ｋ V (S ｔ＋ｋ ) -V(s ｔ ))・(r ｋ +γ ｋ V (S ｔ＋ｋ ) -V(s ｔ )) In other words, in this example, the advantage function A(s, a) is the uncorrected advantage function A'=r ｋ +γ ｋ V (S ｔ＋ｋ ) -V(s ｔ ) correction coefficient ξ β (r ｋ +γ ｋ V (S ｔ＋ｋ ) -V(s ｔ )) into the uncorrected advantage function A' = r ｋ +γ ｋ V (S ｔ＋ｋ ) -V(s ｔ ) is obtained by multiplying

[0048] (Specific Example 3 of Advantage Function) The advantage function A(s, a) may use the following formula: A(s, a)=ξ β ([double discounted sum of TD error δ]) * double discounted sum of TD error δ In this case, the advantage function A(s, a) is calculated from the double discounted sum of TD error δ (before correction advantage function A') by the correction coefficient ξ β is obtained by multiplying the doubly discounted sum of the TD error δ (the uncorrected advantage function A′).

[0049] The policy learning unit 113 updates the policy π using the calculated advantage function A(s, a). As an example, the policy learning unit 113 updates the policy π using a commonly used policy gradient method.

[0050] <Flow of Reinforcement Learning Method> The flow of the reinforcement learning method executed by the information processing device 1A will be described with reference to Fig. 5. Fig. 5 is a flow diagram showing the flow of the reinforcement learning method S2 executed by the information processing device 1A. Note that some steps may be executed in parallel or in a different order.

[0051] (Step S101) In step S101, the correction coefficient calculation unit 112 acquires a risk parameter β. The correction coefficient calculation unit 112 may acquire the risk parameter β input via the input / output unit 40A. Furthermore, the correction coefficient calculation unit 112 may acquire the risk parameter β from a storage destination (which may be within a storage device of the information processing device 1A or may be a storage device external to the information processing device 1A) designated by the user of the information processing device 1A. Furthermore, the correction coefficient calculation unit 112 may acquire the risk parameter β by receiving the risk parameter β from another device via the communication unit 30A.

[0052] (Steps S102 and S103) In step S102, the correction coefficient calculation unit 112 calculates the correction coefficient ξ β In step S103, the policy learning unit 113 initializes the value function and the policy π(s, a).

[0053] (Steps S104 to S108) In step S104, the reward estimation unit 111 sets the initial state to state s. In step S105, the reward estimation unit 111 selects an action a based on the policy π. In step S106, the reward estimation unit 111 acquires a reward x and a next state s' from the simulator 2. In step S107, the reward estimation unit 111 records the pre-transition state s, action a, reward x, and post-transition state s' in the episode storage unit 121. In step S108, the policy learning unit 113 updates the current state s with s'.

[0054] (Step S109) In step S109, the policy learning unit 113 determines whether the state s' is a final state. If the state s' is a final state (YES in step S109), the policy learning unit 113 proceeds to the processing of step S110. On the other hand, if the state s' is not a final state (NO in step S109), the policy learning unit 113 returns to the processing of step S105.

[0055] (Steps S110 and S111) In step S110, the reward estimation unit 111 calculates the risk-corrected reward v from the state s using the state value function V(s). In step S111, the policy learning unit 113 calculates a value estimation error function from the data stored in the episode storage unit 121. As an example, the value estimation error function is calculated using the following equation: Here, StopGrad( ) indicates that the variable in ( ) is differentiated as a constant when calculating the gradient. η>0 is the minimum rate.

[0056] (Steps S112 and S113) In step S112, the policy learning unit 113 compares the data stored in the episode storage unit 121 with the correction coefficient ξ β (A') to calculate the advantage function A(s, a). In step S113, the policy learning unit 113 calculates the policy error term from the advantage function A(s, a). As an example, the policy error term is calculated using the following formula: -logp*stopgrad(A(s, a))

[0057] (Step S114) In step S114, the reward estimator 111 updates the state value function V(s) using the value estimation error function and the policy error term. Also, the policy learning unit 113 updates the policy π using the value estimation error function and the policy error term.

[0058] Furthermore, the policy learning unit 113 outputs the updated policy π. The policy learning unit 113 may output the policy π to an output device (such as a display, printer, or speaker) connected via the input / output unit 40A. Furthermore, the policy learning unit 113 may output the policy π by writing it to a storage destination (which may be in a storage device of the information processing device 1A or a storage device external to the information processing device 1A) designated by the user of the information processing device 1A. Furthermore, the policy learning unit 113 may output the policy π by transmitting it to another device via the communication unit 30A.

[0059] Furthermore, the policy learning unit 113 may display at least one of the state value function V(s) and the advantage function A(s, a) on a display device.

[0060] (Step S115) In step S115, the policy learning unit 113 determines whether an update termination condition is satisfied. An example of the update termination condition is whether the number of update steps or the number of episodes satisfies a predetermined condition (for example, whether it exceeds a threshold). If the update termination condition is satisfied (YES in step S115), the policy learning unit 113 ends the processing. On the other hand, if the update termination condition is not satisfied (NO in step S115), the policy learning unit 113 returns to the processing of step S104.

[0061] <Regarding the validity of the advantage function> Here, the reward N(μ ｋ , σ ｋ ２ ) can be obtained. Here, μ ｋ is the mean value, and σ ｋ is the standard deviation. In this case, the state value function V(S Ｔ－１ ) is u Ｔ +β / 2·σ Ｔ ２ This is described in Non-Patent Document 2. Also, V(S Ｔ－２ )teeth, Since it is updated by V (S Ｔ－２ )teeth, converges to.

[0062] Here, for simplicity, if we assume that the discount rate is γ=1, Due to the reproducibility of the normal distribution, V(S ｋ ) is the state s ｋ It can be seen that this estimates the reward variance and expected value up to the end of the episode.

[0063] (Validity of Specific Example 1 of Advantage Function) Here, we consider the validity of the advantage function when it is the above-mentioned Specific Example 1. In this case, due to the reproducibility of the normal distribution, the total reward until the end is Therefore, A(s ｋ , a ｋ ) is expressed as follows: where x is a sample from the distribution of the above equation (10): It can be expressed as: Advantage function A(s ｋ , a ｋ ) is the expected value of V(s ｋ ) = μ + β / 2 σ ２ The estimated value of is the distribution of rewards obtained by actual selection (the distribution of the above formula (10)) When it is larger than F, it becomes a negative value, when it is smaller it becomes a positive value, and when they are the same it becomes zero. β = μ + β / 2 σ ２ In this way, it can be said that the policy learning process executed by the information processing device 1A allows the action a to be selected more appropriately.

[0064] (Validity of Specific Example 2 of Advantage Function) Next, consider the case where the advantage function is the above specific example 2. For simplicity, the explanation will be given here with γ=1. In this case as well, the k-step reward sum is Since it is distributed as follows, the sample from here is taken as x, and the advantage function A(s ｔ , a ｔ )teeth, This becomes:

[0065] This advantage function A(s ｋ , a ｋ ) is the expected value of V(s ｋ ) = μ + β / 2 σ ２ The estimated value of the distribution of the above formula (14) and the estimated value μ of the state value function V between k and Tｋ：Ｔ +β / 2·σ ２ ｋ：Ｔ and When the value of a is larger than the value of b, the value of b is negative, when the value of b is smaller than the value of b, the value of b is positive, and when the values ​​are the same, the value of b is zero. From this, it can be said that the policy learning executed by the information processing device 1A enables the action a to be selected more appropriately.

[0066] <Example> Fig. 6 is a diagram showing the results of a simulation performed by the information processing device 1A. The plane shown in Fig. 6 includes a plurality of areas divided into a grid, one of which is set as a start point ST, and the other two are set as goals Gx and Gy. In one action, the agent moves to an area adjacent to the current area, and by repeating this movement, it reaches goal Gx or goal Gy. When goal Gx is reached, a reward is obtained according to a certain probability distribution. Furthermore, when goal Gy is reached, a reward is obtained according to a probability distribution different from that of goal Gx.

[0067] Here, the risk parameter β=-1, μ+1 / 2*βσ ２ The evaluation was carried out with the following. In other words, the agent was trained to avoid risk with a variance of -0.5. The agent's behavior was measured under the following conditions of Example 1 to Example 3.

[0068] In Example 1, the probability distribution of x is N(1,0), and the probability distribution of y is N(1.1,1.0). ２ In this case, y becomes smaller by the amount of risk discounting, so the model was trained to select x.

[0069] In Example 2, we define the probability distribution of x as N(0.8, 0.3 ２ ) and the probability distribution of y is N(1.1, (1.0) ２ In this case, 0.8-1 / 2*0.3 ２ >1.1-1 / 2*1.0 ２ Therefore, the learning was carried out to select x.

[0070] In Example 3, we define the probability distribution of x as N(0.5, 0.3 ２ ) and the probability distribution of y is N(1.1, (1.0) ２In this case, 0.5-1 / 2*0.3 ２ <1.1-1 / 2*1.0 ２ Therefore, learning was performed to select y. In this way, the information processing device 1A can perform learning in which the policy is appropriately evaluated without overestimating or underestimating the policy.

[0071] <Effects of Information Processing Device> As described above, the information processing device 1A includes a value function update unit 11 that updates the state value function V(s) using a risk parameter β related to the risk of the reward x, and a correction coefficient ξ that updates the state value function V(s) using the risk parameter β and the pre-correction advantage function A′(s, a) obtained using the state value function V(s). β A correction coefficient calculation unit 12 for calculating (A'), and a correction coefficient ξ β The advantage function A(s, a) is calculated by multiplying (A') by the uncorrected advantage function A'(s, a), and the calculated advantage function A(s, a) is used to update the policy π.

[0072] For example, by applying the technology described in Non-Patent Document 2 to the conventional Actor-Critic method, the advantage function A(s, a) can be calculated as follows: A(s, a) = Q(s, a) - {V(s) - βσ ２}, the problem arises that the policy will be overestimated in the risk-averse case (when β<0), while being underestimated in the risk-seeking case (when β>0). In response to this, the information processing device 1A does not add a value based on the risk parameter to the pre-correction advantage function A', but instead uses a correction coefficient ξ obtained based on the risk parameter β. β By updating the advantage function A by multiplying it by (A'), it is possible to prevent the policy from being overestimated or underestimated. β By calculating the advantage function A(s, a) using (A'), not only the difference used for updating one step but also the sum of rewards in an episode can be used as an advantage in risk-sensitive reinforcement learning.

[0073] The information processing device 1A further includes a reward estimation unit 111 that estimates a risk-corrected reward v, which is a reward that takes risk into consideration, from the state s using a state value function V(s). The policy learning unit 113 calculates a correction coefficient ξ β The information processing device 1A employs a configuration in which the advantage function A(s, a) is updated using (A') and the risk-corrected reward v. Therefore, the information processing device 1A can prevent the policy from being overestimated or underestimated in reinforcement learning.

[0074] In the information processing device 1A, the correction coefficient calculation unit 112 calculates the correction coefficient ξ from the pre-correction advantage function A′(s, a) using a sigmoid function including the risk parameter β as a coefficient. β Therefore, the information processing device 1A can prevent the policy from being overestimated or underestimated in reinforcement learning.

[0075] Furthermore, the information processing device 1A employs a configuration in which the pre-correction advantage function A'(s, a) is the difference between the sum of discounted rewards up to the last step in an episode and the estimated value of the risk-corrected reward. Therefore, the information processing device 1A can prevent the policy from being overestimated or underestimated in reinforcement learning.

[0076] In the information processing device 1A, the pre-correction advantage function A'(s, a) is calculated by dividing the sum of discounted rewards in k steps by r ｋ , discount rate is γ, and the state after the transition of step t is s ｔ , state s ｔ The state value function of V(s ｔ ) then r ｋ +γ ｋ V (S ｔ＋ｋ ) -V(s ｔ ) is adopted. Therefore, the information processing device 1A can prevent the policy from being overestimated or underestimated in reinforcement learning.

[0077] Furthermore, the information processing device 1A employs a configuration in which the pre-correction advantage function A'(s, a) is a double-discounted sum of TD errors. Therefore, the information processing device 1A can prevent the policy from being overestimated or underestimated in reinforcement learning.

[0078] Furthermore, the information processing device 1A employs a configuration in which the policy learning unit 113 outputs the updated policy π. Therefore, according to the information processing device 1A, it is possible to more appropriately determine the next action, etc., using the updated policy π.

[0079] Furthermore, the information processing device 1A employs a configuration in which the policy learning unit 113 displays at least one of the state value function V(s, a) and the advantage function A(s, a). Therefore, according to the information processing device 1A, a user of the information processing device 1A can confirm at least one of the state value function and the advantage function.

[0080] [Example of Software Implementation] Some or all of the functions of the reinforcement learning device 1 and the information processing device 1A (hereinafter also referred to as "each of the above devices") may be implemented by hardware such as an integrated circuit (IC chip), or by software.

[0081] In the latter case, each of the above devices is realized by, for example, a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in Figure 7. Figure 7 is a block diagram showing the hardware configuration of computer C that functions as each of the above devices.

[0082] The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for causing the computer C to function as each of the above-mentioned devices. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing the functions of each of the above-mentioned devices.

[0083] The processor C1 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a tensor processing unit (TPU), a quantum processor, a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0084] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may also include a communication interface for transmitting and receiving data to and from other devices. The computer C may also include an input / output interface for connecting input / output devices such as a keyboard, a mouse, a display, and a printer.

[0085] The program P can also be recorded on a non-transitory, tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.

[0086] [Appendix 1] The present disclosure includes the technologies described in the following appendices. However, the present invention is not limited to the technologies described in the following appendices, and various modifications are possible within the scope of the claims. (Appendix a1) A reinforcement learning device that learns a policy for determining a next action of an agent based on a state of the environment and a reward obtained from the agent's action, the reinforcement learning device comprising: value function updating means that updates a state value function using a risk parameter related to a risk of the reward; correction coefficient calculation means that calculates a correction coefficient using the risk parameter and a first advantage function obtained using the state value function; and policy learning means that calculates a second advantage function by multiplying the correction coefficient by the first advantage function and updates the policy using the calculated second advantage function.

[0087] (Appendix a2) The reinforcement learning device according to Appendix a1, further comprising a reward estimation means for estimating a risk-corrected reward, which is a reward that takes the risk into consideration, from the state using the state value function, wherein the policy learning means updates the second advantage function using the correction coefficient and the risk-corrected reward.

[0088] (Supplementary Note a3) The reinforcement learning device according to Supplementary Note a1 or a2, wherein the correction coefficient calculation means calculates the correction coefficient from the first advantage function using a sigmoid function including the risk parameter as a coefficient.

[0089] (Supplementary Note a4) The reinforcement learning device according to Supplementary Note a2, wherein the first advantage function is a difference between a sum of discounted rewards up to a final step in an episode and an estimate of the risk-corrected reward.

[0090] (Note a5) The first advantage function is a function that calculates the sum of discounted rewards in k steps by r ｋ , discount rate is γ, and the state after the transition of step t is s ｔ , state s ｔ The state value function of V(s ｔ ) then r ｋ +γ ｋ V (S ｔ＋ｋ ) -V(sｔ ) The reinforcement learning device according to appendix a2.

[0091] (Supplementary Note a6) The reinforcement learning device according to Supplementary Note a2, wherein the first advantage function is a double-discounted sum of TD errors.

[0092] (Supplementary Note a7) The reinforcement learning device according to any one of Supplementary Note a1 to a6, further comprising: an output unit configured to output the policy updated by the policy learning unit.

[0093] (Supplementary Note a8) The reinforcement learning device according to any one of Supplementary Notes a1 to a7, further comprising: a display means for displaying at least one of the state value function and the advantage function.

[0094] (Supplementary Note b1) A reinforcement learning method in which at least one processor learns a policy for determining an agent's next action based on the state of the environment and a reward obtained from the agent's action, the reinforcement learning method including the processor updating a state value function using a risk parameter related to the risk of the reward, calculating a correction coefficient using the risk parameter and a first advantage function obtained using the state value function, calculating a second advantage function by multiplying the correction coefficient by the first advantage function, updating the policy using the calculated second advantage function, and outputting the updated policy.

[0095] (Appendix B2) The reinforcement learning method described in Appendix B1 further includes the at least one processor estimating a risk-corrected reward, which is a reward that takes the risk into account, from the state using the state value function, and in the process of updating the policy, the at least one processor updates the second advantage function using the correction coefficient and the risk-corrected reward.

[0096] (Supplementary Note b3) The reinforcement learning method according to Supplementary Note b1 or b2, wherein in the step of calculating the correction coefficient, the at least one processor calculates the correction coefficient from the first advantage function using a sigmoid function including the risk parameter as a coefficient.

[0097] (Supplementary Note b4) The reinforcement learning method according to Supplementary Note b2, wherein the first advantage function is the difference between the sum of discounted rewards up to the last step in an episode and the estimated value of the risk-adjusted reward.

[0098] (Note b5) The first advantage function is a function that calculates the sum of discounted rewards in k steps by r ｋ , discount rate is γ, and the state after the transition of step t is s ｔ , state s ｔ The state value function of V(s ｔ ) then r ｋ +γ ｋ V (S ｔ＋ｋ ) -V(s ｔ ) The reinforcement learning method according to appendix b2.

[0099] (Supplementary Note b6) The reinforcement learning method according to Supplementary Note b2, wherein the first advantage function is a double-discounted sum of TD errors.

[0100] (Supplementary Note a7) The reinforcement learning method according to any one of Supplementary Notes b1 to b6, further comprising the at least one processor outputting the updated policy.

[0101] (Supplementary Note b8) The reinforcement learning method according to any one of Supplementary Notes b1 to b7, further comprising the at least one processor displaying at least one of the state value function and the advantage function.

[0102] (Addendum c1) A program that causes a computer to perform reinforcement learning to learn a policy for determining an agent's next action based on the state of the environment and a reward obtained from the agent's action, the program causing the computer to execute: a value function update process that updates a state value function using a risk parameter related to the risk of the reward; a correction coefficient calculation process that calculates a correction coefficient using the risk parameter and a first advantage function obtained using the state value function; a policy learning process that calculates a second advantage function by multiplying the correction coefficient by the first advantage function and updates the policy using the calculated second advantage function; and an output process that outputs the policy updated in the policy learning process.

[0103] (Appendix c2) The program described in Appendix c1, further causing the computer to execute a reward estimation process that uses the state value function to estimate a risk-corrected reward, which is a reward that takes the risk into consideration, from the state, and in the policy learning process, the computer updates the second advantage function using the correction coefficient and the risk-corrected reward.

[0104] (Additional Note c3) The program according to Additional Note c1 or c2, wherein in the correction coefficient calculation process, the computer calculates the correction coefficient from the first advantage function using a sigmoid function including the risk parameter as a coefficient.

[0105] (Supplementary Note c4) The program according to Supplementary Note c2, wherein the first advantage function is the difference between the sum of discounted rewards up to the last step in an episode and the estimated value of the risk-adjusted reward.

[0106] (Note c5) The first advantage function is a function that calculates the sum of discounted rewards in k steps by r ｋ , discount rate is γ, and the state after the transition of step t is s ｔ , state s ｔ The state value function of V(s ｔ ) then r ｋ +γ ｋ V (S ｔ＋ｋ ) -V(sｔ ) The program according to appendix c2.

[0107] (Supplementary Note c6) The program according to Supplementary Note c2, wherein the first advantage function is a double-discounted sum of TD errors.

[0108] (Supplementary Note a7) The program according to any one of Supplementary Notes c1 to c6, further causing the computer to execute an output process of outputting the policy updated in the policy learning process.

[0109] (Additional Note c8) The program according to any one of Additional Notes c1 to c7, further causing the computer to execute a display process of displaying at least one of the state value function and the advantage function.

[0110] (Appendix d) A reinforcement learning system that learns a policy for determining an agent's next action based on the state of the environment and a reward obtained from the agent's action, comprising: a value function update means that updates a state value function using a risk parameter related to the risk of the reward; a correction coefficient calculation means that calculates a correction coefficient using the risk parameter and a first advantage function obtained using the state value function; and a policy learning means that calculates a second advantage function by multiplying the correction coefficient by the first advantage function, and updates the policy using the calculated second advantage function.

[0111] [Appendix 2] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0112] (Additional Note e1) A reinforcement learning device comprising at least one processor, the at least one processor executing: a value function update process that updates a state value function using a risk parameter related to the risk of a reward obtained from the agent's actions in reinforcement learning; a correction coefficient calculation process that calculates a correction coefficient using the risk parameter and a first advantage function obtained using the state value function; and a policy learning process that calculates a second advantage function by multiplying the first advantage function by the correction coefficient, and updates a policy for determining the agent's next action using the calculated second advantage function.

[0113] The behavior inference device may further include a memory, and the memory may store a program for causing the at least one processor to execute each of the processes.

[0114] (Appendix e2) The reinforcement learning device described in Appendix e1, wherein the at least one processor further performs a reward estimation process to estimate a risk-corrected reward, which is a reward that takes the risk into account, from the state using the state value function, and the at least one processor updates the second advantage function using the correction coefficient and the risk-corrected reward in the policy learning process.

[0115] (Additional Note e3) The reinforcement learning device according to Additional Note e1 or e2, wherein in the correction coefficient calculation process, the at least one processor calculates the correction coefficient from the first advantage function using a sigmoid function including the risk parameter as a coefficient.

[0116] (Supplementary Note e4) The reinforcement learning device according to Supplementary Note e2, wherein the first advantage function is a difference between a sum of discounted rewards up to a final step in an episode and an estimate of the risk-corrected reward.

[0117] (Note e5) The first advantage function is a sum of discounted rewards in k steps, ｋ , discount rate is γ, and the state after the transition of step t is s ｔ, state s ｔ The state value function of V(s ｔ ) then r ｋ +γ ｋ V (S ｔ＋ｋ ) -V(s ｔ ) The reinforcement learning device according to appendix e2.

[0118] (Supplementary Note e6) The reinforcement learning device according to Supplementary Note e2, wherein the first advantage function is a double-discounted sum of TD errors.

[0119] (Additional Note e7) The reinforcement learning device according to any one of Additional Notes e1 to e6, wherein the at least one processor further executes an output process of outputting the policy updated in the policy learning process.

[0120] (Additional Note e8) The reinforcement learning device according to any one of Additional Notes e1 to e7, wherein the at least one processor further executes a display process that displays at least one of the state value function and the advantage function.

[0121] (Supplementary note e9) A reinforcement learning method in which at least one processor learns a policy for determining an agent's next action based on the state of the environment and a reward obtained from the agent's action, the reinforcement learning method including the processor updating a state value function using a risk parameter related to the risk of the reward, calculating a correction coefficient using the risk parameter and a first advantage function obtained using the state value function, calculating a second advantage function by multiplying the correction coefficient by the first advantage function, and updating the policy using the calculated second advantage function.

[0122] (Appendix e10) A recording medium having recorded thereon a program for causing a computer to function as the information processing device described in appendix e1, the recording medium being non-transitory and readable by a computer and having recorded thereon a program for causing a computer to execute the value function update process, the correction coefficient calculation process, the policy learning process, and the output process.

[0123] REFERENCE SIGNS LIST 1 Reinforcement learning device 1A Information processing device 11 Value function update unit 12, 112 Correction coefficient calculation unit 13, 113 Policy learning unit 14 Output unit 111 Reward estimation unit S1, S2 Reinforcement learning method S11 Value function update process S12 Correction coefficient calculation process S13 Policy learning process S14 Output process

Claims

1. A reinforcement learning device that learns a policy for determining an agent's next action based on the state of the environment and a reward obtained from the agent's action, comprising: a value function updating means that updates a state value function using a risk parameter related to the risk of the reward; a correction coefficient calculation means that calculates a correction coefficient using the risk parameter and a first advantage function obtained using the state value function; and a policy learning means that calculates a second advantage function by multiplying the correction coefficient by the first advantage function, and updates the policy using the calculated second advantage function.

2. The reinforcement learning device of claim 1, further comprising a reward estimation means for estimating a risk-corrected reward, which is a reward taking into account the risk, from the state using the state value function, and the policy learning means updates the second advantage function using the correction coefficient and the risk-corrected reward.

3. The reinforcement learning device according to claim 1 or 2, wherein the correction coefficient calculation means calculates the correction coefficient from the first advantage function using a sigmoid function including the risk parameter as a coefficient.

4. The reinforcement learning device according to claim 2, wherein the first advantage function is the difference between the sum of discounted rewards up to the final step in an episode and the estimated value of the risk-corrected reward.

5. The first advantage function is the sum of discounted rewards in k steps as r ｋ , the discount rate is γ, and the state after the transition at step t is s ｔ , state s ｔ The state value function of V(s ｔ ) where r ｋ +γ ｋ V (S ｔ＋ｋ ) -V(s ｔ 3. The reinforcement learning device according to claim 2, wherein:

6. The reinforcement learning device according to claim 2, wherein the first advantage function is a doubly discounted sum of TD errors.

7. The reinforcement learning device according to any one of claims 1 to 6, further comprising an output means for outputting the policy updated by the policy learning means.

8. A reinforcement learning device according to any one of claims 1 to 7, comprising: a display means for displaying at least one of the state value function and the advantage function.

9. A reinforcement learning method in which at least one processor learns a policy for determining an agent's next action based on the state of the environment and a reward obtained from the agent's action, the reinforcement learning method including the processor: updating a state value function using a risk parameter related to the risk of the reward; calculating a correction coefficient using the risk parameter and a first advantage function obtained using the state value function; calculating a second advantage function by multiplying the correction coefficient by the first advantage function, and updating the policy using the calculated second advantage function.

10. A program for causing a computer to execute reinforcement learning to learn a policy for determining an agent's next action based on the state of the environment and a reward obtained from the agent's action, the program causing the computer to execute the following: a value function update process for updating a state value function using a risk parameter related to the risk of the reward; a correction coefficient calculation process for calculating a correction coefficient using the risk parameter and a first advantage function obtained using the state value function; and a policy learning process for calculating a second advantage function by multiplying the correction coefficient by the first advantage function and updating the policy using the calculated second advantage function.

Images