Reinforcement learning device, reinforcement learning method, and recording medium

Abstract

In order to provide reinforcement learning that is suitable when using a single estimation amount, a reinforcement learning device (1) comprises: a setting unit (11) that sets an initial value of a risk corrected reward; a calculation unit (12) that calculates a correction coefficient; an update unit (13) that updates the risk corrected reward; an estimation unit (14) that estimates the risk corrected reward by a process for calculating the correction coefficient and a process for updating the risk corrected reward; and a learning unit (15) that performs learning on the basis of the estimated risk corrected reward.

Description

[Technical field]

[0001] The present invention relates to a reinforcement learning device, a reinforcement learning method, and program Regarding. [Background technology]

[0002] In reinforcement learning, a learning agent takes actions based on the observed environment. Reinforcement learning is performed by sequentially obtaining a strategy for determining actions from the environment so as to increase the expected value of the reward corresponding to the change in the environment caused by the action.

[0003] For example, an agent that moves a cart equipped with an inverted pendulum will be described. This agent is designed so that the longer the time the pendulum stays upside down, the higher the reward it will receive. This agent observes the position of the cart, the speed of the cart, the angle of the pendulum, and the angular speed of the pendulum, and controls the cart according to the observed results. In this case, the agent can maximize the expected value of the time the pendulum stays upside down, thereby lengthening the time the pendulum stays upside down.

[0004] However, the above reinforcement learning does not take into account the variance in rewards. For example, in the agent moving the cart with the inverted pendulum, a large variance in rewards means a large variance in the time it takes for the pendulum to stand up. If the variance in the time it takes for the pendulum to stand up is large, even if the average time is long, the bottom 10% of agents may have a very short time.

[0005] Therefore, a type of reinforcement learning called risk-sensitive reinforcement learning (hereinafter referred to as "risk-sensitive reinforcement learning") is known, which considers the variance in reward by relating it to risk. In risk-sensitive reinforcement learning, where the expected value of reward is Q, the variance of reward is V, and the risk preference / avoidance parameter is β, the goal is to maximize Q+βV. With this configuration, risk-sensitive reinforcement learning can reduce and avoid risk (or increase and prefer when parameter β>0) while increasing the expected value of reward.

[0006] For example, Non-Patent Document 1 describes a configuration that uses a new estimated value update method under the assumption that the reward distribution is a normal distribution in risk-sensitive reinforcement learning, which previously required the estimation of the expected value Q of the reward and the variance V of the reward. Non-Patent Document 1 describes a configuration that uses the new estimated value update method to directly estimate the value of Q+βV with one estimator. Non-Patent Document 1 also discloses through numerical experiments that even if the reward distribution is not a normal distribution, deviations will occur, but learning that estimates the magnitude of risk correctly to a certain extent is possible. [Prior art documents] [Non-patent literature]

[0007] [Object 1] JPEG2024047829000001.jpg12169 Summary of the Invention [Problem to be solved by the invention]

[0008] In the risk-sensitive reinforcement learning described in the above-mentioned Non-Patent Document 1, as in other risk-sensitive reinforcement learning, learning is performed by maximizing Q+βV, where Q is the expected value of the reward, V is the variance of the reward, and β is the risk preference / aversion parameter. Q is the expected value of the reward, so it has the dimension of the reward, and V is the variance, so it has the dimension of the square of the dimension of the reward. Therefore, β has the dimension of the reward to the negative first power. Therefore, in the risk-sensitive reinforcement learning described in the above-mentioned Non-Patent Document 1, when the environment changes, the degree of risk consideration changes even with the same β.

[0009] For example, a case will be described in which an appropriate β is determined in a certain environment, and the risk-sensitive reinforcement learning described in Non-Patent Document 1 is performed in another environment where the reward is doubled by the β. In this case, Q in another environment is only twice as large as Q in a certain environment, while V in another environment is four times as large as V in a certain environment. Therefore, risk is more strongly avoided, and the expected value of the reward is neglected. In this case, if β is also halved in accordance with the dimensions, learning similar to risk-sensitive reinforcement learning in a certain environment can be performed. However, there is a problem that the size of the reward obtained in another environment cannot be known until learning is performed.

[0010] This problem can be solved when estimating Q and V independently by using the standard deviation σ=√V as the variance of reward, since the dimension of the expected value of reward matches the dimension of the variance of reward. In this case, the above problem does not occur if Q+cσ is maximized with c as the dimensionless risk preference / aversion parameter. However, when estimating either Q or V, in other words, when using a single estimator, there is a problem that such a method cannot be adopted.

[0011] One aspect of the present invention has been made in consideration of the above problems, and one example of a purpose thereof is to provide reinforcement learning that is suitable when one estimator is used. [Means for solving the problem]

[0012] A reinforcement learning device according to one aspect of the present invention is a reinforcement learning device that learns a strategy for determining a next action to be taken by an agent that has performed an action based on the result of observing an environment, and includes: a setting means for setting an initial value of a risk-corrected reward, which is a reward that takes risk into consideration; a calculation means for calculating a correction coefficient based on a predetermined parameter related to risk relative to reward and a difference between the risk-corrected reward and the reward obtained by the next action; an update means for updating the risk-corrected reward based on the risk-corrected reward, the correction coefficient, and the difference; an estimation means for estimating the risk-corrected reward by a process in which the calculation means calculates a correction coefficient using the risk-corrected reward updated by the update means, and a process in which the update means updates the risk-corrected reward using the updated risk-corrected reward and the correction coefficient; and a learning means for learning based on the estimated risk-corrected reward.

[0013] A reinforcement learning method according to one aspect of the present invention includes a reinforcement learning device that learns a policy for determining a next action to be taken by an agent that has performed an action based on a result of observing an environment, the method including: setting an initial value of a risk-corrected reward, which is a reward that takes risk into consideration; calculating a correction coefficient based on a predetermined parameter related to risk relative to reward and a difference between the risk-corrected reward and the reward obtained by the next action; updating the risk-corrected reward based on the risk-corrected reward, the correction coefficient, and the difference; estimating a risk-corrected reward by a process of calculating a correction coefficient in the calculating step using the risk-corrected reward updated in the updating step; and a process of updating the risk-corrected reward in the updating step using the updated risk-corrected reward and the correction coefficient; and learning based on the estimated risk-corrected reward.

[0014] According to one aspect of the present invention program is a program that causes a computer to function as a reinforcement learning device that learns a strategy for determining the next action to be taken by an agent that has performed an action based on the results of observing the environment. In There is ,before The computer functions as a setting means for setting an initial value of a risk-corrected reward, which is a reward that takes risk into consideration, a calculation means for calculating a correction coefficient based on a predetermined parameter related to risk relative to reward and the difference between the risk-corrected reward and the reward obtained by the next action, an update means for updating the risk-corrected reward based on the risk-corrected reward, the correction coefficient, and the difference, an estimation means for estimating the risk-corrected reward by a process in which the calculation means calculates a correction coefficient using the risk-corrected reward updated by the update means, and a process in which the update means updates the risk-corrected reward using the updated risk-corrected reward and the correction coefficient, and a learning means for learning based on the estimated risk-corrected reward. Effect of the Invention

[0015] According to one aspect of the present invention, it is possible to provide reinforcement learning that is suitable for the case where one estimator is used. [Brief description of the drawings]

[0016] [Figure 1] 1 is a block diagram showing a configuration of a reinforcement learning device according to a first exemplary embodiment of the present invention. [Diagram 2] 1 is a flow chart showing a flow of a reinforcement learning method according to an exemplary embodiment 1 of the present invention. [Diagram 3] FIG. 11 is a block diagram showing a configuration of a reinforcement learning device according to an exemplary embodiment 2 of the present invention. [Figure 4] FIG. 11 is a diagram showing a data flow in a control unit according to the second exemplary embodiment of the present invention. [Diagram 5] 11 is a flowchart showing a flow of processing executed by a reinforcement learning device according to an exemplary embodiment 2 of the present invention. [Figure 6] FIG. 11 is a schematic diagram showing how an AGV according to an exemplary embodiment 3 of the present invention performs reinforcement learning. [Figure 7] FIG. 11 is a block diagram showing a configuration of an AGV according to an exemplary embodiment 3 of the present invention. [Figure 8]FIG. 2 is a block diagram showing an example of a hardware configuration of a reinforcement learning device and an AGV according to each exemplary embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Example Embodiment 1 A first exemplary embodiment of the present invention will be described in detail with reference to the drawings. This exemplary embodiment is a basic form of the exemplary embodiments described later.

[0018] (Configuration of Reinforcement Learning Device 1) The configuration of a reinforcement learning device 1 according to this exemplary embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of a reinforcement learning device 1 according to this exemplary embodiment.

[0019] The reinforcement learning device 1 is a device that learns a policy for determining a next action to be taken by an agent that has performed an action based on the result of observing an environment in which rewards follow a normal distribution. As shown in Fig. 1, the reinforcement learning device 1 includes a setting unit 11, a calculation unit 12, an update unit 13, an estimation unit 14, and a learning unit 15. In this exemplary embodiment, the setting unit 11, the calculation unit 12, the update unit 13, the estimation unit 14, and the learning unit 15 are configured to realize a setting means, a calculation means, an update means, an estimation means, and a learning means, respectively.

[0020] The setting unit 11 sets an initial value of the risk-corrected reward, which is a reward that takes risk into consideration. The initial value of the risk-corrected reward is not particularly limited, and may be, for example, a random value or a value learned without risk correction.

[0021] The risk-adjusted reward is a value calculated using the following formula (1), where σ is the standard deviation of the reward, c is a specified parameter, and μ is the expected value of the reward obtained by the next action. ν=μ-cσ (1) The calculation unit 12 calculates a correction coefficient based on a predetermined parameter related to risk relative to reward, and the difference between the risk-corrected reward and the reward obtained by the next action.

[0022] Here, the predetermined parameter for reward is a parameter of risk preference / aversion that is a preset real value.

[0023] Furthermore, if the reward obtained by the next action is r, the calculation unit 12 calculates the difference δ using the following formula (2). δ = r - ν (2) Furthermore, the calculation unit 12 calculates the correction coefficient ξ using the following equations (3) to (5).

[0024]

number

[0025]

number

[0026]

number

[0027] That is, the calculation unit 12 calculates the correction coefficient ξ by using the following formula (6) when δ>0, and by using the following formula (7) when δ<0.

number

[0028]

number

[0029] Here, the update unit 13 updates the risk-corrected reward v using the following equation (8). ν←ν+ξδ (8) The estimation unit 14 estimates the risk-corrected reward by a process in which the calculation unit 12 calculates a correction coefficient using the risk-corrected reward updated by the update unit 13, and a process in which the update unit 13 updates the risk-corrected reward using the updated risk-corrected reward and the correction coefficient.

[0030] The learning unit 15 includes: Estimated Learn based on risk-adjusted reward.

[0031] (Proof that the update has a unique and stable fixed point) In the following, when the reward is normally distributed, this risk-corrected reward update has a unique and stable fixed point ν * and fixed point ν * Prove that μ+cσ is equal to

[0032] A fixed point ν with unique and stable updates * means that the expected value of the update J(ν) satisfies the following equations (9) to (11). J(ν)<0 if ν>ν * (9) J(ν)=0 if ν=ν * =μ+cσ (10) J(ν)>0 if ν<ν * (11) That is, the expected value of the update J(ν) is the expected value of the estimated risk-adjusted reward ν when * If the expected value of ν is larger than ν, it means that ν is updated to a smaller expected value. Also, the expected value of the update J(ν) is the time when the estimated risk-adjusted reward ν is smaller than ν * If the expected value of ν is smaller than ν, then ν is updated to a larger expected value. Also, the expected value of the update J(ν) is the expected value of ν when the estimated risk-adjusted reward ν is smaller than ν * If it is equal to, the expected value of the update amount of ν is 0.

[0033] Here, when the above-mentioned formula (8) is rewritten using the following formula (12), formula (13) is obtained.

number

[0034]

number

[0035]

number

[0036]

number

[0037]

number

[0038] In this way, since the expected value of the update J(ν) satisfies equations (9) to (11), the update can be performed at a unique and stable fixed point ν * =μ+cσ.

[0039] As described above, in the reinforcement learning device 1 according to this exemplary embodiment, the reinforcement learning device 1 learns a policy for determining the next action to be taken by an agent that has performed an action based on the result of observing the environment, and includes a setting unit 11 that sets an initial value of a risk-corrected reward, which is a reward that takes risk into consideration, a calculation unit 12 that calculates a correction coefficient based on a predetermined parameter related to the risk relative to the reward, and the difference between the risk-corrected reward and the reward obtained by the next action, an update unit 13 that updates the risk-corrected reward based on the risk-corrected reward, the correction coefficient, and the difference, an estimation unit 14 that estimates the risk-corrected reward by a process in which the calculation unit 12 calculates a correction coefficient using the risk-corrected reward updated by the update unit 13, and a process in which the update unit 13 updates the risk-corrected reward using the updated risk-corrected reward and the correction coefficient, and a learning unit 15 that learns based on the estimated risk-corrected reward.

[0040] Therefore, according to the reinforcement learning device 1 according to this exemplary embodiment, the degree of risk consideration does not change depending on the predetermined parameters even in two environments with different rewards by using the risk-corrected reward. Therefore, according to the reinforcement learning device 1 according to this exemplary embodiment, it is possible to obtain an effect that it is possible to provide reinforcement learning suitable for the case where one estimator is used.

[0041] (Reinforcement learning method flow S1) The flow of the reinforcement learning method S1 according to this exemplary embodiment will be described with reference to Fig. 2. Fig. 2 is a flow diagram showing the flow of the reinforcement learning method S1 according to this exemplary embodiment.

[0042] (Step S11) In step S11, the setting unit 11 sets an initial value of a risk-adjusted reward, which is a reward that takes risk into consideration.

[0043] (Step S12) In step S12, the calculation unit 12 calculates a correction coefficient based on a predetermined parameter related to risk relative to reward, and the difference between the risk-corrected reward and the reward obtained by the next action.

[0044] (Step S13) In step S13, the update unit 13 updates the risk-corrected reward based on the risk-corrected reward, the correction coefficient, and the difference.

[0045] (Step S14) In step S14, the estimation unit 14 estimates the risk-corrected reward by a process in which the calculation unit 12 calculates a correction coefficient using the risk-corrected reward updated by the update unit 13, and a process in which the update unit 13 updates the risk-corrected reward using the updated risk-corrected reward and the correction coefficient.

[0046] (Step S15) In step S15, the learning unit 15 learns based on the estimated risk-corrected reward.

[0047] As described above, in the reinforcement learning method S1 according to this exemplary embodiment, the setting unit 11 sets an initial value of the risk-corrected reward, which is a reward that takes risk into consideration, the calculation unit 12 calculates a correction coefficient based on a predetermined parameter related to the risk to the reward, and the difference between the risk-corrected reward and the reward obtained by the next action, the update unit 13 updates the risk-corrected reward based on the risk-corrected reward, the correction coefficient, and the difference, the estimation unit 14 estimates the risk-corrected reward by the calculation unit 12 calculating the correction coefficient using the risk-corrected reward updated by the update unit 13, and the update unit 13 updating the risk-corrected reward using the updated risk-corrected reward and the correction coefficient, and the learning unit 15 learns based on the estimated risk-corrected reward. Therefore, according to the reinforcement learning method S1 according to this exemplary embodiment, the same effect as that of the reinforcement learning device 1 described above can be obtained.

[0048] Exemplary embodiment 2 A second exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first exemplary embodiment are given the same reference numerals, and the description thereof will be omitted as appropriate.

[0049] (Overview of Reinforcement Learning Device 2) The reinforcement learning device 2 according to this exemplary embodiment is a device that learns a strategy for determining a next action to be taken by an agent that has performed an action based on the result of observing an environment in which rewards follow a normal distribution. As an example, the reinforcement learning device 2 is a device that learns from an environment on a simulator. Examples of simulations include simulations of mobile robots, production and inventory management schedules, and AI for gambling, trading, and games.

[0050] (Configuration of Reinforcement Learning Device 2) The configuration of the reinforcement learning device 2 according to this exemplary embodiment will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the configuration of the reinforcement learning device 2 according to this exemplary embodiment. As shown in Fig. 3, the reinforcement learning device 2 includes a control unit 20, an input / output unit 27, and a storage unit 28.

[0051] The input / output unit 27 is an interface for inputting and outputting data. As an example, a parameter c relating to a risk to a reward is input via the input / output unit 27. The input / output unit 27 supplies the input data to the control unit 20. In addition, the input / output unit 27 outputs the data supplied from the control unit 20 to another device connected thereto. The parameter c is as described above as a predetermined parameter.

[0052] The storage unit 28 stores data referenced by the control unit 20. As an example, the storage unit 28 stores a parameter c.

[0053] (Functions of the control unit 20) The control unit 20 controls each component included in the reinforcement learning device 2. As an example, the control unit 20 acquires a parameter c supplied from the input / output unit 27, and stores it in the storage unit .

[0054] 3, the control unit 20 also functions as a setting unit 11, a calculation unit 12, an update unit 13, an estimation unit 14, a learning unit 15, a policy decision unit 21, and an execution unit 22. In this exemplary embodiment, the setting unit 11, the calculation unit 12, the update unit 13, the estimation unit 14, and the learning unit 15 are configured to realize a setting means, a calculation means, an update means, an estimation means, and a learning means, respectively.

[0055] The setting unit 11 sets initial parameters for the estimation unit 14 to estimate a risk-corrected reward v, which is a reward that takes risk into account. For example, the setting unit 11 sets an initial value of the risk-corrected reward v and weights of a neural network for the estimation unit 14 to estimate the risk-corrected reward v. Set .

[0056] The setting unit 11 also generates a state-action value function of the array v(S, A) of estimated values ​​of risk-corrected rewards for action a in state s.

[0057] The calculation unit 12 calculates the correction coefficient ξ based on the parameter c, the risk-corrected reward ν, and the difference δ between the reward r obtained by the next action. The method of calculating the correction coefficient ξ includes the above-mentioned methods using the formulas (3) to (5). In addition, the calculation unit 12 calculates the correction coefficient ξ + and the correction factor ξ - Calculate the correction coefficient ξ + and the correction factor ξ - The method for calculating the above formula (6) and formula (7) can be used.

[0058] The update unit 13 updates the risk-corrected reward based on the risk-corrected reward v, the correction coefficient ξ, and the difference δ. The method of updating the risk-corrected reward v includes the method using the above-mentioned formula (8). That is, the update unit 13 updates the risk-corrected reward v by adding the risk-corrected reward v to the value obtained by multiplying the correction coefficient ξ by the difference δ.

[0059] As an example, the update unit 13 calculates the difference δ using the following equation (17). δ=r+γν´(s´,a)-ν(s,a) ···(17) Here, γ represents the discount rate.

[0060] The estimation unit 14 estimates the risk-corrected reward v by a process in which the calculation unit 12 calculates the correction coefficient ξ using the risk-corrected reward v updated by the update unit 13, and a process in which the update unit 13 updates the risk-corrected reward v using the risk-corrected reward v and the correction coefficient ξ updated by the update unit 13. As an example, the estimation unit 14 may be an estimation means having parameters such as a neural network.

[0061] The learning unit 15 learns based on the updated risk-corrected reward. As an example, the learning unit 15 trains the policy decision unit 21 based on the updated risk-corrected reward ν.

[0062] The strategy decision unit 21 is trained by the learning unit 15 so as to receive a state s as an input and output an action a to be executed by the execution unit 22 .

[0063] The execution unit 22 executes the action a outputted from the measure decision unit 21 .

[0064] (Data flow in control unit 20) The flow of data in the control unit 20 will be described with reference to Fig. 4. Fig. 4 is a diagram showing the flow of data in the control unit 20 according to this exemplary embodiment.

[0065] First, the data is processed as shown by the dotted line in Fig. 4. That is, the calculation unit 12 obtains the parameter c and calculates the correction coefficient ξ + and the correction factor ξ - The calculation unit 12 calculates the calculated correction coefficient ξ + and the correction factor ξ - is supplied to the update unit 13.

[0066] The setting unit 11 also generates a state-action value function of an array v(S,A) of estimated values ​​of risk-corrected reward v. The setting unit 11 also sets a risk-corrected reward estimate v(s,a), which is the value of the risk-corrected reward for the initial state s and action a, as the initial value of the risk-corrected reward v. The setting unit 11 supplies the set risk-corrected reward estimate v(s,a) to the updating unit 13. The setting unit 11 also supplies the state s to the policy decision unit 21.

[0067] Next, the data is processed as indicated by the solid lines in Fig. 4. That is, the policy decision unit 21 outputs the action a to the execution unit 22. The execution unit 22 executes the action a.

[0068] Then, the execution unit 22 executes the action a, and the update unit 13 obtains the reward r. The update unit 13 also obtains the next state s'. The update unit 13 then calculates the next state s', the risk-corrected reward estimate v(s', a) for the action a, and the difference δ.

[0069] The update unit 13 updates the risk-corrected reward estimate ν(s,a), the difference δ, and the correction coefficient ξ + and the correction factor ξ - Specifically, as described above, the update unit 13 updates the risk-corrected reward ν, which is an estimated amount, based on the difference δ. + and the correction factor ξ - The risk-corrected reward ν is updated using either of the above. With this configuration, the update unit 13 can successively obtain a good estimate. In other words, the calculation unit 12 updates the correction coefficient ξ according to whether the difference δ is positive or negative. + and the correction factor ξ - and supplies it to the update unit 13.

[0070] Then, the update unit 13 supplies the updated risk-corrected reward estimate v(s, a) to the estimation unit 14. The update unit 13 also updates the state s, and supplies the updated state s to the policy decision unit 21.

[0071] The estimation unit 14 obtains multiple risk-corrected reward estimates v(s,a) by the calculation unit 12 calculating the correction coefficient ξ and the update unit 13 updating the risk-corrected reward estimate v(s,a). The estimation unit 14 then estimates the risk-corrected reward v. The estimation unit 14 supplies the estimated risk-corrected reward v to the learning unit 15.

[0072] The learning unit 15 trains the policy decision unit 21 based on the estimated risk-corrected reward ν.

[0073] (Process flow S2 executed by reinforcement learning device 2) The process flow S2 executed by the reinforcement learning device 2 will be described with reference to Fig. 5. Fig. 5 is a flowchart showing the process flow S2 executed by the reinforcement learning device 2 according to this exemplary embodiment.

[0074] (Step S21) In step S21, the calculation unit 12 acquires the parameter c stored in the storage unit .

[0075] (Step S22) In step S22, the calculation unit 12 calculates the correction coefficient ξ based on the parameter c. + and the correction factor ξ - The calculation unit 12 calculates the calculated correction coefficient ξ + and the correction factor ξ - is supplied to the update unit 13.

[0076] (Step S23) In step S23, the setting unit 11 generates an array v(S, A) that stores the estimated value of the risk-corrected reward for each state and each action.

[0077] (Step S24) In step S24, the setting unit 11 sets the state s to the initial state s. Then, the setting unit 11 sets the risk-corrected reward estimate v(s, a) in the initial state s and action a as an initial value. The setting unit 11 supplies the set risk-corrected reward estimate v(s, a) to the update unit 13. In addition, the setting unit 11 supplies the state s to the policy decision unit 21. That is, the reinforcement learning device 2 starts a simulation.

[0078] (Step S25) In step S25, the policy decision unit 21 randomly selects an action a. As an example, the policy decision unit 21 may be configured to select the most valuable action a with a probability of 1-ε, or to select a random action a with a probability ε.

[0079] (Step S26) In step S26, the policy decision unit 21 supplies the action a to the execution unit 22. Based on the action a supplied from the policy decision unit 21, the execution unit 22 executes the action a.

[0080] (Step S27) In step S27, the update unit 13 acquires the reward r obtained by the execution unit 22 executing the action a, and the next state s'.

[0081] (Step S28) In step S28, the update unit 13 calculates the risk-corrected reward estimate v(s', a). Then, the update unit 13 calculates the difference δ using the above-mentioned formula (17).

[0082] (Step S29) In step S29, the update unit 13 calculates the risk-corrected reward estimate v(s', a) and updates the array v(S, A).

[0083] (Step S30) In step S30, the update unit 13 calculates the risk-corrected reward estimate v(s,a), the difference δ, and the correction coefficient ξ using equation (8). + and the correction factor ξ- Based on this, we update the estimator, the risk-adjusted reward v.

[0084] As described above, the update unit 13 updates the correction coefficient ξ depending on whether the difference δ is positive or negative. + and the correction factor ξ - In other words, the calculation unit 12 uses either the correction coefficient ξ depending on whether the difference δ is positive or negative. + and the correction factor ξ - Calculate either one of the following and use the calculated correction coefficient ξ + and the correction factor ξ - The update unit 13 supplies one of the above to the update unit 13.

[0085] Then, the update unit 13 updates the state s from the initial state s to the next state s'.

[0086] (Step S31) In step S31, the update unit 13 determines whether the updated state s is a final state.

[0087] In step S31, if it is determined that the state is not the final state (step S31: NO), the reinforcement learning device 2 returns to the process of step S25.

[0088] (Step S32) If it is determined in step S31 that the state is the final state (step S31: YES), in step S32, the update unit 13 determines whether or not an update end condition is satisfied. As an example, the update unit 13 determines whether or not a condition such as a predetermined number of episodes or a reward is satisfied.

[0089] If it is determined in step S32 that the final condition is not satisfied (step S32: NO), the reinforcement learning device 2 returns to the process of step S24.

[0090] On the other hand, if it is determined in step S32 that the final condition is satisfied (step S32: YES), the reinforcement learning device 2 ends the processing shown in FIG.

[0091] As described above, in the reinforcement learning device 2 according to this exemplary embodiment, the risk-corrected reward can be estimated using one estimator by using the dimensionless parameter c. Therefore, in the reinforcement learning device 2 according to this exemplary embodiment, the effect is obtained that risk-sensitive reinforcement learning can be performed using one estimator, and the degree of risk consideration can be made to be unchanged depending on the magnitude of the reward.

[0092] Moreover, the reinforcement learning device 2 according to this exemplary embodiment employs a configuration in which learning is performed from an environment on a simulator. With this configuration as well, the same effects as those of the above-mentioned reinforcement learning device 1 can be obtained.

[0093] Exemplary embodiment 3 A third exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the above exemplary embodiment are denoted by the same reference numerals, and the description thereof will not be repeated.

[0094] (Outline of AGV3) An overview of an AGV (Automatic Guided Vehicle) 3 according to this exemplary embodiment will be described with reference to Fig. 6. Fig. 6 is a schematic diagram showing how the AGV 3 according to this exemplary embodiment performs reinforcement learning.

[0095] The AGV 3 according to this exemplary embodiment is a mobile robot that acts using a policy obtained by a reinforcement learning device or an estimated risk-corrected reward. As an example, the AGV 3 acts using a policy obtained by the above-mentioned reinforcement learning device 2 or an estimated risk-corrected reward.

[0096] As shown in Fig. 6, the AGV 3 acts using a policy obtained by a reinforcement learning device trained to reach the destination point 52 while avoiding obstacles 53 (obstacles 53A and 53B) or an estimated risk-corrected reward. The shorter the time it takes to reach the destination point 52, the greater the reward. In addition, if the AGV 3 hits an obstacle 53, a penalty (negative reward) is given.

[0097] 6, the state of the AGV 3, the destination point 52, and the positions of the obstacles 53 are observed by an observation device 58. Then, the information observed by the observation device 58 is output to the AGV 3 via a computer 57 capable of communicating with the AGV 3. With this configuration, the AGV 3 can recognize its own state, the destination point 52, and the positions of the obstacles 53.

[0098] (AGV3 configuration) The configuration of the AGV 3 according to this exemplary embodiment will be described with reference to Fig. 7. Fig. 7 is a block diagram showing the configuration of the AGV 3 according to this exemplary embodiment.

[0099] As shown in FIG. 7, the AGV 3 includes a control unit 20, an input / output unit 27, a storage unit 28, a sensor 31, an encoder 32, and a motor 33.

[0100] The control unit 20 controls each component of the AGV 3. 7 2, the control unit 20 also functions as a setting unit 11, a calculation unit 12, an update unit 13, an estimation unit 14, a learning unit 15, a strategy determination unit 21, and an execution unit 22. In this exemplary embodiment, the setting unit 11, the calculation unit 12, the update unit 13, the estimation unit 14, and the learning unit 15 are configured to realize a setting means, a calculation means, an update means, an estimation means, and a learning means, respectively. The functions of the control unit 20 are similar to those of the control unit 20 in the exemplary embodiment described above, and therefore will not be described.

[0101] The input / output unit 27 and the storage unit 28 are similar to the input / output unit 27 and the storage unit 28 in the above-described exemplary embodiment, respectively, and therefore will not be described.

[0102] The sensor 31 acquires sensor information indicating the state of the AGV 3. As an example, the sensor 31 is an acceleration sensor that acquires sensor information indicating the acceleration of the AGV 3.

[0103] The encoder 32 outputs the sensor information acquired by the sensor 31 to the control unit 20 as an electrical signal.

[0104] The motor 33 is an electric motor that drives the AGV 3. As an example, the motor 33 drives the AGV 3 based on a command from the execution unit 22 of the control unit 20.

[0105] (Processing and effects performed by AGV3) The AGV 3 performs reinforcement learning using the state of the AGV 3 itself acquired by the sensor 31 and the encoder 32, the information observed by the observation equipment 58, and the information of the map shown in FIG. 6 as a state s.

[0106] 6, if AGV3 tries to reach destination point 52 in a short time, there is a high risk of hitting obstacle 53 and receiving a negative reward. In this case, the variance of the reward that AGV3 receives increases. Also, if AGV3 tries to reach destination point 52 via shortest route 54, there is a high risk of hitting obstacle 53A by taking route 55 due to an error.

[0107] However, the AGV 3 performs reinforcement learning taking such risks into consideration. That is, the AGV 3 does not learn to increase only the expected value of the reward, but learns a route to the destination point 52 taking into consideration the reward for reaching the destination point 52 quickly and the reward for colliding with the obstacle 53.

[0108] Therefore, AGV3 can more easily learn a strategy to follow a path such as dotted path 56 that has a high reward.

[0109] [Software implementation example] Some or all of the functions of the reinforcement learning devices 1 and 2 and the AGV 3 may be realized by hardware such as an integrated circuit (IC chip), or may be realized by software.

[0110] In the latter case, the reinforcement learning devices 1 and 2, and the AGV 3 are realized, for example, by a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter, referred to as computer C) is shown in FIG. 8. The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for operating the computer C as the reinforcement learning devices 1 and 2, and the AGV 3. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing each function of the reinforcement learning devices 1 and 2, and the AGV 3.

[0111] The processor C1 may be, for example, a central processing unit (CPU), a graphic processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a microcontroller, or a combination of these. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination of these.

[0112] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may further include a communication interface for transmitting and receiving data to and from other devices. The computer C may further include an input / output interface for connecting input / output devices such as a keyboard, a mouse, a display, and a printer.

[0113] Furthermore, the program P can be recorded on a non-transitory tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can obtain the program P via such a recording medium M. Furthermore, the program P can be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also obtain the program P via such a transmission medium.

[0114] [Additional Note 1] The present invention is not limited to the above-described embodiment, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means disclosed in the above-described embodiment are also included in the technical scope of the present invention.

[0115] [Additional Note 2] Some or all of the above-described embodiments can be described as follows. However, the present invention is not limited to the following described aspects.

[0116] (Appendix 1) A reinforcement learning device that learns a strategy for determining a next action to be taken by an agent that has performed an action based on the results of observing an environment, the reinforcement learning device comprising: a setting means for setting an initial value of a risk-corrected reward, which is a reward that takes risk into consideration; a calculation means for calculating a correction coefficient based on a predetermined parameter related to risk relative to reward and a difference between the risk-corrected reward and the reward obtained by the next action; an update means for updating the risk-corrected reward based on the risk-corrected reward, the correction coefficient, and the difference; an estimation means for estimating the risk-corrected reward by a process in which the calculation means calculates a correction coefficient using the risk-corrected reward updated by the update means, and a process in which the update means updates the risk-corrected reward using the updated risk-corrected reward and the correction coefficient; and a learning means for learning based on the estimated risk-corrected reward.

[0117] (Appendix 2) The reinforcement learning device according to claim 1, wherein the update means updates the risk-corrected reward by adding the risk-corrected reward to a value obtained by multiplying the correction coefficient by the difference.

[0118] (Appendix 3) A mobile robot that acts using a policy obtained by the reinforcement learning device described in appendix 1 or 2, or an estimated risk-corrected reward.

[0119] (Appendix 4) A reinforcement learning method comprising: a reinforcement learning device that learns a policy for determining a next action to be taken by an agent that has performed an action based on a result of observing an environment, the method comprising: setting an initial value of a risk-corrected reward, which is a reward that takes risk into consideration; calculating a correction coefficient based on a predetermined parameter related to risk relative to reward and a difference between the risk-corrected reward and the reward obtained by the next action; updating the risk-corrected reward based on the risk-corrected reward, the correction coefficient, and the difference; estimating a risk-corrected reward by a process of calculating a correction coefficient in the calculating step using the risk-corrected reward updated in the updating step; and a process of updating the risk-corrected reward in the updating step using the updated risk-corrected reward and the correction coefficient; and learning based on the estimated risk-corrected reward.

[0120] (Appendix 5) A computer-readable recording medium having a program recorded thereon that causes a computer to function as a reinforcement learning device that learns a strategy for determining a next action to be taken by an agent that has performed an action based on the results of observing an environment, the program causing the computer to function as: a setting means for setting an initial value of a risk-corrected reward, which is a reward that takes risk into account; a calculation means for calculating a correction coefficient based on a predetermined parameter related to risk relative to reward and a difference between the risk-corrected reward and the reward obtained by the next action; an update means for updating the risk-corrected reward based on the risk-corrected reward, the correction coefficient, and the difference; an estimation means for estimating the risk-corrected reward by a process in which the calculation means calculates a correction coefficient using the risk-corrected reward updated by the update means, and a process in which the update means updates the risk-corrected reward using the updated risk-corrected reward and the correction coefficient; and a learning means for learning based on the estimated risk-corrected reward.

[0121] [Additional Note 3] A part or all of the above-described embodiments can be further expressed as follows.

[0122] A reinforcement learning device that learns a strategy for determining a next action to be taken by an agent that has performed an action based on the results of observing an environment, the reinforcement learning device comprising: a setting process that sets an initial value of a risk-corrected reward, which is a reward that takes risk into consideration; a calculation process that calculates a correction coefficient based on a predetermined parameter related to risk relative to reward and a difference between the risk-corrected reward and the reward obtained by the next action; an update process that updates the risk-corrected reward based on the risk-corrected reward, the correction coefficient, and the difference; an estimation process that estimates a risk-corrected reward by a process that calculates a correction coefficient in the calculation process using the risk-corrected reward updated in the update process, and a process that updates the risk-corrected reward in the update process using the updated risk-corrected reward and the correction coefficient; and a learning process that learns based on the estimated risk-corrected reward.

[0123] The reinforcement learning device may further include a memory, and the memory may store a program for causing the processor to execute the setting process, the calculation process, the update process, the estimation process, and the learning process. The program may be recorded in a computer-readable, non-transitory, tangible recording medium. [Explanation of symbols]

[0124] 1, 2 Reinforcement learning device 3. AGV 11 Setting section 12 Calculation section 13 Update section 14 Estimation part 15 Learning Department 21 Policy Decision-Making Department 22 Executive Department

Claims

1. A reinforcement learning device that learns a policy for determining a next action to be taken by an agent that has performed an action based on a result of observing an environment, comprising: A setting means for setting an initial value of a risk-adjusted reward, which is a reward taking into account risk; A calculation means for calculating a correction coefficient based on a predetermined parameter related to risk relative to reward, the risk-corrected reward, and the difference between the reward obtained by the next action; An update means for updating the risk-corrected reward based on the risk-corrected reward, the correction coefficient, and the difference; An estimation means for estimating a risk-corrected reward by a process in which the calculation means calculates a correction coefficient using the risk-corrected reward updated by the update means, and a process in which the update means updates the risk-corrected reward using the updated risk-corrected reward and the correction coefficient; learning means for learning based on the estimated risk-adjusted reward; A reinforcement learning device comprising:

2. The update means updates the risk-corrected reward by adding the risk-corrected reward to a value obtained by multiplying the correction coefficient by the difference. The reinforcement learning device according to claim 1 .

3. A mobile robot that acts using a policy obtained by the reinforcement learning device according to claim 1 or an estimated risk-corrected reward.

4. A reinforcement learning device that learns a policy for determining a next action to be taken by an agent that has performed an action based on a result of observing an environment, Setting an initial value for risk-adjusted reward, which is reward taking into account risk; Calculating a correction factor based on a predetermined parameter related to risk to reward and a difference between the risk-corrected reward and the reward obtained by the next action; updating a risk-corrected compensation based on the risk-corrected compensation, the compensation factor, and the difference; estimating a risk-corrected compensation by calculating a compensation factor in the calculating step using the updated risk-corrected compensation in the updating step, and updating the risk-corrected compensation in the updating step using the updated risk-corrected compensation and the compensation factor; learning based on the estimated risk-adjusted reward; Reinforcement learning methods including

5. A program that causes a computer to function as a reinforcement learning device that learns a strategy for determining a next action to be taken by an agent that has performed an action based on a result of observing an environment, comprising: The computer, A setting means for setting an initial value of a risk-adjusted reward, which is a reward taking into account risk; A calculation means for calculating a correction coefficient based on a predetermined parameter related to risk relative to reward, the risk-corrected reward, and the difference between the reward obtained by the next action; An update means for updating the risk-corrected reward based on the risk-corrected reward, the correction coefficient, and the difference; An estimation means for estimating a risk-corrected reward by a process in which the calculation means calculates a correction coefficient using the risk-corrected reward updated by the update means, and a process in which the update means updates the risk-corrected reward using the updated risk-corrected reward and the correction coefficient; learning means for learning based on the estimated risk-adjusted reward; A program that functions as a

Images