Reinforcement learning device, reinforcement learning method, and program

Abstract

This reinforcement learning device learns a policy for determining the next action of an agent on the basis of an environmental state and a reward earned by the agent for the action thereof, the reinforcement learning device comprising: an estimation unit that uses, as an initial model, a base model which estimates an expected value of a reward and which does not take risk into consideration, to sequentially estimate a risk-corrected reward, which is a reward that takes risk into consideration; and a learning unit that sequentially learns an agent policy on the basis of the risk-corrected reward.

Description

Reinforcement learning device, reinforcement learning method, and program

[0001] This disclosure relates to a reinforcement learning device, a reinforcement learning method, and a program.

[0002] Reinforcement learning is a well-known technique. Reinforcement learning is a technique that learns a policy, which is a guideline for determining the next action of an agent, based on the state observed from the environment and the reward that can be obtained from the agent's actions. In reinforcement learning, for example, the policy is updated to increase the expected value of the reward in response to the environmental changes caused by the agent's actions. Risk-aware reinforcement learning, which takes risk into account, is also known (for example, Patent Document 1).

[0003] Japanese Patent Publication No. 2014-130520

[0004] In the risk-aware reinforcement learning described in Patent Document 1, etc., there is a problem in that the computational cost increases when changing the degree of risk or when wanting to run multiple reinforcement learning programs with different degrees of risk, because the evaluation function takes risk into account.

[0005] This disclosure has been made in view of the above-mentioned issues, and one exemplary purpose is to provide a technique that can suppress the increase in computational cost in risk-aware reinforcement learning.

[0006] An exemplary aspect of the present disclosure is a reinforcement learning device that learns a policy for determining the next action of an agent based on the state of the environment and the reward obtained from the agent's actions, and comprises: estimation means for sequentially estimating a risk-adjusted reward, which is a reward that takes risk into account, using a base model that estimates the expected value of the reward and does not take risk as an initial model; and learning means for sequentially learning the agent's policy based on the risk-adjusted reward.

[0007] An exemplary aspect of the present disclosure relates to a reinforcement learning method in which at least one processor learns a policy for determining the next action of an agent based on the state of the environment and the rewards obtained from the agent's actions, the method comprising: the processor sequentially estimating a risk-adjusted reward, which is a reward that takes risk into account, using a base model that estimates the expected value of the reward and does not take risk as an initial model; and sequentially learning a policy for the agent based on the risk-adjusted reward.

[0008] A program relating to an illustrative aspect of this disclosure is a program that causes a computer to perform reinforcement learning to learn a policy for determining the next action of an agent based on the state of the environment and the reward obtained from the agent's actions, and causes the computer to perform an estimation process that sequentially estimates a risk-adjusted reward, which is a reward that takes risk into account, using a base model that estimates the expected value of the reward and does not take risk into consideration as an initial model, and a learning process that sequentially learns a policy for the agent based on the risk-adjusted reward.

[0009] According to an illustrative aspect of this disclosure, it is possible to suppress the increase in computational cost in risk-aware reinforcement learning.

[0010] This is a block diagram showing the configuration of the reinforcement learning device related to this disclosure. This is a flowchart showing the flow of the reinforcement learning method related to this disclosure. This is a block diagram showing the configuration of the reinforcement learning device related to this disclosure. This is a diagram for explaining the processing performed by the reinforcement learning device related to this disclosure. This is a diagram for explaining the processing performed by the reinforcement learning device related to this disclosure. This is a block diagram showing the configuration of the reinforcement learning device related to this disclosure. This is a block diagram showing a specific configuration example of the reinforcement learning device related to this disclosure. This is a flowchart showing a specific processing example of the reinforcement learning device related to this disclosure. This is a block diagram showing a specific configuration example of the reinforcement learning device related to this disclosure. This is a flowchart showing a specific processing example of the reinforcement learning device related to this disclosure. This is a block diagram showing the configuration of a computer that functions as a reinforcement learning device related to this disclosure.

[0011] The following are examples of embodiments of the present invention. However, the present invention is not limited to the exemplary embodiments shown below, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining some or all of the technologies (things or methods) employed in each of the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, embodiments obtained by appropriately omitting some of the technologies employed in each of the exemplary embodiments shown below may also be included in the scope of the present invention. In addition, the effects mentioned in each of the exemplary embodiments shown below are examples of effects that can be expected in that exemplary embodiment and do not define the scope of the present invention. That is, embodiments that do not produce the effects mentioned in each of the exemplary embodiments shown below may also be included in the scope of the present invention.

[0012] [First Exemplary Embodiment] A first exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. This exemplary embodiment is the basic form for each of the exemplary embodiments described later. The scope of application of each technology adopted in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technology adopted in this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems occur. Furthermore, each technology shown in the drawings referenced to explain this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems occur.

[0013] (Configuration of the Reinforcement Learning Device) The configuration of the reinforcement learning device 1 according to this exemplary embodiment will be described with reference to Figure 1. The reinforcement learning device 1 is a device that learns a policy for determining the next action of an agent based on the state of the environment and the rewards obtained from the agent's actions. Here, the state of the environment may also include the state of the agent. Figure 1 is a block diagram showing the configuration of the reinforcement learning device 1. As shown in Figure 1, the reinforcement learning device 1 comprises an estimation unit 11 and a learning unit 12.

[0014] The estimation unit 11 uses a base model that estimates the expected value of the reward but does not consider risk as an initial model, and sequentially estimates the risk-adjusted reward, which is the reward that takes risk into account. Then, the learning unit 12 sequentially learns the agent's policy based on the risk-adjusted reward.

[0015] As an example, in the first step of the target episode, the estimation unit 11 acquires information indicating the state of the environment and the reward after the agent's actions in that step have been performed, and uses this information to estimate the risk-adjusted reward using the base model as the initial model. This process may also be described as a process that sequentially estimates the risk-adjusted reward, which is the reward that takes risk into account, by referring to the expected value of the reward estimated by the base model, which does not consider risk, as the initial value.

[0016] The estimation unit 11 or the learning unit 12 then determines the agent's next action by referring to the risk-adjusted reward. The learning unit 12 then acquires information indicating the state of the environment and the reward after the agent's next action has been performed, and uses this information to train the target model. Here, the target model is a model that is sequentially trained using the base model as the initial model and referring to the risk-adjusted reward. The target model may also be referred to as a risk-sensitive model, but this term is not limited to this exemplary embodiment.

[0017] The specific estimation process for the risk-adjusted reward described above is not limited to this exemplary embodiment, but as an example, it is estimated from the state s using the state value function V(s). Here, the state value function V(s) represents the expected value of the cumulative reward when the initial state is s under the above policy π.

[0018] In a learning process that considers risk, the state value function V(s) is updated, for example, by referring to a non-zero risk parameter that indicates the degree to which risk is considered, and the risk-adjusted reward is estimated by referring to this state value function V(s). The target model (risk-sensitive model) is sequentially learned by such a learning process, for example.

[0019] On the other hand, in a learning process that does not consider risk, the state value function V(s) is updated, for example, without referring to the risk parameter, or by setting the risk parameter to 0, and the risk-free reward is estimated by referring to this state value function V(s). The base model described above is a model learned by such a learning process, as an example.

[0020] (Effects of the reinforcement learning device) As described above, the reinforcement learning device 1 is a reinforcement learning device that learns a policy for determining the agent's next action based on the state of the environment and the reward obtained from the agent's actions. The reinforcement learning device 1 employs the following configuration: - A base model that estimates the expected value of the reward and does not consider risk is used as the initial model to sequentially estimate a risk-adjusted reward, which is a reward that takes risk into account. - The agent's policy is sequentially learned based on the risk-adjusted reward. In this way, the reinforcement learning device 1 uses the base model as the initial model to estimate a risk-adjusted reward and learns the agent's policy based on the risk-adjusted reward, so it is possible to suppress the increase in computational cost in reinforcement learning that takes risk into account.

[0021] (Flow of Reinforcement Learning Method) The flow of the reinforcement learning method S1 according to this exemplary embodiment will be explained with reference to Figure 2. Figure 2 is a flowchart showing the flow of the reinforcement learning method S1. The reinforcement learning method S1 is a reinforcement learning method in which at least one processor learns a policy for determining the next action of the agent based on the state of the environment and the reward obtained from the agent's actions. Here, the state of the environment may also include the state of the agent. As shown in Figure 2, the reinforcement learning method S1 includes a process (step) S11 for estimating a risk-adjusted reward and a process (step) S12 for learning the agent's policy.

[0022] (Step S11) In step S11, the estimation unit 11 estimates a risk-adjusted reward, which is a reward that takes risk into account. Here, in the first step of the target episode, in order to estimate the risk-adjusted reward, a base model that estimates the expected value of the reward but does not take risk into consideration is used as the initial model. In addition, in this process, as an example, the expected value of the reward estimated by the base model that does not take risk into consideration may be referenced as the initial value.

[0023] As an example, in the first step of the target episode, the estimation unit 11 acquires information indicating the state of the environment and the reward after the agent's actions in that step have been performed, and uses this information to estimate the risk-adjusted reward using the base model as the initial model.

[0024] Furthermore, the estimation unit 11, as an example, acquires information indicating the state of the environment and the reward after the agent's actions in each step of the target episode, other than the first step, and uses this information to estimate the risk-adjusted reward using the target model. Here, the target model is a model that is sequentially trained using the base model as the initial model and referring to the risk-adjusted reward, as described above.

[0025] Then, the estimation unit 11 or the learning unit 12 refers to the risk-adjusted reward and decides what the agent should do in the next step.

[0026] (Process S12) Next, in process S12, the learning unit 12 learns the agent's strategy based on the risk-adjusted reward estimated in process S12. As an example, the learning unit 12 acquires information indicating the state of the environment and the reward after the agent's action in the next step has been performed, and uses this information to train the target model. The specific processing by the learning unit 12 has been described above, so it will not be explained here.

[0027] After processing in step S12, for example, it is determined whether the last step of the target episode has been completed. If it is determined that it has been completed, the processing related to the reinforcement learning method S1 is completed; otherwise, the process returns to step S11 and is repeated. Here, in steps other than the first step of the target episode, the risk-adjusted reward is estimated in step S11 by referring to the target model.

[0028] (Effects of the reinforcement learning method) As described above, the reinforcement learning method S1 is a reinforcement learning method in which at least one processor learns a policy for determining the agent's next action based on the state of the environment and the reward obtained from the agent's actions. In the reinforcement learning method S1, the processor employs the following configuration: - Uses a base model that estimates the expected value of the reward and does not consider risk as an initial model to sequentially estimate a risk-adjusted reward, which is a reward that takes risk into account; and - Sequentially learns the agent's policy based on the risk-adjusted reward. The above configuration produces the same effects as the reinforcement learning device 1.

[0029] [Second Exemplary Embodiment] A second exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same function as those described in the above-described exemplary embodiment are denoted by the same reference numerals, and their descriptions are omitted as appropriate. The scope of application of each technology adopted in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technology adopted in this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems arise. Furthermore, each technology shown in the drawings referenced to describe this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems arise.

[0030] <Overview of Information Processing Device> The information processing device 1A relating to this disclosure is a device that performs reinforcement learning by sequentially updating a policy π for determining the agent's next action a, based on the state s of the environment and the reward x obtained from the agent's action a. The information processing device 1A is an example of a reinforcement learning device relating to this disclosure. The information processing device 1A uses a base model that estimates the expected value Q of the reward and does not consider risk as an initial model, and performs reinforcement learning to construct a risk-sensitive model that considers the variability of the reward x as a risk. Here, "a model that does not consider risk" refers to a model constructed by learning that does not consider statistics related to the reward x, as an example, and "a model that considers risk" refers to a model constructed by learning that considers said statistics, as an example.

[0031] As an example, the information processing device 1A performs the following processes: (Process 1) constructs a base model for estimating the expected value Q of the reward through a learning process that does not consider risk, and (Process 2) updates a risk-sensitive model (target model) using the above base model as the initial model, by referring to the risk parameter β related to risk. The specific processes will be described later, but (Process 2) is performed by (Process 2a) updating the risk-sensitive model using the above base model as the initial model, by referring to the expected value Q of the reward and the variance σ of the reward. ２ , and using the risk parameter β related to risk, the risk-adjusted reward Q + βσ ２ This can be described as a process of training the model to maximize the value, or (process 2b) a process of training a risk-sensitive model using the above base model as the initial model by updating the state value function V(s) that represents the risk-adjusted reward using the risk parameter β. The information processing device 1A can be used, for example, for controlling mobile robots, scheduling production and inventory management, simulating trade and games, etc.

[0032] <Configuration of the Information Processing System> The configuration of the information processing system 100A, which includes an information processing device 1A, will be described with reference to Figure 3. Figure 3 is a block diagram showing the configuration of the information processing system 100A. As shown in Figure 3, the information processing system 100A includes an information processing device 1A and a terminal device 2A. Also, as shown in Figure 3, the information processing device 1A and the terminal device 2A are configured to communicate with each other via a network N. Here, the specific configuration of the network N is not limited to this exemplary embodiment, but as an example, a wireless LAN (Local Area Network), a wired LAN, a WAN (Wide Area Network), a public telephone network, a mobile data communication network, or a combination of these networks can be used.

[0033] <Configuration of the terminal device> As shown in Figure 3, the terminal device 2A comprises a control unit 20A, an execution unit 21, and a communication unit 24.

[0034] (Communication Unit 24) The communication unit 24 communicates with devices outside the terminal device 2A. For example, the communication unit 24 communicates with the information processing device 1A. The communication unit 24 transmits data supplied from the control unit 20A to the information processing device 1A, and supplies data received from the information processing device 1A to the control unit 20A.

[0035] (Execution Unit 21) The execution unit 21 executes action a specified by the policy π derived by the information processing device 1A. The execution unit 21 is an example of an agent that executes action a. The execution unit 21 can be implemented, for example, as a drive unit for a robot to be controlled, an ordering unit that performs production and inventory management, or an execution unit that performs trade and game simulations, but these examples do not limit the present exemplary embodiment.

[0036] Furthermore, the execution unit 21 also functions as an observation unit that observes the state s of the target environment before and after the execution of the above action a. The target environment refers, for example, to the environment in which the agent (the execution unit 21) is placed. The state s of the environment may also include the state of the execution unit 21 itself. The observation unit can be configured, for example, by one or more sensors or an API (Application Programming Interface) for acquiring data, but these examples do not limit this exemplary embodiment.

[0037] (Control Unit 20A) As shown in Figure 3, the control unit 20A includes a reward acquisition unit 22 and a reward provision unit 23. The reward acquisition unit 22 acquires information about the reward x obtained as a result of the action a performed by the execution unit 21. The reward provision unit 23 provides the information about the reward x acquired by the reward acquisition unit 22 to the information processing device 1A via the communication unit 24. Here, the information that the reward provision unit 23 provides to the information processing device 1A may also include information indicating the state s of the environment acquired by the execution unit 21.

[0038] <Configuration of the Information Processing Device> As shown in Figure 3, the information processing device 1A comprises a control unit 10A, a storage unit 17A, a communication unit 30A, and an input / output unit 40A.

[0039] (Communication Unit 30A) The communication unit 30A communicates with external devices of the information processing device 1A via a communication line. The specific configuration of the communication line is not limited to this exemplary embodiment, but examples of communication lines include wireless LAN (Local Area Network), wired LAN, WAN (Wide Area Network), public network, mobile data communication network, or a combination thereof. The communication unit 30A transmits data supplied from the control unit 10A to other devices and supplies data received from other devices to the control unit 10A.

[0040] (Input / Output Unit 40A) Input / output devices such as keyboards, mice, displays, printers, and touch panels are connected to the input / output unit 40A. The input / output unit 40A receives various types of information from the connected input devices to the information processing device 1A. The input / output unit 40A also outputs various types of information to the connected output devices under the control of the control unit 10A. An interface such as USB (Universal Serial Bus) can be used as the input / output unit 40A.

[0041] (Storage Unit 17A) The storage unit 17A stores various data referenced by the control unit 10A, and various data derived by the control unit 10A. For example, the storage unit 17A stores: Episode information EPI, Base model BM, Risk parameter RP, Reference information RI, and Risk-sensitive model RSM.

[0042] (Episode Information EPI) Episode information EPI includes the environmental state s, reward x, and action a at each step of the target episode. Episode information EPI may also include the risk-adjusted reward ν at each step of the target episode. The term "episode information" is used for convenience only and does not limit this exemplary embodiment.

[0043] (State s) State s is, in part, the state of the environment as described above. State s may also be described as a state observed from the environment. For example, in the task of autonomous play of a computer game, state s includes the state of an object that affects the progress of the game. Another example is that in the task of selecting the transport operation of a transport device, state s includes the position, direction of movement, speed, angular velocity of the transport device, the position of the passage, or the position and speed of a static or dynamic obstacle. Another example is that in the task of selecting the excavation operation of an excavator, state s includes the attitude, position of the excavator that excavates soil, the shape of the soil to be excavated, or the amount of soil in the excavator's bucket. However, state s is not limited to the examples described above and may include other states. State s may include, for example, the state of the environment such as temperature or weather.

[0044] (Action a) Action a is the action of an agent in reinforcement learning, as partially described above. Examples include autonomous gameplay control of a computer game, drilling operation control of an excavator, or transport operation control of a transport device. However, actions are not limited to these examples and may include others.

[0045] (Reward x) Reward x is the reward that the agent receives for taking action. Reward x may be, for example, a value added or subtracted depending on the outcome of a game, the score of the game, the amount excavated by the excavator, the time taken for excavation, the time taken for transport, or whether or not an obstacle was encountered during transport. However, reward x is not limited to these examples and may be other than those listed above.

[0046] (Risk-Adjusted Reward) The risk-adjusted reward ν is a reward that takes risk into account, as partially described above, and is calculated from state s using the state value function V(s). The estimation process of the risk-adjusted reward ν by the reward estimation unit 111 will be described later.

[0047] (Base Model BM) The base model BM is a model that estimates the expected value Q of the reward, and is constructed by a learning process that does not consider risk. The base model BM can also be described as a model constructed by a learning process in which the risk parameter β, which indicates the degree to which risk is considered, is set to 0.

[0048] (Risk Parameter RP) The risk parameter RP is a parameter that indicates the extent to which a risk is considered in a learning process that takes risk into account, and in this exemplary embodiment, it is also referred to as the risk parameter β. The risk parameter β can also be described as a parameter related to the variability of rewards. For example, if you want to avoid risk by suppressing the variability of the rewards you receive, you can set a negative value for the risk parameter β. On the other hand, if you want to increase the risk to favor a behavior, you can set a positive value for the risk parameter β. In other words, the smaller the value of the risk parameter β, the smaller the risk can be, that is, the more the variability of rewards can be suppressed.

[0049] (Reference Information RI) Reference Information RI includes various types of information referenced in the model learning (update) process by the control unit 10A. For example, Reference Information RI includes the value of parameter C, which is referenced to set the range of the risk parameter β. The value of parameter C may be predetermined, or it may be determined based on instructions input by the user via the input / output unit 40A. The process for setting the range of the risk parameter β will be described later.

[0050] (Risk-Sensitive Model RSM) The Risk-Sensitive Model RSM is a reinforcement learning model that uses the base model BM as the initial model and is sequentially updated through reinforcement learning that takes risk into account. The Risk-Sensitive Model RSM is sometimes referred to as the target model. The learning (updating) process of the Risk-Sensitive Model RSM will be described later.

[0051] (Control Unit 10A) As shown in Figure 3, the control unit 10A includes an estimation unit 11, a learning unit 12, a range determination unit 13, an episode information management unit 14, a parameter management unit 15, and a correction coefficient calculation unit 16.

[0052] (Episode Information Management Unit 14, Parameter Management Unit 15) The Episode Information Management Unit 14 stores and reads each data included in the Episode Information EPI described above into the Storage Unit 17A. For example, the Episode Information Management Unit 14 stores the state s and reward x acquired from the terminal device 2A into the Storage Unit 17A for each step of the target episode. The Episode Information Management Unit 14 also stores the risk-corrected reward ν derived by the Control Unit 10A into the Storage Unit 17A for each step of the target episode. The Episode Information Management Unit 14 also stores the policy π derived by the Control Unit 10A, or the action a determined by the policy π, into the Storage Unit 17A for each step of the target episode. The Episode Information Management Unit 14 also functions as an acquisition unit that acquires this information stored in the Storage Unit 17A for reference by the Control Unit 10A. The Parameter Management Unit 15 stores and reads the risk parameter β into the Storage Unit 17A.

[0053] (Estimation Unit 11) As described in Exemplary Embodiment 1, the estimation unit 11 uses the base model BM as an initial model to sequentially estimate the risk-adjusted reward ν, which is a reward that takes risk into account. For example, in the first step of the target episode, the estimation unit 11 acquires information indicating the state of the environment s and the reward x in that step, and using this information as a reference, estimates the risk-adjusted reward ν using the base model BM as an initial model. In addition, in steps other than the first step of the target episode, the estimation unit 11 acquires information indicating the state of the environment s and the reward x in that step, and using this information as a reference, estimates the risk-adjusted reward using the target model (risk-sensitive model). Here, the target model is a model that is sequentially trained using the base model BM as an initial model and referring to the risk-adjusted reward ν, as described above.

[0054] In this exemplary embodiment, the estimation unit 11 estimates the risk-adjusted reward ν from state s using a state value function V(s) as an example. The state value function V(s) is the expected value of the cumulative reward when the initial state is s under policy π. In the process of updating the state value function that takes risk into account, the state value function V(s) that represents the risk-adjusted reward ν is updated (estimated) by the following equation (1) as an example. Here, α represents the learning rate, and δ represents the TD error (temporal-difference error). The equation is given by R(s), where R(s) is the reward x corresponding to state s, and γ is the discount rate (0 ≤ γ < 1). Also, σ β For example, This is the sigmoid function obtained by the following equation, where β is the risk parameter.

[0055] (Learning Unit 12) As described in Exemplary Embodiment 1, the Learning Unit 12 sequentially learns the agent's policy based on the risk-adjusted reward ν estimated (updated) by the Estimation Unit 11. As an example, the Learning Unit 12 learns a risk-sensitive model by referring to the risk-adjusted reward ν and determines a policy (action a) that maximizes the risk-adjusted reward ν.

[0056] As partially mentioned above, the learning process of the base model BM can also be understood by referring to the above explanation. That is, in the learning process of the base model BM, the estimation unit 11 only needs to update the state value function V(s) that defines the base model BM by setting β = 0 in equations 1 to 3 above. The state value function V(s) thus updated (generated) has meaning as the expected value of the cumulative reward when risk is not considered. Then, the learning unit 12 refers to the state value function V(s) and sequentially learns the agent's policy when risk is not considered.

[0057] Figure 4A is a diagram illustrating, in this exemplary embodiment, a plurality of risk-sensitive models (Models 1 to 3 in Figure 4A) obtained by processing by the estimation unit 11 and the learning unit 12, and the computational cost required to construct each risk-sensitive model.

[0058] Figure 4B is a diagram illustrating, in part, multiple risk-sensitive models (Models 1 to 3 in Figure 4B) and the computational costs required to construct each risk-sensitive model using the process described in the comparative example.

[0059] In Figures 4A and 4B, Model 1 has a risk parameter β １ This shows a risk-sensitive model defined by, and Model 2 has a risk parameter β ２ This shows a risk-sensitive model defined by, and Model 3 has a risk parameter β ３ This shows a risk-sensitive model defined by [the specified formula / method]. Furthermore, in Figures 4A and 4B, the length of each arrow schematically indicates the magnitude of the computational cost.

[0060] As shown in Figure 4A, in the process according to this exemplary embodiment, first, a base model BM is learned (generated) without considering risk parameters. Then, using the base model BM as the initial model, models 1 to 3 are learned (generated) by the learning process performed by the estimation unit 11 and the learning unit 12.

[0061] On the other hand, as shown in Figure 4B, in the comparative example, models 1 to 3 are learned (generated) separately from the beginning. Therefore, the computational cost of the comparative example is higher than that of the process according to this exemplary embodiment. In other words, in the process according to this exemplary embodiment, the computational cost can be reduced compared to the process according to the comparative example when generating multiple risk-sensitive models.

[0062] Figure 5 shows another example of the generation of multiple risk-sensitive models by the estimation unit 11 and the learning unit 12 according to this exemplary embodiment. As shown in Figure 4A, the estimation unit 11 and the learning unit 12 can generate multiple risk-sensitive models using a base model as an initial model, and these generated risk-sensitive models can also be used as initial models for learning (generating) other risk-sensitive models.

[0063] As an example, in the example shown in Figure 5, a base model (β=0) is first learned (generated) without considering the risk parameter. Then, by using this base model (β=0) as the initial model, the estimation unit 11 and the learning unit 12 learn (generate) models 1 and 2. Here, model 1 is a risk-sensitive model defined by the risk parameter β=1, and model 2 is a risk-sensitive model defined by the risk parameter β=-1.

[0064] Then, the estimation unit 11 and the learning unit 12 learn (generate) model 11 using model 1 as the initial model, and learn (generate) model 21 using model 2 as the initial model. Here, model 11 is a risk-sensitive model defined by the risk parameter β = 2, and model 21 is a risk-sensitive model defined by the risk parameter β = -2.

[0065] Thus, in this exemplary embodiment, the estimation unit 11 may adopt a configuration in which it uses a trained model (e.g., models 1 and 2) as an initial model, which is a model for estimating the expected value of the reward and has a risk parameter of a certain value (e.g., β = 1, -1) set, to sequentially estimate a risk-adjusted reward according to a risk parameter of a different value (e.g., β = 2, -2), and the learning unit 12 sequentially learns the agent's policy based on the risk-adjusted reward (trains a risk-sensitive model).

[0066] It can also be understood that the processing by the estimation unit 11 and the learning unit 12 learns (generates) one or more risk-sensitive models through additional learning based on the initial model. However, in this exemplary embodiment, as an example, a risk-sensitive model is learned (generated) by changing only the risk parameter without changing the settings related to the environment and reward. In other words, a risk-sensitive model is learned (generated) by changing only the objective function (for example, the state value function V(s) described above) without changing the settings related to the environment and reward. In addition to the effect of reducing computational cost as described above, such a learning process has the advantages of: - A suitable risk-sensitive model can be constructed simply by changing the parameters. - Since it starts from a state value function V(s) that accurately gives the expected value of cumulative reward in a situation where risk is not considered, the error in the learning process can be kept small.

[0067] (Range Determination Unit 13) Returning to Figure 3, the range determination unit 13 provided in the control unit 10A determines the range of the risk parameter β that the estimation unit 11 refers to in order to estimate the risk-corrected reward. For example, the range determination unit 13 may determine the range of the risk parameter β by referring to the variance of the update target quantity at the end of learning (generation) of the base model BM. For example, the range determination unit 13 may be configured to determine the range of the risk parameter by referring to the variance of the reward derived by the base model BM, which does not take risk into consideration.

[0068] More specifically, the range determination unit 13 may be configured to determine the range of the risk parameter β by |β| < √(2V)logC so that the absolute value of 1 / (1 - exp(β / √(2V))) becomes sufficiently large. Here, V represents the variance of the update amount (reward) at the end of the learning of the base model BM. Also, the parameter C is a parameter referred to for setting the range of the above-described risk parameter β. The specific value of the parameter C does not limit this exemplary embodiment, but as an example, C = 10 6 It can be set to a value such as (10 to the power of 6).

[0069] By the range determination unit 13 performing the above-described processing, in the update process of the state value function referring to the risk parameter β, the value of the risk parameter β can be preferably set. Thereby, it is possible to prevent a situation where the update amount (change amount) becomes extremely close to 0 and the learning process does not progress accordingly.

[0070] <Example> Using the above-described information processing apparatus 1A, verification of the effect was performed under the following settings. - Game settings: The enemy character and the self character repeat passing so as not to drop the ball. - The enemy character is controlled by an enemy agent of an RNN model having 120 parameters. - The self agent acquires, as the state s, observation values of a total of 12 dimensions including the xy coordinates and the xy-direction speeds of the self character, the enemy character, and the ball, and performs an output in any of the left, up, and right directions as the control of the self character referring to the state s. - As the base model, a base model generated by learning for 1400M steps (1400 × 10 ６ steps) without considering risk was used. - Using the risk-corrected reward set with the risk parameter β = -1 (risk-averse type) with the base model as the initial model, a risk-sensitive model was generated by learning for 24M steps (24 × 10 ６ steps).

[0071] As a result, the score for the base model was 1.48 ± 1.292, while the score for the risk-sensitive model was approximately 0.68 ± 0.87, indicating a reduction in the variance of the score (risk avoidance). Furthermore, while the base model required 1400 million steps to train, it was found that a suitable risk-sensitive model could be generated with an additional 24 million steps of training.

[0072] (Effects of Information Processing Device 1A) As described above, the Information Processing Device 1A is a reinforcement learning device that learns a policy for determining the agent's next action based on the state of the environment and the reward obtained from the agent's actions. The Information Processing Device 1A employs the following configuration: - A base model BM that estimates the expected value of the reward and does not consider risk is used as the initial model to sequentially estimate a risk-adjusted reward ν, which is a reward that takes risk into account. - The agent's policy is sequentially learned based on the risk-adjusted reward ν. In this way, the reinforcement learning device 1A uses the base model BM as the initial model to estimate a risk-adjusted reward ν and learns the agent's policy based on the risk-adjusted reward ν, thus suppressing the increase in computational cost in reinforcement learning that takes risk into account.

[0073] Furthermore, as described above, in the information processing device 1A, the estimation unit 11 determines the range of the risk parameter β that is referenced to estimate the risk-corrected reward ν. Therefore, in the update process of the state value function that references the risk parameter β, the value of the risk parameter β can be set to a suitable value. This prevents the update amount (change amount) from becoming very close to zero, which would prevent the learning process from progressing.

[0074] (Additional notes regarding exemplary embodiment 2) The estimation unit 11, the learning unit 12, or the range determination unit 13 may be configured to present (display) to the user the results of the above-described processing or a part of the information referenced in the above-described processing via a display or the like provided by the input / output unit 40A. For example, the estimation unit 11 or the learning unit 12 may present to the user via the input / output unit 40A the following: (1) parameters defining the base model BM; (2) various values ​​derived from the base model BM (including the expected value of the reward); (3) information identifying other models referenced as initial models in the learning of a certain model. For example, in the case of (3) above, the schematic diagrams shown in Figures 4A and 5 may be presented to the user.

[0075] With this configuration, the estimation unit 11, the learning unit 12, or the range determination unit 13 can suitably inform the user of the results of the above-described processing or the information referenced in the above-described processing.

[0076] [Third Exemplary Embodiment] A third exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same function as those described in the above-described exemplary embodiment are denoted by the same reference numerals, and their descriptions are omitted as appropriate. The scope of application of each technology adopted in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technology adopted in this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems arise. Furthermore, each technology shown in the drawings referenced to describe this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems arise.

[0077] (Configuration of Information Processing System 100B) The configuration of the information processing system 100B according to this exemplary embodiment will be described with reference to Figure 6. Figure 6 is a block diagram showing the configuration of the information processing system 100B. As shown in Figure 6, the information processing system 100B differs from the configuration described in exemplary embodiment 2 in that the control unit 10B of the information processing device 1B includes a correction coefficient calculation unit 16. The storage unit 17B of the information processing device 1B stores the correction coefficient MC calculated by the correction coefficient calculation unit 16. Each unit shown in Figure 6 may perform the processing already described in exemplary embodiment 1 or exemplary embodiment 2, or it may perform the processing described below in place of, or as a more specific processing of, the processing already described in exemplary embodiment 1 or exemplary embodiment 2. In the following description, redundant explanations of matters already described may be omitted.

[0078] (Configuration Example 1) Figure 7 is a block diagram showing an example of the functional configuration and processing flow of the information processing device 1B according to this exemplary embodiment. Note that the unidirectional arrows in Figure 7 simply indicate the direction of the flow of a signal (data) and do not exclude bidirectionality. As shown in Figure 7, the information processing device 1B according to this configuration example includes a hyperparameter holding unit 122, a correction coefficient calculation unit 112, a reward estimation unit 111, an episode holding unit 121, and a policy learning unit 113. In Figure 7, the reward estimation unit 111 corresponds to the estimation unit 11, and is therefore also referred to as the reward estimation unit 111 (11). The policy learning unit 113 corresponds to the learning unit 12, and is therefore also referred to as the policy learning unit 113 (12). The episode holding unit 121 corresponds to the episode information management unit 14, and is therefore also referred to as the episode holding unit 121 (14). The hyperparameter holding unit 122 corresponds to the parameter management unit 15, and is therefore also referred to as the hyperparameter holding unit 122 (15). Since the correction coefficient calculation unit 122 has a configuration corresponding to the correction coefficient calculation unit 16, it is also referred to as the correction coefficient calculation unit 122(16).

[0079] (Reward Estimation Unit and State Value Function) Similar to Exemplary Embodiments 1 and 2, the reward estimation unit 111 uses a base model that estimates the expected value of the reward but does not consider risk as an initial model, and sequentially estimates the risk-adjusted reward, which is the reward that takes risk into account. For example, the reward estimation unit 111 obtains the reward x and state s from a terminal device 2A that functions as a simulator. The reward estimation unit 111 also estimates the risk-adjusted reward ν from state s using the state value function V(s). The state value function V(s), as explained in Exemplary Embodiment 2, is the expected value of the cumulative reward when the initial state is s under policy π. For example, the state value function V(s) is updated by the following equation (A1). Here, α is the learning rate and δ is the TD error. The TD error δ can be calculated, for example, by the following equation (A2). Here, R(s) is the reward released in state s, and γ is the discount rate.

[0080] Furthermore, in equation (A1), the correction coefficient ξ β (δ) is a correction coefficient used to update the state value function V(s), and is calculated, for example, using a sigmoid function. Correction coefficient ξ β (δ) is, for example, That is the case.

[0081] Furthermore, the reward estimation unit 111 calculates the discounted reward sum r. As an example, after the end of one episode, the reward estimation unit 111 calculates r ｔ +γr ｔ＋１ +γ ２ r ｔ＋２ The discounted reward sum r is calculated by calculating +...

[0082] (Correction coefficient calculation unit and correction coefficient) The correction coefficient calculation unit 112 uses the risk-adjusted reward ν and discounted reward sum r estimated by the reward estimation unit 111, as well as the state s, to calculate the correction coefficient ξ β Calculate (A'). Correction coefficient ξ β (A') is a correction coefficient used to adjust the advantage function A(s, a), described later, to account for risk. Correction coefficient ξ β (A') is the correction coefficient ξ used to update the state value function V(s) described above. β(δ) may be calculated using the same function. For example, the correction coefficient calculation unit 112 uses a sigmoid function that includes the risk parameter β as a coefficient to calculate the correction coefficient ξ from the pre-correction advantage function A' described later. β Calculate the correction factor ξ. β (A') can be expressed, for example, by the following formula. The unadjusted advantage function A' is a function that indicates how good an action performed is, calculated using the risk-adjusted reward ν. The unadjusted advantage function A' is an example of the first advantage function relating to this disclosure.

[0083] (Specific example of the unadjusted advantage function 1) As an example, the unadjusted advantage function A' is the difference (r - V) between the discounted sum of rewards r up to the last step in an episode and the estimated risk-adjusted reward V.

[0084] (Specific example of unadjusted advantage 2) Also, as an example, the unadjusted advantage function A' is calculated by taking the discounted sum of rewards in k steps as r ｋ γ is the discount rate, and s is the state after the transition in step t. ｔ , state s ｔ The state value function is V(s ｔ When ), it is expressed by the following formula: r ｋ +γ ｋ V(S) ｔ＋ｋ )-V(s ｔ ) (Specific example of uncorrected advantage 3) Also, the uncorrected advantage function A' may be, for example, the double discounted sum of TD errors. The double discounted sum is the double discounted sum used in the GAE (Generalized Advantage Estimator) method.

[0085] (Policy Learning Unit) The policy learning unit 113 sequentially learns the agent's policy based on the risk-adjusted reward, similar to exemplary embodiments 1 and 2. As an example, the policy learning unit 113 updates the state value function V(s) using the risk parameter β. As an example, the policy learning unit 113 updates the state value function V(s) using the above-described equation (A1).

[0086] Furthermore, the policy learning unit 113 uses the correction coefficient ξ calculated by the correction coefficient calculation unit 112. β The advantage function A(s, a) is updated using (A') and the risk-adjusted reward ν. The advantage function A(s, a) is an example of the second advantage function relating to this disclosure. The policy learning unit 113 also updates the policy π(s, a) using the updated advantage function A(s, a).

[0087] (Advantage Function) The advantage function A(s,a) is a function that indicates how good an action performed is. The advantage function A(s,a) is the difference between the action-value function Q(s,a) and the state-value function V(s). The advantage function A(s,a) can also be estimated as the actual sum of rewards minus the estimated sum of rewards, and the desired policy (a more favorable policy / desirable action) has a larger sum of reward than the estimate, i.e., a positive advantage. On the other hand, if the best policy and accurate reward estimation have already been obtained, the advantage function A(s,a) will be 0 or negative on average. In this case, an advantage function A(s,a) of 0 or negative means that there is no better action (i.e., the best action is already taken), so the ultimately desirable state is when the advantage function A(s,a) is zero or negative.

[0088] As an example, the policy learning unit 113 calculates the advantage function A(s, a) using the uncorrected advantage function A' by the following formula.

[0089] A(s, a) = ξ β (A'(s,a))・(A'(s,a)) In other words, in this example, the advantage function A(s,a) is the correction coefficient ξ, which is a value in the range of 0 to 1 obtained by transforming the uncorrected advantage function A' using the sigmoid function. β It is calculated by multiplying (A') by the uncorrected advantage function A'.

[0090] (Specific example of the advantage function 1) The advantage function A(s, a) can be expressed, for example, by the following equation: A(s, a) = ξ β(r - V) * (r - V) where r is the discounted sum of rewards up to the last step in the episode, and V is the estimated risk-adjusted reward. In other words, in this example, the advantage function A(s, a) is the adjustment coefficient ξ calculated from the unadjusted advantage function A' = r - V. β It is obtained by multiplying (r - V) by the uncorrected advantage function A' = r - V.

[0091] (Specific example of the advantage function 2) Generalizing the advantage function A(s, a) from Specific Example 1 above (to k steps instead of all steps), we get the following equation: A(s, a) = ξ β (r ｋ +γ ｋ V(S) ｔ＋ｋ )-V(s ｔ ))・(r ｋ +γ ｋ V(S) ｔ＋ｋ )-V(s ｔ )) In other words, in this example, the advantage function A(s, a) is equal to the uncorrected advantage function A' = r ｋ +γ ｋ V(S) ｔ＋ｋ )-V(s ｔ Correction coefficient ξ calculated from ) β (r ｋ +γ ｋ V(S) ｔ＋ｋ )-V(s ｔ )) The uncorrected advantage function A' = r ｋ +γ ｋ V(S) ｔ＋ｋ )-V(s ｔ It is obtained by multiplying by ).

[0092] (Specific example of the advantage function 3) Alternatively, the advantage function A(s, a) can be expressed using the following formula: A(s, a) = ξ β ([Double discounted sum of TD error δ]) *Double discounted sum of TD error δ In this case, the advantage function A(s, a) is the correction coefficient ξ calculated from the double discounted sum of TD error δ (advantage function A' before correction). β This is obtained by multiplying it by the double-discounted sum of TD errors δ (uncorrected advantage function A').

[0093] The policy learning unit 113 updates policy π using the calculated advantage function A(s, a). As an example, the policy learning unit 113 updates policy π using the commonly used policy gradient method.

[0094] <Flowchart of Reinforcement Learning Method> Next, the flowchart of the reinforcement learning method executed by the information processing device 1B in this configuration example will be explained with reference to Figure 8. Figure 8 is a flowchart showing the flowchart of the reinforcement learning method executed by the information processing device 1B. Note that some steps may be executed in parallel or in a different order.

[0095] (Step S101) In step S101, the correction coefficient calculation unit 112 acquires the risk parameter β. The correction coefficient calculation unit 112 may acquire the risk parameter β input via the input / output unit 40A. Alternatively, the correction coefficient calculation unit 112 may acquire the risk parameter β from a storage location specified by the user of the information processing device 1B (which may be within the storage device of the information processing device 1B or from a storage device outside the information processing device 1B). Alternatively, the correction coefficient calculation unit 112 may acquire the risk parameter β by receiving it from another device via the communication unit 30A.

[0096] (Steps S102 and S103) In step S102, the correction coefficient calculation unit 112 calculates the correction coefficient ξ β An initial value is set for (A'). Also, in step S103, the policy learning unit 113 initializes the value function and the policy π(s, a).

[0097] (Steps S104 to S108) In step S104, the reward estimation unit 111 sets an initial state for state s. In step S105, the reward estimation unit 111 selects action a based on policy π. In step S106, the reward estimation unit 111 obtains the reward x and the next state s' from the simulator 2. In step S107, the reward estimation unit 111 records the state s before the transition, action a, reward x, and the state s' after the transition in the episode storage unit 121. In step S108, the policy learning unit 113 updates the current state s with s'.

[0098] (Step S109) In step S109, the policy learning unit 113 determines whether state s' is a final state. If state s' is a final state (YES in step S109), the policy learning unit 113 proceeds to the process in step S110. On the other hand, if state s' is not a final state (NO in step S109), the policy learning unit 113 returns to the process in step S105.

[0099] (Steps S110 and S111) In step S110, the reward estimation unit 111 calculates the risk-adjusted reward ν from state s using the state value function V(s). In step S111, the policy learning unit 113 calculates the value estimation error function from the data held in the episode storage unit 121. The value estimation error function is calculated as follows, for example. Here, StopGrad() indicates that the variable inside the parentheses is treated as a constant when calculating the gradient. Therefore, η > 0 is the minimum rate.

[0100] (Steps S112 and S113) In step S112, the policy learning unit 113 uses the data held in the episode holding unit 121 and the correction coefficient ξ β The advantage function A(s, a) is calculated using (A'). In step S113, the policy learning unit 113 calculates the policy error term from the advantage function A(s, a). The policy error term is calculated as follows, for example: -logp*stopgrad(A(s, a))

[0101] (Step S114) In step S114, the reward estimation unit 111 updates the state value function V(s) using the value estimation error function and the policy error term. The policy learning unit 113 also updates the policy π using the value estimation error function and the policy error term.

[0102] Furthermore, the policy learning unit 113 outputs the updated policy π. The policy learning unit 113 may output policy π to an output device (display, printer, speaker, etc.) connected via the input / output unit 40A. Alternatively, the policy learning unit 113 may output policy π by writing it to a storage location specified by the user of the information processing device 1B (which may be within the storage device of the information processing device 1B or a storage device outside the information processing device 1B). Alternatively, the policy learning unit 113 may output policy π by transmitting it to another device via the communication unit 30A.

[0103] Furthermore, the policy learning unit 113 may display at least one of the state value function V(s) and the advantage function A(s,a) on the display device.

[0104] (Step S115) In step S115, the policy learning unit 113 determines whether the update termination condition is met. The update termination condition is, for example, whether the number of update steps or episodes meets a predetermined condition (for example, whether a threshold has been exceeded). If the update termination condition is met (YES in step S115), the policy learning unit 113 terminates the process. On the other hand, if the update termination condition is not met (NO in step S115), the policy learning unit 113 returns to the process in step S104.

[0105] <Regarding the validity of the advantage function> Here, the reward N(μ) sampled from a normal distribution according to the state s and action a is ｋ , σ ｋ ２ Let's consider an environment where we can approximate the result of μ. Here, μ ｋ σ is the average value, ｋ This is the standard deviation. In this case, the state value function V(S) Ｔ－１ ) is u Ｔ +β / 2・σ Ｔ ２ It converges to this. This is described in the document "Deletang, Gregoire, et al. "Model-Free Risk-Sensitive Reinforcement Learning." arXiv:2111.02907(2021)". Also, V(S Ｔ－１Assuming that ) converges, S Ｔ－１ The state one step before that, S Ｔ－２ When considering this, V(S Ｔ－２ )teeth, It will be updated, so V(S Ｔ－２ )teeth, It converges to this. Here, for simplicity, if we set the discount rate γ = 1, Therefore, due to the regeneration property of the normal distribution, V(S ｋ ) is state s ｋ This will allow us to estimate the reward distribution and expected value up to the final episode.

[0106] (Validity of Specific Example 1 of the Advantage Function) Here, we consider the validity of the case where the advantage function is Specific Example 1 as described above. In this case, due to the reproducibility of the normal distribution, the sum of rewards until the end is, It is distributed as follows. Therefore, A(s ｋ , a ｋ ) is calculated by taking x as the sample from the distribution of equation (10) above, It can be expressed as follows. Therefore, the advantage function A(s ｋ , a ｋ The expected value of ) is V(s ｋ ) = μ + β / 2 * σ ２ The estimated value is the distribution of rewards obtained from actual choices (the distribution in equation (A10) above), When it is greater than F, it becomes a negative value; when it is less than F, it becomes a positive value; and when it is the same, it becomes zero. Here, F β = μ + β / 2 * σ ２ Thus, it can be said that the policy learning process performed by the information processing device 1A allows for a more appropriate selection of action a.

[0107] (Justification for Specific Example 2 of the Advantage Function) Next, let's consider the case where the advantage function is Specific Example 2 above. For simplicity, we will explain with γ = 1 here. In this case as well, the k-step reward sum is, Since it is distributed as follows, let x be the sample from here, and the advantage function A(s ｔ , a ｔ )teeth, This is the result.

[0108] The expected value of this advantage function A(s ｋ , a ｋ ) is such that V(s ｋ ) = μ + β / 2 · σ ２ The estimated value is negative when it is larger than the distribution of the above formula (A14) and the estimated value μ of the state value function V between k and T ｋ：Ｔ + β / 2 · σ ２ ｋ：Ｔ and is obtained using the above, becomes a negative value when it is larger, a positive value when it is smaller, and zero when they are the same. From this, according to the policy learning executed by the information processing apparatus 1A, it can be said that the action a is more appropriately selected.

[0109] As described above, in the above configuration example 1, the information processing apparatus 1B uses, as an initial model, a base model that estimates the expected value of the reward and does not consider risk, and sequentially estimates a risk-corrected reward that is a reward considering risk. A reward estimation unit 111(11), and a policy learning unit 113(12) that sequentially learns the agent's policy based on the risk-corrected reward. In the configuration where the policy learning unit 113(12) updates the state value function V(s) using the risk parameter (β), the information processing apparatus 1B uses the risk parameter (β) and a first advantage function (A') obtained using the state value function V(s). It further includes a correction coefficient calculation unit 112(16) that calculates a correction coefficient (ξ β ). The policy learning unit 113(12) multiplies the correction coefficient (ξ β ) by the first advantage function (A') to calculate a second advantage function (A), and updates the policy using the calculated second advantage function (A).

[0110] According to the information processing apparatus 1B configured as described above, in addition to the effects exhibited by the information processing apparatuses 1 and 1A according to exemplary embodiments 1 and 2, it has the effect of being able to perform learning that more appropriately evaluates the policy in reinforcement learning.

[0111] (Configuration Example 2) Figure 9 is a block diagram showing another example of the functional configuration and processing flow of the information processing device 1B according to this exemplary embodiment. Note that the unidirectional arrows in Figure 9 simply indicate the direction of the flow of a signal (data) and do not exclude bidirectionality. As shown in Figure 9, the information processing device 1B according to this configuration example includes a calculation unit 212, an update unit 213, an estimation unit 214, a learning unit 215, a policy determination unit 221, and a setting unit 211. In Figure 9, the calculation unit 212 corresponds to the correction coefficient calculation unit 16, and is therefore also referred to as the calculation unit 212(16). Also, as shown in Figure 9, the update unit 213 and the estimation unit 214 correspond to the estimation unit 11 described above, as an example. Also, as shown in Figure 9, the learning unit 215 and the policy determination unit 221 correspond to the learning unit 12 described above, as an example. However, these corresponding examples do not limit this exemplary embodiment.

[0112] (Setting Unit) The setting unit 211 sets initial parameters for the estimation unit 214 to estimate the risk-adjusted reward ν, which is a reward that takes risk into account. Examples of initial parameters set by the setting unit 211 include the initial value of the risk-adjusted reward ν and the weights of the neural network used by the estimation unit 214 to estimate the risk-adjusted reward ν. Here, as an example, the expected value of the reward estimated by the base model that does not consider risk is used as the initial value of the risk-adjusted reward ν.

[0113] In this configuration example, the risk-adjusted reward ν is a value calculated using the following equation (B1), where σ is the standard deviation of the reward, c is a predetermined parameter, and μ is the expected value of the reward obtained by the next action: ν = μ - cσ ... (Equation B1) The setting unit 211 also generates a state-action value function for the array ν(S, A) of estimated risk-adjusted rewards for action a in state s.

[0114] (Calculation Unit) The calculation unit 212 calculates a correction coefficient ξ based on the parameter c, the risk-adjusted reward ν, and the difference δ between the reward r obtained by the next action. Here, the predetermined parameter for the reward is, for example, a pre-set real-valued parameter for risk preference / avoidance.

[0115] Furthermore, if r is the reward obtained from the next action, the calculation unit 212 calculates the difference δ using the following equation (B2): δ = r - ν ... (Equation B2) In addition, the calculation unit 212 calculates the correction coefficient ξ using the following equations (B3) to (B5). Here, erf is the error function. The calculation unit 212 also calculates the following equation (B4) for a predetermined parameter c. In this case, the correction coefficient ξ is expressed by the following equation (B5). Here, sgn is the sign function.

[0116] In other words, the calculation unit 212 calculates the correction coefficient ξ using the following equation (B6) when δ > 0, and using the following equation (B7) when δ < 0.

[0117] (Update Unit) The update unit 213 updates the risk-adjusted reward based on the risk-adjusted reward ν, the correction coefficient ξ, and the difference δ. As an example, the update unit 213 updates the risk-adjusted reward ν using the following formula (B8): ν ← ν + ξ δ ... (B8) That is, the update unit 213 updates the risk-adjusted reward ν by adding the risk-adjusted reward ν to the value obtained by multiplying the correction coefficient ξ by the difference δ.

[0118] As an example, the update unit 213 may be configured to calculate the difference δ using the following equation (C1): δ = r + γν'(s', a) - ν(s, a) ... (Equation C1) Here, γ represents the discount rate.

[0119] (Estimation Unit) The estimation unit 214 sequentially estimates the risk-corrected reward ν, which is a reward considering risk, using as an initial model a base model that estimates the expected value of the reward and does not consider risk. As an example, the estimation unit 214 uses the risk-corrected reward ν updated by the update unit 213 to perform the process of the calculation unit 212 calculating the correction coefficient ξ, and uses the risk-corrected reward ν updated by the update unit 213 and the correction coefficient ξ to perform the process of the update unit 213 updating the risk-corrected reward ν, thereby estimating the risk-corrected reward ν. As an example, the estimation unit 214 may be an estimation means having parameters such as a neural network.

[0120] (Learning Unit, Policy Decision Unit, Execution Unit) The learning unit 215 sequentially learns the agent's policy based on the risk-corrected reward ν. In other words, the learning unit 215 learns based on the updated risk-corrected reward. As an example, the learning unit 215 causes the policy decision unit 221 to learn based on the updated risk-corrected reward ν.

[0121] The policy decision unit 221 is learned by the learning unit 215 so as to output the action a executed by the execution unit 21 with the state s as an input.

[0122] The execution unit 21 executes the action a output from the policy decision unit 221.

[0123] (Proof that the Update Has a Unique and Stable Fixed Point) In the following, in this configuration example, when the reward follows a normal distribution, it is proved that the update of this risk-corrected reward has a unique and stable fixed point ν ＊ and the fixed point ν ＊ coincides with μ + cσ.

[0124] Having a unique and stable fixed point ν ＊ means that the expected value J(ν) of the update satisfies the following equations (B9) to (B11). J(ν) < 0 if ν > ν ＊ ... (Equation B9) J(ν) = 0 if ν = ν ＊ = μ + cσ... (Equation B10) J(ν) > 0 if ν < ν ＊... (Equation B11) That is, the expected value of the update J(ν) is the value of the estimated risk-adjusted reward ν. ＊ A value greater than ν indicates that ν will be updated to a smaller expected value. Furthermore, the expected value of the update J(ν) is the estimated risk-adjusted reward ν. ＊ A smaller value indicates that ν will be updated to a larger expected value. Furthermore, the expected value of the update J(ν) is the estimated risk-adjusted reward ν. ＊ If this is equal to the given value, it indicates that the expected value of the update amount of ν is 0.

[0125] Now, if we rewrite equation (8) above using equation (B12) below, we get equation (B13).

[0126] In equation (B13), the second term on the right-hand side represents the update amount, so the expected value of the update amount is given by equation (B14). Here, if we perform the variable transformation ν → t in equation (B14) to ν = μ + tσ, we obtain the following equation (B15). Here, using equation (B15) and the following equation (B16), we get ν = ν ＊ When = μ + cσ, J(ν ＊ ) = 0. Thus, equation (B10) is proven.

[0127] Furthermore, J is continuous and differentiable with respect to ν, and dJ / dν < 0. That is, J is strictly monotonically decreasing, and from the proven equation (B10), equations (B9) and (B11) are proven.

[0128] Thus, since the expected value J(ν) of the update satisfies equations (B9) to (B11), the update is unique and stable at a fixed point ν. ＊ It has the property = μ + cσ.

[0129] (Data flow in the control unit 10B) Next, the data flow in the control unit 10B according to this configuration example will be explained with reference to Figure 9. Figure 9 is a diagram showing the data flow in the control unit 10B according to this exemplary embodiment.

[0130] First, as shown by the dotted arrow in Figure 9, the data is processed. That is, the calculation unit 212 obtains the parameter c and the correction coefficient ξ ＋ and correction coefficient ξ － The calculation unit 212 calculates the correction coefficient ξ. ＋ and correction coefficient ξ － This is supplied to the update unit 213.

[0131] Furthermore, the setting unit 211 generates a state-action value function for an array of estimated values ​​of the risk-adjusted reward ν, ν(S, A). The setting unit 211 also sets the initial value of the risk-adjusted reward ν as the risk-adjusted reward estimate ν(s, a), which is the value of the risk-adjusted reward for the initial state s and action a. As described above, the setting unit 211 may, for example, use the expected value of the reward estimated by the base model that does not consider risk as the initial value of the risk-adjusted reward ν. The setting unit 211 supplies the set risk-adjusted reward estimate ν(s, a) to the update unit 213. The setting unit 211 also supplies the state s to the policy decision unit 221.

[0132] Next, as shown by the solid arrows in Figure 9, the data is processed. Specifically, the policy decision unit 221 outputs action a to the execution unit 21. The execution unit 21 then executes action a.

[0133] Then, when the execution unit 21 executes action a, the update unit 213 obtains the reward r. The update unit 213 also obtains the next state s'. The update unit 213 then calculates the next state s', the risk-adjusted reward estimate ν(s', a) for action a, and the difference δ.

[0134] The update unit 213 calculates the risk-adjusted reward estimate ν(s, a), the difference δ, and the correction coefficient ξ. ＋ and correction coefficient ξ － Based on this, the estimated risk-adjusted reward ν is updated. Specifically, as described above, the update unit 213 adjusts the correction coefficient ξ according to the sign of the difference δ. ＋ and correction coefficient ξ －The risk-adjusted reward ν is updated using one of the following. With this configuration, the update unit 213 can successively obtain a better estimator. In other words, the calculation unit 212 adjusts the correction coefficient ξ according to the sign of the difference δ. ＋ and correction coefficient ξ － One of the following is calculated and supplied to the update unit 213.

[0135] The update unit 213 then supplies the updated risk-adjusted reward estimate ν(s, a) to the estimation unit 214. The update unit 213 also updates the state s and supplies the updated state s to the policy decision unit 221.

[0136] The estimation unit 214 obtains multiple risk-corrected reward estimates ν(s, a) through the process in which the calculation unit 212 calculates a correction coefficient ξ and the update unit 213 updates the risk-corrected reward estimate ν(s, a). The estimation unit 214 then estimates the risk-corrected reward ν. The estimation unit 214 supplies the estimated risk-corrected reward ν to the learning unit 215.

[0137] The learning unit 215 learns the policy decision unit 221 based on the estimated risk-adjusted reward ν.

[0138] <Flow of Reinforcement Learning Method> Next, the flow of the reinforcement learning method executed by the information processing device 1B in this configuration example will be explained with reference to Figure 10. Figure 10 is a flowchart showing the flow of the reinforcement learning method executed by the information processing device 1B in this configuration example.

[0139] (Step S21) In step S21, the calculation unit 212 obtains the parameter c stored in the storage unit 17B.

[0140] (Step S22) In step S22, the calculation unit 212 calculates a correction coefficient ξ based on the parameter c. ＋ and correction coefficient ξ － The calculation unit 212 calculates the correction coefficient ξ. ＋ and correction coefficient ξ － This is supplied to the update unit 213.

[0141] (Step S23) In step S23, the setting unit 211 generates an array ν(S, A) that stores the estimated values ​​of the risk-adjusted reward for each state and each action.

[0142] (Step S24) In step S24, the setting unit 211 sets the state s to the initial state s. The setting unit 211 then sets the initial value of the initial state s and the risk-adjusted reward estimate ν(s,a) for action a. As described above, the setting unit 211 may, for example, use the expected value of the reward estimated by the base model that does not consider risk as the initial value of the risk-adjusted reward ν. The setting unit 211 supplies the set risk-adjusted reward estimate ν(s,a) to the update unit 213. The setting unit 211 also supplies the state s to the policy decision unit 221. That is, the information processing device 1B starts the simulation.

[0143] (Step S25) In step S25, the policy decision unit 221 randomly selects action a. As an example, the policy decision unit 221 may be configured to select the action a with the highest value with a probability of 1-ε, or to select a random action a with a probability of ε.

[0144] (Step S26) In step S26, the policy decision unit 221 supplies action a to the execution unit 21. The execution unit 21 executes action a based on the action a supplied from the policy decision unit 221.

[0145] (Step S27) In step S27, the update unit 213 obtains the reward r obtained by the execution unit 21 performing action a, and the next state s'.

[0146] (Step S28) In step S28, the update unit 213 calculates the risk-adjusted reward estimate ν(s', a). Then, the update unit 213 calculates the difference δ using the formula (C1) described above.

[0147] (Step S29) In step S29, the update unit 213 calculates the risk-adjusted reward estimate ν(s', a) and updates the array ν(S, A).

[0148] (Step S30) In step S30, the update unit 213 uses equation (B8) to obtain the risk-adjusted reward estimate ν(s, a), the difference δ, and the correction coefficient ξ ＋ and correction coefficient ξ － Based on this, the estimated risk-adjusted reward ν is updated.

[0149] Here, as described above, the update unit 213 adjusts the correction coefficient ξ according to the sign of the difference δ. ＋ and correction coefficient ξ － One of the following is used. In other words, the calculation unit 212 uses a correction coefficient ξ depending on whether the difference δ is positive or negative. ＋ and correction coefficient ξ － Calculate one of the following, and use the calculated correction coefficient ξ ＋ and correction coefficient ξ － One of these is supplied to the update unit 213.

[0150] Then, the update unit 213 updates the state s from the initial state s to the next state s'.

[0151] (Step S31) In step S31, the update unit 213 determines whether the updated state s is the final state.

[0152] If it is determined in step S31 that the final state is not reached (step S31: NO), the information processing device 1B returns to the process in step S25.

[0153] (Step S32) If it is determined in step S31 that the final state is reached (Step S31: YES), in step S32 the update unit 213 determines whether or not the update termination conditions are met. For example, the update unit 213 determines whether or not predetermined conditions such as the number of episodes or rewards are met.

[0154] If it is determined in step S32 that the final condition is not met (step S32: NO), the information processing device 1B returns to the process in step S24.

[0155] On the other hand, if it is determined in step S32 that the final condition is met (step S32: YES), the information processing device 1B terminates the process shown in Figure 10.

[0156] Thus, in the above configuration example 2, the information processing device 1B is configured to include: an estimation unit 214 (11) that sequentially estimates a risk-adjusted reward, which is a reward that takes risk into account, using a base model that estimates the expected value of the reward and does not take risk as an initial model; and a learning unit 215 (12) that sequentially learns the agent's policy based on the risk-adjusted reward. The information processing device 1B is further configured to include a calculation unit 212 (16) that calculates a correction coefficient based on the risk parameter, the risk-adjusted reward, and the difference between the two rewards obtained by the next action. The estimation unit 214 (11) updates the risk-adjusted reward based on the risk-adjusted reward, the correction coefficient, and the difference. The learning unit 215 (11) sequentially learns the agent's policy based on the updated risk-adjusted reward.

[0157] The information processing device 1B configured as described above provides the following effects in addition to those achieved by the information processing devices 1 and 1A according to exemplary embodiments 1 and 2. Specifically, in the information processing device 1B according to this configuration, the risk-corrected reward can be estimated using a single estimator by using a dimensionless parameter c. Therefore, the information processing device 1B according to this configuration provides the effect that risk-sensitive reinforcement learning can be performed with a single estimator, and the degree of risk consideration does not change depending on the size of the reward.

[0158] The information processing device 1B can also be expressed as follows:

[0159] A reinforcement learning device that learns a strategy for determining the next action of an agent based on the state of the environment and the rewards obtained from the agent's actions, comprising: an estimation means (estimation unit 11) that sequentially estimates a risk-adjusted reward, which is a reward that takes risk into account; and a range determination means (range determination unit 13) that determines the range of risk parameters that the estimation means refers to in order to estimate the risk-adjusted reward.

[0160] With the above configuration, the range determination unit 13 performs the above processing, which allows the value of the risk parameter to be set appropriately in the update process that references the risk parameter. This prevents the update amount (change amount) from becoming very close to zero, which would prevent the learning process from progressing.

[0161] (Application Examples) The information processing devices (reinforcement learning devices) 1, 1A, and 1B according to the exemplary embodiments described above can, for example, be applied to the control of an AGV (Automatic Guided Vehicle). The AGV in this application example is a mobile robot that acts using a policy obtained by the reinforcement learning device or an estimated risk-corrected reward. For example, the AGV acts using a policy obtained by the information processing devices (reinforcement learning devices) 1, 1A, and 1B described above or an estimated risk-corrected reward.

[0162] The AGV in this application example acts using a strategy obtained by a reinforcement learning device that has learned to avoid obstacles and reach a destination, or using an estimated risk-adjusted reward. The shorter the time it takes to reach the destination, the greater the reward. In addition, if it hits an obstacle, a penalty (negative reward) is given.

[0163] Furthermore, in this application example, the status of the AGV, the destination point, and the location of obstacles are observed by the observation equipment provided in the information processing system related to this application example. The information observed by the observation equipment is then acquired by the AGV. With this configuration, the AGV can recognize its own status, the destination point, and the location of obstacles.

[0164] (AGV Configuration) The AGV in this application example may be configured as a separate unit from the information processing devices (reinforcement learning devices) 1, 1A, and 1B as a control target of the information processing devices (reinforcement learning devices) 1, 1A, and 1B described above, or it may be configured to include the information processing devices (reinforcement learning devices) 1, 1A, and 1B described above. The AGV may also be configured to include a sensor, encoder, and motor. The sensor, encoder, and motor are, for example, an example of the configuration included in the execution unit 21 described above.

[0165] The above-mentioned sensor acquires sensor information indicating the status of the AGV. For example, the sensor is an accelerometer that acquires sensor information indicating the acceleration of the AGV. The encoder supplies the sensor information acquired by the sensor as an electrical signal to the control units of the information processing devices (reinforcement learning devices) 1, 1A, and 1B. The motor is an electric motor that drives the AGV. For example, the motor drives the AGV based on a command from the execution unit 21.

[0166] (Processing and effects performed by the AGV) The AGV performs reinforcement learning using its own state acquired by sensors and encoders, information observed by observation instruments, and map information as state s. For example, if the AGV tries to reach the destination in a short amount of time, the risk of colliding with an obstacle and receiving a negative reward increases. In this case, the variability of the reward acquired by the AGV increases. Also, if the AGV tries to reach the destination via the shortest path, the risk of taking an alternative path due to errors and colliding with an obstacle increases.

[0167] However, AGVs perform reinforcement learning while taking these risks into consideration. That is, AGVs do not learn to maximize only the expected value of the reward, but rather learn the path to the destination by considering the reward for reaching the destination quickly and the reward for colliding with obstacles. Therefore, AGVs can more easily learn policies that take the path that yields the highest reward.

[0168] Furthermore, as described above, the information processing devices (reinforcement learning devices) 1, 1A, and 1B employ a configuration in which: • A base model that estimates the expected value of the reward but does not consider risk is used as the initial model to sequentially estimate the risk-adjusted reward, which is the reward that takes risk into account; and • The agent's policy is sequentially learned based on the risk-adjusted reward. In this way, by using the base model as the initial model to estimate the risk-adjusted reward and learning the agent's policy based on the risk-adjusted reward, the increase in computational cost can be suppressed in reinforcement learning that takes risk into account.

[0169] [Example of implementation by software] Some or all of the functions of the information processing devices (reinforcement learning devices) 1, 1A, and 1B (hereinafter also referred to as "each of the above devices") may be implemented by hardware such as integrated circuits (IC chips) or by software.

[0170] In the latter case, each of the above devices is implemented, for example, by a computer that executes instructions for a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in Figure 11. Figure 11 is a block diagram showing the hardware configuration of computer C, which functions as each of the above devices.

[0171] Computer C comprises at least one processor C1 and at least one memory C2. Memory C2 stores a program P for operating Computer C as each of the above-mentioned devices. In Computer C, the processor C1 reads and executes the program P from memory C2, thereby realizing each of the above-mentioned devices.

[0172] For processor C1, for example, a CPU (Central Processing Unit), GPU (Graphic Processing Unit), DSP (Digital Signal Processor), MPU (Micro Processing Unit), FPU (Floating Point Number Processing Unit), PPU (Physics Processing Unit), TPU (Tensor Processing Unit), quantum processor, microcontroller, or a combination thereof can be used. For memory C2, for example, flash memory, HDD (Hard Disk Drive), SSD (Solid State Drive), or a combination thereof can be used.

[0173] Furthermore, computer C may also be equipped with RAM (Random Access Memory) for loading program P at runtime and for temporarily storing various data. Computer C may also be equipped with a communication interface for sending and receiving data with other devices. Furthermore, computer C may also be equipped with an input / output interface for connecting input / output devices such as a keyboard, mouse, display, and printer.

[0174] Furthermore, program P can be recorded on a non-temporary, tangible recording medium M that is readable by computer C. Such recording medium M can include, for example, tape, disk, card, semiconductor memory, or programmable logic circuitry. Computer C can acquire program P via such recording medium M. Program P can also be transmitted via a transmission medium. Such transmission mediums can include, for example, a communication network or broadcast waves. Computer C can also acquire program P via such transmission medium.

[0175] Furthermore, each of the above functions of each of the above devices may be implemented by a single processor in a single computer, by multiple processors in a single computer working together, or by multiple processors in each of multiple computers working together. In addition, the programs for implementing each of the above functions in each of the above devices may be stored in a single memory in a single computer, distributed and stored in multiple memories in a single computer, or distributed and stored in multiple memories in each of multiple computers.

[0176] [Addendum A] This disclosure includes the technologies described in the following addendums. However, the present invention is not limited to the technologies described in the following addendums, and various modifications are possible within the scope of the claims.

[0177] (Appendix A1) A reinforcement learning device that learns a policy for determining the next action of an agent based on the state of the environment and the reward obtained from the agent's actions, comprising: estimation means for sequentially estimating a risk-adjusted reward, which is a reward that takes risk into account, using a base model that estimates the expected value of the reward and does not take risk into consideration as an initial model; and learning means for sequentially learning the agent's policy based on the risk-adjusted reward.

[0178] (Appendix A2) The reinforcement learning apparatus according to Appendix A1, further comprising a range determination means for determining the range of risk parameters that the estimation means references to estimate the risk-adjusted reward.

[0179] (Appendix A3) The reinforcement learning apparatus according to Appendix A2, wherein the range determination means determines the range of the risk parameter by referring to the reward derived by the base model, which is the reward variance without considering the risk.

[0180] (Appendix A4) The reinforcement learning device according to any one of Appendix A1 to A3, wherein the estimation means uses a trained model as an initial model, which is a model for estimating the expected value of a reward and has a risk parameter of a certain value set, to sequentially estimate a risk-corrected reward according to a risk parameter of a different value.

[0181] (Note A5) The reinforcement learning device according to any one of Notes A1 to A4, wherein the learning means updates the state value function using the risk parameter, and the reinforcement learning device further comprises a correction coefficient calculation means that calculates a correction coefficient using the risk parameter and a first advantage function obtained using the state value function, and the learning means calculates a second advantage function by multiplying the correction coefficient by the first advantage function, and updates the policy using the calculated second advantage function.

[0182] (Appendix A6) The reinforcement learning device according to any one of Appendix A1 to A4, further comprising a calculation means for calculating a correction coefficient based on the difference between the risk parameter, the risk-corrected reward, and the reward obtained by the next action, wherein the estimation means updates the risk-corrected reward based on the risk-corrected reward, the correction coefficient, and the difference, and the learning means sequentially learns the agent's policy based on the updated risk-corrected reward.

[0183] [Addendum B] This disclosure includes the technologies described in the following addendums. However, the present invention is not limited to the technologies described in the following addendums, and various modifications are possible within the scope of the claims.

[0184] (Appendix B1) A reinforcement learning method for learning a policy to determine the next action of an agent based on the state of the environment and the reward obtained from the agent's actions, comprising: an estimation process in which at least one processor sequentially estimates a risk-adjusted reward, which is a reward that takes risk into account, using a base model that estimates the expected value of the reward and does not take risk as an initial model; and a learning process in which at least one processor sequentially learns a policy for the agent based on the risk-adjusted reward.

[0185] (Appendix B2) The reinforcement learning method according to Appendix B1, wherein the at least one processor further includes a range determination process that determines a range of risk parameters that the estimation process refers to in order to estimate the risk-adjusted reward.

[0186] (Appendix B3) The reinforcement learning method according to Appendix B2, wherein in the range determination process, the at least one processor determines the range of the risk parameter by referring to the reward derived by the base model, which is the reward variance without considering the risk.

[0187] (Appendix B4) The reinforcement learning method according to any one of Appendix B1 to B3, wherein in the estimation process, at least one processor uses a trained model as an initial model, which is a model for estimating the expected value of a reward and has a risk parameter of a certain value set, to sequentially estimate a risk-corrected reward according to a risk parameter of a value different from the said certain value.

[0188] (Note B5) The reinforcement learning method according to any one of Notes B1 to B4, wherein in the learning process, the at least one processor updates the state value function using the risk parameter, the at least one processor further includes a correction coefficient calculation process which calculates a correction coefficient using the risk parameter and a first advantage function obtained using the state value function, the at least one processor calculates a second advantage function by multiplying the correction coefficient by the first advantage function, and updates the policy using the calculated second advantage function.

[0189] (Appendix B6) The reinforcement learning method according to any one of Appendix B1 to B4, further comprising a calculation process in which at least one processor calculates a correction coefficient based on the difference between the risk parameter, the risk-corrected reward, and the reward obtained by the next action, in the estimation process, the at least one processor updates the risk-corrected reward based on the risk-corrected reward, the correction coefficient, and the difference, and in the learning process, the at least one processor sequentially learns the agent's policy based on the updated risk-corrected reward.

[0190] [Addendum C] This disclosure includes the technologies described in the following addendums. However, the present invention is not limited to the technologies described in the following addendums, and various modifications are possible within the scope of the claims.

[0191] (Appendix C1) A reinforcement learning program that causes a computer to function as a reinforcement learning device that learns a strategy for determining the next action of an agent based on the state of the environment and the reward obtained from the agent's actions, wherein the computer functions as an estimation means that sequentially estimates a risk-adjusted reward, which is a reward that takes risk into account, using a base model that estimates the expected value of the reward and does not take risk into consideration as an initial model, and as a learning means that sequentially learns a strategy for the agent based on the risk-adjusted reward.

[0192] (Appendix C2) The reinforcement learning program according to Appendix C1, wherein the computer further functions as a range determination means for determining the range of risk parameters that the estimation means references to estimate the risk-adjusted reward.

[0193] (Appendix C3) The range determination means is a reinforcement learning program according to Appendix C2, which determines the range of the risk parameter by referring to the reward derived by the base model, which is the reward variance without considering the risk.

[0194] (Appendix C4) The estimation means is a reinforcement learning program according to any one of Appendix C1 to C3, which uses a trained model as an initial model, which is a model for estimating the expected value of a reward and has a risk parameter of a certain value set, to sequentially estimate a risk-corrected reward according to a risk parameter of a different value.

[0195] (Note C5) The reinforcement learning program according to any one of Notes C1 to C4, wherein the learning means updates the state value function using the risk parameter, the computer further functions as a correction coefficient calculation means that calculates a correction coefficient using the risk parameter and a first advantage function obtained using the state value function, the learning means calculates a second advantage function by multiplying the correction coefficient by the first advantage function, and updates the policy using the calculated second advantage function.

[0196] (Appendix C6) The reinforcement learning program according to any one of Appendices C1 to C4, wherein the computer further functions as a calculation means for calculating a correction coefficient based on the risk parameter, the risk-corrected reward, and the difference between the risk parameter and the reward obtained by the next action, the estimation means updates the risk-corrected reward based on the risk-corrected reward, the correction coefficient, and the difference, and the learning means sequentially learns the agent's policy based on the updated risk-corrected reward.

[0197] [Addendum D] This disclosure includes the technologies described in the following addendums. However, the present invention is not limited to the technologies described in the following addendums, and various modifications are possible within the scope of the claims.

[0198] (Appendix D1) A reinforcement learning device that learns a policy for determining the next action of an agent based on the state of the environment and the reward obtained from the agent's actions, comprising at least one processor, the at least one processor performing an estimation process that sequentially estimates a risk-adjusted reward, which is a reward that takes risk into account, using a base model that estimates the expected value of the reward and does not take risk into consideration as an initial model, and a learning process that sequentially learns a policy for the agent based on the risk-adjusted reward.

[0199] The reinforcement learning device may also include memory. Furthermore, the memory may store a program that causes at least one processor to execute each of the aforementioned processes.

[0200] (Appendix D2) The reinforcement learning apparatus according to Appendix D1, wherein the at least one processor further performs a range determination process that determines the range of risk parameters that the estimation process refers to in order to estimate the risk-adjusted reward.

[0201] (Appendix D3) The reinforcement learning apparatus according to Appendix D2, wherein in the range determination process, the at least one processor determines the range of the risk parameter by referring to the reward derived by the base model, which is the reward variance without considering the risk.

[0202] (Appendix D4) The reinforcement learning device according to any one of Appendix D1 to D3, wherein in the estimation process, at least one processor uses a trained model as an initial model, which is a model for estimating the expected value of a reward and has a risk parameter of a certain value set, to sequentially estimate a risk-corrected reward according to a risk parameter of a value different from the said certain value.

[0203] (Note D5) The reinforcement learning device according to any one of Notes D1 to D4, wherein in the learning process, the at least one processor updates the state value function using the risk parameter, the at least one processor further performs a correction coefficient calculation process which calculates a correction coefficient using the risk parameter and a first advantage function obtained using the state value function, and in the learning process, the at least one processor calculates a second advantage function by multiplying the correction coefficient by the first advantage function, and updates the policy using the calculated second advantage function.

[0204] (Note D6) The reinforcement learning device according to any one of Notes D1 to D4, wherein the at least one processor further performs a calculation process to calculate a correction coefficient based on the risk parameter, the risk-corrected reward, and the difference between the two rewards obtained by the next action; in the estimation process, the at least one processor updates the risk-corrected reward based on the risk-corrected reward, the correction coefficient, and the difference; and in the learning process, the at least one processor sequentially learns the agent's policy based on the updated risk-corrected reward.

[0205] [Addendum E] This disclosure includes the technologies described in the following addendums. However, the present invention is not limited to the technologies described in the following addendums, and various modifications are possible within the scope of the claims.

[0206] (Appendix E1) A non-temporary recording medium that records a reinforcement learning program which causes a computer to function as a reinforcement learning device that learns a policy for determining the next action of an agent based on the state of the environment and the rewards obtained from the agent's actions, the program which causes the computer to perform an estimation process that sequentially estimates a risk-adjusted reward, which is a reward that takes risk into account, using a base model that estimates the expected value of the reward and does not take risk into consideration as an initial model, and a learning process that sequentially learns a policy for the agent based on the risk-adjusted reward.

[0207] 1, 1A, 1B ... Reinforcement learning device (information processing device) 11 ... Estimation unit 12 ... Learning unit 13 ... Range determination unit

Claims

1. A reinforcement learning device that learns a policy for determining the next action of an agent based on the state of the environment and the reward obtained from the agent's actions, comprising: an estimation means for sequentially estimating a risk-adjusted reward, which is a reward that takes risk into account, using a base model that estimates the expected value of the reward and does not consider risk as an initial model; and a learning means for sequentially learning the agent's policy based on the risk-adjusted reward.

2. The reinforcement learning device according to claim 1, further comprising a range determination means for determining a range of risk parameters that the estimation means references for estimating the risk-adjusted reward.

3. The reinforcement learning device according to claim 2, wherein the range determination means determines the range of the risk parameter by referring to the reward derived by the base model, which is the reward variance without considering the risk.

4. The reinforcement learning device according to any one of claims 1 to 3, wherein the estimation means uses a trained model as an initial model, which is a model for estimating the expected value of a reward and has a risk parameter of a certain value set, to sequentially estimate a risk-corrected reward according to a risk parameter of a value different from the said certain value.

5. The reinforcement learning device according to claim 2 or 3, wherein the learning means updates the state value function using the risk parameter, the reinforcement learning device further comprises a correction coefficient calculation means for calculating a correction coefficient using the risk parameter and a first advantage function obtained using the state value function, the learning means calculates a second advantage function by multiplying the correction coefficient by the first advantage function, and updates the policy using the calculated second advantage function.

6. The reinforcement learning device according to claim 2 or 3, further comprising a calculation means for calculating a correction coefficient based on the difference between the risk parameter, the risk-corrected reward, and the reward obtained by the next action, wherein the estimation means updates the risk-corrected reward based on the risk-corrected reward, the correction coefficient, and the difference, and the learning means sequentially learns the agent's policy based on the updated risk-corrected reward.

7. A reinforcement learning method in which at least one processor learns a policy for determining the next action of an agent based on the state of the environment and the rewards obtained from the agent's actions, the method comprising: the processor sequentially estimating a risk-adjusted reward, which is a reward that takes risk into account, using a base model that estimates the expected value of the reward and does not take risk as an initial model; and sequentially learning a policy for the agent based on the risk-adjusted reward.

8. The reinforcement learning method according to claim 7, wherein the at least one processor further includes a range determination process that determines a range of risk parameters that the estimation process refers to in order to estimate the risk-adjusted reward.

9. The reinforcement learning method according to claim 8, wherein in the range determination process, the at least one processor determines the range of the risk parameter by referring to the reward derived by the base model, which is the variance of the reward without considering the risk.

10. A program that causes a computer to perform reinforcement learning to learn a policy for determining the next action of an agent based on the state of the environment and the rewards obtained from the agent's actions, the program causing the computer to perform an estimation process that sequentially estimates a risk-adjusted reward, which is a reward that takes risk into account, using a base model that estimates the expected value of the reward and does not consider risk as an initial model, and a learning process that sequentially learns a policy for the agent based on the risk-adjusted reward.

Images