Optimization device, optimization method, and optimization program

Abstract

To improve efficiency related to calculation.SOLUTION: An optimization apparatus holds a unique hidden state obtained by modifying a hidden state to a predetermined mode and a current state estimation in the unique hidden state. An observation state rule uses observation at a time t as conditions for conditional probability. A state transition rule uses a unique hidden state at a time t and a unique hidden state at a time t+1 as conditions for conditional probability so as to obtain a unique hidden state, and defines the state transition rule and the observation state rule so as to obtain behavior of an agent. A distribution is updated in accordance with procedures of the agent using the above rules.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an optimization device, an optimization method, and an optimization program. [Background technology]

[0002] Techniques such as hidden Markov models, Bayesian estimation, and reinforcement learning are used as approaches to solving the problem.

[0003] For example, Patent Document 1 discloses a hidden variable model estimation device that can realize high-speed model selection even if the number of model candidates increases exponentially along with the number of hidden states and types of observation probabilities. This hidden variable model estimation device has a variational probability calculation unit that calculates the variational probability by maximizing a reference value defined as a lower bound of an approximation obtained by Laplace approximating a marginal log-likelihood function with respect to an estimate for a complete variable. The hidden variable model estimation device also has a model estimation unit that estimates an optimal hidden variable model by estimating the type of observation probability and parameters for each hidden state, and a convergence determination unit that determines whether the reference value used when the variational probability calculation unit calculates the variational probability has converged.

[0004] Patent Document 2 discloses a system for selecting an action to be performed by a reinforcement learning agent that interacts with an environment. The system includes a process configured to process according to a current hidden state of a goal regression neural network (NN) to generate an initial goal vector in a goal space for a time step and update the internal state of the goal regression NN.

[0005] In addition, reinforcement learning is a method in which a reward and a next state are determined for each combination of state and action. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Re-tabled publication 2013 / 179579 [Patent Document 2] Special Publication No. 2020-508524 Summary of the Invention [Problem to be solved by the invention]

[0007] In the framework of reinforcement learning, a method called Q-learning is typically used. Q-learning is defined for discrete states, but in real problems, there are a huge number of observation states, so learning is often difficult using standard Q-learning.

[0008] In recent years, Deep Q Network (DQN), which combines an advanced version of Q-learning with a multi-layered neural network, has been shown to be effective in a variety of tasks. Although trained DQNs show very high performance, it is unclear how the internal state of the behavior is interpreted in a black box given the environment. In addition, although the performance after training is high, many iterations are required for learning, and little consideration is given to effectively obtain rewards during learning.

[0009] On the other hand, algorithms using Bayesian estimation that effectively learn state transitions while estimating important hidden states from a large number of observations are also widely known. However, this method treats reward prediction and state transitions separately, which means that states unrelated to rewards are analyzed in detail, resulting in a large waste of memory and computational resources depending on the problem setting. In addition, Thompson sampling is known as a general-purpose method that balances learning and reward acquisition in a framework called the multi-armed bandit problem, but it cannot be directly applied to complex problems.

[0010] The present invention has been made in consideration of the above circumstances, and has an object to provide an optimization device, an optimization method, and an optimization program that make it possible to improve the efficiency of calculations. [Means for solving the problem]

[0011] In order to achieve the above object, the optimization device according to the present invention uses a system in which each law based on a state transition law, an observation state law, and a reward law is defined, and in a model in which an agent's actions are repeated to learn each of the laws and obtain a reward, the optimization device holds an original hidden state in which a hidden state has been changed to a predetermined mode, and an estimate of a current state in the original hidden state, and the observation state law uses an observation at time t as a condition for conditional probability to obtain the original hidden state, and the state transition law uses the original hidden state at time t and the original hidden state at time t+1 as conditions for conditional probability to obtain the agent's action, and is configured to include: a setting unit that defines the state transition law and the observation state law such that the original hidden state is obtained by changing the hidden state in a predetermined manner and an estimate of a current state in the original hidden state, and the observation state law uses an observation at time t as a condition for conditional probability to obtain the original hidden state, and the state transition law uses the original hidden state at time t and the original hidden state at time t+1 as conditions for conditional probability to obtain the agent's action; and an update unit that determines an optimal action of the agent based on a Bellman equation, assuming a distribution of each parameter representing a probability sampled based on the laws, and an estimate of the current state, and repeatedly updates the estimate of the current state and the distribution using the laws by a posterior distribution obtained by applying Bayes' theorem to the laws, a predetermined prior distribution, and observation information including the optimal action.

[0012] The optimization method according to the present invention uses a system in which the state transition law, the observation state law, and the reward law are defined, and in a model in which an agent's actions are repeated to learn each of the laws and obtain a reward, the system holds an original hidden state in which the hidden state has been changed to a predetermined mode, and an estimate of the current state in the original hidden state, the observation state law uses an observation at time t as a condition for conditional probability to obtain the original hidden state, and the state transition law defines the state transition law and the observation state law such that the original hidden state is obtained by using the original hidden state at time t and the original hidden state at time t+1 as conditions for conditional probability to obtain the agent's action, the system determines the agent's optimal action based on the Bellman equation, assuming a distribution of each parameter representing the probability sampled based on each of the laws and an estimate of the current state, and causes a computer to execute the following processes: updating the estimate of the current state and the distribution using each of the laws using a posterior distribution obtained by applying Bayes' theorem to observation information including each of the laws, a predetermined prior distribution, and the optimal action, and repeating the process of updating the estimate of the current state and the distribution using each of the laws. Effect of the Invention

[0013] According to the optimization device, optimization method, and optimization program of the present invention, it is possible to obtain an effect of improving efficiency of calculations. [Brief description of the drawings]

[0014] [Figure 1] FIG. 13 is a diagram showing an example of a transition diagram of a conventional method and a transition diagram of the method of the present embodiment regarding estimation of states and rules. [Diagram 2] FIG. 2 is a diagram illustrating the functional configuration of an optimization device according to an embodiment of the present invention. [Diagram 3] FIG. 2 is a block diagram showing a hardware configuration of the optimization device. [Figure 4] FIG. 4 is a diagram showing an optimization processing routine of the optimization device according to the embodiment of the present invention. [Diagram 5] 11 is a graph showing an example of experimental results of the method of this embodiment and other methods. [Figure 6] This is a table showing the number of hidden states at convergence in an experiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0016] First, the principles of the embodiment of the present invention will be described.

[0017] FIG. 1 is a diagram showing an example of a transition diagram of a conventional method and a transition diagram of the method of this embodiment regarding the estimation of states and rules. First, as a basic principle, the estimation of states and rules of the conventional method will be explained. The conventional method and the method of this embodiment have in common an internal model that uses a system (transition diagram) in which each rule is defined by the state transition rule, the observed state rule, and the reward rule, and repeats the actions of the agent to learn each rule and acquire rewards. The upper part of FIG. 1 is a transition diagram of the estimation of states and rules of the conventional method. For a time t, the observed state o t is obtained, and action a t If you select it, you will receive a reward t and the next observation t+1 In this case, the long-term reward Σ τ=0 ∞ Gamma τ r t+τ We want to make the selection so that the observation o is as large as possible. t There is a hidden state behind , and the hidden state changes probabilistically depending on the action, and the reward is also determined probabilistically. t Let p(s t+1 │s t ,a t ), the law of observation state p(o t │s t ) and the law of reward p(r t │s t ,a t) is defined. Hereafter, these are called the transition law, observation law, and reward law. However, these laws are unknown to the algorithm (agent) that is trying to achieve the goal. It is necessary for the agent to learn the laws by repeating actions and to obtain high rewards in parallel.

[0018] On the other hand, in the principle according to the embodiment of the present invention, the agent uses its own hidden state s as an internal model for the environment. t ' and the current state estimate q(s t '), where s t ' is t The simplified version focuses on the elements related to the compensation of s. t Although the existence and transition rules of are assumed, they are not actually estimated. Assuming that there are some complex states and transition rules, s is the state that eliminates unnecessary states from the perspective of reward, without directly considering them. t ' is.

[0019] In the method of the present embodiment, the internal model is also provided for the transition rule, observation rule, and reward rule, but the observation rule and the transition rule have different p(s t '│o t ) and p(a t │s t ',s t+1 The reward law p(r t │s t ',a t ) is the same as an actual rule. With this format, the transition diagram of the state and rule estimation in this embodiment can be made as shown in the lower part of Figure 1. The agent holds the parameters of each rule as a probability distribution and learns according to the observation results. Specifically, the parameter M'(s t ',s t+1 ',a t )=p(a t │s t ',s t+1 '),N'(s t ',o t )=p(o t │s t '),L(st ',a t ,r t )=p(r t │s t ',a t ) the parameter predictions q(M'), q(N'), and q(L) are set. q(M') and q(N') can be easily converted to the actual laws and corresponding q(M) and q(N).

[0020] Here, we will explain the parameters. For example, in p(r|s,L), if the possible values ​​of r in state s1 are r1, r2, and r3, then L(s1,r1) represents the probability that r1 is obtained, and L can be expressed as a matrix, L(s1,r1)+L(s1,r2)+L(s1,r3)=1. M is a tensor that takes the indexes of s, s', and a, and the sum of a is 1.

[0021] The agent operates in the following steps: [1] Sample M, N, and L according to the probability distributions q(M), q(N), and q(L). [2] Calculate the sampled M, N, and L and the state estimate q(s t ') is correct, the optimal action a t is determined based on the Bellman equation and output. t As a result of outputting, new information is t ,o t+1 [3] Law p(a t │s t ',s t+1 '),p(o t │s t '),p(r t │s t ',a t ) and the prior distribution q(s t '), q(M'), q(N), q(L) and the observed information o t ,a t ,r t ,o t+1 Apply Bayes' theorem to the new knowledge q(s t+1 '), q(M'), q(N), q(L). [4] Then, return to [1] and repeat.

[0022] The Bellman equation calculates the expected long-term reward q when action a is taken, so we simply take the action that maximizes q. Applying Bayes' theorem means calculating the posterior distribution p(s,s',L,M,N|o,o',r,r') using each rule p(a|s,s',M),p(s|o,L),p(r|s,N) and the prior distribution p(L,M,N). Below is an example of applying the calculation of the posterior distribution.

[0023] p(s,s',L,M,N|o,o',r,r',a) ∝p(r,r',a,s,s',L,M,N|o,o') =p(r|s,N)p(r'|s',N)p(a|s,s',M)p(s|o,L),p(s'|o',L)p(L,M,N)

[0024] In addition, the calculation method uses variational Bayes to calculate the posterior distribution until it converges each time, but in the conventional method, the posterior distribution is not calculated until it converges each time, and only one step of variational Bayes is updated. In addition, although the amount of calculation increases with the method of this embodiment, a significant improvement in online performance is expected.

[0025] The features of the above procedure are explained below. In procedure [1], conventional methods have not attempted to apply Thompson sampling to state transitions. Conventional methods mainly combine [1] and [2] to predict q using a neural network. Regarding procedure [2], it is a standard method to calculate q when M, N, and L are known, and it is also a standard method to determine an action probabilistically using softmax based on Q. In contrast, the method of this embodiment is characterized in that sampling is performed in [1], and softmax is not used, but the action with the largest q is simply selected. Regarding procedure [3], conventional methods use a probability model p(s t+1 |s t ,a t ),p(o t |s t ), whereas in the method of this embodiment, p(a t |s t ',st+1 '),p(s t '|o t ) is assumed. t is assumed to be a simple categorical distribution, whereas the method of the present embodiment extends it to a direct product of categorical distributions.

[0026] In the above example, variational Bayesian calculations are performed; however, variational Bayesian calculations are not essential to the procedure itself, and other calculation methods may be used.

[0027] <Configuration of an optimization device according to an embodiment of the present invention> Next, the configuration of the optimization device according to the embodiment of the present invention will be described.

[0028] 2 is a diagram showing the functional configuration of the optimization device 100 according to an embodiment of the present invention. As shown in FIG. 2, the optimization device 100 functionally includes a setting unit 110, an update unit 112, and a storage unit 120.

[0029] The setting unit 110 sets the state transition rules and the definitions of the rules for the observed states, and stores the settings in the storage unit 120. Hereinafter, the settings are read out as appropriate for each rule and processing is performed.

[0030] As shown in the above principle, the hidden state s t A unique hidden state s t ', and the current state estimate q(s t By setting, the law of observation states holds that the condition of the conditional probability is the observation o at time t. t Using the unique hidden state s t ' to obtain (p(s t '│o t )). By setting, the state transition law is the unique hidden state s at time t as a condition of the conditional probability. t ' and the unique hidden state s at time t+1 t+1 ', and the agent's action a t (p(at │s t ',s t+1 ')).

[0031] The update unit 112 updates the optimal action a t The distribution is updated repeatedly until the predetermined condition is met. t First, following the procedure [1] of the agent's behavior, we sample the parameters M, N, and L that represent probabilities based on each rule. Then, following procedure [2], we calculate the sampled parameters M, N, and L and the current state estimate q(s t '), and the agent's optimal action a is calculated based on the Bellman equation. t Determine and output the optimal action a t As a result of outputting, new information is t ,o t+1 Following the procedure [3], we obtain each law p(a t │s t ',s t+1 '),p(o t │s t '),p(r t │s t ',a t ), given prior distribution q(s t '), q(M'), q(N), q(L), and the observed information o including the optimal action t ,a t ,r t ,o t+1 Then, the update unit 112 uses the obtained posterior distribution to estimate the current state and to calculate the distribution q(s t+1 By repeatedly updating q(M'), q(N), and q(L), the final converged distribution is output.

[0032] The storage unit 120 stores the settings related to each rule set by the setting unit 110, calculation data in the calculation process by the update unit 112, and the calculation results.

[0033] FIG. 3 is a block diagram showing a hardware configuration of the optimization device 100. As shown in FIG.

[0034] 3, the optimization device 100 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and a communication interface (I / F) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

[0035] The CPU 11 is a central processing unit, and executes various programs and controls each part. That is, the CPU 11 reads a program from the ROM 12 or the storage 14, and executes the program using the RAM 13 as a working area. The CPU 11 controls each of the above components and performs various arithmetic processing according to the program stored in the ROM 12 or the storage 14. In this embodiment, an optimization program is stored in the ROM 12 or the storage 14.

[0036] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is configured with a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and stores various programs including an operating system, and various data.

[0037] The input unit 15 includes a pointing device such as a mouse, and a keyboard, and is used to perform various input operations.

[0038] The display unit 16 is, for example, a liquid crystal display, and displays various information. The display unit 16 may function as the input unit 15 by adopting a touch panel system.

[0039] The communication interface 17 is an interface for communicating with other devices such as terminals, and uses standards such as Ethernet (registered trademark), FDDI, and Wi-Fi (registered trademark).

[0040] <Operation of the optimization device according to the embodiment of the present invention> Next, a description will be given of the operation of the optimization device 100 according to the embodiment of the present invention. As each part of the optimization device 100, the CPU 11 executes an optimization processing routine shown in FIG.

[0041] In step S100, the CPU 11 sets the state transition rule and the definition of the rule of the observed state, and stores the settings in the storage unit 120.

[0042] In step S102, the CPU 11 samples the parameters M, N, and L according to the agent operation procedure [1].

[0043] In step S104, the CPU 11 calculates, in accordance with the procedure [2], each of the sampled parameters M, N, and L and the current state estimate q(s t '), and the agent's optimal action a is calculated based on the Bellman equation. t Determine.

[0044] In step S105, the CPU 11 selects the optimal action a t As a result of outputting, new information is t ,o t+1 get.

[0045] In step S106, the CPU 11 calculates each rule, a predetermined prior distribution, and an optimal action a according to the procedure [3]. t Bayes' theorem is applied to the observed information, including

[0046] In step S108, the CPU 11 determines whether or not the update condition is satisfied. If it is determined that the condition is satisfied, the process proceeds to step S110, and if it is determined that the condition is not satisfied, the process returns to step S102 and repeats the process (step [4]).

[0047] In step S110, the finally obtained distribution is output, and the process ends.

[0048] According to the embodiment of the present invention described above, it is possible to improve the efficiency of calculations. The key features of the technology are three-fold. The first is that state estimation using a probabilistic model is integrated with the decision-making problem, the second is that the observation rule and transition rule are unique rules rather than normal rules, and the third is the use of sampling in the procedure.

[0049] The first point, integrating state estimation by a probabilistic model with a decision-making problem, will be described. Various methods have been proposed so far for estimating models of hidden states from observations, such as the Kalman filter. In addition, there have been many studies on decision-making problems in state transition environments, typified by Q-learning. However, despite being an important problem in reality, only limited efforts have been made on the problem of integrating the two. In terms of the first point, the technology of this embodiment is a method of integrating state estimation with a decision-making problem.

[0050] The second point is that the law has a unique format. In this embodiment, the observation law and the transition law are expressed as p(s t '│ot) and p(a t │s t ',s t+1 This formulation allows us to model the probability of observations t can be treated as a given decision, not a value to be estimated. In the usual formulation, the observation o t On the other hand, in the method of this embodiment, the observation o t Instead, it is enough to predict the reward, so the hidden state s required for predicting the reward is used instead of the true transition rule. t ', the transition rule is reconstructed only by the hidden states. This reduces the number of hidden states, making learning more efficient. In human cognition, for example, only a small number of states are truly required for decision-making, compared to a huge number of observations such as those of vision and hearing. By using this rule, it is possible to effectively realize such advanced mechanisms with a simple model.

[0051] The third point, utilization of sampling in the procedure, will be explained. The important issue is whether to utilize an uncertain internal model to obtain as much reward as possible based on the current estimation, or to search to obtain information for future rewards. In this embodiment, utilization is performed according to the procedures (2) and (3) described in the principle, but since the uncertainty of the original estimation is taken into account in the sampling of (1), the most effective search can be performed. Therefore, the efficiency is improved by combining decision-making by a state estimation model and sampling, which is a novel point.

[0052] As described above, the technology of the embodiment of the present invention makes it possible to improve the efficiency of calculations. That is, for a hidden Markov model in which a hidden state transitions depending on a state and an action and an observed state is generated, and a reward is added to the hidden Markov model, an algorithm is provided that can correctly estimate state transitions in a small number of iterations while reducing opportunity loss and that provides high explainability of the results.

[0053] Opportunity loss is the loss related to the agent's actions. For example, if taking action a1 would earn r1 as a reward, but taking action a2 earns r2, then r1-r2 is the opportunity loss. If the information is incomplete at the initial stage, the opportunity loss cannot be reduced to 0, so the opportunity loss tends to be large when acting according to incomplete information or taking exploratory actions despite having sufficient information. It is important to have a small average opportunity loss when the algorithm is repeatedly executed over a long period of time, and reducing opportunity loss is roughly the same as increasing the total reward obtained.

[0054] [Experimental Results] The experimental results of the method of this embodiment will be described. FIG. 5 is a graph showing an example of the experimental results of the method of this embodiment and other methods. FIG. 5 shows the cumulative reward for each trial number for each estimation method. The method proposed in this embodiment is O mul→S→R. The method proposed in this embodiment is able to obtain the optimal reward most quickly. Also, FIG. 6 is a table showing the number of hidden states at the time of convergence in the experiment. In the method of this embodiment, it is possible to estimate 8 types of hidden states related to the reward for 32 types of observed states. This is the reason why learning can be performed at high speed.

[0055] A supplementary note on the rules of this embodiment is provided. The rules shown in FIG. 1 are "true rules" that are not known to the agent, and in these rules, observations are generated from hidden states. Normally, in a probabilistic model, it is common to find true rules, but in the problem setting assumed in this embodiment, it is not necessary to estimate all of the complex true rules. In particular, for the part where an observation o is obtained from a state s, a rule is learned as to what kind of observation will be obtained next time. However, since the state o can actually be observed, prediction is not necessary, and it is sufficient to only predict the hidden state s and the reward r. Therefore, the method of this embodiment is the result of considering a formulation that does not require the deliberate prediction of o.

[0056] For example, in the true law, four states {s0, s1, s2, s3} correspond one-to-one to observations {o0, o1, o2, o3}, but if the four states are equivalent from the viewpoint of reward, any observation can be said to be equivalent for the purpose of obtaining reward. In the conventional method, it was learned by distinguishing that o0 is generated from a state called s0 and o2 is generated from a state called s1, and accordingly a large number of states were required. In the method of this embodiment, it is learned that when o0, o1, o2, and o3 are observed, they are common states. The true law is not necessarily fixed to one view, and it can also be interpreted that the state is the same and the observations are generated probabilistically. In that sense, in this example, s and s' correspond completely equivalently.

[0057] In general, agents in conventional methods always construct a state and its transition model that can explain the observation o. On the other hand, the method of this embodiment reconstructs the state simply from the perspective of reward, without being caught up in seemingly diverse and complex observations, which may have various interpretations. Since p(o|s) can be found from p(s|o) and p(o) by Bayes' theorem, it can be said that s and s' are equivalent models.

[0058] The technology of the embodiment of the present invention can be applied to a wide range of problem settings in which observed states and rewards are given, and can learn faster than conventional methods, so it is expected to have a wide range of applications. In particular, it is expected to be used in the field of robotics and for improving the performance of agent systems using AI.

[0059] The present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the spirit and scope of the present invention. [Explanation of symbols]

[0060] 100 Optimizer 110 Setting section 112 Update Department 120 Storage section

Claims

1. In a model in which a system is used in which each law is defined based on a state transition law, an observed state law, and a reward law, and an agent repeats actions to learn each of the laws and obtain a reward, maintaining a unique hidden state in which the hidden state has been modified in a predetermined manner, and an estimate of the current state in the unique hidden state; The law of the observation state is such that, using the observation at time t as the condition of the conditional probability to obtain the unique hidden state, The state transition rule uses the unique hidden state at time t and the unique hidden state at time t+1 as the condition of the conditional probability to obtain the agent's action: A setting unit that defines a transition rule of the state and a rule of the observed state; determining an optimal action for the agent based on a Bellman equation assuming a distribution of each parameter representing a probability of sampling based on each of the laws and an estimate of the current state; an update unit that repeats estimating the current state and updating the distribution using each of the laws based on a posterior distribution obtained by applying Bayes' theorem to observation information including each of the laws, a predetermined prior distribution, and the optimal action; An optimization device comprising:

2. 2. The optimization device according to claim 1, wherein the update unit selects an action that maximizes a long-term reward q based on a Bellman equation by sampling each parameter in a procedure of the agent's action.

3. In a model in which a system is used in which each law is defined based on a state transition law, an observed state law, and a reward law, and an agent repeats actions to learn each of the laws and obtain a reward, maintaining a unique hidden state in which the hidden state has been modified in a predetermined manner, and an estimate of the current state in the unique hidden state; The law of the observation state is such that, using the observation at time t as the condition of the conditional probability to obtain the unique hidden state, The state transition rule uses the unique hidden state at time t and the unique hidden state at time t+1 as the condition of the conditional probability to obtain the agent's action: defining transition laws for said states and laws for said observed states; determining an optimal action for the agent based on a Bellman equation assuming a distribution of each parameter representing a probability of sampling based on each of the laws and an estimate of the current state; repeating the estimation of the current state and updating the distribution using each of the laws based on a posterior distribution obtained by applying Bayes' theorem to the observed information including each of the laws, a predetermined prior distribution, and the optimal action; An optimization method for having a computer execute a process.

4. A program for causing a computer to function as each unit of the optimization device according to claim 1 or 2.

Images