Planning device, planning method, and program

Abstract

To provide a planning device, a planning method, and a program which can create a plan at a higher speed than in the conventional art.SOLUTION: A planning device comprises a creation update unit, a read-ahead calculation unit, an action probability calculation unit, a simulation unit, and a determination unit. The creation update unit creates or updates a search tree in which each state in an action plan is made to correspond to a node. The read-ahead calculation unit alternately executes action selection and simulation, and performs read-ahead calculation of a preceding state and reward from a first state. The action probability calculation unit calculates an action probability being a selection probability of action in the read-ahead calculation. The simulation unit performs the simulation in the read-ahead calculation, based on the action probability. The determination unit determines whether or not to branch the search tree depending on the result of the read-ahead calculation.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] An embodiment of the present invention relates to a plan creation device, a plan creation method, and a program. [Background technology]

[0002] Planning problems such as capital investment can be automated and the quality of the plan improved by formulating the necessary requirements and solving them as an optimization problem. In real planning problems, it is necessary to consider multiple evaluation indicators such as cost, reliability, and delivery date. Many of these are in a trade-off relationship, and in order to select a plan with an excellent indicator, it is often necessary to compromise on other indicators.

[0003] When multiple evaluation indexes need to be considered, weights must be assigned to the evaluation indexes in advance and the objective function must be scalarized using a weighted sum or similar method in order to treat the problem as a simple maximization or minimization problem. The importance of multiple evaluation indexes is often not clear in advance. Pareto optimization is known as a framework for finding a solution set for multiple conflicting evaluation indexes. Multi-objective Monte Carlo tree search is a method for performing Pareto optimization in planning problems, but in complex problems, efficient action selection cannot be performed and the search time can be enormous. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Weijia Wang, Michele Sebag, “Multi-objective Monte-Carlo Tree Search”, Asian Conference on Machine Learning, 2012, PMLR 25:p507-522 [Non-Patent Document 2] Weizhe Chen, Lantao Liu, “Pareto Monte Carlo Tree Search for Multi-Objective Informative Planning”, Robotics: Science and Systems 2019 Summary of the Invention [Problem to be solved by the invention]

[0005] An object of the present invention is to provide a planning device, a planning method, and a program that are capable of creating a plan faster than ever before. [Means for solving the problem]

[0006] The plan creation device of the embodiment has a creating / updating unit, a look-ahead calculation unit, an action probability calculation unit, a simulation unit, and a judgment unit. The creating / updating unit creates or updates a search tree in which each state in the action plan corresponds to a node. The look-ahead calculation unit alternately executes action selection and simulation, and performs look-ahead calculation of a future state and a reward from a first state. The action probability calculation unit calculates an action probability that is a probability of action selection in the look-ahead calculation. The simulation unit performs simulation in the look-ahead calculation based on the action probability. The judgment unit judges whether or not to branch the search tree depending on the result of the look-ahead calculation. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of a plan creation device according to an embodiment. [Diagram 2] FIG. 1 is a diagram showing an example of a power grid system. [Diagram 3] FIG. 13 is a diagram showing an example of a search tree to be created or updated. [Figure 4] FIG. 13 is an image diagram showing how a probability distribution is assigned according to the embodiment. [Diagram 5] FIG. 13 is a diagram showing an example of weight distribution of rewards according to the Dirichlet distribution. [Figure 6]11 is a diagram showing an example of processing performed by a look-ahead calculation unit according to the embodiment; [Figure 7] FIG. 13 is an image diagram showing an example of branching determination based on the expected value of reward. [Figure 8] FIG. 13 is a diagram showing an example of branch determination based on conditional arrival probability for weights. [Figure 9] 4 is a flowchart of a processing procedure performed by the plan creation device according to the embodiment. [Figure 10] FIG. 1 is a diagram showing an example of the configuration of a plan creation device according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, a plan creation device, a plan creation method, and a program according to an embodiment will be described with reference to the drawings. In the following description, an action plan such as a facility change plan will be described as an example of processing handled by the plan creation device. Note that the application is not limited to facility change plan problems for social infrastructure systems. In the following example, a power system will be described as an example of a social infrastructure system, but the social infrastructure system is not limited to this. The social infrastructure system may be, for example, a water supply, gas, road, communication, or other system.

[0009] Fig. 1 is a block diagram showing an example of the configuration of a plan creation device according to this embodiment. As shown in Fig. 1, the plan creation device 1 includes an information processing unit 10, a search tree storage unit 12, a setting unit 14, and a display unit 16. The information processing unit 10 includes a simulation unit 101, a creation / update unit 102, a determination unit 103, a look-ahead calculation unit 104, and an action probability calculation unit 105.

[0010] The setting unit 14 is, for example, a touch panel sensor provided on the display unit 16, a keyboard, a mouse, a data input / output terminal such as a USB (Universal Serial Bus), etc. The setting unit 14 acquires the operation results of the user and data input by the user, and outputs the acquired operation results and data to the information processing unit 10. The operation results include, for example, various conditions for creating a plan. The data includes, for example, data on a system diagram of the power grid system.

[0011] The display unit 16 is, for example, a liquid crystal image display device, an organic EL (Electro Luminescence) image display device, etc. The display unit 16 displays a plan based on the search tree information created by the plan creation device 1. The plan is created by the information processing unit 10.

[0012] The plan creation device 1 considers the balance of multiple rewards and creates and presents a plan that corresponds to a Pareto solution. Note that a Pareto solution is a set of solutions in which the balance between multiple objective functions is different.

[0013] The information processing unit 10 creates a search tree in which each state of the plan corresponds to a node, assigns a probability distribution of multiple reward weights to each node, and calculates the action probability based on the weights sampled from the probability distribution of the reward weights, thereby creating multiple proposed plans with different balances of importance.

[0014] The simulation unit 101 performs a simulation in the look-ahead calculation, i.e., a simulation of a state beyond the first state, using the operation result and data acquired by the setting unit 14. The simulation unit 101 performs the simulation in the look-ahead calculation using, for example, a tree-structured search algorithm. The simulation unit 101, for example, executes a power flow simulator to calculate the load on the system devices. The simulation unit 101 also evaluates the state of the system using, for example, a predetermined evaluation index, and calculates the expected value of the reward.

[0015] The creating and updating unit 102 creates a search tree in which an initial state in the action plan corresponds to a node. The creating and updating unit 102 updates the search tree in which each state in the action plan corresponds to a node. The created search tree will be described later. The creating and updating unit 102 stores the created or updated search tree information in the search tree storage unit 12. Each state in the plan is, for example, for each predetermined time period (for example, every year).

[0016] The determination unit 103 determines whether or not to branch the search tree depending on the result of the look-ahead calculation, and outputs the determination result to the creating / updating unit 102. The determination unit 103 compares the case where the search tree is branched with the case where it is not branched, and performs branching only when, for example, the effect is sufficiently large. Alternatively, the determination unit 103 focuses on the probability of achievement for multiple reward weights, and performs branching when the maximum value is equal to or greater than a certain value, and does not perform branching when the maximum value is not equal to or greater than a certain value.

[0017] The look-ahead calculation unit 104 alternately executes action selection and simulation, and performs look-ahead of a state beyond the first state and look-ahead of a reward, for example. In look-ahead of rewards, a policy function previously learned by reinforcement learning is used, for example. The look-ahead calculation unit 104 calculates or acquires the weight importance of a state beyond the first state. After the look-ahead simulation is completed, the look-ahead calculation unit 104 checks, for each action, a set of weight samples that maximizes the weighted sum of rewards when the action is selected. The look-ahead calculation unit 104 calculates the expected value of rewards based on the simulated results. The look-ahead calculation unit 104 records the results of the look-ahead and creates a look-ahead trial table. The look-ahead calculation unit 104 calculates the expected value of rewards for each first action selection and creates an individual reward expected value table. The look-ahead trial table, the individual reward expected value table, and the like will be described later.

[0018] The action probability calculation unit 105 calculates an action probability, which is the selection probability of an action in the look-ahead calculation. The action probability is the probability that an action is selected. The action probability calculation unit 105 assigns a probability distribution of reward weights to the state of the system represented by the nodes of the search tree based on the balance of multiple evaluation indexes.

[0019] The search tree storage unit 12 stores the search tree information created by the creating / updating unit 102. The search tree storage unit 12 also stores a probability distribution of weights that represents the weights with which multiple evaluation indexes are considered for each node of the search tree. The probability distribution of reward weights will be described later.

[0020] Here, an example of a power grid system for creating an action plan will be described. FIG. 2 is a diagram showing an example of a power system. The illustrated structural example includes buses 1 to 4. A transformer for transforming 220 kV to 110 kV is provided between bus 1 and bus 2. A 60 MW consumer is connected to bus 2. Bus 2 and bus 3 are connected by a 70 km power line. A generator and a 70 MW consumer are connected to bus 3. Bus 2 and bus 4 are connected by a 40 km power line, and bus 3 and bus 4 are connected by a 50 km power line. A generator and a 10 MW consumer are connected to bus 4.

[0021] Next, an example of a search tree to be created or updated will be described. 3 is a diagram showing an example of a search tree to be created or updated, in which each node of the search tree corresponds to a state of the target system. The state of the system to be planned at a certain point in time is the 0th state (initial state) S t=0 Then, the state at that time corresponds to the root node of the search tree.

[0022] 0th state (initial state) S t=0 If you take an action at time , the first state S t=1 n (n is an integer greater than or equal to 1), the state of the system changes according to the action. The state after the change becomes a child node of the search tree. In the example of Figure 3, the 0th state (initial state) S t=0 Action A t=1 1 When you perform action A, t=1 2 There are two ways to act when doing this.

[0023] When multiple candidate actions are possible, the state to which the action is to be transitioned will differ depending on the action selected. In the search tree, this is represented by a parent node having multiple child nodes. For example, the first state S t=1 2 If is the parent node, then the state S of the second state t=22 and state S t=2 3 is a child node.

[0024] First state S t=1 n Now, let's say the state after each action is S t=1 1 and state S t=1 2 In the example of Figure 3, there are two nodes in state S t=1 1 There is one action from A t=2 1 ) and state S t=1 2 There are two actions from t=2 2 , A t=2 3 ). In the example of Figure 3, the final state is the 0th state (initial state) S t=0 Each set of paths (tr1, tr2, ...) from a node to the Tth state corresponds to a plan.

[0025] The creating / updating unit 102 creates the 0th state based on, for example, data of a system diagram of a power grid system. Next, the simulation unit 101 and the look-ahead calculation unit 104 calculate the action A from the 0th state based on the probability distribution of the reward weights of the parent node and the child node. t=1 1 The first state S t=1 1 Further, the simulation unit 101 and the look-ahead calculation unit 104 generate an action A from the 0th state based on the probability distribution of the reward weights of the parent node and the child node. t=1 2 The first state S t=1 2 is created through simulation.

[0026] Next, the creating / updating unit 102 creates the first state S t=1 1 The simulation unit 101 and the look-ahead calculation unit 104 select action A.t=2 1 The second state S t=2 1 The creating and updating unit 102 creates the first state S t=1 2 The simulation unit 101 and the look-ahead calculation unit 104 select action A. t=2 2 The second state S t=2 2 And Action A t=2 3 The second state S t=2 3 In this manner, the creating / updating unit 102 creates and updates the search tree.

[0027] Next, an example of the process performed by the behavior probability calculation unit 105 will be described. The action probability calculation unit 105 calculates the weight w∈R of the balance of the multiple evaluation indexes in the state of the system represented by the nodes of the search tree. ≧0 D (R ≧0 is a set of 0 or positive real numbers, and D is the number of evaluation indexes), a probability distribution (hereinafter also referred to as "probability distribution of reward weight" or "probability distribution of weight") indicating whether the evaluation indexes are taken into consideration is assigned. FIG. 4 is an image diagram of assigning a probability distribution according to this embodiment. Note that the example of the probability distribution in FIG. 4 is an example, and is not limited to this. Also, the evaluation index may be two or more, and may be three or more. Note that, regarding the weight w of the balance of multiple evaluation indexes described here, since it indicates that an evaluation index corresponding to a large weight has a greater influence on action selection than an index with a small weight, that is, it is more important, the weight w of each evaluation index may be referred to as "weight importance" hereinafter.

[0028] The example in Fig. 4 shows the results of a search conducted in consideration of two indices, evaluation index 1 and evaluation index 2. The evaluation indexes are, for example, cost, reliability, etc. The prior distribution g11 of the importance of the weights in the 0th state, which is the initial state, has a uniform probability density, that is, various weights w are considered with equal probability.

[0029] The look-ahead calculation unit 104 calculates or obtains the weight importance of the next state. The determination unit 103 aggregates the results of calculations by the look-ahead calculation unit 104 and selects an action from among the multiple actions based on the balance of the weights of the two indices. For example, when transitioning from the 0th state to the 1st state, the determination unit 103 selects action A t=1 1 Is it better to choose action A? t=1 2 Determine whether it is better to select

[0030] In the example of Figure 4, the first state is state S t=1 1 and state S t=1 2 There are two states: t=1 1 The probability distribution g12 of the weights of the evaluation index 2 has a larger value than the evaluation index 1, that is, the evaluation index 2 is more important than the evaluation index 1. t=1 2 In the probability distribution g13 of the weights, the value of the probability distribution of evaluation index 1 is larger than that of evaluation index 2, that is, evaluation index 1 is given more importance than evaluation index 2. From state 0 to state S t=1 1 If the state reaches , it is highly likely that the weighting is set to emphasize evaluation index 2. Similarly, from the 0th state to state S t=1 2 If the result is 0, it is highly likely that the weighting is set to emphasize evaluation index 1.

[0031] For example, when the evaluation index 2 is emphasized, the judgment unit 103 judges that the behavior A t=1 1 If you select and prioritize evaluation index 1, then take action A. t=1 2 The determination unit 103 selects an action according to the weight of the evaluation index and aggregates the results. As a result, according to the embodiment, when reaching a state beyond the first state, by assigning the weights for multiple evaluation indexes as a probability distribution of the weights, it becomes easy to create a plan that corresponds to multiple weights.

[0032] Here, an example of the probability distribution of the importance of the weights will be described. The probability distribution of weights shown in Fig. 4 is, for example, the weights w ∈ R ≧0 D may follow a Dirichlet distribution. The probability density function p of the Dirichlet distribution is defined by the following equation (1). In the following equations, the bold characters represent vectors.

[0033]

number

[0034] In equation (1), w is the weight and α(∈R ≧0 D ) is a parameter, and B(α) is given by the following equation (2).

[0035]

number

[0036] In equation (2), Γ is the gamma function and Π is the product. In this case, the look-ahead calculation unit 104 performs look-ahead based on weights sampled from the Dirichlet distribution of the parent node and tally up the results. At that time, the look-ahead calculation unit 104 stores, for each sampled weight, which action will maximize the weighted sum of multiple rewards.

[0037] After completing the look-ahead simulation, the look-ahead calculation unit 104 checks, for each action, a set of weight samples that maximizes the weighted sum of rewards when that action is selected.

[0038] Here, the assignment of weight distribution will be described. FIG. 5 is a diagram showing an example of weight distribution of rewards according to the Dirichlet distribution. The look-ahead calculation unit 104 estimates the parameters of the Dirichlet distribution by, for example, a maximum likelihood estimation method. The look-ahead calculation unit 104 uses, for example, the method of P. Minka (Reference 1) as an estimation method. Assuming that the confirmed set of weight samples also follows the Dirichlet distribution, the parameter α ∈ R D Furthermore, the look-ahead calculation unit 104 assumes that the distribution of reward weights in child nodes of the search tree follows a Dirichlet distribution of the estimated parameters, and assigns this probability distribution to the child nodes.

[0039] Reference 1; Thomas P. Minka, “Estimating a Dirichlet distribution”, 2003, [Searched 2020.10.9], Internet<URL;https: / / tminka.github.io / papers / dirichlet / minka-dirichlet.pdf>

[0040] 5, the determination unit 103 determines, for each transition destination node, from the weight prior distribution g11, whether the weight probability distribution g21 of evaluation index 2 or the weight probability distribution g23 of evaluation index 1 is selected. The look-ahead calculation unit 104 approximates the weight probability distribution g21 of evaluation index 2 with a Dirichlet distribution to estimate parameters of the weight probability distribution g22. The look-ahead calculation unit 104 approximates the weight probability distribution g23 of evaluation index 1 with a Dirichlet distribution to estimate parameters of the weight probability distribution g24.

[0041] Next, an example of the process performed by the look-ahead calculation unit 104 will be described. FIG. 6 is a diagram showing an example of processing performed by the look-ahead calculation unit 104 according to this embodiment. The simulation unit 101 performs a simulation for multiple future states. Based on the simulation results, the look-ahead calculation unit 104 calculates the expected value Q of the reward in the following formula (3) in order to confirm how much of each of the multiple rewards can be obtained. d Calculate (g101).

[0042]

number

[0043] In addition, equation (3) in Figure 6 is the expected reward Q for the next three steps. d In the formula (3), d indicates the evaluation index, and A indicates S. t From S t+1 A' indicates the action to S t+1 From S t+2 A'' indicates the action to S t+2 From S t+3 In Fig. 6, "act" indicates an action.

[0044] Next, the look-ahead calculation unit 104 records the look-ahead result and creates a look-ahead trial table (table g102). The look-ahead calculation unit 104 may or may not store the created look-ahead trial table in its own portion. Next, the look-ahead calculation unit 104 calculates the expected value of the reward for each initial action selection, and creates an expected value table of individual rewards (table g103). d Calculate the average value of the reward to calculate the expected reward.

[0045] There are multiple methods for selecting an action when reading ahead. For example, the first action selection method randomly selects an available action, while the second action selection method samples actions according to the action probability indicated by a policy function previously learned by reinforcement learning.

[0046] A policy function that takes into account multiple evaluation indexes is expressed, for example, by the following formula (4): Note that formula (4) represents the probability of selecting action A when state S and reward weight w are given.

[0047]

number

[0048] This policy function can be modeled by a neural network. If we consider the combination of state and reward weight settings as a broad state, we can learn the model parameters using a policy-based reinforcement learning algorithm. For example, the policy gradient method by S. Sutton et al. is used as a policy-based reinforcement learning algorithm (Reference 2).

[0049] Reference 2: Richard S. Sutton, David McAllester, Satinder Singh, Yishay Mansour, “Policy Gradient Methods for Reinforcement Learning with Function Approximation”, [Searched 2020.10.9], Internet <URL; https: / / papers.nips.cc / paper / 1713-policy-gradient-methods-for-reinforcement-learning-with-function-approximation.pdf>

[0050] The neural network that models the policy function may have any structure, for example, a fully connected neural network (multilayer perceptron) or a graph neural network.

[0051] The input of the policy function is the feature expressing the state S and the reward weight w. When a fully connected neural network is used, the input is a combined vector of the feature vector of the state S and the reward weight w. When a graph neural network is used, the input is the structure of the system that creates the plan and the graph structure data expressing the state S of the parent node of the search tree, to which the reward weight w data has been added. When using a graph neural network, for example, the method described in JP 2019-204294 A may be used. In addition, the policy function of equation (2) may be realized by using multiple models according to the reward weight w, with the input of the neural network being only the features representing the state S.

[0052] When performing a simulation for multiple steps ahead, a move selection algorithm such as the Pareto UCB (Upper Confidence Bound) algorithm may be used for the first action. The Pareto UCB algorithm is an algorithm that selects moves based on multiple evaluation indexes, and the method proposed by M. Drugan et al. may be used (Reference 3). The move selection algorithm is an algorithm that makes a balanced selection between actions that are known to be likely to yield high rewards based on the results of previous exploration, and actions that have not yet been sufficiently explored.

[0053] Reference 3; Madalina M. Drugan, Ann Nowe, “Designing multi-objective multi-armed bandits algorithms: a study”, The 2013 International Joint Conference on Neural Networks (IJCNN), IEEE, 2013, p2358-2365

[0054] Next, an example of the process performed by the determination unit 103 will be described. If the search tree is branched for all selectable actions, the number of nodes becomes so large that it may not be possible to perform the search up to the Tth state (FIG. 3). Since it is necessary to branch the search tree only at effective points, in this embodiment, the judgment unit 103 performs the branch judgment.

[0055] There are several possible methods for determining branching of a search tree. The first branching judgment method compares the case where the search tree is branched with the case where it is not branched, and branches only if the effect is sufficiently large. Specifically, the look-ahead calculation unit 104 calculates the expected value of the reward when not branching and the expected value of the reward when branching. Then, the judgment unit 103 judges to branch only if the expected value is improved by branching and the difference between the expected values ​​is greater than a predetermined threshold value ε. This makes it possible to branch only in effective places.

[0056] An example of branching decision based on the expected value of reward is shown in FIG. 7. FIG. 7 is an image diagram showing an example of branching decision based on the expected value of reward. In the example of FIG. 7, action A is selected as one of the selectable actions. 1 and A 2 The following are two examples.

[0057] The look-ahead calculation unit 104 calculates the weights of the action A for each weight sampled from the parent node. 1 If you select (A 1 If you want to do this only, then Action A 2 If you select (A 2 If you select the optimal one according to the probability distribution of weights (A 1 and A 2 In table g112, the expected value of the reward for the three cases w = (w 1 ,w 2 ) are the weights of w1 and w2. For example, action A 1 When the above is selected, the look-ahead calculation unit 104 selects A in table g103. 1 Q 1 and Q 2 The expected value of the reward is calculated using the value of and w=(0.9, 0.1). The look-ahead calculation unit 104 calculates the average value of the expected value for each weight.

[0058] If the search tree is not branched, action A is taken regardless of the weight. 2When the optimal action is selected, an expected reward value of 0.55 is obtained. In contrast, when the search tree is branched to select the optimal action, an expected reward value of 0.60 is obtained. In this example, the expected reward value is improved by 0.05 by branching. The determination unit 103 branches only if this score improvement is greater than a threshold value ε (0.60-max(0.47,0.55)>ε).

[0059] When there are three or more selectable actions, the determination unit 103 may select which two actions to select and branch from. Also, the determination unit 103 may select three or more actions according to weights and branch to three child nodes.

[0060] The second branch judgment method focuses on the probability of arrival for multiple reward weights, and branches if the maximum value is equal to or greater than a certain value, and does not branch if not. An example of branch judgment using the probability of arrival for multiple reward weights is shown in FIG. 8. FIG. 8 is a diagram showing an example of branch judgment based on conditional probability of arrival for weights. Note that FIG. 8 shows a state in which three state transition destinations are expected at most when making a branch judgment for t=2. Here, the achievement probability for multiple reward weights is expressed by the following equation (5).

[0061]

number

[0062] Equation (5) shows the probability of which state is selected for the tth action, given the reward weight w. If this probability is low for any weight, then the optimal plan will not include state S. t Since it is considered that there is a low possibility that the state is included, it is considered that there is no need to search for a plan following this state. In such a case, the determination unit 103 does not branch the search tree and add this state to the child node. According to Bayes' law, the probability of reaching multiple reward weights follows the following equation (6).

[0063]

number

[0064] In equation (6), p(S t ) is the probability of reaching each node state, and can be estimated by a look-ahead simulation based on the prior distribution p(S0) set in the 0th action. p(w|S t ) is the weighted distribution of rewards corresponding to each state, and is assigned to child nodes by assigning the probability distribution described above.

[0065] As a result, the value of formula (6) is normalized by multiplying these two probability values ​​for each possible state. Note that the graph g120 shows the conditional attainment probability for the weight in the first action. The hatched area g121 shows the state S t=1 1 The hatched area g122 indicates the conditional probability of reaching state S t=1 2 The graph g130 shows the conditional probability of reaching the state S with respect to the weights in the second action. The hatched area g131 shows the conditional probability of reaching the state S t=2 1 The hatched area g132 indicates the conditional probability of reaching state S t=2 2 The hatched area g133 indicates the conditional probability of reaching state S t=2 3 The dashed line g134 indicates the threshold.

[0066] The determination unit 103 excludes actions that are not included in the optimal plan from the search while taking into account the weighted probability density of the reward. For example, no matter how the evaluation indexes are weighted, for example, in the state S t=2 3 does not reach state S t=2 3 If the conditional probability of reaching the destination path is less than the threshold, the decision unit 103 decides not to branch.

[0067] Next, an example of a processing procedure performed by the plan creation device 1 will be described. FIG. 9 is a flowchart of a processing procedure performed by the plan creation device 1 according to this embodiment.

[0068] The setting unit 14 obtains the operation result and data to obtain the simulation conditions. Next, the simulation unit 101 sets the simulation conditions (step S1).

[0069] The simulation unit 101 initializes the search tree, the weight distribution of rewards, etc. (step S2).

[0070] The creating / updating unit 102 selects a node for performing a simulation (step S3).

[0071] The simulation unit 101 and the look-ahead calculation unit 104 repeat the look-ahead simulation K times (K is an integer equal to or greater than 1) and the processes of steps S5 to S7 from k=1 to K (step S4).

[0072] The look-ahead calculation unit 104 samples weights from the weight distribution of the rewards of the parent nodes (step S5).

[0073] The simulation unit 101 and the look-ahead calculation unit 104 repeat the look-ahead process up to P states ahead and the process of step S7 from τ=t+1 to t+P (step S6).

[0074] The action probability calculation unit 105 calculates the action probability in the look-ahead calculation. Next, the simulation unit 101 and the look-ahead calculation unit 104 perform a simulation of the look-ahead reward (step S7).

[0075] The determination unit 103 determines whether or not the search tree should be branched, and if so, to which node the search tree should be branched (step S8).

[0076] The look-ahead calculation unit 104 calculates the probability distribution of the reward weight of the child node. Next, the creating / updating unit 102 assigns the probability distribution to the child node (step S8).

[0077] The creating / updating unit 102 judges whether or not the selection of all nodes to be simulated is completed (step S10). If the selection of all nodes is completed (step S10; YES), the creating / updating unit 102 ends the process. If the selection of all nodes is not completed (step S10; NO), the creating / updating unit 102 returns to the process of step S3.

[0078] Next, a specific configuration example of the plan creation device 1 will be described. Fig. 10 is a diagram showing an example of the configuration of the plan creation device 1 according to this embodiment. As shown in Fig. 10, the information processing unit 10 includes, for example, a display unit 151, an input unit 152, a communication unit 153, a CPU (Central Processing Unit) 154, a memory 155, and a storage unit 156. The display unit 151, the input unit 152, the communication unit 153, the CPU (Central Processing Unit) 154, the memory 155, and the storage unit 156 are connected via a bus 157.

[0079] The display unit 151 is, for example, a liquid crystal display device, an organic EL display device, or the like, and corresponds to, for example, the display unit 16 . The input unit 152 is, for example, a touch panel sensor, a keyboard, a mouse, or the like, and corresponds to, for example, the setting unit 14 .

[0080] The communication unit 153 is a network interface such as a LAN (Local Area Network) board or a wireless network circuit, and communicates with, for example, an external device according to the control of the CPU 154. The CPU 154 corresponds to, for example, the information processing unit 10 .

[0081] The memory 155 includes, for example, a RAM (Random Access Memory) and a ROM (Read only memory). The storage unit 156 includes, for example, an SSD (Solid State Drive) or an HDD (Hard Disc Drive), and corresponds to, for example, the search tree storage unit 12. The storage unit 156 also stores a control program. The configuration shown in FIG. 10 is just an example, and the present invention is not limited to this.

[0082] All or part of the functions of the plan creation device 1 or the information processing unit 10 may be, for example, a software function unit.

[0083] According to at least one of the embodiments described above, by having the information processing unit 10, it is possible to create a plan faster than before.

[0084] As described above, according to this embodiment, the conventional drawback of it being difficult to select an appropriate action when searching ahead can be compensated for, and multiple proposed plans with different balances of importance can be quickly created. Furthermore, according to this embodiment, when a state beyond the first state is reached, the weights for multiple evaluation indexes are assigned as a probability distribution, making it easy to create a plan that corresponds to multiple weights. Moreover, according to this embodiment, it is possible to exclude actions that are not included in the optimal plan from the search while taking into account the probability distribution of the reward weights.

[0085] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope of the invention and its equivalents described in the claims, as well as in the scope and spirit of the invention. [Explanation of symbols]

[0086] 1... plan creation device, 10... information processing unit, 12... search tree storage unit, 14... setting unit, 16... display unit, 101... simulation unit, 102... creation / update unit, 103... judgment unit, 104... look-ahead calculation unit, 105... action probability calculation unit

Claims

1. a creating / updating unit that creates or updates a search tree in which each state in the action plan corresponds to a node; a look-ahead calculation unit that alternately executes action selection and simulation and performs look-ahead calculation of future states and rewards from a first state; an action probability calculation unit that calculates an action probability, which is a selection probability of an action in the look-ahead calculation; a simulation unit that performs a simulation in the look-ahead calculation based on the action probability; a determination unit that determines whether or not to branch the search tree depending on a result of the look-ahead calculation; Equipped with The weight probability distribution represents the balance of multiple evaluation indicators for each node of the search tree, The look-ahead calculation unit estimates a balance of a plurality of rewards when an action is selected based on a result of the simulation; and the creating / updating unit assigns a probability distribution of the weights to a child node of a transition destination of the action; Planning device.

2. the weight probability distribution is a Dirichlet distribution, the look-ahead calculation unit calculates whether an expected value of reward will be large when each action is selected for a plurality of weight samples sampled from the Dirichlet distribution of the parent node, and estimates parameters of the Dirichlet distribution of the child node from the plurality of weights assigned to each action by a maximum likelihood estimation method; The creating and updating unit assigns a Dirichlet probability distribution to the child node. The plan creation device according to claim 1 .

3. the action probability calculation unit reads a policy function that has been learned in advance, and calculates the action probability according to a state of a parent node of the search tree and a weight sample sampled from the weight probability distribution; the simulation unit performs a look-ahead simulation of the behavior selected by sampling from the distribution of the behavior probability by the look-ahead calculation unit; The plan creation device according to claim 1 .

4. the action probability calculation unit reads a policy function modeled by a graph neural network, and inputs graph structure data representing the structure of the system that creates the plan and the state of a parent node of the search tree to calculate an action probability; the simulation unit performs a look-ahead simulation of the behavior selected by sampling from the distribution of the behavior probability by the look-ahead calculation unit; The plan creation device according to claim 3.

5. When performing a simulation of a plurality of future states, the look-ahead calculation unit selects a first action by a Pareto UCB (Upper Confidence Bound) algorithm. The plan creation device according to claim 1 .

6. the determination unit compares an expected value of a reward obtained when the search tree is branched and an expected value of a reward obtained when the search tree is not branched according to the simulation result, and determines to branch when a certain value or more is obtained as a result of the comparison; the creating / updating unit branches and updates the search tree in accordance with the determined result; The plan creation device according to claim 1 .

7. a search tree recorder for storing the created search tree and for storing, for each node of the search tree, a probability of the node being selected as part of the optimal plan; The determination unit calculates a conditional arrival probability for a node for each reward weight, and determines not to branch to a node having a low arrival probability for any weight. The plan creation device according to claim 1 .

8. A creating / updating unit creates or updates a search tree in which each state in the action plan corresponds to a node; A look-ahead calculation unit alternately executes action selection and simulation, and performs look-ahead calculation of future states and rewards from the first state; an action probability calculation unit calculates an action probability which is a selection probability of an action in the look-ahead calculation; a simulation unit performing a simulation in the look-ahead calculation based on the action probability; a determination unit that determines whether or not to branch a search tree according to a result of the look-ahead calculation; The weight probability distribution represents the balance of multiple evaluation indicators for each node of the search tree, the look-ahead calculation unit estimates a balance of a plurality of rewards when an action is selected based on a result of the simulation; the creating / updating unit assigns a probability distribution of the weights to a child node of a transition destination of the action; How to create a plan.

9. On the computer, creating or updating a search tree in which each state in the action plan corresponds to a node; Alternate between action selection and simulation, and perform look-ahead calculations of future states and rewards from the first state. Calculating an action probability, which is a selection probability of an action in the look-ahead calculation; performing a simulation in the look-ahead calculation based on the action probability; determining whether or not to branch the search tree depending on a result of the look-ahead calculation; The weight probability distribution represents the balance of multiple evaluation indicators for each node of the search tree, Based on the results of the simulation, we estimate the balance of multiple rewards when an action is selected, A probability distribution of the weights is assigned to a child node of a transition destination of the action. program.

Images