Information processing device, information processing method, and program

Abstract

The present invention provides an information processing device, an information processing method, and a program capable of making a change plan for social infrastructure. The information processing device has a definition unit, a determination unit, and a reinforcement learning unit. The definition unit defines attributes by establishing correspondences with nodes and edges, and defines a convolution function related to a model of the data representing the graph structure based on the data of the graph structure representing the structure of the system. The evaluation unit inputs the state of the model system, and for a system of one or more changed models that make the model produce conceivable structural changes at each time step, calculates a policy function and a state value function at each time step, and evaluates the structural change of the system based on the policy function. The reinforcement learning unit performs reinforcement learning by using the cost generated when the structural change is applied to the system, i.e., the reward value, the state value function, and the model, so as to optimize the structural change of the system.

Description

Technical Field

[0001] Embodiments of the present invention relate to an information processing device, an information processing method, and a program. Background Art

[0002] In recent years, the aging problem has been listed as a major issue in social infrastructure systems. For example, in the power system, the aging of substation equipment worldwide is becoming increasingly serious, and it is important to formulate an equipment investment plan. Experts in various fields have developed solutions to such equipment investment plan issues. Regarding the planning method applicable to social infrastructure systems, it is sometimes necessary to meet the necessary conditions of large-scale, diversity, and variability. However, in the existing technology, there is a problem that changes in composition cannot be handled.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2007-80260

[0006] Non-patent literature

[0007] Non-patent document 1: Nagata Masayuki and Takehara Arisa, Averaging support tool for updating energized equipment taking into account supply reliability constraints - Development of a prototype -, Research report R08001, Electric Power Research Institute, February 2009 Summary of the invention

[0008] Problems to be solved by the invention

[0009] An object of the present invention is to provide an information processing device, an information processing method, and a program capable of creating a change proposal for social infrastructure.

[0010] Means used to solve problems

[0011] The information processing device of the embodiment has a definition unit, a determination unit, and a reinforcement learning unit. The definition unit defines attributes by establishing correspondence with nodes and edges, and defines a convolution function related to a model of the data representing the graph structure based on the data of the graph structure representing the structure of the system. The evaluation unit inputs the state of the system to the model, and for a system of one or more changed models that cause the model to produce a conceivable structural change at each time step, obtains a policy function given as a probability distribution of the structural change and a state value function required for reinforcement learning at each time step, and evaluates the structural change of the system based on the policy function. The reinforcement learning unit performs reinforcement learning by using the cost generated when the structural change is applied to the system, i.e., the reward value, the state value function, and the model, so as to optimize the structural change of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 FIG. 1 is a diagram showing an example of an evaluation power system model.

[0013] Figure 2 This is a diagram showing a configuration example of an actual system.

[0014] Figure 3 This is a diagram showing an example of definition of the types of assumed nodes AN.

[0015] Figure 4 It is used to illustrate Figure 3 In the configuration of ＊ Figure 1 shows an example of a

[0016] Figure 5 It means according to Figure 4 A graph of a neural network generated from graph-structured data.

[0017] Figure 6 It is a block diagram of a neural network generation device.

[0018] Figure 7 This is a diagram showing how a neural network is generated from graph-structured data.

[0019] Figure 8 It is used to determine the coefficient α of the neural network generation device i，j FIG. 1 is a diagram illustrating the method.

[0020] Fig. 9 It is a block diagram showing a configuration example of an information processing device according to an embodiment.

[0021] Fig.10 This is a diagram showing a mapping example of convolution processing and attention processing in an implementation manner.

[0022] Fig.11 This is a diagram for explaining an example of selection management of changes performed by a meta-graph structure sequence management function unit according to the embodiment.

[0023] Fig.12 This is a diagram showing the flow of information in an example of a learning method performed by the information processing device according to the first embodiment.

[0024] Fig.13 This is a diagram for explaining an example of the candidate node processing function according to the second embodiment.

[0025] Fig.14 This is a diagram for explaining parallel value inference using candidate nodes.

[0026] Fig.15 This is a diagram for explaining the flow of facility change plan (inference) calculation according to the third embodiment.

[0027] Fig.16 It is a diagram for explaining parallel inference processing.

[0028] Fig.17 This is a diagram showing an example of the overall functional configuration of inference.

[0029] Fig.18 This is a diagram showing cost examples for disposal, new installation, and replacement of equipment in an equipment change plan of an electric power system.

[0030] Fig.19 It is a diagram showing a learning curve for the equipment change planning problem of the power system.

[0031] Fig. 20 It is a graph showing the entropy evaluation for each learning step.

[0032] Fig.21 This is a diagram showing a specific plan that minimizes the cumulative cost among the generated plans.

[0033] Fig. 22 The diagram shows an example of an image displayed on a display device.

[0034] Description of Reference Numerals

[0035] 100…Neural network generation device, 1…Information processing device, 11…Management function unit, 12…Graph Convolution Neural Network, 13…Reinforcement learning unit, 14…Operation unit, 15…Image processing unit, 16…Prompt unit, 111…Metagraph structure sequence management function unit, 112…Convolution function management function unit, 113…Neural network management function unit, 2…Environment, 3…Display device, S…State of the system, S'…New state of the system, A…Action DETAILED DESCRIPTION

[0036] Hereinafter, the information processing device, information processing method and program of the embodiment will be described with reference to the accompanying drawings. Hereinafter, in the following description, the processing processed by the information processing device is described by taking the equipment change plan as an example. In addition, the present embodiment is not limited to the equipment change plan problem for the social infrastructure system.

[0037] First, an example of a power system will be described.

[0038] Figure 1 is a diagram showing an example of an evaluation power system model. Figure 1Thus, the evaluation power system model includes AC power sources V_0 to V_3, transformers T_0 to T_8, and buses B1 to B14. The bus is a concept of a "location" that connects power supply sources and power demanders.

[0039] In the equipment change here, for transformer T_0 between bus B4 and bus B7, transformer T_1 between bus B4 and bus B9, transformer T_2 between bus B5 and bus B6, transformer T_3 between bus B7 and bus B8, transformer T_4 between bus B7 and bus B9, transformer T_5 between bus B4 and bus B7, transformer T_6 between bus B4 and bus B9, transformer T_7 between bus B5 and bus B6, and transformer T8 between bus B7 and bus B9, it is assumed that one of the three options of "addition", "discard", and "maintenance" is selected. Since there are three options for each transformer, the number of combinations when there are n (n is an integer greater than 1) transformers is 3. n When considering such equipment changes, it is necessary to consider the operating cost (maintenance cost) of the transformer equipment, the installation cost, and the risk cost caused by system downtime, etc.

[0040] In the implementation mode, in order to perform equipment changes, the actual system is first represented by a graph structure.

[0041] Figure 2 This is a diagram showing an example of the structure of an actual system. The example of the structure shown in the figure includes bus 1 to bus 4. A transformer that transforms 220 [kV] to 110 [kV] is provided between bus 1 and bus 2. A demander of 60 [MW] is connected to bus 2. Bus 2 and bus 3 are connected by a 70 [km] power line. A generator and a demander of 70 [MW] are connected to bus 3. Bus 2 and bus 4 are connected by a 40 [km] power line, and bus 3 and bus 4 are connected by a 50 [km] power line. A generator and a demander of 10 [MW] are connected to bus 4.

[0042] exist Figure 2 In such a configuration, if the bus is considered as an actual node, the transformer is considered as an actual edge of type "T", and the power line is considered as an actual edge of type "L", then it can be as follows Figure 3 Express it that way. Figure 3 This is a diagram showing an example of the definition of the type of the hypothetical node AN. Reference symbol g1 represents an example of the content of the graph structure data, and reference symbol g2 schematically represents the case of converting the actual node RN and the actual edge RE into the hypothetical node AN. In reference symbol g1, RN (Bx) (x is an integer from 1 to 4) represents the actual node, and RE (Ly) (y is an integer from 1 to 3) and RE (T1) represent the actual edge.

[0043] In the embodiment, the data of the graph structure of reference numeral g1 is converted into a hypothetical node meta-graph (reference numeral g3) as shown in reference numeral g2. In addition, the method of converting the data of the graph structure into the hypothetical node meta-graph will be described later. In reference numeral g2, AN(Bx), AN(T1), and AN(Ly) represent actual nodes. In the following description, the graph of reference numeral g2 is referred to as a meta-graph.

[0044] Next, in Figure 3 In the configuration of ＊ The following example is used to illustrate this. Figure 4 It is used to illustrate Figure 3 In the configuration of ＊ In addition, the device T1 is added ＊ The same type as the device T1. Reference numeral g5 indicates the added device T1. ＊ .

[0045] When expressed as a neural network structure Figure 4 When the meta-graph is Figure 5 Express it that way. Figure 5 It means according to Figure 4 The graph structure data is a graph of a neural network generated by the graph structure data. Reference numeral g11 indicates that the device T1 is not added. ＊ The neural network of the system, reference numeral g12 represents the additional device T1 ＊ Thus, in the embodiment, the convolution function corresponding to the added device is added to the network. Since the deletion of the device is the reverse action of the addition, the corresponding node of the meta-node and its connection link are deleted. In addition, since the added device T1 ＊ Same type as T1, so device T1 ＊ The convolution function of W is the same as that of T1. L (1) With W B (1) is the propagation matrix of the first intermediate layer, W L (2) With W B (2) is the propagation matrix of the second intermediate layer. L is the propagation matrix from the assumed node of node L. The propagation matrix W B is a propagation matrix from the assumed node of the node B. In addition, for example, B4' represents an assumed node of the first intermediate layer, and B4" represents an assumed node of the second intermediate layer.

[0046] Thus, the change of a device is equivalent to the change of the convolution function corresponding to the device (local processing). The addition of a device is equivalent to the addition of a convolution function. The abandonment of a device is equivalent to the deletion of a convolution function.

[0047] Next, a configuration example of the neural network generation device 100 will be described.

[0048] Figure 6 : is a block diagram of the neural network generation device 100. The neural network generation device 100 includes, for example, a data acquisition unit 101, a storage unit 102, a network processing unit 103, and an output unit 104.

[0049] The data acquisition unit 101 acquires graph structure data from an external device, for example, and stores it in the storage unit 102. Instead of acquiring graph structure data from an external device, the data acquisition unit 101 may acquire (read) graph structure data pre-stored in the storage unit 102, or may acquire graph structure data input by a user using an input device.

[0050] The storage unit 102 is implemented, for example, by a RAM (Random Access Memory), a HDD, a flash memory, etc. The data of the graph structure stored in the storage unit 102 is, for example, data that represents the graph structure as records of actual nodes RN and actual edges RE. In addition, the data of the graph structure may be given a feature quantity as the initial state of each actual node RN. In addition, the feature quantity as the initial state of the actual node RN may also be prepared as a data set different from the data of the graph structure.

[0051] The network processing unit 103 includes, for example, an actual node / actual edge adjacency relationship extraction unit 1031 , a tentative node meta-graphization unit 1032 , and a meta-graph convolution unit 1033 .

[0052] The actual node·actual edge adjacency relationship extraction unit 1031 refers to the data of the reference structure to extract the actual nodes RN and actual edge RE in an adjacency relationship (connection relationship). For example, the actual node·actual edge adjacency relationship extraction unit 1031 comprehensively extracts the actual nodes RN or actual edge RE in an adjacency relationship (connection relationship) for each actual node RN and actual edge RE, and stores them in the storage unit 102 in a corresponding form.

[0053] The assumed node meta-graphing unit 1032 generates a neural network that connects the states of the assumed nodes AN in layers to connect the actual nodes RN and the actual edges RE extracted by the actual node and actual edge adjacency relationship extraction unit 1031. At this time, the assumed node meta-graphing unit 1032 follows the rules based on the above-mentioned graph attention network to determine the propagation matrix W and the coefficient α in a manner consistent with the purpose of the above-mentioned neural network.i，j .

[0054] The meta-graph convolution unit 1033 inputs the feature value of the initial value of the actual node RN in the assumed node AN into the neural network to derive the state (feature value) of the assumed node AN in each layer. By repeatedly executing this step, the output unit 104 outputs the feature value of the assumed node AN to the outside.

[0055] The provisional node feature storage unit 1034 stores the feature that is the initial value of the actual node RN. The provisional node feature storage unit 1034 stores the feature derived by the meta-graph convolution unit 1033.

[0056] Next, a method for generating a neural network from graph-structured data is described.

[0057] Figure 7 This is a diagram showing how a neural network is generated from graph-structured data. Figure 7 In FIG. 1 , reference numeral g7 represents a graph structure. Reference numeral g8 represents a neural network. In addition, the neural network generation device 100 generates a neural network.

[0058] As shown in the figure, the neural network generation device 100 sets not only the actual node RN but also the hypothetical node AN including the actual edge RE, and generates a neural network that propagates the feature quantity of the k-1th layer of the hypothetical node AN to other hypothetical nodes AN in a connection relationship and the feature quantity of the kth layer of the hypothetical node AN itself. k is a natural number greater than 1, and the layer with k=0 refers to, for example, the input layer.

[0059] The neural network generation device 100 determines the feature value of the first intermediate layer based on, for example, the following formula (1). Formula (1) corresponds to a method for calculating the feature value h1# of the first intermediate layer of the provisional node (RN1).

[0060] As an example, α 1，12 is a coefficient indicating the degree of propagation between the assumed node (RN1) and the assumed node (RE12). The feature quantity h1## of the second intermediate layer of the assumed node (RN1) is expressed by the following equation (2). The feature quantities of the third intermediate layer and thereafter are determined in the same manner.

[0061] [Formula 1]

[0062] h1#=α 1，1 ·W·h1+α 1，12 ·W·h 12 +α 1，13 ·W·h 13 +α 1，14 ·W·h 14 …(1)

[0063] [Formula 2]

[0064] h1##=α 1，1 ·W·h1#+α 1，12 ·W·h 12 #+α 1，13 ·W·h 13 #+α 1，14 ·W·h 14 #…(2)

[0065] The neural network generation device 100 determines the coefficient α according to the rule based on the graph attention network, for example. i，j . Figure 8 is used to explain the determination coefficient α of the neural network generation device 100 i，j The neural network generation device 100 combines the feature value h of the assumed node RNi of the propagation source i The vector Wh obtained by multiplying the propagation matrix W i The vector (Whj) obtained by multiplying the feature amount hj of the assumed node RNj of the propagation destination by the propagation matrix W is i , Whj) inputs a separate neural network a (attention), inputs the vector of the output layer into the activation function such as sigmoid function, ReLU, softmax function, normalizes it, and adds it, thereby deriving the coefficient α i，j The individual neural network a obtains parameters and the like in advance for the phenomenon to be analyzed.

[0066] The neural network generation device 100 follows the above rules to determine the parameters (W, α) of the neural network in accordance with the purpose of the neural network. i，j ). The purpose of the neural network is to output a future state, output an indicator for evaluating a state, or classify the current state when assuming that the node AN is in the current state.

[0067] Next, a configuration example of the information processing device 1 will be described.

[0068] Fig. 9 1 is a block diagram showing a configuration example of the information processing device 1 according to the embodiment. Fig. 9 In this way, the information processing device 1 includes a management function unit 11, a graph convolutional neural network 12, a reinforcement learning unit 13, an operation unit 14, an image processing unit 15, and a prompt unit 16. The management function unit 11 includes a meta-graph structure sequence management function unit 111, a convolution function management function unit 112, and a neural network management function unit 113. In addition, the environment 2 and the display device 3 are connected to the information processing device 1.

[0069] The environment 2 is, for example, a simulator, a server device, a database, a personal computer, etc. The environment 2 receives a change proposal as an action from the information processing device 1. The environment calculates a state incorporating the change, calculates a report, and returns it to the information processing device 1.

[0070] The display device 3 is, for example, a liquid crystal display device, and displays the image output by the information processing device 1 .

[0071] The information processing device 1 has the functions of the above-mentioned neural network generation device 100, and constructs a graph neural network and updates it based on machine learning. For example, the management function unit 11 can also have the function of the neural network generation device 100. In addition, the graph neural network can also be generated in advance. The information processing device 1 implements a neural network change based on the change plan obtained from the environment 2, estimates the value of the value function (Value), and performs reinforcement learning processing such as TD (Temporal Difference) calculation based on the reward fed back from the environment. The information processing device 1 updates coefficient parameters such as the convolution function based on the results of reinforcement learning. In addition, the convolution network can also be a multi-layer neural network composed of connecting the convolution functions corresponding to each device. In addition, each convolution function can also include attention processing as needed. In addition, the model is not limited to a neural network, for example, it can also be a support vector machine.

[0072] The meta-graph structure sequence management function unit 111 obtains a change information signal reflecting the equipment change as a part of the "state signal" from the environment 2. When the meta-graph structure sequence management function unit 111 obtains the change information signal, it defines the meta-graph structure corresponding to the corresponding new system configuration and formulates the corresponding neural network structure. At this time, the meta-graph structure sequence management function unit 111 formulates a neural network structure for inferring and calculating the evaluation value of the value function and the policy function required to efficiently process the change plan. In addition, the meta-graph structure sequence management function unit 111 refers to the convolution function corresponding to the changed part from the convolution function management function unit 112, and constructs a meta-graph equivalent to the actual system configuration from the convolution function set. Then, the meta-graph structure management function unit 111 performs a change in the meta-graph structure equivalent to the equipment change (updates the graph structure in accordance with the action, sets the "candidate node", etc.). The meta-graph structure sequence management function unit 111 defines and manages attributes by establishing correspondence with nodes and edges. In addition, the meta-graph structure management function unit 111 has a part of the functions of the neural network generation device 100 described above. The meta-graph structure sequence management function unit 111 is an example of a “definition unit”.

[0073] The convolution function management functional unit 112 has a function of defining convolution functions corresponding to device types and a function of updating parameters of convolution functions. The convolution function management functional unit 112 manages convolution modules or attention modules corresponding to partial meta-graph structures. The convolution function management functional unit 112 defines convolution functions related to the model of data representing the graph structure based on data of a graph structure representing the structure of the system. In addition, the partial meta-graph structure is a library function of separate convolution functions corresponding to nodes or edges of each device type. The convolution function management functional unit 112 updates the parameters of each convolution function during the learning process. In addition, the convolution function management functional unit 112 has a part of the functions of the above-mentioned neural network generation device 100. In addition, the convolution function management functional unit 112 is an example of a "definition unit".

[0074] The neural network management function unit 113 obtains the convolution module or attention module corresponding to the neural network structure formulated by the meta-graph structure sequence management function unit 111 and the partial meta-graph structure managed by the convolution function management function unit 112. The neural network management function unit 113 has the function of converting the meta-graph into a multi-layer neural network, the output function definition function of the neural network required for reinforcement learning, and the update function of the parameter set of the above-mentioned convolution function or neural network. In addition, the functions required for reinforcement learning are, for example, reward functions, policy functions, etc. In addition, the output function definition refers to, for example, a fully connected multi-layer neural network with the output of the convolution function as input. In addition, full connection refers to the way in which each input is connected to all other inputs. In addition, the neural network management function unit 113 has a part of the functions of the above-mentioned neural network generation device 100. In addition, the neural network management function unit 113 is an example of an "evaluation unit".

[0075] The graph convolutional neural network 12 stores, for example, an attention-type graph convolutional network composed of multiple convolutions as a deep neural network.

[0076] The reinforcement learning unit 13 performs reinforcement learning using the graph convolutional neural network constructed by the graph convolutional neural network 12 and the state and reward output by the environment. The reinforcement learning unit 13 changes the parameters based on the results after reinforcement learning and outputs the changed parameters to the convolution function management function unit 112. In addition, the reinforcement learning method will be described later.

[0077] The operation unit 14 is a keyboard, a mouse, a touch panel sensor provided on the display device 3 , etc. The operation unit 14 detects an operation by a user and outputs the detected operation result to the image processing unit 15 .

[0078] The image processing unit 15 generates an image related to the evaluation environment and an image related to the evaluation result based on the operation result, and outputs the generated images, namely, the image related to the evaluation environment and the image related to the evaluation result, to the presentation unit 16. The image related to the evaluation environment and the image related to the evaluation result will be described later.

[0079] The presentation unit 16 outputs the image output by the image processing unit 15 to the environment 2 and the display device 3 .

[0080] Next, we will explain how to standardize the equipment change plan sequence based on the equipment's attention and convolutional models. Fig.10 This is a diagram showing a mapping example of convolution processing and attention processing in this embodiment.

[0081] First, the actual system is represented by a graph structure (S1). Next, the edge type and function attribute are set according to the graph structure (S2). Next, it is represented by a metagraph (S3). Next, network mapping is performed (S4).

[0082] Reference numeral g20 is an example of network mapping. Reference numeral g21 is an edge convolution module. Reference numeral g22 is a graph attention module. Reference numeral g23 is a time series recognition module. Reference numeral g24 is a state value function V(s) inference module. Reference numeral g25 is an action probability p(a|s) calculation module.

[0083] Here, the equipment change planning problem can be defined as a reinforcement learning problem. That is, the equipment change planning problem can be defined as a reinforcement learning problem by setting the graph structure and the parameters of each node and edge (equipment) as states, the addition and deletion of equipment as actions, and the income and cost obtained as rewards.

[0084] An example of selection management of changes performed by the meta-graph structure sequence management function unit 111 will be described. Fig.11 This is a diagram for explaining an example of selection management of changes performed by the meta-graph structure sequence management function unit 111 .

[0085] Here, as an initial state (t=0), a graph structure with four nodes such as that denoted by reference numeral g31 is considered.

[0086] From this state, as change candidates at the next time t=1, n (n is an integer greater than or equal to 1) options are considered, as indicated by the reference numerals g41, g42, ..., g4n in the middle section.

[0087] According to each of these options, an option for the next time t = 2 is derived. Reference numerals g51, g52, ... denote example options from the graph structure of reference numeral g43.

[0088] In this way, the selection sequence is expressed as a sequence of metagraphs that reflect the changes, in other words, a sequence of node changes. In the embodiment, reinforcement learning is used as a means of extracting a sequence suitable for the strategy from such a sequence.

[0089] In the embodiment, the graph neural network formed by the information processing device 1 always corresponds to the system configuration on the environment side. In addition, the information processing device 1 promotes reinforcement learning through the new state S, the reward value obtained based on it, the value function inferred by the neural network side, and the policy function as the evaluation result on the environment side.

[0090] (First Embodiment)

[0091] An example of a learning method performed by the information processing device 1 is described. Here, as a learning method, an example using A3C (Asynchronous Advantage Actor-Critic) is described, but the learning method is not limited to this. In addition, in the embodiment, reinforcement learning is used as a means of extracting a sequence suitable for reward from a selection sequence. In addition, reinforcement learning may also be deep reinforcement learning, for example.

[0092] Fig.12 1 is a diagram showing the flow of information in an example of a learning method performed by the information processing device 1 of this embodiment. Fig.12 In the example, environment 2 includes an external environment DB (database) 21 and a system environment 22. System environment 22 includes a physical model simulator 221, a reward calculation unit 222, and an output unit 223. In addition, the device type is represented by a convolution function. Furthermore, the graph structure of the system is represented by the graph structure of the convolution function group.

[0093] The data stored in the external environment DB 21 is external environment data, etc. Environment data includes, for example, specifications of device nodes, demand data in power systems, information related to graph structures, and is a parameter that affects action decisions regardless of environmental conditions or behavior.

[0094] The physical model simulator 221 includes, for example, a flow simulator, a traffic simulator, a physical model, a function, an equation, a simulator, a real machine, etc. The physical model simulator 221 obtains data stored in the external environment DB 21 as needed, and performs simulation using the obtained data and the physical model. The physical model simulator 221 outputs the simulation results (S, A, S') to the feedback calculation unit 222. S is the state of the system (Last State), A is the extracted action, and S' is the new state of the system.

[0095] The reward calculation unit 222 calculates the reward value R using the simulation results (S, A, S') obtained from the physical model simulator 221. The calculation method of the reward value R will be described later. The reward value R is, for example, {(R1, a1), ..., (R T , a T )}. Here, T is the equipment planning study period. In addition, a p (p is an integer from 1 to T) for each node, for example, a1 is the first node, a p is the pth node.

[0096] The output unit 223 uses the new state S′ of the system as the state S of the system, and outputs the state S of the system and the feedback value R to the information processing device 1 .

[0097] The neural network management function unit 113 of the management function unit 11 inputs the state S of the system output by the environment 2 into the neural network stored in the graph convolutional neural network 12, and obtains the policy function π(·|S, θ) and the state value function V(S, w). Here, w is a weight coefficient matrix (also called a convolution term) corresponding to the attribute dimension of the node. The neural network management function unit 113 uses the following formula (3) to determine the action (equipment change) A in the next step.

[0098] [Formula 3]

[0099] A~π(·|S,θ)…(3)

[0100] The neural network management function unit 113 outputs the action (equipment change) A in the next step determined to the environment 2. That is, the policy function π(·|S, θ) is input to the state S of the system of the research object and outputs the action (action). In addition, the neural network management function unit 113 outputs the obtained state value function V(S, w) to the reinforcement learning unit 13. In addition, the policy function π(·|S, θ) for selecting the action is given as the probability distribution of the action candidate for the metagraph structure change.

[0101] In this way, the neural network management function unit 113 obtains the policy function and the state value function required for reinforcement learning for each time step for the system of one or more changed models that cause the neural network to produce a conceivable structural change at each time step, and evaluates the structural change of the system based on the policy function. In addition, the neural network management function unit 113 can also evaluate the plan of structural change of the system or the plan of equipment change.

[0102] The reinforcement learning unit 13 receives the state value function V(S, w) output by the management function unit 11 and the reward value R output by the environment 2. The reinforcement learning unit 13 uses the input state value function V(S, w) and the reward value R to perform reinforcement machine learning by a machine learning method such as A3C, and repeats the sequence of actions (movements) a number of times equivalent to the equipment planning research period (T). The reinforcement learning unit 13 outputs the parameters <ΔW>π and <Δθ>π obtained as a result of the reinforcement machine learning to the management function unit 11.

[0103] The convolution function management functional unit 112 updates the parameters of the convolution function based on the parameters output by the reinforcement learning unit 13 .

[0104] The neural network management function unit 113 reflects the updated parameters <ΔW>π, <Δθ>π in the neural network, and evaluates the neural network in which the parameters are reflected.

[0105] In addition, in selecting the next action, the management function unit 11 can use the candidate nodes (see Figure 4 , Figure 5 ), or not to use it.

[0106] Next, an example of the reward function is described.

[0107] The first example of a reward function is (bias) - (equipment setup, decommissioning, operation, and maintenance costs).

[0108] The first example of the reward function may be to model the cost for each device (function) and define a positive reward value by subtracting the cost from the bias. The bias is a parameter appropriately set to a certain positive value in order to make the reward function value a positive value.

[0109] The second example of the reward function is (bias) - (risk cost). Depending on the device configuration, the physical system conditions may not be met. The situation where the conditions are not met is, for example, the connection conditions are not met, the process is unbalanced, the output conditions are not met, etc. In the case of such a large risk, a negative large reward (risk) can also be imposed.

[0110] The third example of the reward function may be a combination of the first to third examples of the reward function.

[0111] In this way, in this embodiment, various reward functions can be designed as in the first to third examples.

[0112] (Second Embodiment)

[0113] In this embodiment, an example of selecting the next action using the candidate nodes will be described.

[0114] The meta-graph structure sequence management function unit 111 may also use a candidate node processing function. In this embodiment, a method for connecting a function candidate with the possibility of adding a device node as a next action (action) candidate to a meta-graph and performing value inference on multiple action candidates in parallel is described. In addition, the configuration of the information processing device 1 is the same as that of the first embodiment.

[0115] As a feature of the attention-type neural network, even if a node is added, by adding the learned convolution function corresponding to the node to the neural network, the effect of the addition can be effectively analyzed and evaluated without re-learning. The reason is that the constituent elements of the graph structure neural network based on the graph attention network are represented as convolution functions, and the whole is represented as a graph connection of the function group. That is, when using candidate nodes, it can be decomposed into a neural network that represents the entire system and a convolution function that constitutes the added node for management.

[0116] Fig.13 1 is a diagram for explaining an example of the candidate node processing function of the present embodiment. Reference numeral g101 is a metagraph in step length t, and reference numeral g102 is a neural network in step length t. Reference numeral g111 is a metagraph in step length t+1, and reference numeral g112 is a neural network in step length t+1.

[0117] The management function unit 11 evaluates the possibility of adding as a change candidate. Fig.13 As shown in the reference numeral g111, the candidate node is connected to the meta graph using a unidirectional connection candidate. Thus, the management function unit 11 processes the candidate node as a convolution function of a unidirectional connection.

[0118] The management function unit 11 adds node T1 for evaluation ＊ The value of the case is as shown in the reference numeral g112, by moving from nodes B1 and B2 to T1 ＊ unidirectional connection and execute in parallel with T1 and T1 ＊ Node-related value calculation (strategy function, state value function). In addition, reference numeral g1121 is the reward difference of T1, and reference numeral g1122 is the reward difference of T1. ＊ Additional Reward Difference The estimation of the reward value of the two-dimensional action indicated by reference numeral g112 can be performed in parallel.

[0119] Therefore, in this embodiment, as the node (T1, T1 ＊ ), the four combinations of {(yes, yes), (yes, no), (no, yes), (no, no)} can be evaluated simultaneously. As a result, according to this embodiment, since evaluation can be performed in parallel, calculations can be performed at a high speed.

[0120] Fig.14 It is a diagram for illustrating parallel value inference using candidate nodes. Reference numeral g151 is a meta-graph of state S in step t. Reference numeral g161 is a meta-graph of state S1 (yes, no) involved in action A1 in step t+1. Reference numeral g162 is a meta-graph of state S2 (yes, yes) involved in action A2 in step t+1. Reference numeral g163 is a meta-graph of state S3 (no, yes) involved in action A3 in step t+1. Reference numeral g164 is a meta-graph of state S4 (no, no) involved in action A4 in step t+1. Reference numeral g171 is a meta-graph of candidate node T1. ＊ Virtually connected to the metagraph of state S.

[0121] In addition, Fig.14 In FIG. 1 , it is assumed that in a system in state S at step length t, an action of adding or maintaining nodes between B1 and B2 is selected. The management function unit 11 determines an option based on which option can obtain a higher reward under the condition.

[0122] Here, in the case of S4 (none, none) among the four combinations, B1 and B2 are not connected in the system, and the system does not exist. In this case, the management function unit 11 generates a large risk cost (penalty). In addition, in this case, the management function unit 11 performs reinforcement learning in parallel based on the value function value from the neural network and the policy function for states S1 to S4.

[0123] (Third Embodiment)

[0124] In this embodiment, an example of performing parallel processing of the processing of sampling the plan sequence scenarios will be described. In addition, the configuration of the information processing device 1 is the same as that of the first embodiment.

[0125] Fig.15 This is a diagram for explaining the flow of facility change plan (inference) calculation according to the present embodiment. Fig.15 The main calculation processes and signal flows in the case of creating and learning different external environment data using the policy function obtained by the A3C learning function to plan a facility change plan (change sequence) are exemplified.

[0126] The information processing device 1 samples the plans using the convolution function of each device obtained. Then, the information processing device 1 outputs the plans in the order of cumulative scores, for example, the order of lowest cost.

[0127] The external environment DB 21 stores, for example, demand data in the power system, data related to equipment specifications, or external environment data sets different from learning data such as a system graph structure.

[0128] The policy function is constructed by using a graph neural network constructed using learned convolution functions (learned parameters: θπ).

[0129] The system state S is used as input and the following formula (4) is used to determine the next action (device node change).

[0130] [Formula 4]

[0131] A~π(·|S,θπ)…(4)

[0132] The management function unit 11 extracts a strategy based on the strategy function (probability distribution of each action) corresponding to the state according to formula (4). The management function unit 11 inputs the extracted action A into the system environment and calculates the new state S' and the reward value R associated with it. The new state S' is used as an input for determining the next step. Accumulated during the reward study period. The management function unit 11 repeatedly performs this operation for a step length equivalent to the study period and obtains each cumulative reward score (G).

[0133] Fig.16 It is a diagram for explaining parallel inference processing.

[0134] A series of change plans during the study period is equivalent to an equipment change plan. The cumulative reward score corresponding to the plan is calculated. The set of combinations of the plan solutions and their scores calculated in this way becomes the plan solution candidate set.

[0135] First, the management function unit 11 samples a plan (action sequence {at}t) from the policy function obtained by learning in episode units, and obtains a score.

[0136] Next, the management function unit 11 selects, for example, using the argmax function to extract the plan {A1, ..., AT} corresponding to the largest test among the G values ​​of the trial (test) results. The management function unit 11 can further extract a higher-level plan.

[0137] According to this embodiment, the processing of sampling each plan sequence scheme can be processed in parallel (in Fig.16 is N times).

[0138] In order to process the policy function in parallel, normalization is required in the output layer. For example, the following formula (5) is used in normalization.

[0139] [Formula 5]

[0140]

[0141] In formula (5), the preference function is the output node of the object, and the product of coefficient θ and vector xπ(s t , a, θ).

[0142] Here, a case of handling multi-dimensional actions will be described.

[0143] If the action space is a two-dimensional space, a=(a1, a2) can be considered as the direct product of the two spaces, which can be expressed as the following equation (6): a1 is the first node, and a2 is the second node.

[0144] [Formula 6]

[0145] h(s t ,a,θ)=h(s t ,a1,θ)+h(s t , a2, θ)…(6)

[0146] That is, the preference function only needs to be calculated and added for each space. t The same, then each preference function can be calculated in parallel.

[0147] Fig.17 is a diagram showing an example of the overall functional configuration of the inference. Fig.15 .

[0148] The device node update strategy model g201 is a learned strategy function, which represents the probability distribution of action selection for each step learned in the above process.

[0149] The subject setting function g202 is a function for defining and setting subjects such as the initial system configuration, initialization of each node parameter, external environment data, test data, and cost model.

[0150] The task formalization function g203 is a function that formalizes the task defined by the task setting function and the learned policy function used as the update policy model with reinforcement learning. It includes the study period (plot), strategy (cumulative cost minimization, averaging), action space, environment state space, evaluation score function formalization (definition), etc.

[0151] The change sequence sample extraction and cumulative score evaluation function g204 generates a required number of action sequences as samples from the learned policy function in the defined environment and agent environment.

[0152] The best cumulative score plan display function g205 selects the sample with the best score from the sample set, or arranges the samples in order of scores for presentation.

[0153] The function setting UIg 206 is a user interface for setting each function unit.

[0154] Next, a specific calculation example of an equipment change plan is described.

[0155] Here, an example of a method in which the embodiment is applied to the following problems will be described. As an evaluation power system model, a Figure 1 IEEE Case 14 (Electrical Engineering, U. of Washington) shown.

[0156] The task is to search for a plan with the minimum cumulative cost in a series of 30-step equipment update sequences. The initial state is Figure 1 In this way, a total of 9 transformers (T_x) of the same specification are equipped between the buses. Figure 1 As shown in the figure, for the transformers between buses B5-B6, B4-B9, B7-B9, and B4-B7, one of the three actions "add", "discard", and "keep as is" can be selected for each node at each step. That is, there are 3×3×3×3=81 action spaces.

[0157] Regarding the cost to be considered, the installation cost is charged for each equipment node of the transformer, the cost corresponding to the time elapsed and the load power value is charged, and if the equipment changes make the conditions for the establishment of the environment difficult, a large penalty value is charged as the cost. In addition, the conditions for the establishment of the environment are, for example, power flow balance.

[0158] The main points of the topic are as follows.

[0159] I. System of Systems Model; IEEE Case 14

[0160] II. Topic: Develop an equipment change plan for the installation and deletion of IEEE Case 14 transformers in a way that minimizes cost during the planning period (30 update opportunities).

[0161] III. Conditions;

[0162] III-1; Initial state: Transformers (V_x) of the same specifications are provided between the buses.

[0163] III-2; The operating cost of each transformer equipment is set as the (weighted) sum of the following three costs (installation cost, maintenance cost, and risk cost).

[0164] ·Setup costs; temporary costs

[0165] Maintenance cost: cost corresponding to the passage of time and load power value

[0166] Risk cost: the cost of damage in case of system downtime (large)

[0167] IV. Rewards of reinforcement learning; (reward) = (reward bias) - (operating cost)

[0168] Reinforcement learning action: Periodically select one of the equipment strategy options (add, abandon, or do nothing) for each transformer.

[0169] V. Demand load curve for year Y data

[0170] VI. Specifications of generators and lines are based on the IEEE model

[0171] VII. Evaluation (inference); formulate equipment change plan corresponding to the power demand data of the year after year Y

[0172] Fig.18 It is a diagram showing examples of the costs of scrapping, installing, and replacing equipment in the equipment change plan of the power system. Each cost can also be further classified in this way, and a cost coefficient can be set for each cost. For example, the transformer addition cost is a temporary cost, and the cost coefficient is 0.1. In addition, the transformer removal cost is a temporary cost, and the cost coefficient is 0.01. Such cost classification and cost coefficient setting are pre-set. The classification and setting of costs can also be set by the designer of the system based on actual operations performed in the past. In the embodiment, the installation cost and operation and maintenance cost of each device are incorporated as a function in this way.

[0173] The learning curve of the A3C learning result is shown in Fig.19 . Fig.19 is a graph showing the learning curve for the equipment change planning problem in the power system. Fig.19 In the figure, the horizontal axis is the number of learning update steps, and the vertical axis is equivalent to the above-mentioned cumulative reward value. In addition, reference numeral g301 is a learning curve of the average value. Reference numeral g302 is a learning curve of the central value. Reference numeral g303 is the average value of the random plan for comparison. Reference numeral g304 is the central value of the random plan for comparison. Fig.19 The equipment change plan is generated as a sample based on the policy function updated at each learning step, and the average and median value of the cumulative reward value of the sample set are shown. Fig.19 In this way, it can be seen that strategies that can get higher scores can be obtained through learning.

[0174] Fig. 20is a graph showing the entropy evaluation for each learning step. In addition, Fig.10 The entropy shown is the mutual entropy of a random strategy with the same system composition. Fig. 20 In the figure, the horizontal axis is the number of learning update steps, and the vertical axis is the average entropy. After the number of learning steps exceeds 100,000 times, the average entropy converges to a range of about -0.05 to -0.09.

[0175] In addition, although the progress of the learning process can be understood through the learning curve, the actual equipment change plan needs to be generated using the policy function obtained from the learning process. To this end, the following selection criteria can be set: calculate 1,000 plans and the cumulative reward value of each plan, and extract the plan that can achieve the minimum cumulative reward value as the selection strategy from the sequence, or extract the top three from the minimum cumulative reward value.

[0176] In addition, when the information processing device 1 creates a plan based on the strategy, it generates a plan change plan for the research period based on the strategy function and manages it in association with the accumulated reward value (for example, Plan k :{At~π(·|S t )} t →G k ).

[0177] Fig.21 This is a diagram showing a specific plan that minimizes the cumulative cost among the generated plans. Each row is a separate device node, and each column indicates the change timing (for example, every week). Fig.21 In addition, the "rightward arrow" means doing nothing, "remove" means abandoning or removing the equipment, and "newly install" means adding the equipment.

[0178] exist Fig.21 In the figure, a series of action sequences for each device are shown from the initial state 0 to the 29th update opportunity (29 cycles). As the initial state, there are 9 devices whose nodes are deleted and added as the sequence progresses. Fig.21 As in the example shown, by also presenting the cost of the entire system at each timing, the user can easily understand that the cumulative value is smaller than the cumulative values ​​of other plans.

[0179] Fig. 22 The diagram shows an example of an image displayed on the display device 3.

[0180] The image with reference numeral g401 is an example of an image representing the evaluation object system using a meta-graph. The image with reference numeral g402 is an image of a circuit diagram corresponding to the actual system. The image with reference numeral g403 is an example of an image representing the evaluation object system using a neural network structure. The image with reference numeral g404 is an example of an image representing the top three plans with the lowest cost in the cumulative cost. The image with reference numeral g405 is an example of an image representing a specific equipment change plan with the first cumulative minimum cost (for example, Fig.21 ).

[0181] Thus, in the embodiment, samples that meet the conditions and have good scores (low-cost samples) are extracted from the sample plan set. Fig. 22 In this way, multiple items are selected and displayed. In addition, as a plan, equipment change plans are displayed in sequence for each sample.

[0182] In this way, the information processing device 1 displays the system meta-diagram and the plan on the display device 3 ( Figure 1 ). The information processing device 1 may also extract samples that meet the conditions and have good scores from the sample plan set, and select the top multiple items for display. The information processing device 1 may also display the equipment change plan as a plan for each sample sequence. The information processing device 1 displays the conditions from problem setting to environment setting, learning function setting, acquisition of learning-based strategy function, inference using the acquired strategy function, i.e., formulation of the equipment change plan, based on the operation result of the user operating the operation unit 14. In addition, the displayed image may also be an image of a graph, table, or the like.

[0183] The user can adopt the optimal plan according to the environment and situation by checking the displayed images or graphs of the plan, cost, etc.

[0184] Next, extraction filters such as averaging and parameter change will be described. The information processing device 1 may use extraction filters such as averaging and parameter change in extracting the optimal plan.

[0185] The first extraction example is to select a plan that satisfies the set level of averaging from the set M. The second extraction example is to make a plan by changing the coefficient of the cost function. In the second extraction example, for example, the coefficient dependency is evaluated. The third extraction example is to change the initial state of each device and make a plan. In the third extraction example, for example, the initial state dependency (the initial years of study period, etc.) is evaluated.

[0186] According to at least one of the embodiments described above, by having a convolution function management function unit, a meta-graph structure sequence management function unit, a neural network management function unit and a reinforcement learning unit, it is possible to create a change plan for social infrastructure.

[0187] In addition, according to at least one of the embodiments described above, a neural network to which candidate nodes are connected is used in the system, and a combination of the connected nodes and the candidate nodes is evaluated by parallel processing, thereby enabling faster processing.

[0188] Furthermore, according to at least one of the above-described embodiments, since plans with good scores are presented on the display device 3 , it is easy for the user to study the plans.

[0189] In addition, the functional parts of the neural network generation device 100 and the information processing device 1 are implemented by executing programs (software) through hardware processors such as CPU (Central Processing Unit). Some or all of these components can be implemented by hardware (circuit parts; including circuit systems (circuitry)) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit), etc., or they can be implemented by the cooperation of software and hardware. The program can be pre-stored in a storage device such as HDD (Hard Disk Drive) or flash memory, or it can be installed by storing it in a removable storage medium such as DVD, CD-ROM, etc. and installing the storage medium in a drive device.

[0190] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other ways, and various omissions, substitutions, and changes can be made without departing from the scope of the subject matter of the invention. These embodiments and their variations are included in the scope and subject matter of the invention, and are also included in the invention described in the claims and their equivalents.

Claims

1. An information processing device comprising: A definition unit that defines attributes by establishing correspondence between nodes and edges, and defines a convolution function associated with a model representing the graph-structured data based on the graph-structured data representing the structure of the power system; an evaluation unit that inputs the state of the power system into the model, obtains a policy function given as a probability distribution of the change and a state value function required for reinforcement learning for each time step for one or more changed models, and evaluates the structural change of the power system based on the policy function, wherein the changed model produces the conceivable change compared to the model before the change at each time step; and The reinforcement learning unit optimizes the structural change of the power system by performing reinforcement learning using a reward value which is a cost generated when the structural change is applied to the power system, the state value function, and the model after the change.

2. The information processing device according to claim 1, The definition unit defines a convolution function according to a device type of a device included in the power system.

3. The information processing device according to claim 1, The reinforcement learning unit outputs the coefficients, i.e., parameters, of the convolution function obtained as a result of the reinforcement learning to the definition unit, The definition unit updates the parameters of the convolution function based on the parameters output by the reinforcement learning unit, The evaluation unit reflects the updated parameters in the model, and evaluates the model reflecting the parameters.

4. The information processing device according to claim 1, The definition unit includes, in the power system, a candidate for changing the graph structure as a candidate node in the graph structure, and configures the candidate node as the convolution function with a unidirectional connection. The evaluation unit constructs the model using the convolution function of the unidirectional connection.

5. The information processing device according to claim 4, The evaluation unit evaluates each combination of a node connected to the candidate node and the candidate node by parallel processing for the model to which the candidate node is connected in the graph structure.

6. The information processing device according to claim 1, The information processing device further includes a presenting unit configured to present the configuration change of the power system evaluated by the evaluating unit together with a cost associated with the configuration change of the power system.

7. An information processing method, causing a computer to execute the following steps: defining attributes by establishing correspondence with nodes and edges, and defining a convolution function associated with a model representing the graph structure data based on the graph structure data representing the structure of the power system; The state of the power system is input to the model, and for one or more changed models, a policy function given as a probability distribution of the change and a state value function required for reinforcement learning are obtained at each time step, and the structural change of the power system is evaluated based on the policy function, wherein the changed model produces the conceivable change at each time step compared to the model before the change; The structural change of the power system is optimized by performing reinforcement learning using a reward value, which is a cost generated when the structural change is applied to the power system, the state value function, and the changed model.

8. A computer-readable storage medium storing a program, the program causing a computer to execute the following steps: defining attributes by establishing correspondence with nodes and edges, and defining a convolution function associated with a model representing the graph structure data based on the graph structure data representing the structure of the power system; The state of the power system is input to the model, and for one or more changed models, a policy function given as a probability distribution of the change and a state value function required for reinforcement learning are obtained at each time step, and the structural change of the power system is evaluated based on the policy function, wherein the changed model produces the conceivable change at each time step compared to the model before the change; The structural change of the power system is optimized by performing reinforcement learning using a reward value, which is a cost generated when the structural change is applied to the power system, the state value function, and the changed model.

Images