Neural network computing device and computing method

Abstract

To provide a calculation system of a neural network capable of modeling an object of modeling using an expression based on a physical rule.SOLUTION: A calculation system 1 according to the present invention includes a structure optimization unit 16 that is connected to an apparatus 3 in which first and second sensors 31 are mounted, and that inputs a detection value of the first sensor 31, and a structural formula transformation unit 14 that deduces a prediction formula, which is an expression representing a relationship between the detection value of the first sensor 31 and a prediction value of a detection value of the second sensor 31, using a neural network (NN). The NN includes a node having a value represented by an expression which has an output value of the node as a base and contains a power whose exponent is a weight coefficient set to an edge linked to the node. The structure optimization unit 16 calculates the structure of the NN and the weight coefficient using the detection value of the first sensor 31. The structural formula transformation unit 14 uses the structure of the NN and the weight coefficient, which are calculated by the structure optimization unit 16, to deduce as the prediction formula an expression having an item containing a power.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a computing device and a computing method for modeling plants, industrial equipment, etc. using a neural network.

Background Art

[0002] In plants, industrial equipment, etc., control and diagnosis may be performed by modeling them. When modeling plants, industrial equipment, etc., modeling in a form according to physical laws based on their characteristics is often performed. However, physical modeling requires a large amount of labor and its accuracy greatly depends on the expertise of the operator. Therefore, as a means to replace physical modeling, there is a modeling method using a mathematical model such as a neural network. Using this method, modeling can be automatically performed if a sufficient amount of data is prepared, and high-precision prediction results can be expected.

[0003] However, different from physical modeling, a neural network has poor explicability and interpretability for the calculated prediction results, and it is difficult to easily understand and explain the reason why the prediction results are obtained. Therefore, the process related to prediction has become a bottleneck when introducing a neural network in industrial fields dealing with plants, industrial equipment, etc. where an explanation responsibility is required.

[0004] From such a background, a neural network with high interpretability that is easy for general technicians to understand is desired.

[0005] A neural network consists of multiple layers, each composed of one or more nodes, with nodes in different layers connected by edges. In a typical neural network, each node uses a nonlinear function called an activation function to transform the input. Furthermore, the nodes defined in each layer are connected by edges to all nodes defined in the next layer. Due to these characteristics, the mathematical formulas that represent neural networks are complex nonlinear functions. This is the main reason why neural networks are considered a black box.

[0006] An example of a conventional neural network is disclosed in Patent Document 1. The neural network disclosed in Patent Document 1 simplifies the nonlinear and complex mathematical formulas of conventional neural networks, making it possible to model them with mathematical formulas that are easy for users to understand. [Prior art documents] [Patent Documents]

[0007] [Patent Document 1] Japanese Patent Publication No. 2002-42106 [Overview of the Initiative] [Problems that the invention aims to solve]

[0008] Conventional neural networks, such as the technology disclosed in Patent Document 1, have not solved the problem of the neural network being a black box. For example, in the neural network disclosed in Patent Document 1, the mathematical formulas are simplified to make them easier for users to understand, but the formulas that represent the output values ​​are merely approximations and do not necessarily perform modeling in accordance with physical laws. Therefore, it is difficult to ensure consistency with the physical laws underlying the object being modeled in the neural network in Patent Document 1.

[0009] The objective of the present invention is to provide a neural network computing device and method that can model a target object using mathematical formulas based on physical laws. [Means for solving the problem]

[0010] The neural network computing device according to the present invention is connected to a device on which a first sensor and a second sensor are installed, and comprises a structure optimization unit that inputs the detected value of the first sensor, and a structure formula conversion unit that uses a neural network to derive a prediction formula which is a mathematical formula representing the relationship between the detected value of the first sensor and the predicted value of the detected value of the second sensor. The neural network includes nodes whose values ​​are expressed by a mathematical formula that includes exponentiation, with the output value of the node as the base and the weight coefficient set on the edge connected to the node as the exponent. The structure optimization unit calculates the structure of the neural network and the weight coefficient from the detected value of the first sensor. The structure formula conversion unit uses the structure of the neural network and the weight coefficient calculated by the structure optimization unit to derive a mathematical formula that includes the term with exponentiation as the prediction formula.

[0011] The neural network calculation method according to the present invention is performed on a computing device connected to equipment on which a first sensor and a second sensor are installed, and comprises a structure optimization step in which the computing device inputs the detected value of the first sensor, and a structure formula conversion step in which the computing device uses the neural network to derive a prediction formula which is a mathematical formula that represents the relationship between the detected value of the first sensor and the predicted value of the detected value of the second sensor. The neural network includes nodes whose values ​​are expressed by a mathematical formula that includes a power, with the output value of the node as the base and the weight coefficient set on the edge connected to the node as the exponent. In the structure optimization step, the computing device calculates the structure of the neural network and the weight coefficient from the detected value of the first sensor. In the structure formula conversion step, the computing device uses the structure of the neural network and the weight coefficient calculated in the structure optimization step to derive a mathematical formula that includes a term with a power as the prediction formula. [Effects of the Invention]

[0012] According to the present invention, it is possible to provide a neural network computing device and computing method that can model a target of modeling using mathematical formulas based on physical laws. [Brief explanation of the drawing]

[0013] [Figure 1] A diagram showing the configuration of a neural network computing device according to Embodiment 1 of the present invention. [Figure 2] A schematic diagram showing the structure of the neural network used by the computing device to derive mathematical formulas (prediction formulas). [Figure 3] A flowchart illustrating the processing flow performed by a computing device. [Figure 4] This diagram shows an example of the configuration of some of the data (constraints related to input / output signals) stored in the structural constraint memory unit. [Figure 5A] This diagram shows an example of the configuration of some of the data stored in the structural constraint memory (constraints regarding the presence or absence of edges in nodes responsible for sum calculations in the intermediate layer). [Figure 5B] This diagram shows an example of the configuration of some of the data stored in the structural constraint memory unit (constraints regarding the presence or absence of edges in nodes that are responsible for calculating the product in the intermediate layer). [Figure 5C] This diagram shows an example of the configuration of some of the data stored in the structural constraint memory unit (constraints regarding the presence or absence of edges connecting the nodes of the intermediate layer and the nodes of the output layer). [Figure 6] A diagram showing an example of the configuration of structural optimization conditions stored in the structural optimization condition memory unit. [Figure 7] A diagram showing an example of the configuration of sensor data stored in the sensor data storage unit. [Figure 8] A flowchart of the process performed by the NN structure optimization unit in S6 of Figure 3. [Figure 9] A diagram showing an example of the data structure stored in the analysis results storage unit. [Figure 10]An example of a screen for a user to set constraints on input and output signals of a neural network. [Figure 11] An example of a screen for a user to set constraints on the presence or absence of edges between each node. [Figure 12] An example of a screen for a user to set structure optimization conditions. [Figure 13] An example of a screen for outputting a mathematical formula (prediction formula) derived by a computing device in Example 1. [Figure 14] In Example 2 of the present invention, a schematic diagram showing the structure of a neural network used by a computing device for deriving a mathematical formula (prediction formula). [Figure 15] An example of a screen for outputting a mathematical formula (prediction formula) derived by a computing device in Example 2.

Mode for Carrying Out the Invention

[0014] In the neural network computing device and computing method according to the present invention, for a modeling target such as a plant or industrial equipment, the input-output relationship can be modeled using a mathematical formula based on physical laws. Therefore, the user can confirm whether the neural network obtained by the present invention matches the physical characteristics of the modeling target, and can improve the explicability and interpretability of the prediction results calculated by the neural network.

[0015] Hereinafter, the neural network computing device and computing method according to an embodiment of the present invention will be described with reference to the drawings. In the drawings used in this specification, the same or corresponding components are denoted by the same reference numerals, and repeated explanations of these components may be omitted.

Examples

[0016] This section describes a neural network computing device and calculation method according to Embodiment 1 of the present invention. The computing device according to this embodiment uses equipment such as plants and industrial machinery as the target of modeling, and uses data of values ​​detected by sensors installed on this equipment (sensor detection values) to derive a mathematical formula that represents the relationship between the sensor detection values ​​and the predicted detection values ​​of other sensors installed on the equipment (predicted detection values ​​of other sensors) using a neural network. The equipment to be modeled is modeled using this mathematical formula. The calculation method according to this embodiment is a neural network calculation method executed by the computing device according to this embodiment.

[0017] In the following explanation, neural networks may sometimes be abbreviated as "NN".

[0018] Figure 1 shows the configuration of the neural network computing device according to this embodiment. The computing device 1 is connected to the device 3 via a control system 4 that controls the device 3 to be modeled.

[0019] Device 3 is modeled using mathematical formulas derived by computing device 1. Device 3 is equipped with multiple sensors 31.

[0020] The control system 4 comprises a control device 41 and a sensor data storage unit 42. The control device 41 is a device that implements control logic for receiving data from a sensor 31 installed in the device 3 (for example, the detected value of the sensor 31) and outputting a control signal for the device 3. The sensor data storage unit 42 is a storage device that stores the data from the sensor 31 (sensor data) input by the control device 41.

[0021] The computing device 1 is composed of a computer and is equipped with an arithmetic processing unit such as a CPU and a storage device such as memory, and executes the calculation method according to this embodiment. The computing device 1 is connected to the input / output device 2 and the device 3, and its functional components include an input / output control unit 11, a structural constraint condition storage unit 12, an NN structural constraint setting unit 15, a structural optimization condition storage unit 13, an NN structural optimization unit 16, an analysis result storage unit 17, and a structural formula conversion unit 14.

[0022] The computing device 1 receives data from the device 3, which is equipped with multiple sensors 31. The computing device 1 receives the detected values ​​of the sensors 31 as data (sensor data) from the sensors 31 installed in the device 3, and derives a formula that represents the relationship between the detected values ​​of sensor 31 and the predicted detected values ​​of the other sensors 31 (predicted detected values ​​of the other sensors 31). The other sensors 31 are sensors 31 installed in the device 3 that are different from the sensors 31 from which the detected values ​​were input as sensor data.

[0023] Input / output device 2 is an input device for the user to input information into computing device 1, and is also an output device equipped with a screen that displays data calculated by computing device 1 and information entered by the user.

[0024] The components of computing device 1 are briefly described below, with a more detailed explanation to follow later.

[0025] The input / output control unit 11 processes information entered by the user using the input / output device 2, and processes data and information to be displayed on the input / output device 2.

[0026] The structural constraint memory unit 12 is composed of a memory device and stores constraints regarding the input and output signals of the neural network and constraints regarding the presence or absence of edges between each node, which are input by the user using the input / output device 2.

[0027] The NN structure constraint setting unit 15 includes an input / output signal setting unit 151 and an edge constraint setting unit 152, and performs the process of setting constraint conditions for the structure of the NN used in calculations. The input / output signal setting unit 151 takes the constraint conditions for the input / output signals of the NN stored in the structure constraint condition storage unit 12 as input and sets these constraint conditions as constraint conditions during NN calculations. The edge constraint setting unit 152 takes the constraint conditions for the presence or absence of edges between each node stored in the structure constraint condition storage unit 12 as input and sets these constraint conditions as constraint conditions when determining the structure of the NN.

[0028] The structure optimization condition storage unit 13 is composed of a memory device and stores the structure optimization conditions entered by the user using the input / output device 2. The structure optimization conditions include, for example, the upper limit of the number of terms in the formula determined by the computing device 1, the range of initial values ​​for the NN weight coefficients, and the number of NN structures to try.

[0029] The NN structure optimization unit 16 receives structure optimization conditions from the structure optimization condition storage unit 13 and sensor data from the device 3 from the sensor data storage unit 42, and uses the constraint conditions set by the NN structure constraint setting unit 15 to calculate the structure of the NN and the weight coefficients. In calculating the weight coefficients, the NN structure optimization unit 16 updates the weight coefficients until they converge, as will be described later.

[0030] The analysis result storage unit 17 is composed of a memory device and stores the structure of the NN (for example, the presence or absence of edges between each node) and the weight coefficient values ​​calculated by the NN structure optimization unit 16. The analysis result storage unit 17 stores the structure of the NN and the weight coefficient values ​​each time the weight coefficient values ​​converge during the calculation of the weight coefficients by the NN structure optimization unit 16. The analysis result storage unit 17 can also store the structure of the NN and the weight coefficients obtained during the convergence process (i.e., when the weight coefficient values ​​have not yet converged).

[0031] The structural formula conversion unit 14 uses the NN structure and weight coefficient values ​​stored in the analysis result storage unit 17 to derive a formula that represents the relationship between the input signal and the output signal, that is, a formula (prediction formula) that represents the relationship between the detected value of sensor 31 and the predicted detected value of other sensors 31. The structural formula conversion unit 14 derives the prediction formula each time the NN structure optimization unit 16 updates the weight coefficients. In other words, the structural formula conversion unit 14 can derive the prediction formula not only using the weight coefficient values ​​when the weight coefficients have converged, but also using the weight coefficient values ​​when the weight coefficients have not converged.

[0032] The computing device 1, equipped with the above-described components, can derive a mathematical formula (prediction formula) expressed by the neural network, that is, a prediction formula that models the device 3 and represents the relationship between the detected value of sensor 31 and the predicted detected value of other sensors 31.

[0033] Here, we will explain the configuration of the neural network (NN) used by computing device 1 to derive the mathematical formula (prediction formula).

[0034] Figure 2 is a schematic diagram showing the structure of the neural network (NN) used by the computing device 1 to derive the mathematical formula (prediction formula). The NN consists of three layers: an input layer, a hidden layer, and an output layer. The elements that make up the NN are represented by lines called edges and by circles called nodes. In the example shown in Figure 2, the input layer has n nodes indicated by the subscript j (j=1 to n), the hidden layer has (m+k) nodes indicated by the subscript i (i=1 to (m+k)), and the output layer has 1 node.

[0035] In Figure 2, the edges represented by thick lines are the edges that the NN in this embodiment possesses, while the edges represented by thin lines are edges that the NN in this embodiment does not possess. The NN in this embodiment has a structure in which only some nodes are connected by edges between layers, and only the edges represented by thick lines are present. Conventional NNs have the edges represented by thick lines and the edges represented by thin lines in Figure 2, and all nodes are connected by edges between layers. The NN in this embodiment can be considered to have a structure in which the edges represented by thin lines in conventional NNs have been removed.

[0036] Node j in the input layer is set with n input signal values ​​xj (j=1 to n) as input values. The input signals are the values ​​detected by sensor 31. Then, node j in the input layer outputs xj as an output value.

[0037] Next, we will explain the calculation from the input layer to the hidden layer. Each edge is assigned a weight coefficient w(j,i). j and i represent the nodes j and i located at both ends of the edge. That is, w(j,i) represents the weight coefficient set for the edge connecting node j in the input layer and node i in the hidden layer. For each edge, a power is calculated with the output xj (J=1~n) of the node in the input layer as the base and the weight coefficient w(j,i) as the exponent.

[0038] In the hidden layer, the values ​​calculated at each edge are input, and their sum h or product z is calculated. This can be expressed mathematically as follows:

[0039]

number

[0040]

number

[0041] δ is a function that outputs 0 if there are no edges connecting the nodes, and 1 if there are. In this embodiment, according to the constraints set by the edge constraint setting unit 152 (Figure 1) regarding the presence or absence of edges between each node, not all nodes in the layers are connected by edges, but only some nodes are connected by edges.

[0042] Node i in the hidden layer takes the output value xj of node j in the input layer as its input value, and has a value that can be expressed by a formula involving a power, with this input value xj as the base and the weight coefficient w set on the edge connecting node j and node i as the exponent. Furthermore, as shown in formulas (1) and (2), node i in the hidden layer has a node that calculates the sum h of these powers and a node that calculates the product z of these powers. In the example shown in Figure 2, the first m nodes i (i=1 to m) in the hidden layer calculate the sum h using formula (1), and the latter k nodes i (i=(m+1) to (m+k)) calculate the product z using formula (2). The number of nodes m that calculate the sum h and the number of nodes k that calculate the product z can be arbitrarily determined, and a sufficient number should be set to represent the formula.

[0043] Next, we will explain the calculation from the hidden layer to the output layer. The output layer has only one node. The weight coefficient set on the edge connecting node i in the hidden layer and the node in the output layer is denoted by w'j. The node in the output layer outputs an output value y. The output value y is a value calculated using the mathematical formula (prediction formula) expressed by the NN.

[0044] The mathematical formula (prediction formula) expressed by the NN having the structure described above can be expressed in the form of formula (3), that is, in the form of a linear sum of terms that include exponents.

[0045]

number

[0046] In equation (3), the output value y is determined by calculating not only the sum h but also the product z for node i in the hidden layer. By calculating the product z as well, a more accurate equation based on physical laws can be obtained.

[0047] The main differences between the neural network (NN) used in the computing device 1 of this embodiment and conventional general neural networks are as follows:

[0048] The first point is that, as shown in equations (1) to (3), the output value y is obtained by performing a power calculation with the input value xj of the NN as the base and the weight coefficient w(j,i) as the exponent. Physical laws are generally often expressed in mathematical formulas that include a power term. In this embodiment, by expressing the mathematical formula represented by the NN in a mathematical formula that includes a power term with the input value as the base and the weight coefficient as the exponent, it is possible to perform modeling using a mathematical formula based on physical laws.

[0049] The second point is that, as shown in equation (2), the node in the hidden layer calculates the product z, which is input to the values ​​calculated at each edge (the values ​​obtained by exponentiation). As mentioned above, by calculating the product z as well as the sum h, we can obtain a more accurate equation based on physical laws.

[0050] The third point is that only some nodes are connected by edges between layers, and not all nodes are connected by edges. The structure of the NN, that is, which nodes are connected to which nodes between layers, is determined by δ in equations (1) to (3). As the number of edges connecting the nodes in each layer increases, the mathematical formulas that the NN represents also become more complex. Therefore, the formulas can be simplified by reducing the number of edges. In this embodiment, since nodes are connected to only some nodes between layers, it is possible to simplify the mathematical formulas that the NN represents by reducing the number of edges and making it easier to ensure consistency with physical laws.

[0051] The computing device 1 according to this embodiment has the above-described features and can express mathematical formulas (physical formulas) representing physical characteristics as shown in equation (3) as prediction formulas. Therefore, with the computing device 1 according to this embodiment, the user can easily understand the reason why the output value of NN (predicted detection value of sensor 31) was obtained, and the explainability and interpretability of the output value can be improved.

[0052] Figure 3 is a flowchart showing the processing flow of computing device 1. Using Figure 3, we will explain the processing flow of computing device 1 in deriving mathematical formulas using the NN described above.

[0053] In S1, when the user uses the input / output device 2 to instruct the computing device 1 to perform calculations to derive a mathematical formula, the NN structure constraint setting unit 15 obtains this calculation execution instruction through the input / output control unit 11.

[0054] In S2, the input / output signal setting unit 151 of the NN structure constraint setting unit 15 receives constraint conditions related to the input / output signals of the NN from the structure constraint condition storage unit 12, and sets the input constraint conditions as constraint conditions for NN calculation.

[0055] Figure 4 shows an example of the configuration of some of the data stored in the structural constraint memory unit 12. As an example, Figure 4 shows the constraints related to the input and output signals of the neural network (NN) among the data stored in the structural constraint memory unit 12. These constraints identify which sensor 31 each input and output signal belongs to by the name of the sensor 31. In the example shown in Figure 4, the constraint is that the NN calculates the predicted detected value of sensor Z, which is the output signal, from the detected values ​​(measured values) of sensors A to D, which are the input signals. The input signals are signals from multiple sensors, and the output signals are signals from a single sensor. This corresponds to the input and output layer nodes of the NN shown in Figure 2.

[0056] The constraint data for the input and output signals of the neural network is set in advance by the user using the input / output device 2 and stored in the structural constraint storage unit 12.

[0057] Let's return to the explanation of Figure 3.

[0058] In S3, the edge constraint setting unit 152 of the NN structure constraint setting unit 15 receives constraint conditions regarding the presence or absence of edges between each node from the structure constraint condition storage unit 12, and sets the input constraint conditions as constraint conditions when determining the structure of the NN.

[0059] Figures 5A to 5C show examples of the configuration of some of the data stored in the structural constraint memory unit 12. As an example, Figures 5A to 5C show constraints regarding the presence or absence of edges between each node (constraints for determining the structure of the neural network) among the data stored in the structural constraint memory unit 12. In Figures 5A to 5C, the symbol "×" in the table indicates that there are no edges between nodes. Edges exist between nodes that do not have the symbol "×".

[0060] Figures 5A and 5B show examples of constraints regarding the presence or absence of edges between nodes, specifically which nodes have edges connecting the input layer nodes and the hidden layer nodes, and which do not. Figure 5A shows the nodes responsible for calculating the sum h shown in equation (1), i.e., the hidden layer nodes i (i=1 to m) shown in Figure 2. Figure 5B shows the nodes responsible for calculating the product z shown in equation (2), i.e., the hidden layer nodes i (i=(m+1) to (m+k)) shown in Figure 2.

[0061] Figure 5C shows an example of constraints regarding the presence or absence of edges between nodes, specifically which nodes have edges connecting the nodes in the hidden layer and the nodes in the output layer, and which do not.

[0062] The constraint data regarding the presence or absence of edges between each node is set in advance by the user using the input / output device 2 and stored in the structural constraint storage unit 12. The user can set these constraints, for example, in response to signals detected by the sensor 31. As described above, in conventional neural networks, all nodes are connected by edges between layers. This constraint determines which edges are removed from the conventional neural network in the neural network of this embodiment in order to facilitate consistency with physical laws.

[0063] Let's return to the explanation of Figure 3.

[0064] In S4, the NN structure optimization unit 16 receives structure optimization conditions from the structure optimization condition storage unit 13. The structure optimization conditions are learning conditions for the NN that have been set in advance by the user.

[0065] Figure 6 shows an example of the configuration of structural optimization conditions stored in the structural optimization condition storage unit 13. Figure 6 shows the upper limit of the number of mathematical terms, the initial range of the weight coefficients, and the number of NN structures to try as structural optimization conditions.

[0066] The upper limit of the number of formula terms represents the upper limit of the number of terms in the formula (prediction formula) derived by the computing device 1.

[0067] The initial range of the weight coefficients represents the range of initial values ​​for the weight coefficients w that are set during the NN learning process (convergence calculation of the weight coefficients w). The NN structure optimization unit 16 sets the initial values ​​of the weight coefficients w according to this initial range of weight coefficients in the learning process described later.

[0068] The number of NN structures to try represents the number of times the computing device 1 derives the formula. The NN structure optimization unit 16 derives the formula by changing the pattern of whether or not there are edges between each node (i.e., changing the structure of the NN), as will be described later. The number of times the formula is derived by changing this pattern is determined by the number of NN structures to try.

[0069] Let's return to the explanation of Figure 3.

[0070] In S5, the NN structure optimization unit 16 inputs sensor data from the device 3 by receiving time-series data from the sensor data storage unit 42 for the input signals stored in the structure constraint condition storage unit 12.

[0071] Figure 7 shows an example of the configuration of sensor data stored in the sensor data storage unit 42. The sensor data storage unit 42 stores the detected values ​​of each sensor 31, which become the input signals for NN, in a time-series format.

[0072] Let's return to the explanation of Figure 3.

[0073] In S6, the NN structure optimization unit 16 determines the structure of the NN through a learning process and calculates weight coefficients w corresponding to the determined NN structure, thereby determining the optimal NN that represents the relationship between the input signal and the output signal. For learning the NN, for example, backpropagation is used. The details of the process performed in S6 will be explained below with reference to Figure 8.

[0074] Figure 8 is a flowchart of the process performed by the NN structure optimization unit 16 in S6 of Figure 3. The NN learning process in the computing device 1 according to this embodiment, that is, the process of deriving the mathematical formula, is a flow consisting of a double loop, as shown in Figure 8.

[0075] The outer loop (S11-S16) is the process of determining the structure of the NN, that is, whether or not there are edges between each node. In this process, for edges that are specified to exist in the constraint conditions (Figures 5A-5C) regarding the presence or absence of edges between each node stored in the structure constraint condition storage unit 12, it is decided whether or not to make the edge exist (i.e., whether or not to delete the edge). The number of times the NN structure optimization unit 16 changes the pattern of the presence or absence of edges between each node (i.e., the NN structure) follows the structure optimization condition, which is the number of NN structures to try (Figure 6).

[0076] The inner loop (S14-S15) is the process of determining the weight coefficients w for the NN structure determined in the outer loop. The convergence condition for these weight coefficients w is that the rate of change of the difference (loss) between the predicted detected value of sensor 31 output by the NN and the detected value (actual value) of sensor 31 becomes less than or equal to a threshold. This threshold can be arbitrarily determined by the user in advance.

[0077] The NN structure and weight coefficients w ultimately determined by the NN structure optimization unit 16 are obtained under the condition that minimizes the loss.

[0078] The following describes the process by which the NN structure optimization unit 16 determines the structure of the NN and calculates the weight coefficients w, following the flow chart in Figure 8.

[0079] In S11, the NN structure optimization unit 16 determines whether the number of NN structures tried so far (i.e., the number of times the computing device 1 has derived a formula) is less than the number of NN structures to try set by the user as a structure optimization condition. As long as the number of NN structures tried so far is less than the number of NN structures to try set by the user, the processes in S12 to S16 are performed.

[0080] In S12, the NN structure optimization unit 16 uses Bayesian optimization to determine the structure of the NN (i.e., the presence or absence of edges between each node) so as to satisfy the upper limit of the number of mathematical terms set by the user as a structure optimization condition. The NN structure optimization unit 16 then decides whether or not to include (not delete or not include) the edges that are considered to exist in the constraint conditions regarding the presence or absence of edges between each node, which are specified by the user and stored in the structure constraint condition storage unit 12. Bayesian optimization is a processing method that uses statistical methods to efficiently search for a solution. In this embodiment, the optimization process is made more efficient by statistically determining the search range by changing the pattern of the presence or absence of edges between each node (i.e., changing the structure of the NN) using the relationship between the structure of the NN, which is the search solution, and its evaluation value, loss.

[0081] The NN structure optimization unit 16 can determine which edges to remove from the NN by deciding whether or not there are edges between each node so as to satisfy the set upper limit of the number of mathematical terms. In other words, the edges to be removed from the NN to reduce the number can be specified by the user as constraints on the presence or absence of edges between each node (structure constraint memory unit 12), or they can be determined by the NN structure optimization unit 16.

[0082] The relationship between the presence or absence of edges between each node and the number of terms in the final derived formula depends on the number of edges between the hidden layer and the output layer. For the first m nodes (i=1 to m) of the hidden layer shown in Figure 2, the sum h of the input values ​​is calculated, so the number of terms in the formula is the total number of edges connecting the hidden layer nodes connected to the output layer and the input layer nodes. For the latter k nodes (i=(m+1) to (m+k)), the product z of the input values ​​is calculated, so the number of terms in the formula is the number of hidden layer nodes connected to the output layer.

[0083] As described above, once the structure of the neural network (NN) is determined, the number of terms in the derived mathematical formula is also uniquely determined. Utilizing this property, the NN structure optimization unit 16 uses Bayesian optimization to determine the structure of the NN (the presence or absence of edges between nodes) so as to satisfy the set upper limit on the number of mathematical terms.

[0084] In S13, the NN structure optimization unit 16 sets the initial value of the weight coefficient w assigned to each edge for the NN structure determined in S12. This initial value can be set to any value, but as mentioned above, it is a value that follows the range of initial values ​​of the weight coefficient set by the user as a structure optimization condition.

[0085] Next, the NN structure optimization unit 16 performs the learning process of the NN, that is, the convergence calculation of the weight coefficients w. This convergence calculation consists of a process (S15) to update the weight coefficients w using equations (4) to (7) described later, and a process (S14) to determine convergence based on the rate of change of the loss.

[0086] The following describes the process of updating the weight coefficient w in S15. There are four equations used to update the weight coefficient w: equations (4) to (7). This is because there are two types of nodes for each of the two pairs of nodes: the output layer and the hidden layer, and the hidden layer and the input layer. These nodes include two types of nodes: i (i=1 to m) that calculate the sum h in the hidden layer, and i (i=(m+1) to (m+k)) that calculate the product z. Therefore, there are four equations depending on the combination of these. Equations (4) to (7) and equations (8) to (9), which will be described later, indicate that the value of w before the update is substituted for w on the right side of the arrow, and the value obtained from the equation on the right side of the arrow is the updated w (w on the left side of the arrow).

[0087] At the edges connected to node i (i=1~m) in the pair of output layer and hidden layer nodes, equation (4) is used as the equation for updating the weight coefficient w. As mentioned above, w is the weight coefficient, h is the output value of node i (i=1~m) in the hidden layer shown in equation (1), and loss is the difference between the predicted detected value of sensor 31 output by the NN and the detected value (actual value) of sensor 31. η is an arbitrary constant called the learning rate, and its value is fixed to, for example, 0.001.

[0088]

number

[0089] For the edges connected to node i(i=(m+1)~(m+k)) in the pair of output layer and hidden layer nodes, equation (5) is used as the equation for updating the weight coefficient w. As mentioned above, z is the output value of the hidden layer node i(i=(m+1)~(m+k)) shown in equation (2).

[0090]

number

[0091] For the edges connected to node i (i=1 to m) in the pair of nodes in the hidden layer and the input layer, equation (6) is used as the equation for updating the weight coefficient w. As mentioned above, x is the output value of node j (j=1 to n) in the input layer.

[0092]

number

[0093] For the edges connected to node i (i=(m+1)~(m+k)) in the pair of nodes in the hidden layer and input layer, equation (7) is used as the formula for updating the weight coefficient w.

[0094]

number

[0095] The NN structure optimization unit 16 repeatedly updates the weight coefficients w using equations (4) to (7) shown above to determine the final value of the weight coefficients w.

[0096] Equations (6) and (7) cannot be used when the output value x of the node in the input layer, i.e., the detected value of sensor 31, is negative. Therefore, equation (8) may be used instead of equation (6), and equation (9) may be used instead of equation (7). Equations (8) and (9) can also handle cases where the detected value of sensor 31 is negative.

[0097]

number

[0098]

number

[0099] Since the above equations (4) to (9) can be derived from the equations used in the backpropagation process that the NN uses for training, a detailed explanation is omitted.

[0100] After updating the weight coefficient w in S15, the NN structure optimization unit 16 determines in S14 whether the value of the weight coefficient w has converged.

[0101] In S14, the NN structure optimization unit 16 determines, as described above, whether the rate of change of the difference (loss) between the predicted detected value of sensor 31 output by the NN and the detected value (actual value) of sensor 31 is below a threshold. The rate of change of loss is the rate of change of loss before and after updating the weight coefficient w. If the rate of change of loss is below the threshold, the NN structure optimization unit 16 determines that the value of the weight coefficient w has converged and executes the process in S16.

[0102] In S16, the NN structure optimization unit 16 stores the data and weight coefficients w of the NN structure when the loss is minimized, based on the loss obtained for the NN structures determined so far. The NN structure optimization unit 16 determines that the NN structure that minimizes the loss is the optimal structure and stores the data and weight coefficients w of the NN structure at that time. In addition, the NN structure optimization unit 16 stores not only the data and weight coefficients w of the final NN structure obtained when the values ​​of the weight coefficients w converge, but also the data and weight coefficients w of the NN structure obtained during the convergence process.

[0103] The process described above using Figure 8 is the process performed in S6 of Figure 3.

[0104] Let's return to the explanation of Figure 3.

[0105] In S7, the NN structure optimization unit 16 stores the data on the NN structure (presence or absence of edges between nodes) and the weight coefficients w obtained in the processing of S6 into the analysis result storage unit 17.

[0106] Figure 9 shows an example of the data structure stored in the analysis result storage unit 17. The analysis result storage unit 17 stores the NN structure data and the weight coefficient w values ​​obtained when the weight coefficient w values ​​converge and when they do not converge. In Figure 9, the numbers entered in the table represent the weight coefficient w corresponding to each edge, and the symbol "×" in the table indicates that there are no edges between nodes.

[0107] Let's return to the explanation of Figure 3.

[0108] In S8, the structural formula conversion unit 14 uses the NN structure data and the weight coefficient w values ​​stored in the analysis result storage unit 17 to derive a formula representing the relationship between the input signal and the output signal, i.e., a formula representing the relationship between the detected value of sensor 31 and the predicted detected value of other sensors 31 (prediction formula), according to equation (3). Equation (3) includes power terms for h and z, as shown in equations (1) and (2). Therefore, the computing device 1 can perform modeling using a formula based on physical laws using equation (3).

[0109] Furthermore, the structural formula conversion unit 14 derives a prediction formula each time the NN structure optimization unit 16 updates the weight coefficients. That is, the structural formula conversion unit 14 can derive a prediction formula not only using the weight coefficient values ​​when the weight coefficients have converged, but also using the weight coefficient values ​​when the weight coefficients have not converged.

[0110] The above is a description of the processing flow performed by computing device 1.

[0111] Next, we will explain an example of the screen that input / output device 2 displays when a user uses computing device 1. Input / output device 2 displays the following screen in response to a command from computing device 1.

[0112] Figure 10 shows an example of a screen where the user sets constraints regarding the input and output signals of the neural network (NN). The constraints regarding the input and output signals of the NN are stored in the structural constraint memory unit 12 and are conditions that identify which sensor 31 each input signal and output signal belongs to by the name of the sensor 31.

[0113] The user specifies which sensor 31 the output signal belongs to in item 101 of the output value used. For example, the computing device 1 displays a list of sensors 31 installed on the equipment 3 on the screen of the input / output device 2, and the user selects one sensor 31 from this list to be designated as an output. The user also specifies which sensor 31 the input signal belongs to in item 102 of the input value used. For input signals, the user can select multiple sensors 31. As shown in Figure 10, the selected sensors 31 are displayed in a different color.

[0114] Figure 11 shows an example of a screen where the user sets constraints regarding the presence or absence of edges between each node. These constraints are stored in the structural constraint memory unit 12 and specify the edges that will be removed from the neural network.

[0115] The user specifies whether or not an edge exists for the edges connecting the input layer nodes and the hidden layer nodes in items 201 and 202. Item 201 concerns the edges related to the node responsible for calculating the sum h. Item 202 concerns the edges related to the node responsible for calculating the product z. The user also specifies whether or not an edge exists for the edges connecting the hidden layer nodes and the output layer nodes in item 203. In Figure 11, a checkmark in the table indicates that an edge to be deleted has been specified. Checkmarks can be added or removed, for example, by the user clicking on the table with the mouse cursor.

[0116] Figure 12 shows an example of a screen where the user sets structural optimization conditions. The structural optimization conditions are stored in the structural optimization condition storage unit 13 and include the upper limit of the number of mathematical terms, the initial range of weight coefficients, and the number of NN structures to try.

[0117] In item 301, the user specifies the upper limit of the number of terms in the formula derived by the computing device 1. In item 302, the user specifies the range of initial values ​​for the weight coefficients w to be set during the NN learning process (convergence calculation of weight coefficients w). In item 303, the user specifies the number of times (number of patterns) that the computing device 1 will derive the formula by changing the pattern of whether or not there are edges between each node.

[0118] Figure 13 shows an example of a screen that outputs the formula (prediction formula) derived by the computing device 1. Item 401 displays the signal value (detection value) of sensor 31 specified by the user in the constraints regarding the input and output signals of the NN. Item 402 displays the formula (prediction formula) derived when the convergence calculation of the NN's weight coefficients w is completed. This formula is the formula that minimizes the loss throughout the entire processing flow performed by the computing device 1. Item 403 displays the formula derived during the convergence calculation of the weight coefficients w, with each iteration of the convergence calculation.

[0119] In item 403, we display the formula derived during the convergence calculation, that is, the formula derived using the weight coefficient values ​​when the weight coefficients have not converged, because the formula that minimizes the loss is not necessarily the optimal one. The details of this phenomenon are explained below.

[0120] Neural Networks (NNs) and other modeling techniques generally suffer from a problem called overfitting. Overfitting occurs when a model is excessively fitted to the training data, resulting in larger errors for data other than the training data. Therefore, while the prediction error (for example, loss in this example) may be minimized, the resulting formula may not conform to physical laws due to overfitting.

[0121] In the computing device 1 according to this embodiment, the emphasis is on deriving mathematical formulas that conform to the physical laws underlying the modeling target, rather than obtaining a neural network with small prediction errors. Therefore, in the computing device 1 according to this embodiment, the problem of overfitting can be resolved by also displaying the mathematical formulas derived during the convergence calculation process, i.e., the formulas when the loss is not minimized.

[0122] As described above, according to this embodiment, the relationship between the detected values ​​of sensors 31 installed on equipment 3 such as plants and facilities and the predicted detected values ​​of other sensors 31 can be modeled as a mathematical formula based on physical laws. This allows users to easily verify whether the modeled neural network (NN) matches the physical characteristics of the target equipment 3, and improves the explainability and interpretability of the prediction results calculated by the NN. [Examples]

[0123] This section describes a neural network (NN) computing device and computing method according to Embodiment 2 of the present invention. The difference between Embodiment 1 and Embodiment 2 lies in the mathematical formula (prediction formula) that the NN can represent. In Embodiment 1, a formula in the form of a linear sum of terms containing exponents is derived, as shown in formula (3), while in Embodiment 2, a formula in the form of a power of a linear sum is derived.

[0124] Below, we will mainly describe the differences between Example 2 and Example 1. The differences lie in the structure of the neural network (NN), the computation and learning processes of the NN, and the screen that outputs the computation results of the computing device 1.

[0125] First, we will explain the differences between the structure of Example 2 and Example 1.

[0126] Figure 14 is a schematic diagram showing the structure of the neural network (NN) used by the computing device 1 to derive the mathematical formula (prediction formula) in this embodiment. The structure of the NN in this embodiment differs from the structure of the NN in Embodiment 1 (Figure 2) in that the computational processing performed by the edge portions from the input layer to the hidden layer and the computational processing performed by the edge portions from the hidden layer to the output layer are swapped.

[0127] First, let's explain the calculation from the input layer to the hidden layer. Each edge is assigned a weight coefficient w(j,i). j and i represent the nodes j and i located at both ends of the edge. In other words, w(j,i) represents the weight coefficient set for the edge connecting node j in the input layer and node i in the hidden layer. At each edge, the product of the output xj (J=1~n) of the node in the input layer and the weight coefficient w(j,i) is calculated.

[0128] In the hidden layer, the values ​​calculated at each edge are used as input, and their sum h or product z is calculated. This can be expressed mathematically as follows:

[0129]

number

[0130]

number

[0131] δ is a function that outputs 0 if there are no edges connecting the nodes, and 1 if there are. In this example, as in Example 1, not all nodes in the layers are connected by edges; only some nodes are connected by edges.

[0132] In the example shown in Figure 14, the first m nodes (i=1 to m) of the intermediate layer calculate the sum h using equation (10), and the latter k nodes (i=(m+1) to (m+k)) calculate the product z using equation (11). The number of nodes m for calculating the sum h and the number of nodes k for calculating the product z can be arbitrarily determined, and a sufficient number should be set to represent the formula.

[0133] Next, we will explain the calculation from the hidden layer to the output layer. The output layer has only one node. The weight coefficient set on the edge connecting node i in the hidden layer and the node in the output layer is denoted by w'j. The node in the output layer outputs an output value y. At each edge, a power is calculated with the outputs hi (i=1~m) and zi (i=(m+1)~(m+k)) of the node in the hidden layer as the base and the weight coefficient w'j as the exponent.

[0134] The mathematical formula (prediction formula) expressed by the NN having the structure described above can be expressed in the form of formula (12), that is, in the form of a power of a linear sum.

[0135]

number

[0136] Next, we will explain the differences between Example 2 and Example 1 regarding the computation and learning processes of the neural network. The difference lies in the formula used for calculating the convergence of the weight coefficients w. In Example 2, the weight coefficients w are updated using equations (13) to (18), which will be described later.

[0137] The process for updating the weight coefficient w (corresponding to S15 in Figure 8 of Example 1) is described below in detail. There are four equations used to update the weight coefficient w: equations (13) to (16). There are four equations for the same reason as explained in Example 1.

[0138] For the edges connected to node i (i=1~m) in the pair of output layer and hidden layer nodes, equation (13) is used as the formula for updating the weight coefficient w. As mentioned above, w is the weight coefficient, h is the output value of node i (i=1~m) in the hidden layer as shown in equation (10), and loss is the difference between the predicted detected value of sensor 31 output by the NN and the detected value (actual value) of sensor 31. η is the learning rate.

[0139]

number

[0140] For the edges connected to node i(i=(m+1)~(m+k)) in the pair of output layer and hidden layer nodes, equation (14) is used as the equation for updating the weight coefficient w. As mentioned above, z is the output value of the hidden layer node i(i=(m+1)~(m+k)) shown in equation (11).

[0141]

number

[0142] For the edges connected to node i (i=1 to m) in the pair of nodes in the hidden layer and the input layer, equation (15) is used as the formula for updating the weight coefficient w. x is the output value of node j (j=1 to n) in the input layer.

[0143]

number

[0144] For the edges connected to node i (i=(m+1)~(m+k)) in the pair of nodes in the hidden layer and input layer, equation (16) is used as the formula for updating the weight coefficient w.

[0145]

number

[0146] Equation (17) may be used instead of equation (13), and equation (18) may be used instead of equation (14). Equations (17) and (18) can also be used when the output value h or z of the hidden layer node is negative.

[0147]

number

[0148]

number

[0149] Next, we will explain the differences between Example 2 and Example 1 regarding the screen that outputs the calculation results of the computing device 1.

[0150] Figure 15 shows an example of a screen that outputs the formula (prediction formula) derived by the computing device 1 in this embodiment. Item 502 displays the formula (prediction formula) derived when the convergence calculation of the weight coefficients w of the NN is completed. Item 503 displays the formula derived during the convergence calculation of the weight coefficients w, with each iteration of the convergence calculation.

[0151] The format of the formulas displayed in items 502 and 503 differs from the format of the formulas displayed in items 402 and 403 (Figure 13) in Example 1. In Example 2, the formulas are displayed in a form that includes a power of a linear sum.

[0152] As described above, this embodiment makes it possible to derive mathematical formulas of a form that cannot be expressed in Embodiment 1, namely, mathematical formulas that have powers of linear sums. For this reason, in this embodiment as well, the relationship between the detected values ​​of sensors 31 installed on equipment 3 such as plants and equipment and the predicted detected values ​​of other sensors 31 can be modeled as a mathematical formula based on physical laws. This allows the user to easily verify whether the modeled neural network (NN) matches the physical characteristics of the target equipment 3, and improves the explainability and interpretability of the prediction results calculated by the NN.

[0153] It should be noted that the present invention is not limited to the embodiments described above, and various modifications are possible. For example, the embodiments described above are explained in detail to make the present invention easier to understand, and the present invention is not necessarily limited to embodiments having all the configurations described. Furthermore, it is possible to replace parts of the configuration of one embodiment with the configuration of another embodiment. It is also possible to add configurations from other embodiments to the configuration of one embodiment. Furthermore, it is possible to delete parts of the configuration of each embodiment, or to add or replace other configurations. [Explanation of Symbols]

[0154] 1...Computing device, 2...Input / output device, 3...Equipment, 4...Control system, 11...Input / output control unit, 12...Structural constraint condition storage unit, 13...Structural optimization condition storage unit, 14...Structural formula conversion unit, 15...NN structural constraint setting unit, 16...NN structural optimization unit, 17...Analysis result storage unit, 31...Sensor, 41...Control device, 42...Sensor data storage unit, 101...Item for output value used, 102...Item for input value used, 151...Input / output signal setting unit, 152...Edge constraint setting unit, 201, 202...Input layer 203...Items concerning edges connecting the hidden layer and the output layer, 301...Items concerning the upper limit of the number of mathematical terms, 302...Items concerning the initial range of weight coefficients, 303...Items concerning the number of NN structures to try, 401...Items to display sensor signal values, 402...Items to display derived mathematical formulas, 403...Items to display mathematical formulas derived during the convergence calculation, 502...Items to display derived mathematical formulas, 503...Items to display mathematical formulas derived during the convergence calculation.

Claims

1. The first and second sensors are connected to the device in which they are installed. A structure optimization unit that inputs the detected value of the first sensor, A structural formula conversion unit that uses a neural network to derive a prediction formula, which is a mathematical formula representing the relationship between the detected value of the first sensor and the predicted value of the detected value of the second sensor, Equipped with, The neural network includes nodes whose values ​​are expressed by a formula that includes a power, with the output value of the node as the base and the weight coefficient set on the edge connected to the node as the exponent. The structure optimization unit calculates the structure of the neural network and the weight coefficients from the detected values ​​of the first sensor. The structural formula conversion unit uses the neural network structure and weight coefficients calculated by the structural optimization unit to derive a formula having a term containing the exponentiation as the prediction formula. The neural network comprises an input layer, a hidden layer, and an output layer. The node in the input layer receives the detected value from the first sensor and outputs it as an output value. The node in the intermediate layer takes the output value of the node in the input layer as its input value, and has a value that can be expressed by a formula including the power, with the input value as the base and the weight coefficient set on the edge connecting the node in the input layer and the node in the intermediate layer as the exponent. The node in the output layer outputs the value calculated by the prediction formula, The node in the intermediate layer has a node that calculates the sum of the powers and a node that calculates the product of the powers. A neural network computing device characterized by the following.

2. Backpropagation is used to train the aforementioned neural network. The structure optimization unit calculates the weight coefficient using a formula obtained from the formula used in the backpropagation process. A neural network computing device according to claim 1.

3. The system includes a storage unit that stores the upper limit of the number of terms in the prediction formula, the range of initial values ​​for the weight coefficients, and conditions regarding the presence or absence of edges between nodes, as entered by the user. A neural network computing device according to claim 1.

4. The structure optimization unit uses Bayesian optimization to determine whether or not there are edges between each node so as to satisfy the upper limit of the number of terms in the prediction formula stored in the memory unit. A neural network computing device according to claim 3.

5. The structure optimization unit updates the weight coefficients in the calculation of the weight coefficients until the weight coefficients converge. The structural formula conversion unit derives the prediction formula each time the structural optimization unit updates the weight coefficients. A neural network computing device according to claim 1.

6. The first and second sensors are installed in a computer connected to a computing device. The calculation device performs a structural optimization step in which it inputs the detected value of the first sensor, The computing device performs a structural formula transformation step in which it uses a neural network to derive a prediction formula, which is a mathematical formula representing the relationship between the detected value of the first sensor and the predicted value of the detected value of the second sensor. It has, The neural network includes nodes whose values ​​are expressed by a formula that includes a power, with the output value of the node as the base and the weight coefficient set on the edge connected to the node as the exponent. In the structure optimization step, the computing device calculates the structure of the neural network and the weight coefficients from the detected values ​​of the first sensor. In the structural formula conversion step, the computing device uses the structure of the neural network calculated in the structural optimization step and the weight coefficients to derive a formula having a term containing the power as the prediction formula. The neural network comprises an input layer, a hidden layer, and an output layer. The node in the input layer receives the detected value from the first sensor and outputs it as an output value. The node in the intermediate layer takes the output value of the node in the input layer as its input value, and has a value that can be expressed by a formula including the power, with the input value as the base and the weight coefficient set on the edge connecting the node in the input layer and the node in the intermediate layer as the exponent. The node in the output layer outputs the value calculated by the prediction formula, The node in the intermediate layer has a node that calculates the sum of the powers and a node that calculates the product of the powers. A method for calculating neural networks, characterized by the following:

Images