An industrial causal mining method based on neural network weight comparison

Abstract

The present invention discloses an industrial causal mining method based on neural network weight comparison, which uses process data to train a group of single hidden layer neural networks, calculates the path weight matrix and the symmetric path matrix through the weights of these networks, and then generates an undirected graph skeleton representing the correlation between variables by comparing the elements of the symmetric path weight matrix with the size of the threshold; on this basis, the causal direction of each edge in the undirected graph is judged by comparing the relative size of the complexity of the effective model through the network weight, and finally a directed graph representing the causal relationship between variables is obtained. The method of the present invention breaks through the requirements of some current causal mining methods that data must meet linear relationships, additive noise or stationarity, and can be used in complex industrial scenarios. Moreover, the method can mine the undirected skeleton directly from the observed data without relying on artificially given undirected graph skeletons, and judge the causal direction of each pair of undirected relationships with high accuracy.

Description

Technical Field

[0001] The invention belongs to the field of causal mining and industrial intelligent modeling, and relates to an industrial causal mining method based on neural network weight comparison. Background Art

[0002] Causal mining is gradually becoming a cutting-edge hotspot and is gradually intersecting with the machine learning community. At present, many causal mining algorithms based on observational data are being widely used in biology, medicine, economics and other fields. The relationships between variables mined often reveal the causal mechanisms behind the phenomena and promote people's cognitive development in these fields. The success of machine learning and deep learning has also driven the development and application of data-driven industrial intelligent modeling, including soft measurement and process monitoring. Multiple industrial process units are often connected and involve complex reaction mechanisms. Causal mining algorithms based on process data can mine knowledge about the relationship between variables to guide process modeling and optimal control.

[0003] Despite the availability of observational data, the application of causal mining in the industrial field has not received enough attention. The main reason why causal mining is less used in the industrial field is that the widely used causal mining algorithms need to comply with strict assumptions when used, and these assumptions are often too strict for most industrial scenarios. For example, the PC algorithm assumes that the relationship between continuous variables is linear, the ANM algorithm assumes that the noise is additive, Granger causality analysis is only applicable to stationary data, and so on. Deep learning does not require assumptions such as linearity and additiveness, and is more suitable for modeling complex industrial scenarios. Therefore, in response to these problems, it is a practical and reliable method to propose an algorithm for causal mining in industrial scenarios based on the framework of deep learning. Summary of the invention

[0004] The purpose of the present invention is to provide an industrial causal mining method based on neural network weight comparison in view of the assumption limitations of the existing methods. The specific technical scheme is as follows:

[0005] An industrial causal mining method based on neural network weight comparison includes the following steps:

[0006] Step 1: Collect a large number of industrial variable data samples, the data samples contain p industrial variables, denoted as x1, x2, ..., x p ;

[0007] Step 2: Build p single hidden layer neural networks, the single hidden layer neural network includes an input layer, a hidden layer and an output layer, the input layer includes p neurons, the hidden layer includes n h neurons, the output layer includes 1 neuron;

[0008] Step 3: Use the collected data samples to train the p single hidden layer neural networks, use the other variable data in the data samples except the variable to be predicted as the input of the single hidden layer neural network, and the variable to be predicted as the output, and train the p single hidden layer neural networks by minimizing the loss function. After the network parameters are stable, continue training for a certain period, and calculate the contribution of the input neuron to the output neuron based on the average value of the network weights of the period after the network parameters are stable, that is, the path weight, and obtain the path weight matrix Q; the calculation formulas of the path weight q and the path weight matrix Q are as follows:

[0009]

[0010] in, represents the path weight of the i-th input neuron to the predicted output neuron j, is the weight of the input neuron to the hidden layer neuron, is the weight of the hidden layer neuron to be predicted by the output neuron j; i≠j, 1≤i≤p, i∈N + ;

[0011] Step 4: Symmetrize the path weight matrix Q to obtain a symmetric path weight matrix S;

[0012] Step 5: Arrange the upper or lower triangular elements of S from small to large, take a certain quantile of the sequence as the threshold t, and connect the two nodes corresponding to each element greater than t with an edge; thus, an undirected graph representing the connection relationship between variables is obtained;

[0013] Step 6: Calculate the sum of the L2 norms of the average values ​​of the network weights in the period after the network parameters are stabilized, and obtain the effective model complexity EMC of an input node i predicting an output node j i，j And the effective model complexity EMC of input node j predicting output node i j，i ;

[0014] Step 7: If EMC i，j <EMC j，i , then node i is the cause of node j; if EMC j，i <EMC i，j , then node j is the cause of node i; finally, a directed graph representing the causal relationship between industrial process variables is obtained.

[0015] Furthermore, in step 2, the input of the single hidden layer neural network is j And add Gaussian noise e j The original data matrix of the single hidden layer neural network is j=1,2,...,p,indicates the use of the variable x jAll variables other than x j The predicted value obtained by prediction; the hidden layer of the single hidden layer neural network is H -j , the weight matrix from the input layer neurons to the hidden layer neurons is W1, and the weight matrix from the hidden layer neurons to the output layer neurons is W2:

[0016]

[0017]

[0018]

[0019]

[0020] Among them, b1 represents the bias vector from the input layer to the hidden layer; σ is the activation function, and b2 represents the bias vector from the hidden layer to the output layer.

[0021] Furthermore, in step 3, MMD loss is used as the loss function, and its specific form is:

[0022]

[0023] Among them, x j is the true value of the variable, is the predicted value of the variable, D is the matrix of the true value of the variable, is the matrix of variable prediction values; n represents the number of data, and k() represents the Gaussian kernel function.

[0024] Furthermore, in step 3, each hidden layer neural network can obtain a total of (p-1) path weights The path weights of all hidden layer neural networks form a path weight matrix Q, which is expressed as follows:

[0025]

[0026] Furthermore, the calculation formula of the symmetric path weight matrix S in step 4 is as follows:

[0027]

[0028] Where ⊙ represents the multiplication of corresponding elements of the matrix, and the element s of S i，j Describes the closeness of the relationship between node i and node j.

[0029] Furthermore, in step five, the 45% to 65% quantile of the sequence is taken as the threshold t.

[0030] Furthermore, the effective model complexity EMC in step 6 is i，j and EMCj，i The calculation formula is as follows:

[0031]

[0032]

[0033] in, is x i and the weight vector composed of the weights between the neurons in the hidden layer, is the hidden layer neuron and output The weight vector composed of the weights between .

[0034] The beneficial effects of the present invention are as follows:

[0035] (1) The causal mining method proposed in the present invention breaks through the requirements of some current causal mining algorithms that data must meet linear relationships, additive noise or stationarity, and can be used in complex industrial scenarios.

[0036] (2) The present invention does not rely on artificially given undirected graph skeletons, but can directly mine undirected skeletons from observed data and determine the causal direction of each pair of undirected relationships. In addition, the accuracy is greatly improved compared with the currently widely used causal mining algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a structural diagram of a single hidden layer neural network used in the present invention;

[0038] Figure 2 It is a schematic diagram of the causal pointing judgment criteria;

[0039] Figure 3 It is the process flow chart of the supply process of carbon dioxide and ammonia and the steam condensation process;

[0040] Figure 4 It is the directed graph mining result of the supply process of carbon dioxide and ammonia; wherein, the left figure is the real directed graph of the sub-case, and the right figure is the directed graph predicted by the method of the present invention;

[0041] Figure 5 It is the directed graph mining result of the steam condensation process; wherein, the left figure is the real directed graph of the sub-case, and the right figure is the directed graph predicted by the method of the present invention. DETAILED DESCRIPTION

[0042] The present invention will be described in detail below based on the accompanying drawings and preferred embodiments, and the purpose and effects of the present invention will become more clear. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0043] The causal mining method based on neural network weight comparison of the present invention comprises the following steps:

[0044] Step 1: Collect industrial variable data samples. A data sample contains p industrial variables, denoted as x1, x2, ..., x p .

[0045] As one implementation method, an industrial control system may be used to collect industrial variable data samples.

[0046] Step 2: Build p structures based on the open source deep learning platform Pytorch Figure 1 The single hidden layer neural network shown. Set the number of hidden layer neurons n h , learning rate lr, training batch data size bs, number of training cycles ep1, number of test cycles ep2.

[0047] Each single hidden layer network can be expressed as: j=1, 2, ..., p.

[0048] X -j Indicates that node x is removed j , and add Gaussian noise e j The original data matrix is ​​also the input layer of the single hidden layer neural network.

[0049] is the hidden layer of the model. Where W1 is the weight matrix from the input layer neurons to the hidden layer neurons, and b1 represents the bias vector of this layer. σ is the activation function, which enables the model to model nonlinear relationships.

[0050] Is the output layer of the model. The output layer has a single output, and the output value Indicates the use of the variable x j All variables other than x j The predicted value obtained by prediction. W2 is the weight matrix from the hidden layer neurons to the output layer neurons, and b2 represents the bias vector of this layer.

[0051] The specific forms of W1 and W2 are as follows:

[0052]

[0053]

[0054] In order to make the distribution of predicted values ​​as close as possible to the distribution of true values, the MMD loss that describes the similarity between the generated data distribution and the original data distribution is used as the loss function during training. Its specific form is:

[0055]

[0056] Among them, x j is the true value of the variable, is the predicted value of the variable.

[0057] Step 3: Use the collected data samples to train the p single hidden layer neural networks. A total of (ep1+ep2) cycles are trained, where ep2 is the number of cycles for continued training after the network parameters are stable. The average value of the network weights in these ep2 cycles is taken to calculate the path weights and obtain the path weight matrix Q.

[0058] like Figure 1 As shown, the input node x1 is sent to each hidden layer neuron and then to the output node There are n h Paths. x1 to The path weight The calculation method is as follows:

[0059]

[0060] Figure 1 The single hidden layer neural network shown can obtain a total of (p-1) path weights

[0061]

[0062] sum(W i ′) represents the sum of the elements in the i-th column of W′

[0063] A total of p training Figure 1 The network shown in the figure shows the weights of all paths obtained. A path weight matrix Q can be formed. The elements of the jth row of Q come from the network f j , is all nodes x i (i≠j, 1≤i≤p, i∈N + ) points to node x j The diagonal elements of Q are 0, because there is no path from the node to itself in the network, and the path weight from the node to itself is recorded as 0.

[0064]

[0065] Step 4: The path weight matrix Q is an asymmetric matrix. Symmetricalize Q to obtain the symmetric path weight matrix S:

[0066]

[0067] Where ⊙ represents the multiplication of corresponding elements of the matrix. i，j (sj，i ) describes the closeness of the relationship between node i and node j.

[0068] Step 5: Arrange the upper or lower triangular elements of S from small to large, take a certain quantile of the sequence (after many experiments, it is recommended to take the 45% to 65% quantile of the sequence) as the threshold t, and consider the element s greater than t i，j (or s j，i ) The corresponding nodes i and j are connected, and nodes i and j are connected by an edge in the undirected graph. Similarly, all nodes corresponding to elements greater than the threshold are connected to obtain an undirected graph representing the connection relationship between variables.

[0069] Step 6: An undirected graph only represents correlations and does not contain causal information. In order to obtain a causal graph, the causal direction of each edge must also be determined. The present invention measures the effective model complexity of the single hidden layer neural network based on the L2 norm. The calculation formula of this indicator is as follows:

[0070]

[0071]

[0072] like Figure 2 As shown, is the weight vector consisting of the weights between x1 and the hidden layer neurons. is the hidden layer neuron and the output The weight vector is composed of the weights between . EMC i，j Is using x i Predict x j The effective model complexity generated by w u and w v Similarly, EMC j，i Is using x j Predict x i The effective model complexity generated by and It is calculated by the sum of the L2 norms of .

[0073] Step 7: x i and x j The causal relationship between them is determined based on the following criteria: If EMC i，j <EMC j，i , then x i is x j Cause; If EMC j，i <EMC i，j , then x j is x iUsing this method, we can further construct a directed graph based on the undirected graph relationship between variables.

[0074] Example

[0075] In order to verify the effectiveness of the present invention in an industrial scenario, the present invention is described below in conjunction with two continuous processes in a urea synthesis production line of a fertilizer production company. Table 1 and Table 2 are the variables involved in the two process cases and their descriptions. The flowcharts of the two processes are shown in Figure 1. Figure 3 As shown, Figure 3 The left half of the graph shows the supply process of carbon dioxide and ammonia, and the right half shows the steam condensation process.

[0076] Table 1 Description of process variables for supply of carbon dioxide and ammonia

[0077] Variable Symbols Variable Description X1 Compressor four-stage outlet flow X2 Carbon dioxide minus added air flow X3 Ammonia to Carbon Ratio X4 Ammonia pump outlet ammonia quantity indicator X5 Ammonia pump main line outlet flow X6 Ammonia quantity control at ammonia pump outlet

[0078] Table 2 Description of variables in steam condensation process

[0079] Variable Symbols Variable Description X7 Low pressure steam main flow X8 Condenser inlet flow X9 Sink inlet flow control X10 Tank level X11 Low pressure steam drum level control X12 Low pressure steam drum inlet flow X13 Flow from medium pressure steam drum to low pressure steam drum

[0080] 10,000 samples were extracted from each process for training a single hidden layer neural network. The number of hidden layer neurons was set to 20, the batch size was set to 10% of the total data volume, the learning rate was set to 0.01, the training cycle was set to 1,000, and the test cycle was set to 500. After 1,000 rounds of training, the network has gradually converged and the network parameters have been basically stable. After another 500 rounds of training, the average value of the network weights in the last 500 rounds was taken to calculate the symmetric pathway matrix. After obtaining the symmetric pathway matrix, its upper triangular elements are arranged in sequence, and the 45% to 65% quantile of the sequence is taken as the threshold. The two variable nodes corresponding to the elements with symmetric pathway weights greater than the threshold are connected to obtain an undirected graph skeleton. Then, on the basis of the undirected graph, EMC is further used to determine the direction of the connection relationship to obtain a directed graph representing the causal relationship. The experimental results of causal mining for the two processes are as follows:

[0081] (1) Carbon dioxide and ammonia supply process: X1 to X6

[0082] Figure 4 It is the directed graph (causal graph) of this sub-case mined by Ground Truth and the method WCCM of the present invention. The thick solid line with an arrow in the figure represents the edge with correct connection relationship and causal direction; the dotted line with an arrow represents the edge with correct connection relationship but wrong causal direction; the thin solid line without an arrow represents redundant or wrong edge. Table 3 compares the results of WCCM and other four commonly used causal mining algorithms (PC, K2-ANM, ANM, FSGNN-CGNN) on the supply process of carbon dioxide and ammonia.

[0083] Table 3 Comparison of the results of various causal mining algorithms on the supply process of carbon dioxide and ammonia

[0084] algorithm PC K2-ANM ANM FSGNN-CGNN WCCM Correct number of undirected edges 5 6 5 6 6 Undirected edge recall 5 / 6＝0.83 6 / 6＝1 5 / 6＝0.83 6 / 6＝1 6 / 6＝1 Undirected edge accuracy 5 / 10 6 / 11＝0.55 5 / 6＝0.83 6 / 9＝0.67 6 / 7＝0.86 Correct number of directed edges 3 3 1 3 5 Directed edge recall 3 / 6＝0.50 3 / 6＝0.50 1 / 6＝0.17 3 / 6＝0.50 5 / 6＝0.83 Directed edge accuracy 3 / 5＝0.6 3 / 6＝0.5 1 / 5＝0.2 3 / 6＝0.5 5 / 6＝0.83

[0085] from Figure 4 As can be seen from the results in Table 3, 6 out of the 7 undirected edges mined by WCCM are correct, and the directions of 5 of them are correctly judged, which is significantly better than other commonly used algorithms in this field.

[0086] (2) Steam condensation process: X7~X13

[0087] Figure 5 It is the directed graph (causal graph) of this sub-case mined by Ground Truth and WCCM. In the figure, the thick solid line with an arrow indicates the edge with correct connection relationship and causal direction; the dotted line with an arrow indicates the edge with correct connection relationship but wrong causal direction; the thin solid line without an arrow indicates redundant or wrong edge; the dotted line without an arrow indicates the missed edge. Table 4 compares the results of WCCM and other four commonly used causal mining algorithms (PC, K2-ANM, ANM, FSGNN-CGNN) on the steam condensation process.

[0088] Table 4 Comparison of results of various causal mining algorithms on steam condensation process

[0089]

[0090]

[0091] from Figure 5 As can be seen from the results in Table 4, 7 out of 9 undirected edges mined by WCCM are correct, and the directions of 5 of them are correctly judged, which is significantly better than other commonly used algorithms in this field.

[0092] Based on the two process cases, WCCM far outperforms other methods in the accuracy of undirected edges and directed edges. The two process cases have a total of 14 directed edges, and WCCM successfully mined 10 of them, with an overall accuracy of 0.71. Among the currently commonly used methods, the best one is FSGNN-CGNN, which only successfully mined 6 with an accuracy of 0.43. In summary, compared with existing methods, the industrial causal mining method based on neural network weight comparison of the present invention can effectively mine the correlation and causal relationship between industrial process variables, and is a causal mining strategy that is more suitable for industrial scenarios.

[0093] Those skilled in the art can understand that the above are only preferred examples of the invention and are not intended to limit the invention. Although the invention is described in detail with reference to the above examples, those skilled in the art can still modify the technical solutions recorded in the above examples or replace some of the technical features therein with equivalents. Any modification, equivalent replacement, etc. made within the spirit and principle of the invention shall be included in the protection scope of the invention.

Claims

1. An industrial causal mining method based on neural network weight comparison, characterized in that: The following steps are involved: Step 1: Collect industrial variable data samples, the data samples contain p industrial variables, denoted as x1, x2, ..., x p ; Step 2: Build p single hidden layer neural networks, the single hidden layer neural network includes an input layer, a hidden layer and an output layer, the input layer includes p neurons, the hidden layer includes n h neurons, the output layer includes 1 neuron; Step 3: Use the collected data samples to train the p single hidden layer neural networks, use the other variable data in the data samples except the variable to be predicted as the input of the single hidden layer neural network, and the variable to be predicted as the output, and train the p single hidden layer neural networks by minimizing the loss function. After the network parameters are stable, continue training for a certain period, and calculate the contribution of the input neuron to the output neuron based on the average value of the network weights of the period after the network parameters are stable, that is, the path weight, and obtain the path weight matrix Q; the calculation formulas of the path weight q and the path weight matrix Q are as follows: in, represents the path weight of the i-th input neuron to the predicted output neuron j, is the weight of the input neuron to the hidden layer neuron, is the weight of the hidden layer neuron to be predicted by the output neuron j; i≠j, 1≤i≤p, i∈N + ; Step 4: Symmetrize the path weight matrix Q to obtain a symmetric path weight matrix S; Step 5: Arrange the upper or lower triangular elements of S from small to large, take a certain quantile of the sequence as the threshold t, and connect the two nodes corresponding to each element greater than t with an edge; thus, an undirected graph representing the connection relationship between variables is obtained; Step 6: Calculate the sum of the L2 norms of the average values ​​of the network weights in the period after the network parameters are stabilized, and obtain the effective model complexity EMC of the input node i predicting the output node j i，j And the effective model complexity EEC of input node j predicting output node i j，i ; Step 7: If EMC i，j <EMC j，i , then node i is the cause of node j; if EMC j，i <EMC i，j , then node j is the cause of node i; finally, a directed graph representing the causal relationship between industrial process variables is obtained.

2. The industrial causal mining method based on neural network weight comparison according to claim 1 is characterized in that: In step 2, the input of the single hidden layer neural network is j And add Gaussian noise e j The original data matrix of the single hidden layer neural network is Indicates the use of the variable x j All variables other than x j The predicted value obtained by prediction; the hidden layer of the single hidden layer neural network is H -j , the weight matrix from the input layer neurons to the hidden layer neurons is W1, and the weight matrix from the hidden layer neurons to the output layer neurons is W2: Among them, b1 represents the bias vector from the input layer to the hidden layer; σ is the activation function, and b2 represents the bias vector from the hidden layer to the output layer.

3. The industrial causal mining method based on neural network weight comparison according to claim 1 is characterized in that: In step 3, MMD loss is used as the loss function, and its specific form is: Among them, x j is the true value of the variable, is the predicted value of the variable, D is the matrix of the true value of the variable, is the matrix of variable prediction values; n represents the number of data, and k() represents the Gaussian kernel function.

4. The industrial causal mining method based on neural network weight comparison according to claim 1 is characterized in that: In step 3, each hidden layer neural network can obtain (p-1) path weights. The path weights of all hidden layer neural networks form a path weight matrix Q, which is expressed as follows:

5. The industrial causal mining method based on neural network weight comparison according to claim 1 is characterized in that: The calculation formula of the symmetric path weight matrix S in step 4 is as follows: Where ⊙ represents the multiplication of corresponding elements of the matrix, and the element s of S i，j Describes the closeness of the relationship between node i and node j.

6. The industrial causal mining method based on neural network weight comparison according to claim 1 is characterized in that: In the step 5, the 45% to 65% quantile of the sequence is taken as the threshold t.

7. The industrial causal mining method based on neural network weight comparison according to claim 1 is characterized in that: The effective model complexity EMC in step 6 i，j and EMC j，i The calculation formula is as follows: in, is x i The weight vector composed of the weights between neurons in the hidden layer, is the hidden layer neuron and the output The weight vector composed of the weights between .

Images