Electronic equipment fault prediction method based on LSTM enhanced model

Abstract

The electronic equipment fault prediction method based on the LSTM enhanced model of the present invention relates to a method for memory enhancement of a long-short time neural network, and the purpose is to overcome the problem that when the existing LSTM model is used to predict equipment faults, the accuracy gradually decreases with the length of the time series. The method is specifically as follows: Step 1, collecting monitoring data during the operation of the electronic equipment, and preprocessing the monitoring data; Step 2, building an LSTM enhanced model, inputting the preprocessed monitoring data into the LSTM enhanced model for training, and obtaining a trained LSTM enhanced model; Step 3, predicting electronic equipment faults based on the trained LSTM enhanced model and the monitoring data over a period of time.

Description

Technical Field

[0001] The invention relates to a neural network improvement method, in particular to a method for memory enhancement of long- and short-time neural networks. Background Art

[0002] Time series prediction technology has important application value in many fields such as economy, meteorology, geology, hydrology, science, military, and medicine. In industrial control, the uninterrupted operation of machines leads to the occurrence of failures. By processing and predicting industrial data, the automated production process can be effectively supervised and risks can be prevented. In this field, the application problems in actual scenarios have complex nonlinear characteristics, and the linear models used for time series analysis in the early stage have limitations. Traditional machine learning methods such as matrix decomposition, support vector machine SVM (Support Vector Machine), and Bayesian network rely heavily on the effectiveness of manually designed data representation and feature extraction. In recent years, neural networks have been more widely used in the prediction of time series problems. Deep learning can often automatically complete feature extraction by establishing deep neural networks, and build associations at different levels and dimensions through data to improve the explanatory power between independent variables and dependent variables.

[0003] The most commonly used one is the recurrent neural network (RNN). Unlike traditional neural networks, RNN considers both the current input and the result of the previous layer in the hidden layer. This gives the recurrent neural network the ability to remember. However, since traditional RNN uses the back-propagation through time (BPTT) algorithm, there are gradient vanishing and gradient exploding problems during the training process, which limits the long-term prediction ability of RNN. In this scenario, the long short-term memory (LSTM) neural network is proposed to effectively solve the problem of gradient vanishing in RNN.

[0004] In industrial environments, the time series data for predicting equipment failures are collected from sensors with high sampling rates. Usually, the intervals between these data are very short, so long-term dependency problems are more likely to be exposed. Even though the existing LSTM model can predict the changing trends of a large amount of data, the prediction accuracy will gradually decrease with the length of the time series. Summary of the invention

[0005] The purpose of the present invention is to overcome the problem that the accuracy of equipment failure prediction using the existing LSTM model gradually decreases with the length of the time series, and to provide an electronic equipment failure prediction method based on the LSTM enhanced model.

[0006] The electronic equipment fault prediction method based on the LSTM enhanced model of the present invention is as follows:

[0007] Step 1: Collect monitoring data during the operation of the electronic equipment and pre-process the monitoring data;

[0008] Step 2: Build an LSTM enhanced model, input the preprocessed monitoring data into the LSTM enhanced model for training, and obtain the trained LSTM enhanced model;

[0009] The LSTM enhancement model is built based on LSTM. The specific steps are as follows:

[0010] Step 21: The cell state at the previous moment passes through the forget gate and outputs the part to be remembered and the forgotten part. The forgotten part is obtained according to the part to be remembered. The cell state at the previous moment - the part to be remembered = the forgotten part.

[0011] Step 22: Assign weights to the part to be remembered and the part to be forgotten respectively;

[0012] Step 2 and 3: add the weighted forgotten part and the part to be remembered to the current cell state, and update the current cell state;

[0013] Step 3: Predict electronic equipment failures based on the trained LSTM enhancement model and monitoring data over a period of time.

[0014] Among them, in the LSTM enhanced model, the part to be remembered is C t-1(r) , the forgotten part is C t-1(f) ; and wait for memory part C t-1(r) Belongs to the forget gate, forget part C t-1(f) It is a reinforced door;

[0015] The current cell state is:

[0016]

[0017]

[0018] In the formula, C t-1 is the cell state at the previous moment, C t is the cell state at the current moment, W1 is the weight given to the forgotten part, W2 is the weight given to the part to be remembered, i t It is a control unit implemented using the sigmoid activation function in the input gate. is a memory cell using the tanh activation function in the input gate, f t is the value of the forget gate, Θ is the Hadamard product.

[0019] Among them, in the memory unit of LSTM, a reinforcement gate is added on the basis of the original forget gate, and the value range of the reinforcement gate is [0,1]. The above memory unit is the basic computing unit of LSTM.

[0020] The beneficial effects of the present invention are as follows: the present invention adopts a fault time series prediction method of a memory-enhanced LSTM recurrent neural network, proposes to construct a classification extraction model of information in the existing LSTM memory unit, and realizes dual-channel transmission of forgotten information and memory information by adding a reinforcement gate on the basis of the original forgotten information, and selects, fuses, and re-enters the memory unit in proportion. This mechanism enables the network to remain sensitive to relatively distant information to improve memory capacity.

[0021] The LSTM enhanced model, i.e. the improved LSTM model, was compared with the existing LSTM model using industrial failure data. The results showed that the root mean square error (RSME) and mean absolute error (MAE) of the improved LSTM model were 0.145 and 0.083, respectively, which were 53.8% and 69.9% higher than the existing LSTM model, and the AUC reached 0.959. In the problem of long time series prediction, the memory capacity of the memory-enhanced LSTM is improved, which can effectively guarantee the prediction performance while extending the time step. That is, when the improved LSTM model is used to predict equipment failures, the accuracy will not gradually decrease with the length of the time series. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 The schematic diagram of the existing LSTM method for prediction problems; in the internal structure of the existing LSTM model, σ and Tanh represent the sigmoid activation function and the Tanh activation function respectively; X represents element-wise multiplication;

[0023] Figure 2 It is a schematic diagram of the structure of an improved LSTM model in the electronic equipment fault prediction method based on the LSTM enhanced model of the present invention;

[0024] Figure 3 for Figure 2 The equivalent schematic diagram of the reinforcement gate, where C t-1(r) represents the part to be memorized at time t-1, C t-1(f) represents the forgotten part at time t-1, where time t-1 is the previous time and t is a positive integer;

[0025] Figure 4 It is the numerical curve graph of the dependent variable 18S of the training set;

[0026] Figure 5It is the improved LSTM model prediction curve in the method of the present invention;

[0027] Figure 6 The existing LSTM model prediction curve;

[0028] Figure 7 The figure is a comparison of the RMSE results of the improved LSTM model and the existing LSTM model in the method of the present invention; wherein the upper curve is the RMSE curve of the existing LSTM model, and the lower curve is the RMSE curve of the improved LSTM model;

[0029] Figure 8 The figure is a comparison of the MAE results of the improved LSTM model and the existing LSTM model in the method of the present invention; wherein the upper curve is the MAE curve of the existing LSTM model, and the lower curve is the MAE curve of the improved LSTM model;

[0030] Fig. 9 The ROC curves of the existing LSTM model, the improved LSTM model 1 (W1=0.9), and the improved LSTM model 2 (W1=0.4) are compared;

[0031] Fig.10 It is the training error curve of the improved LSTM model in the method of the present invention. DETAILED DESCRIPTION

[0032] 1. Implementation principle of existing LSTM model:

[0033] RNN is a neural network with a self-connected hidden layer. Each node calculates the output at the current moment through the current state, and the current state is determined by the state of the previous moment and the input of the current moment, thereby realizing the memory of time series data. LSTM continues the chain conduction structure of RNN and adds four interactive gates on the basis of RNN. Figure 1 As shown in the figure, input (input gate), forget (forget gate), output (output gate) and cell (cell state) are used to solve the gradient explosion problem. Each LSTM node contains three inputs, namely the node state C at the previous moment. t-1 、The node output h at the previous moment t-1 and the current input x t The unique gate structure of LSTM contains a nonlinear activation function sigmoid, which determines the amount of information transmitted between nodes. The gate structure LSTM is used to control the transmission of forgotten information.

[0034] (a) The forget gate is used to control the preservation of the state information in the previous process; (b) The input gate is used to control the input size of the sequence; (c) The cell state is used to record the state of the LSTM neural network at the current moment; (d) The output gate is used to control the output of the input information after calculation; (e) The hidden state is predicted through the output activation function. The specific process is as follows:

[0035] (a) The forget gate controls the preservation of state information in the previous process by controlling the self-loop weight. At the same time, the sigmoid activation function limits the value of the forget gate to between 0 and 1 to control the cell state to prevent memory saturation. The calculation formula is: t =σ(W f h t-1 +U f x t +b f ) (1)

[0036] Where W f is the forget gate recurrent weight, U f Input weight for the forget gate, b f is the forget gate bias, h t-1 and x t are the output and input of the hidden layer respectively.

[0037] (b) The input gate is implemented by the control unit i using the sigmoid activation function t Memory cells with tanh activation function Composition to control the input of the sequence. The calculation formula is:

[0038] i t =σ(W i h t-1 +U i x t +b i ) (2)

[0039]

[0040] Among them U i and U c are the input weights of the input gate control unit and the input weights of the input gate memory unit, respectively. i and W c are the input gate control unit recurrent weight and the input gate memory unit recurrent weight, respectively, i and b c They are the input gate control unit bias and the input gate memory unit bias respectively.

[0041] (c) Cell state update. Cell state C t Used to record the current state of the LSTM neural network.t-1 And the forget gate output f t The product of, and input gate i t and The calculation formula is:

[0042]

[0043] where Θ is the Hadamard product.

[0044] (d) The output gate calculates the input data through the hidden layer and then compresses it to 0 to 1 through the activation function for output. The calculation formula is:

[0045] o t =σ(W o h t-1 +U o x t +b o ) (5)

[0046] h t =o t Θtanh(C t ) (6)

[0047] Where W o and U o are the output gate loop weight and the output gate input weight, respectively, o Bias for the output gate.

[0048] (e) The hidden state is predicted through the output activation function. Let the input sequence X = (x0, x1, ..., x t ), the predicted value is Indicates that the calculation formula is:

[0049]

[0050] where w y is the weight, b y is the bias, represents the output activation function of LSTM. When the input sequence (x0, x1, ..., x t ), the LSTM model will generate the corresponding hidden states (h0,h1,…,h t ), the hidden state is activated by the data function Convert to predicted value.

[0051] 2. Memory enhancement mechanism for LSTM:

[0052] In this method, a model for re - extracting forgotten information is constructed in the most basic unit (memory unit) of the long - short - term neural network. By setting up a reinforcement gate mechanism, the network's memory ability for historical information is enhanced. This mechanism is for the forgotten information C t-1(f) and the memory information C t-1(r) to be transmitted through a dual - channel, and the two are selected, fused, and re - input into the memory unit in proportion. This enables the network to remain sensitive to relatively distant information to improve its memory ability. The memory reinforcement process is as Figure 2 shown.

[0053] 2.1 Theoretical Analysis

[0054] After introducing the memory enhancement unit, the memory unit is:[[]]

[0055]

[0056] After simplification, we get:[[]]

[0057]

[0058]

[0059] where W3 = W2 - W1. By weighting the forgotten information for memory, the LSTM can maintain a certain degree of sensitivity to the forgotten information. From (10), it can be seen that by setting the value of W1, the LSTM can retain a minimum amount of memory for the information passed from the previous time step.

[0060] Expanding formula (4) gives:[[]]

[0061]

[0062] From (11), it can be known that the unit memory at the t - th time step is obtained by summing the input memories at the i - th step after experiencing t - i times of forgetting. Therefore, by strengthening the input memory at each time step, the strengthening of the unit memory can be achieved.

[0063] Expanding from formula (9) gives:[[]]

[0064]

[0065] Given that 0 < f t < 1, when 0 < W1 < 1 and W1+W3 = 1, So is always greater than C t , that is, W2 = W1+W3 = 1. W2 retains the information passed down by the original forgetting gate, and on this basis, additional historical information is added through W1 to achieve memory enhancement.

[0066] LSTM allows to store information at arbitrary time lags and can transmit error signals to further time series. However, this potential advantage may lead to reduced performance in some cases: the cell state C t Usually, in the process of learning time series, the unit state tends to grow linearly. In the case of continuous input, the unit state may grow infinitely, leading to the saturation of the output gate. The saturation of the output gate will make h t The derivative of disappears, so the input error cannot be obtained, and the unit output is equal to the activation function of the output gate, causing the cell to stop the memory function. Therefore, a forget gate with a value between 0 and 1 is introduced to control the unit memory saturation. Therefore, the improved LSTM model will not cause unit memory saturation due to the increase in memory of input information.

[0067] 3. Experimental results analysis

[0068] The experiment selected the root mean square error (RMSE), mean absolute error (MAE), receiver operating characteristic curve (ROC) and its area under curve (AUC) to evaluate the effect of the improved LSTM model. The prediction ability of the improved LSTM model and the original model were compared by the three indicators of prediction fitting curve, RMSE and MAE. The training error curve was compared with the test error curve to analyze the fitting of the model training process. The ROC curve and AUC table were used to show the authenticity of the comparison model in this fault detection problem.

[0069] 3.1 Dataset

[0070] The experimental data set is taken from the monitoring data of a certain device. A single sensor records a value at a frequency of once every 20ms. The set of values ​​recorded by 160 sensors at the same time is recorded as one piece of data. The data set contains 4,000,686 pieces of normal data collected for 29 consecutive hours from a certain moment, and 56,954 pieces of fault data collected within 0.3 hours. The data set has a total of 160 dimensions, and each dimension represents the corresponding sensor monitoring value. In order to balance the positive and negative samples, this experiment selects normal data at intervals of 20, compresses the data to 200,000, and directly selects fault data, with a total of 256,954 pieces of data. The indicator recorded by sensor number 18S is set as the dependent variable 18S. When the value of 18S is greater than 170, the machine is abnormal (18S is one of the main data that can identify device failures. After observation, it is the most obvious feature distinction). The remaining indicators are used as independent variables. The data set is divided into 8:2 according to the training set and the test set. The LSTM model is built based on the keras framework to realize the prediction of the fault feature parameter 18S. The value of the dependent variable 18S in the training set is as follows Figure 6 As shown, the horizontal axis is the number of samples, and the vertical axis represents the numerical value. This data distribution simulates a training cycle in the actual scenario.

[0071] 3.2 Metrics

[0072] The experiment selected the root mean square error RMSE, mean absolute error MAE and area under the receiver operating characteristic line AUC to evaluate the model effect.

[0073] (1) The calculation formula of root mean square error RMSE is expressed as:

[0074]

[0075] where y t and are the observed value and predicted value of the data time series at time t, T is the number of data points. When RMSE reaches the minimum, the optimal regression model can be obtained.

[0076] (2) The calculation formula of mean absolute error MAE is:

[0077]

[0078] where y t and f t are the observed value and model output value of the data time series at time t, respectively, and T is the number of data points.

[0079] (3) Area Under Curve (AUC). The receiver operating characteristic curve (ROC) is a curve drawn with the true positive rate (sensitivity) as the ordinate and the false positive rate (1-specificity) as the abscissa. AUC is the area under the ROC curve and the coordinate axis. The closer the AUC is to 1.0, the higher the authenticity of the detection method. This paper quantitatively evaluates the model fitting and prediction accuracy by calculating the RMSE, MAE and AUC values ​​on the test set.

[0080] 3.3 Data analysis and preprocessing

[0081] During the data collection process, the quality of the data set will be affected by environmental factors, improper operation of data collection, and problems with the equipment itself, which will lead to anomalies in the collected data. Preprocess the data before training.

[0082] Data normalization is widely used in engineering data processing. Normalization can make data uniform and comparable. This paper adopts min-max normalization, which is a linear transformation of the original data. The transformation formula is shown in formula (15):

[0083]

[0084] Among them, Maxvalue is the maximum value in the data, Minvalue is the minimum value in the data, Maxvalue-Minvalue is the range, x i is the data taken from the dataset, is the size of the normalized data.

[0085] 3.4 Results Analysis

[0086] Figure 5 and Figure 6 The prediction results of the improved LSTM model and the existing LSTM model are shown respectively. The horizontal axis represents the data volume and the vertical axis represents the parameter value. The light-colored line represents the value of the dependent variable 18S in the test set, and the dark-colored line represents the value of the dependent variable predicted by the model. The dependent variable changes from 93 to 163 when there is a potential fault, and increases from 163 to more than the threshold of 170 when a fault occurs. In the normal time series range, the prediction curve of the improved LSTM model is closer to the original curve than the prediction curve of the existing LSTM model. However, in the fault time series area, the prediction error of the improved LSTM model is 5, which is 61.5% less than the prediction error (13) of the existing LSTM model.

[0087] Figure 7 and Figure 8The changing trends of RMSE and MAE with the input sequence length in the prediction process of the existing LSTM model (LSTM) and the improved LSTM model (S_LSTM) are given respectively. The horizontal axis represents the sequence length and the vertical axis represents the error. Figure 7 As shown in the figure, the root mean square error of the existing LSTM model is the largest when the input sequence length is 50, and it decreases to the minimum as the sequence length increases to 130. This shows that there is an upper limit to the continuous input sequence length that the original LSTM model can accept, and the upper limit of the memory capacity established by a single forget gate structure is 130. However, the improved LSTM model does not have this problem. The error curve fluctuates very little in the sequence length range of 0 to 160, and continues to drop after 160. Figure 8 As shown in the figure, the trend of the mean absolute error MAE is very similar to the root mean square error RMSE, which confirms that the upper limit of the memory capacity of the original LSTM model is 130. The comparison of the root mean square error RMSE and the mean absolute error MAE at different input sequence lengths shows that the upper limit of LSTM memory is significantly improved after the introduction of the reinforcement gate memory mechanism.

[0088] Table 1 shows the effect of the W1 parameter on the prediction accuracy of the model, where the sequence length is set to 160, the learning rate is 0.01, and the L2 regularization coefficient is 0.01. W1 takes values ​​between 0 and 1 with an interval of 0.1. The changes of the three evaluation indicators RMSE, MAE and AUC with the parameter W1 are shown in Table 1. The closer the AUC is to 1.0, the higher the authenticity of the detection method. When W1 is 0.2 and 0.8, the RMSE and MAE are the smallest, which are 0.145 and 0.085 respectively, which are much lower than the original LSTM of 0.314 and 0.276. When W1 is 0.9, the AUC value is the highest, 0.959. It can be seen that the prediction results of the improved LSTM model are better than the existing LSTM model in terms of RMSE, MAE and AUC.

[0089] Table 1. Effect of parameter W1 on the model

[0090]

[0091]

[0092] Fig. 9 The ROC curves of the existing LSTM model, the improved LSTM model 1 (W1 = 0.9), and the improved LSTM model 2 (W1 = 0.4) are compared. The horizontal axis represents the false positive rate, and the vertical axis represents the true positive rate. The data when the AUC is the maximum (W1 = 0.9) and the median (W1 = 0.4) are selected to draw the ROC curve. Fig. 9In the results, the improved LSTM model 1 and the improved LSTM model 2 have outstanding prediction capabilities. In comparison, the existing LSTM model (optimal configuration) is still far inferior to the improved LSTM model.

[0093] Fig.10 The figure shows the training error curve of the improved LSTM model, with the number of training iterations as the horizontal axis and the mean absolute error as the vertical axis. During the training process, the training set error (dark color) and the test set error (light color) both decrease as the number of training rounds increases. The model reaches an inflection point at epoch 7, and the error no longer changes significantly. The training ends at 50 rounds of training.

[0094] In summary, the improved LSTM performs better in this prediction. It can not only improve the memory limit, but also improve the prediction accuracy of medium and short sequences.

Claims

1. An electronic equipment fault prediction method based on LSTM enhanced model, characterized in that: The method is specifically as follows: Step 1: Collect monitoring data during the operation of the electronic device and pre-process the monitoring data; Step 2: Build an LSTM enhanced model, input the preprocessed monitoring data into the LSTM enhanced model for training, and obtain a trained LSTM enhanced model; The LSTM enhancement model is built based on LSTM. The specific steps are as follows: Step 21: The cell state at the previous moment passes through the forget gate and outputs the part to be remembered and the forgotten part. The forgotten part is obtained according to the part to be remembered. The cell state at the previous moment - the part to be remembered = the forgotten part. Step 22: assign weights to the part to be remembered and the part to be forgotten respectively; Step 2 and 3: add the weighted forgotten part and the part to be remembered to the current cell state, and update the current cell state; Step 3: Predict electronic equipment failures based on the trained LSTM enhancement model and monitoring data over a period of time; In the memory unit of LSTM, a reinforcement gate is added based on the original forget gate; the part to be remembered C t-1(r) Belongs to the forget gate, forget part C t-1f) It is a reinforced door; The part to be memorized is: C t-1(r) =f t ΘC t-1 Formula 1 The forgotten parts are: C t-1(f) =(1-f t )ΘC t-1 Formula 2 The current cell state is: Substituting Formula 1 and Formula 2 into Formula 3 and setting W2 = 1, we get: In the formula, C t-1 is the cell state at the previous moment, C t is the cell state at the current moment, W1 is the weight given to the forgotten part, W2 is the weight given to the part to be remembered, i t It is a control unit implemented using the sigmoid activation function in the input gate. is a memory cell using the tanh activation function in the input gate, f t is the value of the forget gate, Θ is the Hadamard product; f t =σ(W f h t-1 +U f x t +b f ) Among them, W f is the forget gate recurrent weight, U f Input weight for the forget gate, b f is the forget gate bias, ht-1 and xt are the output and input of the hidden layer respectively, and σ is the sigmoid activation function.

2. The electronic equipment fault prediction method based on the LSTM enhanced model according to claim 1 is characterized in that: The value range of the reinforcement gate is [0,1].

Images