A residual life prediction method based on convolutional neural network and long short-term memory network

Abstract

The application discloses a residual life prediction method based on a convolutional neural network and a long short-term memory network, and relates to the field of residual life prediction.The method mainly comprises two parts of data preprocessing and model training.In the preprocessing stage, working condition information identification is mainly completed on the monitoring data of the failed equipment, normalization and standardization are carried out according to the working condition category, and the preprocessed monitoring data and the corresponding RUL are simultaneously subjected to sliding window processing to obtain input samples and output labels.In the model training stage, the training samples are mainly input into the CNN-LSTM model for time sequence feature extraction and degradation correlation modeling, the predicted RUL of the input information is obtained through forward propagation, the error between the true value and the predicted value is calculated, and the model parameters are updated through backward propagation using a loss function, and the process is repeated until the prediction loss value decreases to a certain range and tends to be stable.The application introduces Dropout and early stopping method to reduce the negative influence of the overfitting problem on the prediction performance of the model, improves the accuracy of the RUL prediction, and provides a better solution for modeling the degradation process of the RUL prediction.

Description

Technical Field

[0001] This invention relates to the field of remaining lifetime prediction, and more specifically to the application of remaining lifetime prediction based on convolutional neural networks and long short-term memory networks. Existing technology

[0002] Data-driven Remaining Useful Life (RUL) prediction methods have become the most widely used prediction and classification methods due to their powerful data processing capabilities. However, existing research mainly focuses on deep feature extraction and RUL prediction tasks from sensor data of monitoring equipment, but lacks consideration for prediction problems under complex operating conditions. In fact, various noise factors often exist in the actual industrial environment, and considering the impact of operating conditions on RUL prediction results has important application value.

[0003] A review of current literature reveals that while various artificial intelligence-based neural networks have demonstrated excellent performance in modeling the degradation process and residual lifetime (RUL) prediction of complex systems, exhibiting powerful nonlinear approximation capabilities, most algorithms only obtain the mapping relationship between data and RUL, lacking consideration of uncertainties in the modeling process. Sbarufatti et al., in "Sequential Monte Carlo sampling based on a committee of artificial neural networks for posterior state estimation and residual lifetime prediction," combined feedforward neural networks with Monte Carlo methods to apply to the RUL probability distribution of parts affected by fatigue cracks, enabling real-time detection of the part's damage state. Yang et al., in "Remaining Useful Life Prediction Based on a Double-Convolutional Neural Network Architecture," proposed a model combining two convolutional neural networks (CNNs) for RUL prediction. The first CNN identifies the initial failure points of different components, while the second CNN establishes a reliable mapping relationship between intermediate variables and RUL. Deep neural networks can also be used to model Health Indicators (HIs). Chen et al., in their paper "An integrated deep learning-based approach for automobile maintenance prediction with GIS data," introduced the Cox PHM proportional hazards model to construct HIs and combined it with multiple proposed Long Short-Term Memory (LSTM) deep neural network structures to model HIs, demonstrating the superior performance of LSTM models in predictive accuracy. To establish a RUL prediction model based on deep neural networks, the key lies in accurately establishing the mapping function between the input monitoring data and the target health status indicator. Furthermore, neural networks integrate data feature processing and modeling analysis into a single network structure, achieving end-to-end RUL prediction, ensuring prediction accuracy while greatly simplifying the traditional RUL prediction process.

[0004] Currently, deep learning-based lifespan prediction methods mainly focus on deep feature extraction and near-life (RUL) prediction based on equipment monitoring data. However, there is little research on prediction under complex operating conditions. Considering the variability and randomness of the actual operating environment of equipment, it is necessary and valuable to consider the impact of operating conditions on lifespan prediction results. Therefore, a deep learning-based RUL method is needed to solve the RUL prediction problem under complex operating conditions. Summary of the Invention

[0005] To address existing problems, this invention proposes a Restricted Low Utility (RUL) prediction method based on convolutional neural networks and long short-term memory networks. First, the K-means method is used to identify the operating conditions of the collected monitoring data to reduce the impact of equipment environmental conditions on model performance. In the data preprocessing stage, a sliding window is used to generate three-dimensional sample formats that LSTM is well-suited for processing. Then, a CNN is used to extract deep features from the data after the operating condition analysis. Finally, LSTM is applied to fit the extracted features to establish a temporal degradation model, which is then extrapolated to complete the RUL prediction task.

[0006] The overall data flow framework diagram of this invention is as follows: Figure 1 As shown, it is mainly divided into two parts: data preprocessing and model training.

[0007] (1) The preprocessing stage mainly completes the identification of operating condition information of the monitoring data of the failed equipment, normalizes and standardizes it according to the operating condition category, and simultaneously performs sliding window processing on the preprocessed monitoring data and the corresponding RUL to obtain input samples and output labels.

[0008] (2) In the model training stage, the training samples are mainly input into the CNN-LSTM model to extract temporal features and model degradation correlation. The predicted RUL of the input information is obtained through forward propagation. The error between the true value and the predicted value is calculated and the model parameters are updated by backpropagation using the loss function. This process is repeated until the predicted loss value drops to a certain range and tends to stabilize.

[0009] Step 1: Select the original dataset

[0010] Based on the actual condition of the maintenance equipment, multiple sensors are selected to collect data and obtain the raw dataset.

[0011] Step 2: Dataset Preprocessing

[0012] (1) Data operating condition information identification

[0013] Since the same equipment operates differently under different configurations, it is necessary to identify the operating condition information of the equipment under different operating conditions. Several data points are randomly selected from the original dataset as initial cluster centroids. The K-means algorithm is then used to learn the distribution of the training set data and classify the operating conditions of the training and test sets, thus adding a new label representing the operating condition information to the data.

[0014] (2) Data normalization and standardization processing

[0015] Equipment operating under different working conditions will inevitably have different operating states. Therefore, it is necessary to normalize and standardize the data by combining the labels obtained in step 1.

[0016]

[0017] Formula (1) mainly performs the data normalization operation, where x d Let x represent the raw value of the d-th sensor. d-max and x d-min x represents the maximum and minimum values ​​of this sensor across all training sets. d This represents the normalized value.

[0018]

[0019] Formula (2) completes the data standardization process, preserving the statistical distribution information of the data. Where u d and σ d Let represent the mean and standard deviation of the d-th sensor, respectively.

[0020] (3) Time series sample generation

[0021] To convert the original two-dimensional signal into a three-dimensional matrix in LSTM input data format, a sliding window process is required. The principle of generating multivariable time-series samples using a sliding time window is as follows: Figure 2 As shown.

[0022]

[0023] Formula (3) represents the processing procedure for the sliding window. This represents all the monitoring data collected by the i-th device. n represents the number of sensors. N tw Let represent the size of the sliding window. Then, the total information obtained by the sliding window at the k-th time point is: Slide the window from left to right in increments of 1 until the final time point T.

[0024] (4) Generating RUL Labels

[0025] Based on the real-world operating conditions of equipment, we know that each device has an initial normal operating state, meaning that its RUL (Relative Limiting Parameter) can be considered constant during this period. According to a piecewise linear model, such as... Figure 3 As shown, a threshold is set to represent the device's RUL health status. When the device's RUL is lower than the threshold, the device is considered to have started to degrade. When the RUL is lower than the threshold, the device is considered to have started to degrade linearly.

[0026] Step 3: Construct the CNN-LSTM network model

[0027] The network structure based on the CNN-LSTM model proposed in this invention is as follows: Figure 4 As shown, the input sample is a two-dimensional matrix, with one dimension representing the number of features and the other representing the time series length. Weight-shared convolutional kernels slide along the time dimension of the sample matrix with a stride of 1 to extract features, achieving a 1D convolution operation. Furthermore, to ensure that the input and output sizes are the same, the feature vector representation for each time point is obtained by padding the convolutional input with zeros along the time dimension. Each convolutional layer is followed by an activation layer, and to avoid information loss, no pooling layers are added for downsampling. After feature filtering by three layers of CNN, feature vectors that fully describe the degradation information are obtained. Then, a one-layer LSTM network is used to model the temporal correlation of the health indicators of the degradation system. A multilayer perceptron (MLP) is used to fit the health state representation vectors extracted by the deep neural network. To reduce the possibility of overfitting during training, the Dropout method is introduced, such as... Figure 5 As shown, during the training process, some neurons are randomly deactivated with a certain probability to prevent them from participating in network training and enhance the model's generalization ability.

[0028] Step 4: Model Training

[0029] (1) Parameter update

[0030] The mini-batch gradient descent algorithm is used in the parameter update process. This leverages the efficient matrix operations of deep learning while avoiding the fluctuations that may be caused by updating parameters using the gradient of a single sample. First, the chain rule of the backpropagation algorithm is used to calculate the gradient of each layer, and then the weight parameters w of that layer are updated using the MBGD algorithm. (l) .

[0031]

[0032]

[0033] In the formula, δ (l) Let η represent the gradient of the l-th layer, and η be the learning rate.

[0034] When updating the parameters of a convolutional layer, it is necessary to pad the kernel with zeros and rotate it 180 degrees to obtain the gradient error, and then calculate the updated weight matrix W based on the gradient error. l The specific steps are as shown in equation (6-10):

[0035] α (l) =f l (net l (6)

[0036] net (l+1) =conv(W l+1 ,α (l) (7)

[0037] δ (l) =δ (l+1) rot180(W l+1 )⊙f l (net (l) (8)

[0038]

[0039]

[0040] In the formula, f l (·) represents the activation function used after the l-th convolutional layer, W l+1 Let α represent the weight matrix of the convolutional layer. (l) This represents the output of the l-th layer, conv(·) represents the convolution operation, and net (l) This represents the feature map obtained after convolution, and rot180 indicates a 180-degree rotation.

[0041] Early stopping can be used to improve model training efficiency, such as Figure 6 As shown, after multiple epochs of training, the loss error decreases on the training set, but the loss shows an upward trend on the validation set. At this point, it can be considered that the model has overfitted. Therefore, network training should be stopped here, and the model weight parameters W should be saved. l The prediction model was then established.

[0042] (2) Model parameter optimization

[0043] The Adam adaptive optimization algorithm is used to optimize the model parameters. The first moment controls the direction of model updates, and the second moment controls the learning rate. Compared with other optimization algorithms, it is not sensitive to gradient scale and is suitable for optimizing deep models with sparse parameters or high complexity.

[0044] Step 5: Model Evaluation

[0045] To more directly observe the model's predictive performance optimization process, a model loss function is constructed based on the root mean square error (RMSE):

[0046]

[0047] Where n is the number of test units, RUL pred_i and RUL true_i Let RUL and RUL represent the predicted RUL and the actual RUL of the i-th test unit, respectively.

[0048] The effectiveness of the model is evaluated using the Score function, as shown in (12).

[0049]

[0050] Where Score is the overall prediction score function, n represents the number of test units, and d i This represents the prediction error of the i-th unit.

[0051] Step Six: RUL Prediction

[0052] The test samples are input into the prediction model and forward computation is performed to output the final RUL prediction results of the test samples at each time step.

[0053] This invention proposes a CNN-LSTM-based RUL prediction method. It analyzes the characteristics of data collected under complex operating conditions and, based on the proposed network structure and features, proposes a method for preprocessing multi-condition data. A specific CNN-LSTM network model is implemented, outlining the detailed steps for equipment lifespan prediction. Due to the risk of overfitting during neural network training, Dropout and early stopping techniques are introduced to mitigate the negative impact of overfitting on model prediction performance, thereby improving the accuracy of RUL prediction and providing a better solution for modeling the degradation process of RUL prediction. Attached Figure Description

[0054] Figure 1 Flowchart of RUL prediction method based on CNN-LSTM

[0055] Figure 2 This is a schematic diagram of generating sequence samples using a sliding window.

[0056] Figure 3 It is a piecewise linearized RUL label model

[0057] Figure 4 This is a diagram of the CNN-LSTM network structure.

[0058] Figure 5 This is a diagram of the dropout principle.

[0059] Figure 6 This is a diagram of the early cessation method.

[0060] Figure 7 This is a schematic diagram of the original degradation trends of 21 sensors in the FD002 dataset.

[0061] Figure 8 This is a clustering result diagram of the FD002 engine data.

[0062] Figure 9 This is a schematic diagram illustrating the degradation trend of 21 sensors in the FD002 dataset after standardization.

[0063] Figure 10 This is a graph showing the experimental results of network structure parameters.

[0064] Figure 11 This is a graph showing the experimental results with input sample parameters.

[0065] Figure 12 These are the engine prediction results for 100 test sets in the FD001 dataset.

[0066] Figure 13 This is the prediction result for a single engine in a subset of the C-MAPSS test. Specific Implementation

[0067] The remaining lifetime prediction method based on convolutional neural networks and long short-term memory networks proposed in this invention is verified through a specific case. The specific process includes the following steps:

[0068] Step 1: Select the original dataset

[0069] The effectiveness of this invention is demonstrated and verified using the Commercial Modular Aero-Propulsion System Simulation (C-MAPSS) dataset.

[0070] There are four subsets in the C-MAPSS dataset, as shown in Table 1. Datasets FD001 and FD003 are engine degradation data under a single operating condition, while FD002 and FD004 are engine degradation data under six different operating conditions. Therefore, the performance of the proposed CNN-LSTM lifetime prediction method will be verified based on datasets with different operating conditions.

[0071] Table 1. Introduction to the C-MAPSS Engine Dataset

[0072]

[0073] Each data subset contains 21 sensor data and 3 operational data. Depending on the operation settings, the engine's operating conditions can be further refined into 6 types, which in turn affect the monitoring data of the corresponding sensors. Figure 7 The original degradation data of 21 sensors from one engine in the FD002 dataset is presented. As can be seen from the figure, due to the influence of operating conditions on engine performance, the original degradation data of the 21 sensors included in FD002 fluctuated significantly throughout the degradation process. Therefore, further analysis based on operating conditions is needed to filter the sensors and extract effective degradation information.

[0074] Step 2: Data Preprocessing

[0075] (1) Data operating condition information identification

[0076] First, the K-means algorithm is used to identify and label the operating conditions of the FD002 and FD004 datasets, so that the monitoring data can be standardized according to the current operating status. Figure 8 The results of clustering equipment operating conditions using the operation information of FD002 are presented. It can be seen that the distance between the operation information of different operating conditions is large, thus achieving a good clustering effect.

[0077] (2) Data normalization and standardization processing

[0078] Then the data is standardized and normalized. Figure 9 The diagram illustrates the degradation trajectories of 21 standardized sensors. It is clear from the diagram that some sensors, such as T24, T30, and T50, show significant degradation trends after standardization, while effective degradation information cannot be obtained for sensors such as T2, P20, and P15.

[0079] (3) Time series sample generation

[0080] The sliding window technique is used to process the sensors retained after the initial screening and generate time-series samples.

[0081] (4) Generating RUL Labels

[0082] The threshold for RUL tags is set to 120 based on a piecewise linear function.

[0083] Step 3: Build and train the CNN-LSTM network model

[0084] The main parameters affecting prediction performance are: the number of convolutional layers in the CNN, the number of hidden units in the LSTM network, the sequence length of the samples, and the training batch size. These parameters were adjusted using a single-variable control method. Furthermore, to eliminate the influence of randomness in network parameter initialization on the experimental results, each parameter setting was repeated five times, and the average of the evaluation results was taken.

[0085] Figure 10 The experiment demonstrates the impact of the number of convolutional neural network layers and the dimension of LSTM hidden units on the model's prediction performance. The figures show that as the number of convolutional layers increases, the model's feature extraction capability further improves, while the score and RMSE values ​​gradually decrease. When the number of convolutional layers reaches 3, the model's prediction performance reaches its optimal state. Further increases in the number of convolutional layers increase model complexity, posing a risk of overfitting, and causing a downward trend in prediction performance. With a fixed number of convolutional layers of 3, similar changes occur as the dimension of the LSTM hidden units increases. When the dimension of the LSTM hidden units reaches 64, the model's prediction error drops to its lowest point.

[0086] Besides network structure affecting model prediction performance, training sample size is also a crucial parameter influencing the prediction performance of the proposed method, particularly its impact on model convergence speed. Figure 11 It can be seen that as the sample sequence length increases, the network requires more time to analyze and process this input information, thus reducing the model's convergence speed. Furthermore, as the sequence length increases, the amount of information contained in a single sample increases, the prediction error gradually decreases, and the prediction accuracy gradually improves. Especially when the sequence length increases from 20 to 25, the prediction accuracy improves significantly, suggesting that longer sample sequences contain sufficient historical information to help predict degradation trends. The batch size of the training samples is also another important factor affecting the model's prediction performance. Figure 11 As shown in the right figure, with the increase of training batch, the speed of processing the same data increases, the gradient descent direction tends to be stable, and the model convergence speed increases.

[0087] In summary, the network structure parameter settings are shown in Table 2.

[0088] Table 2 Network Parameter Settings

[0089]

[0090] Step 4: RUL Prediction Results and Analysis

[0091] The CNN-LSTM-based lifetime prediction method first obtains the device degradation model through supervised network training in the offline stage. Then, in the online RUL prediction stage, the processed data is input into the training and model is saved, and the RUL can be directly obtained. Figure 12The RUL prediction results for the last monitoring point of 100 engines in the FD001 test set are given. It can be seen that the predicted RUL basically overlaps with the real RUL label. The RUL prediction model based on neural network has a much higher prediction accuracy than the particle filter-based method.

[0092] One engine was randomly selected from each of the four test sets for RUL prediction. Figure 13 The figure shows the predicted RUL (Rapid Underlying Value) for engines 76, 190, 99, and 126 over time. It is evident that during the entire degradation prediction period for these four engines under different operating conditions, the predicted RUL values ​​from the proposed method generally follow the same trend as the corresponding actual RUL values, with minimal error between the predicted and actual values. The figure also indicates that:

[0093] (1) The proposed algorithm provides accurate lifetime estimates for equipment under four different operating conditions, demonstrating strong model generalization performance. Furthermore, as the amount of input information increases, the estimated RUL further converges to the true value. This is because in the initial monitoring phase, the equipment can be considered to be in normal operation, thus the predicted value fluctuates around a constant value. As the monitoring time increases, the equipment enters the degradation phase, and the failure trend becomes more significant. At this point, the predicted RUL value gradually approaches the true label value, and the model's prediction accuracy gradually improves. Providing a highly accurate health status assessment in the early stages of equipment failure is of significant guiding importance for subsequent predictive maintenance planning. This predictive performance is highly valuable in practical industrial applications.

[0094] (2) The complexity of operating conditions has a significant impact on the model's predictive performance. The figure shows the RUL prediction results of engines under randomly selected operating conditions. From the comparison of the fluctuation of the RUL prediction curves, engines No. 76 and No. 99 in FD001 and FD003 under single operating conditions are smaller than engines No. 190 and No. 126 in FD002 and FD004 under complex operating conditions. Especially in the near-failure stage, the RUL measurement values ​​of simple operating conditions gradually converge and basically coincide with the true label values, while the RUL prediction values ​​of complex operating conditions still fluctuate around the true label values. The CNN-LSTM algorithm proposed in this chapter performs cluster analysis of operating conditions before inputting data into the deep network model, which helps to reduce the impact of operating condition noise on the prediction model and improve the prediction accuracy of the prediction method in real-world scenarios.

[0095] In addition, the model's predictive performance was measured using RMSE and Score, and the results are shown in Table 3.

[0096] Table 3 RUL prediction performance based on CNN-LSTM

[0097]

[0098] Based on the combined results of RUL prediction and the evaluations of the two model performance assessment methods, the proposed CNN-LSTM-based RUL prediction method can provide relatively accurate RUL prediction results. The algorithm has strong robustness and exhibits stable and effective prediction performance under different working conditions.

Claims

1. A method for predicting remaining lifetime based on convolutional neural networks and long short-term memory networks, characterized in that, Includes the following steps: Step 1: Select the original dataset Based on the actual condition of the maintenance equipment, select multiple sensors to collect data and obtain the raw dataset; Step 2: Dataset Preprocessing (1) Data operating condition information identification Several data points are randomly selected from the original dataset as initial cluster centroids. The K-means algorithm is called to learn the distribution of the training set data and classify the working conditions of the training and test set data, so that the data adds a new label representing working condition information. (2) Data normalization and standardization processing Equipment operating under different working conditions will inevitably have different operating states. Therefore, it is necessary to normalize and standardize the data by combining the labels obtained in step 1. Formula (1) mainly performs the data normalization operation, where Indicates the first The raw values ​​of each sensor, and This represents the maximum and minimum values ​​of the sensor across all training sets. This represents the normalized value; Formula (2) completes the data standardization process, preserving the statistical distribution information of the data; where and They represent the first The mean and standard deviation of each sensor; (3) Time series sample generation To convert the original two-dimensional signal into a three-dimensional matrix in LSTM input data format, sliding window processing is required; Formula (3) represents the processing procedure for the sliding window. Indicates the first All monitoring data collected by each device , Indicates the number of sensors; Indicates the size of the sliding window, then in the... All the information obtained by the sliding window within each time point is Slide the window from left to right in increments of 1 until the last time point. ; (4) Actual RUL label generation Based on practical experience, each device has an initial normal operating state, meaning that its RUL is considered constant during this period. Therefore, a threshold representing the health status of the device's RUL is set according to the piecewise linear model. When the device is operating normally, the RUL is considered constant. When the device's RUL falls below the threshold, the device is considered to begin linear degradation. Step 3: Construct the CNN-LSTM network model The input is a two-dimensional matrix sample, with one dimension representing the number of features and the other representing the time series length. A weighted convolutional kernel slides along the time dimension of the sample matrix with a stride of 1 to extract features, achieving a 1D convolution operation. Furthermore, to ensure the input and output sizes are the same, feature vectors are obtained for each time point. The convolutional input is padded with zeros along the time dimension, and each convolutional layer is followed by an activation layer. Feature vectors are obtained after feature filtering by three CNN layers. Then, a single LSTM network is used to model the health indicators of the degraded system. An MLP is selected to fit the extracted health state representation vectors. To reduce the possibility of overfitting during training, the Dropout method is introduced. During training, some neurons are randomly deactivated with a certain probability to prevent them from participating in network training, thereby enhancing the model's generalization ability. Complete the construction of the CNN-LSTM network model; Step 4: Model Training (1) Parameter update The mini-batch gradient descent algorithm is used in the parameter update process. First, the gradient of each layer is calculated using the chain rule of the backpropagation algorithm, and then the weight parameters of that layer are updated using the MBGD algorithm. ; In the formula, Indicates the first The gradient of the layer, The learning rate; When updating the parameters of a convolutional layer, the kernel is padded with zeros and then rotated 180 degrees to obtain the gradient error. The updated weight parameter matrix is ​​then calculated based on the gradient error. The specific steps are as shown in equation (6-10): In the formula, Indicates the first The activation function used after each convolutional layer This represents the weight matrix of the convolutional layer. Indicates the first The output of the layer, This represents the convolution operation. This represents the feature map obtained after the convolution operation. This indicates a 180-degree rotation; Early stopping is used to improve model training efficiency. After multiple epochs of training, the loss error on the training set decreases, but the loss on the validation set shows an upward trend. At this point, it can be considered that the model has overfitted. Therefore, network training is stopped at this point, and the model weight parameters are saved. To complete the establishment of the prediction model; (2) Model parameter optimization The Adam adaptive optimization algorithm is used to optimize the model parameters. The first moment controls the direction of model updates, and the second moment controls the learning rate. Compared with other optimization algorithms, it is not sensitive to gradient scale and is suitable for the optimization of deep models with sparse parameters or high complexity. Step 5: Model Evaluation To more directly observe the model's predictive performance optimization process, a model loss function is constructed based on the root mean square error (RMSE): in, This is the number of test units. and They represent the first Predicted RUL and actual RUL for each test unit; The effectiveness of the model is evaluated using the Score function, as shown in (12). Where Score is the overall prediction score function, and n represents the number of test units. Indicates the first The prediction error of each unit; Step Six: RUL Prediction The test samples are input into the prediction model and forward computation is performed to output the final RUL prediction results of the test samples at each time step.

Images